Data processing method and system based on AI large model

Abstract

The application relates to an AI large model-based data processing method and system, wherein the method comprises the following steps: receiving a user request or original data, and compiling into a machine executable semantic instruction set through a task compiler; a strategy controller generates a model cooperation topology graph according to the semantic instruction set, defines a called heterogeneous large model instance and a cooperation relationship and a data flow direction thereof; inference is performed according to a corresponding model instance in a scheduling large model array, semantic calibration is performed when an intermediate result is transmitted between models, aligned semantic data is generated, and the aligned semantic data are transmitted to a downstream model with a confidence label; after each model is executed, a final processing result and a reasoning chain with a confidence label are input into an evidence network fusioner, an evidence network graph is constructed, global optimization is performed, and a final processing result and a corresponding explainability traceability path are output, so that the accuracy, efficiency and explainability of a complex data processing task are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method and system based on a large AI model. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful capabilities in fields such as natural language processing and data analysis. However, single models, limited by their architecture design, training data, and parameter scale, struggle to achieve optimal performance across all tasks. To address the demands of complex data processing, multi-model collaboration has become increasingly common in practice, involving the invocation of multiple large models to jointly complete a task. Current multi-model collaboration methods typically employ fixed-process orchestration or simply merge the outputs of each model through voting. However, in existing technologies, different models exhibit varying understandings of the same semantic concept, leading to semantic drift in intermediate results transmitted between models, and downstream models failing to accurately understand the outputs of upstream models. Furthermore, fixed processes cannot dynamically adapt to the optimal model combination and invocation method based on task characteristics, resulting in low collaboration efficiency. The outputs of multiple models may support or contradict each other, lacking an effective comprehensive judgment mechanism, making it difficult to assess the credibility of the final result and resulting in a lack of interpretability. Summary of the Invention

[0003] The main objective of this invention is to provide a data processing method and system based on large AI models. In the collaborative reasoning process of multiple heterogeneous large models, this method improves the accuracy, efficiency, and interpretability of complex data processing tasks by dynamically arranging model collaboration relationships, real-time calibrating semantic consistency between models, and constructing a globally optimized evidence network. To achieve the above objectives, this invention provides a data processing method based on large AI models, comprising the following steps: Receive user requests or raw data, and compile the user requests or raw data into a set of semantic instructions consisting of machine-executable semantic instructions through a task compiler; The semantic instruction set is input into the policy controller, which dynamically generates a model collaboration topology graph. The model collaboration topology graph defines the multiple heterogeneous large model instances that need to be called and the collaboration relationships and data flow between the large model instances. Based on the model collaboration topology graph, multiple heterogeneous large model instances in the large model array are scheduled to perform inference. When passing semantic intermediate results between models, the semantic intermediate results are semantically calibrated to generate aligned semantic data. The semantic data and the attached confidence labels are then passed to downstream model instances for continued execution. The final processing results output after each model instance is executed, as well as the inference chains with confidence labels generated during the execution of each model instance, are input into the evidence network fusion unit. The evidence network fusion device constructs an evidence network graph, and graph computation is used to perform global optimization on the evidence network graph, outputting the final processing result and the corresponding interpretable tracing path.

[0004] Furthermore, the steps for dynamically generating the model collaboration topology graph by the policy controller include: The policy controller extracts features from the semantic instruction set to obtain the complexity features, domain features, and data dependencies of the current task. The complexity features, domain features, and data dependencies are input into a pre-trained meta-learning prediction model, which then outputs multiple candidate model combinations and the execution confidence of each candidate model combination. Based on the execution confidence of each candidate model combination, and combined with the preset heuristic search algorithm, the calling order and parallel relationship of model instances in each candidate model combination are searched and optimized to generate a model collaboration topology graph containing model instance identifiers, calling order, parallel relationship and data flow path.

[0005] Furthermore, before scheduling multiple heterogeneous large model instances corresponding to the large model array to perform inference, the following steps are also included: Pre-deploy a large model array, which contains multiple large model instances that are heterogeneous in terms of architecture type, parameter scale, or professional domain; Configure a unified input / output interface for each model instance in the large model array and assign a unique instance identifier; Establish a mapping table between model instances and executable semantic instruction types, and store the mapping table in the association database of the policy controller.

[0006] Further, the steps of semantically calibrating the semantic intermediate results to generate aligned semantic data, and passing the semantic data and accompanying confidence labels to downstream model instances for continued execution include: The semantic calibration module intercepts the semantic intermediate results and the accompanying confidence labels; The semantic intermediate result is mapped to a preset public semantic space benchmark. By comparing it with the corresponding standard semantic representation in the public semantic space benchmark, the offset between the representation vector of the semantic intermediate result and the standard semantic representation is calculated. If the offset exceeds a preset threshold, it is determined that there is semantic drift, and the representation vector of the semantic intermediate result is adjusted according to the public semantic space benchmark to generate aligned semantic data. Extract the reasoning path corresponding to the semantic intermediate result, perform consistency verification between the reasoning path and the preset standard reasoning logic in the public semantic space benchmark, and identify logical conflict nodes in the reasoning path that conflict with the standard reasoning logic. The logical conflict nodes are marked and corrected, and the corrected reasoning path is associated with the aligned semantic data and then passed to the downstream model instance.

[0007] Furthermore, the confidence label includes at least one of the following information: the original confidence before semantic calibration, the calibration offset during the semantic calibration process, the corrected confidence after calibration, the logical strength of each node in the inference path, and the identifier and version information of the upstream model instance.

[0008] Furthermore, the step of constructing an evidence network graph through the evidence network fusion device includes: The final processing results output by each model instance are extracted as evidence nodes through the evidence network fusion device. Extract the logical causal relationships that exist in the reasoning chain of each model instance, and use the logical causal relationships as directed edges between evidence nodes; After normalizing the confidence labels attached to the inference chain of each model instance, they are used as the node weights of the corresponding evidence nodes, and the logical strengths corresponding to the directed edges are used as the edge weights. The evidence nodes, directed edges, node weights, and edge weights form a weighted directed graph, which serves as the evidence network graph.

[0009] Furthermore, the step of performing global optimization on the evidence network graph through graph computation includes: In the evidence network diagram, the evidence node corresponding to the initial data is taken as the starting point, and each end evidence node is taken as the ending point. Execute a preset graph search algorithm to traverse all paths from the starting point to each ending point, and calculate the comprehensive confidence of each path based on the weighted accumulation of node weights and edge weights on each path; The final processing result corresponding to the endpoint on the path with the highest overall confidence is selected as the output result, and all evidence nodes and corresponding reasoning chains on the path with the highest overall confidence are output as interpretable tracing paths.

[0010] Furthermore, after the step of constructing the evidence network diagram, it also includes: Conflict detection is performed on evidence nodes in the aforementioned evidence network graph that exhibit logical contradictions; If the reasoning chains of two or more evidence nodes are detected to point to mutually exclusive conclusions, the evidence node with the highest confidence is selected as the dominant node based on the confidence label and node weight of each evidence node, and the remaining conflicting nodes are marked as nodes to be verified. The information of the node to be verified is fed back to the policy controller, which then triggers the rescheduling of the relevant model instances for secondary verification.

[0011] Furthermore, after outputting the final processing result and the corresponding interpretable tracing path, the process also includes: The final processing result and the corresponding interpretable tracing path are input into the performance feedback loop; The performance feedback loop verifies the correctness of the final processing result, and performs backpropagation optimization on the model collaboration topology graph, the calibration parameters of the semantic calibration module, and the graph calculation weights of the evidence network fusion device involved in the interpretability tracing path based on the verification result. The optimized parameters are updated in the policy controller, semantic calibration module, and evidence network fusion unit.

[0012] This invention also provides a data processing system based on a large AI model, comprising: The task compilation unit is used to receive user requests or raw data, and compile the user requests or raw data into a semantic instruction set consisting of machine-executable semantic instructions through the task compiler. The policy control unit is used to input the semantic instruction set into the policy controller, which then dynamically generates the model collaboration topology graph. The large model array unit is used to schedule multiple heterogeneous large model instances in the large model array to perform inference according to the model collaboration topology graph, and when passing semantic intermediate results between models, perform semantic calibration on the semantic intermediate results, generate aligned semantic data, and pass the semantic data and the attached confidence label to the downstream model instance to continue execution. The semantic calibration unit is used to input the final processing results output by each model instance after execution, as well as the inference chain with confidence labels generated by each model instance during execution, into the evidence network fusion unit. The network fusion unit is used to construct an evidence network graph through the evidence network fusion device, perform global optimization on the evidence network graph through graph computation, and output the final processing result and the corresponding interpretable tracing path.

[0013] The data processing method and system based on large AI models provided by this invention have the following beneficial effects: This invention dynamically generates a model collaboration topology graph based on meta-learning prediction and heuristic search using a policy controller. This enables the system to adaptively select the optimal model combination and invocation method according to different task characteristics, improving the flexibility and efficiency of multi-model collaboration. By introducing a semantic calibration module when transmitting semantic intermediate results between models, the intermediate results are mapped to a common semantic space benchmark for real-time calibration, effectively eliminating the semantic drift problem in multi-model collaboration and ensuring that downstream models can accurately understand the output of upstream models. At the same time, the calibration process is made transparent and traceable through the transmission of confidence labels. An evidence network fusion device constructs a weighted evidence network graph from the output results of each model and the complete inference chain, and uses a graph computing algorithm for global optimization. While outputting the final result with the highest comprehensive confidence, it also provides a complete interpretable traceability path, making the decision-making process clear and credible. Through performance feedback loop, backpropagation optimization of the entire link parameters is performed to form self-optimization, enabling the system to continuously learn and evolve from historical execution. This invention achieves simultaneous improvement in accuracy, efficiency, and interpretability in the collaborative inference process of multiple heterogeneous large models. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a data processing method based on a large AI model in one embodiment of the present invention; Figure 2 This is a structural block diagram of a data processing system based on a large AI model in one embodiment of the present invention.

[0015] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Reference Figure 1 This is a flowchart illustrating a data processing method based on a large AI model proposed in this invention, including the following steps: S1, receive user requests or raw data, and compile the user requests or raw data into a semantic instruction set consisting of machine-executable semantic instructions through a task compiler; S2, The semantic instruction set is input into the policy controller, and the policy controller dynamically generates a model collaboration topology graph. The model collaboration topology graph defines the multiple heterogeneous large model instances that need to be called and the collaboration relationship and data flow between the large model instances. S3, according to the model collaboration topology graph, schedule multiple heterogeneous large model instances in the large model array to perform inference, and when passing semantic intermediate results between models, perform semantic calibration on the semantic intermediate results to generate aligned semantic data, and pass the semantic data and the attached confidence label to the downstream model instance to continue execution; S4, input the final processing results output after each model instance is executed, and the inference chain with confidence labels generated during the execution of each model instance, into the evidence network fusion unit; S5. Construct an evidence network graph through the evidence network fusion device, and perform global optimization on the evidence network graph through graph computation to output the final processing result and the corresponding interpretable tracing path.

[0018] In one embodiment, a user request or raw data is received. The user request may be natural language text, such as "Analyze the financial statements of a listed company for the past three years and compare its profitability with the average level of the same industry" in this embodiment; the raw data may be structured data, unstructured text, images, etc.

[0019] To address the issues of ambiguous user requests and inaccurate understanding by large models, this technical solution employs a task compiler. The task compiler performs natural language understanding processing on user requests, including: identifying the core intent of the user request using a pre-trained language model. For example, in the above request, the core intent is to analyze financial statements and compare profitability; the entity objects include a listed company, the past three years, and the industry average. The task compiler has a pre-set instruction template library containing various atomic semantic instruction templates, such as data extraction instructions, indicator calculation instructions, comparative analysis instructions, and trend prediction instructions. Based on the identified intent and entities, the compiler matches the corresponding instruction templates to generate a semantic instruction set composed of several atomic semantic instructions.

[0020] For example, the above request can generate the following set of semantic instructions: Data extraction instruction A: Extract financial statement data for the past three years from a listed company; Indicator Calculation Instruction B: Calculate the profitability indicators of a listed company over the past three years, including ROE, net profit margin, etc.; Data Extraction Instruction C: Extract the average profitability indicators of the same industry over the past three years. Comparison Analysis Instruction D: Compares the results of Instruction B and Instruction C, and generates a comparison analysis report.

[0021] The task compiler assigns a unique instruction identifier to each atomic semantic instruction and marks the required input data type, output data type, and dependent preceding instruction identifiers for that instruction, forming a complete semantic instruction set. In this example, instruction D depends on the outputs of instructions B and C, so the dependency relationship of instruction D is marked as "[B, C]". Through the above processing, the originally ambiguous natural language request is transformed into a structured, machine-executable semantic instruction set.

[0022] This embodiment addresses the problem in existing technologies that typically employ fixed processes, failing to dynamically adapt optimal collaboration schemes based on task characteristics, and requiring different model collaboration methods for different tasks. By setting up a policy controller to receive semantic instruction sets and dynamically generate model collaboration topology graphs, this approach solves the issue. The policy controller analyzes the semantic instruction set obtained in step S1, extracting complexity features (e.g., number of instructions, dependency depth), domain features (e.g., finance, healthcare, law), and data dependencies (serial dependency, parallel dependency). The extracted features are then input into a pre-trained meta-learning prediction model.

[0023] The meta-learning prediction model employs a graph neural network (GNN) architecture, using task feature vectors as input and outputting candidate model combinations and their confidence scores. Historical execution records are collected for model training; each record includes: task features, the actual model combination invoked, and the execution result (success / failure, time consumption, accuracy). The Model-Ambilicate (MAML) algorithm from meta-learning is used for training, aiming to enable the model to quickly adapt to new tasks—that is, to accurately predict the optimal model combination based on a limited set of task features. Historical data is grouped by task type, with each group serving as a meta-task. The model is trained on multiple meta-tasks, learning the mapping between task features and the optimal model combination. For the input financial statement analysis task, the meta-learning prediction model outputs multiple candidate model combinations. Each combination contains identifiers for several model instances and an execution confidence score. For example: Candidate Combination 1: Financial-Specific Model A (Data Extraction) + Financial-Specific Model B (Indicator Calculation) + General Comparative Model C (Comparative Analysis), Confidence Level 0.92 Candidate combination 2: General large model D (full-process processing), confidence level 0.75 Candidate combination 3: Financial-specific model A + Statistical model E + Visualization model F, confidence level 0.68 The policy controller optimizes the order of calling and parallel relationships of model instances in each candidate combination based on the candidate combinations and their confidence levels, combined with a pre-defined heuristic search algorithm.

[0024] The heuristic search algorithm described in this embodiment uses A. The search algorithm serves as the foundation, combined with domain-specific heuristic functions. These heuristic functions comprehensively consider the following factors: model compatibility (historical success rate), estimated model execution time (based on historical data), computational resource consumption (GPU memory, CPU cores), and data transfer overhead (data transfer volume between models). Based on this, the search space is defined by all possible model call sequences and parallel combinations. Under the premise of satisfying task dependencies, the search identifies the calling method with the highest overall score. The overall score is calculated as: Overall Score = Confidence × (1 / Estimated Execution Time) × Compatibility Coefficient. The heuristic function comprehensively considers the following factors: model compatibility (historical success rate), estimated model execution time (based on historical data), computational resource consumption (GPU memory, CPU cores), and data transfer overhead (data transfer volume between models), combining all possible model call sequences and parallel combinations. Taking candidate combination 1 as an example, possible calling methods include: ABC (serial), A-(B||C) (B and C in parallel), (A||B)-C (A and B in parallel followed by C), etc. Under the premise of satisfying task dependencies, the calling method with the highest overall score is found. The policy controller outputs a complete model collaboration topology graph, a directed graph containing key information including nodes, edges, parallel relationships, and semantic instruction mappings. Nodes identify model instances, such as "Financial Specific Model AV2.1"; edges indicate data flow, such as "Model A - Model B," meaning the output of Model A serves as the input of Model B; parallel relationships indicate that if multiple models can be executed in parallel, they are grouped together in the graph; each node is associated with the atomic semantic instructions it needs to execute. This model collaboration topology graph structurally stores the policy controller's decisions, serving as the basis for subsequent scheduling. For example, the aforementioned financial statement analysis task might generate a topology graph of "Model A - [Model B, Model C] - Model D," where Model B and Model C execute in parallel.

[0025] This embodiment pre-deploys a large model array containing multiple heterogeneous large model instances in terms of architecture type (e.g., Transformer, MoE), parameter size (e.g., 7B, 13B, 70B), or professional domain (e.g., finance, healthcare, law). Each model instance is configured with a unified input / output interface and assigned a unique instance identifier. Based on the model collaboration topology, the corresponding multiple heterogeneous large model instances are scheduled to perform inference. During this process, different models may have different understandings of the same semantic concept, which can easily lead to semantic drift. For example, "apple" in a general model may refer to fruit, while in a financial model it may refer to Apple Inc.; the output of an upstream model may be misunderstood by a downstream model. To address this, this embodiment introduces a semantic calibration module for real-time calibration when transmitting semantic intermediate results between models. When any upstream model instance completes execution and outputs a semantic intermediate result, this result is not directly transmitted to the downstream model but is intercepted by the semantic calibration module. The semantic calibration module receives data including: semantic intermediate results, such as model A extracting a company's net profit of 10 billion in 2023; and confidence labels, supplementary information generated during model A's execution, including the confidence level of the result (e.g., 0.95) and the inference path (e.g., "extracting from the third paragraph of the financial statement text - numerical normalization processing - unit conversion"). The semantic calibration module internally uses a pre-defined public semantic space benchmark. This benchmark is a unified knowledge representation system comprising three levels: a concept library, a relation library, and an inference rule library. The concept library defines standardized concepts and their unique identifiers. For example, the standard definition of "net profit" is "net profit attributable to the parent company's owners," with the concept ID "FIN-PROFIT-NET," and the concept ID of "operating revenue" is "FIN-REVENUE." The relation library defines standard relationships between concepts; for example, "net profit" is part of the "income statement," and "operating revenue - operating costs = gross profit." The relation library contains over 500 relation types. The inference rule library defines standard inference logic. For example, the rule for calculating the net profit growth rate is "(current period net profit - previous period net profit) / previous period net profit × 100%". The public semantic space benchmark is constructed by integrating multiple general knowledge graphs (such as Wikidata and DBpedia) and domain knowledge bases, undergoing review and standardization by domain experts, and continuously updated in a semi-automatic manner. It is stored in the form of a vector database, with each concept corresponding to a standard representation vector; simultaneously, it stores the relationships and inference rules between concepts in the form of a graph database.

[0026] The received semantic intermediate results are mapped to a public semantic space benchmark to find the corresponding standard semantic concepts. The mapping process includes: linking entities in the results (e.g., "net profit") to standard concept IDs in the concept library; obtaining the standard representation vector corresponding to the standard concept; calculating the cosine distance between the model's output representation vector and the standard representation vector as the offset; and setting a preset threshold. If the offset exceeds this threshold, semantic drift is considered to exist. The preset threshold is set differently for different domains and concept types. For example, for key financial indicators in the financial field, the threshold is set to 0.1 (strict); for everyday concepts in general domains, the threshold is set to 0.3 (lenient). The thresholds are based on the statistical distribution of historical calibration data. A performance feedback loop is used to adjust the thresholds based on the calibration effect. If frequent missed or false detections are found, the thresholds are adjusted accordingly. The threshold adjustment mechanism periodically calculates the missed detection rate (the proportion of concepts that should have been calibrated but were not) and the false detection rate (the proportion of concepts that should not have been calibrated but were calibrated) based on the performance feedback loop. If the false negative rate of a concept exceeds 5% for three consecutive statistical periods, the threshold for that concept will be lowered by 0.05 (not lower than 0.05); if the false positive rate exceeds 5% for three consecutive statistical periods, the threshold for that concept will be raised by 0.05 (not exceeding 0.5). The adjusted threshold takes effect immediately and is recorded in the threshold configuration table of the public semantic space benchmark. For example, if the "net profit" output by model A is actually "operating profit", the offset of its representation vector from the standard "net profit" representation vector may reach 0.6, far exceeding the threshold of 0.1, triggering calibration. If semantic drift is detected, the semantic calibration module adjusts the representation vector according to the public semantic space benchmark to generate aligned semantic data. Adjustment methods include: concept replacement, replacing "operating profit" with "net profit"; vector correction, moving towards the standard representation vector while retaining some original semantic information; recording the adjustment magnitude and generating a calibration offset, such as "offset 0.6, adjusted to 0.1".

[0027] The semantic calibration module extracts the inference path corresponding to the semantic intermediate results and performs consistency verification with the pre-set standard inference logic in the public semantic space benchmark. The standard inference logic is stored in the inference rule base of the public semantic space benchmark. For example, for the inference process of calculating net profit, the standard logic is to extract total operating revenue from the profit statement, extract total operating costs, calculate total profit, deduct income tax, and obtain net profit. The actual inference path executed by the model (e.g., extracting net profit from the balance sheet) is compared with the standard inference logic to identify discrepancies (e.g., the balance sheet should be the profit statement). If there are logically conflicting nodes in the inference path, these nodes are marked and corrected, generating a corrected inference path. Correction methods include node replacement, node deletion, and node addition. The corrected inference path is associated with the aligned semantic data to form a new semantic data package. This data package contains: aligned semantic content, original confidence labels, and calibration process records (including calibration offset and corrected node information). This data package is then passed to downstream model instances.

[0028] In this embodiment, the confidence label is continuously transmitted and enhanced throughout the entire processing chain. The confidence label includes at least the original confidence level before semantic calibration (the upstream model's own confidence assessment of the output), the calibration offset during semantic calibration (reflecting the adjustment magnitude of the calibration), the corrected confidence level after calibration (a new confidence level that comprehensively considers the original confidence level and the calibration offset; the calculation formula is: Corrected Confidence Level = Original Confidence Level × (1 - Calibration Offset)), the logical strength of each node in the inference path (e.g., the reliability of the "Extract from Financial Statements" node, with a value range of [0,1]), and the identifier and version information of the upstream model instance (e.g., "Financial Specific Model A-V2.1"). Through the transmission of the confidence label, the downstream model can understand the specific process and reliability of the received data. For example, if the downstream model receives data that has undergone significant calibration (calibration offset of 0.6) and has a low corrected confidence level (0.35), it can appropriately reduce the dependency weight on that data or give it a lower weight in subsequent fusion. After each model instance has completed execution, the final processing result and a complete inference chain with confidence labels are output and input into the evidence network fusion unit. The evidence network fusion unit is designed to address the technical challenge of comprehensively judging multiple model outputs that may support or contradict each other.

[0029] The evidence network fusion processor uses the final processed results of each model instance as evidence nodes. For example, Model A outputs "The company's profitability is higher than the industry average," Model B outputs "The company's profitability is lower than the industry average," and Model C outputs "The company's net profit growth rate is 15%." Each of these results becomes an evidence node. The inference chain of each model instance is analyzed to extract the logical causal relationships. For example, the inference chain of Model A is "Extract net profit data - Calculate growth rate - Compare with industry average - Draw conclusion," where there is a causal relationship between extracting net profit data and calculating growth rate. The evidence network fusion processor uses these logical causal relationships as directed edges between evidence nodes; it normalizes the confidence labels attached to each inference chain and performs softmax processing on the modified confidence of all evidence nodes to make the sum equal to 1. The higher the confidence, the greater the node weight; this serves as the node weight for the corresponding evidence node. The normalization method uses the logical strength corresponding to the logical causal relationship as the edge weight. The logical strength is calculated as follows: if there is a direct inference relationship between two nodes, the edge weight is set to 0.9; if it is an indirect inference (passing through other nodes), the edge weight is set to 0.7; and if it is a weak association (such as semantic similarity but not a direct inference), the edge weight is set to 0.3. An evidence network fusion processor constructs a weighted directed graph, i.e., the evidence network graph, using evidence nodes, directed edges, node weights, and edge weights to reflect the logical relationships between the outputs of each model.

[0030] The evidence network fusion unit performs global optimization on the constructed evidence network graph using a preset graph search algorithm. In this embodiment, the graph search algorithm combines an improved Dijkstra algorithm with a confidence propagation algorithm. It starts with the evidence node corresponding to the initial data (e.g., the "original financial report data" node) and ends with each final evidence node (e.g., the final conclusion node). The improved Dijkstra algorithm is executed, but the traditional "distance minimization" is replaced with "confidence maximization," meaning the weight of each edge is no longer distance but rather the transferred confidence (edge ​​weight × target node weight).

[0031] Considering the impact of path length on confidence, this embodiment introduces a path length attenuation factor. The actual formula for calculating the overall confidence is as follows: Overall confidence score of a path = Starting point weight × π(edge ​​weight) × Ending point weight × (1 / √path length) The path length is the number of edges traversed from the starting point to the ending point. The purpose of introducing a decay factor is to avoid falsely high confidence levels due to the cumulative effect of excessively long paths. The algorithm traverses all possible paths from the starting point to each ending point, calculating the overall confidence level of each path. The final processing result corresponding to the ending point on the path with the highest overall confidence level is selected as the system's output. Simultaneously, all evidence nodes on that path and their corresponding reasoning chains are output as interpretable tracing paths. For example, if the overall confidence level of the path for the conclusion "the company's profitability is higher than the industry average" is 0.95, while the overall confidence level of the path for "lower than the industry average" is only 0.60, the system selects the former as the final output, along with the complete reasoning path: from original financial report data - extracting net profit data - calculating the growth rate - comparing with the industry average - drawing a conclusion, including the model output and confidence label for each step.

[0032] This embodiment addresses the potential logical contradictions in the outputs of multiple models by implementing conflict detection. After constructing the evidence network graph, logically contradictory evidence nodes are detected: the logical relationships between evidence nodes are examined, and a pre-defined conflict relationship type library is used, including mutually exclusive pairs such as profit growth vs. profit decline, above average vs. below average, and positive vs. negative. If the reasoning chains of node A and node B respectively point to mutually exclusive conclusions pre-defined in the conflict relationship library, a conflict is determined. For example, if model A outputs "The company's profitability is higher than the industry average," and model B outputs "The company's profitability is lower than the industry average," they constitute a direct contradiction. For multiple evidence nodes determined to be in conflict, the corrected confidence level (considering both the original confidence level and calibration offset) and node weight (normalized confidence value) for each node are extracted. A weighted voting mechanism is used to calculate the comprehensive score for each node: Comprehensive score = Corrected confidence level × Node weight × (1 / Path length of the node). The node with the highest comprehensive score is selected as the dominant node, and the remaining nodes are marked as "nodes to be verified." The complete information of the node to be verified is packaged and fed back to the policy controller. This package includes at least: node identifier, corresponding model instance identifier, original input data, inference chain, confidence label, conflict type, and difference analysis from the dominant node. Upon receiving the node information, the policy controller selects a verification method from a pre-defined verification strategy library based on the conflict type. The verification strategy library includes strategies such as increasing the sampling count (repeating the original model three times and observing output stability), changing the model version (calling different versions of the same model), and introducing an arbitration model (calling a third-party independent model for cross-validation). Relevant model instances are scheduled to perform secondary verification. This verification process also undergoes a semantic calibration module to ensure the semantic consistency of the verification data. The secondary validation results are collected and compared with the original results. If the secondary validation results are consistent with the original results (e.g., Model A outputs "above average" again) and the confidence level is stable, the original judgment is maintained, and the trust weight of the model instance corresponding to that node is appropriately increased. If the secondary validation results are inconsistent with the original results (e.g., Model A outputs "below average" in the secondary validation), an anomaly marker is triggered, the model instance is added to the observation list, and calibration parameter adjustment or model update procedures may be triggered. If the secondary validation results are consistent with the dominant node, the original conflicting node is marked as "misjudged," and the calibration parameters of its corresponding model instance are adjusted (e.g., the calibration threshold of the model is appropriately increased). This embodiment continuously optimizes system performance by setting up a performance feedback loop. After outputting the final processing result and the corresponding interpretable traceability path, the performance feedback loop is used to verify the correctness of the final processing result. The verification method can be manual review, comparison with known correct answers, or cross-validation.

[0033] Based on the verification results, backpropagation optimization was performed on the model collaboration topology graph, the calibration parameters of the semantic calibration module, and the graph computation weights of the evidence network fusion unit involved in the interpretability tracing path. The backpropagation optimization employed a variant of the gradient descent method. Specifically, for the generation strategy of the model collaboration topology graph, the verification result (correct / incorrect) was used as a reward signal, and the parameters of the meta-learning prediction model were updated using the policy gradient method. For the calibration parameters of the semantic calibration module, the success rate of subsequent tasks using the calibrated data was used as the loss function, and the baseline values ​​of each concept threshold were adjusted using gradient descent. For the graph computation weights of the evidence network fusion unit, the difference between the verification result and the overall confidence of the path was used as the error, and the baseline values ​​of the edge weights were adjusted using backpropagation. The optimization process adopted an asynchronous update method, triggering a batch update every 100 accumulated verification records to avoid system oscillations caused by frequent updates. For example: if verification shows that a certain path frequently produces incorrect results, the edge weights corresponding to that path are reduced; if it is found that the output of a certain model instance frequently requires significant calibration, the calibration parameters (such as thresholds and adjustment magnitudes) for that model in the semantic calibration module are adjusted; if it is found that the optimal model combination for a certain type of task changes, the parameters of the meta-learning prediction model are updated; if it is found that there are missing or incorrect concept definitions in the public semantic space benchmark, the benchmark update process is triggered; the optimized parameters are updated in the policy controller, semantic calibration module, and evidence network fusion unit to form a closed-loop optimization.

[0034] Reference Figure 2 Here is a structural block diagram of a data processing system based on a large AI model according to an embodiment of the present invention, comprising: The task compilation unit is used to receive user requests or raw data, and compile the user requests or raw data into a semantic instruction set consisting of machine-executable semantic instructions through the task compiler. The policy control unit is used to input the semantic instruction set into the policy controller, which then dynamically generates the model collaboration topology graph. The large model array unit is used to schedule multiple heterogeneous large model instances in the large model array to perform inference according to the model collaboration topology graph, and when passing semantic intermediate results between models, perform semantic calibration on the semantic intermediate results, generate aligned semantic data, and pass the semantic data and the attached confidence label to the downstream model instance to continue execution. The semantic calibration unit is used to input the final processing results output by each model instance after execution, as well as the inference chain with confidence labels generated by each model instance during execution, into the evidence network fusion unit. The network fusion unit is used to construct an evidence network graph through the evidence network fusion device, perform global optimization on the evidence network graph through graph computation, and output the final processing result and the corresponding interpretable tracing path.

[0035] For the specific implementation of each unit in the above device example, please refer to the method embodiments described above, and will not be repeated here.

[0036] In summary, this invention receives user requests or raw data and compiles them into a semantic instruction set consisting of machine-executable semantic instructions using a task compiler. This semantic instruction set is then input into a policy controller, which dynamically generates a model collaboration topology graph. This graph defines the multiple heterogeneous large model instances to be invoked and the collaboration relationships and data flows between them. Based on this model collaboration topology graph, multiple heterogeneous large model instances in the large model array are scheduled to perform inference. When transmitting semantic intermediate results between models, semantic calibration is performed on the semantic intermediate results to generate aligned semantic data. This semantic data, along with accompanying confidence labels, is then passed to downstream model instances for further execution. The final processing results output by each model instance after execution, along with the inference chains with accompanying confidence labels generated during execution, are input into an evidence network fusion unit. The evidence network fusion unit constructs an evidence network graph, and graph computation is used to perform global optimization on this graph, outputting the final processing result and the corresponding interpretable tracing path. This aims to improve the accuracy, efficiency, and interpretability of complex data processing tasks.

[0037] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A data processing method based on a large AI model, characterized in that, Includes the following steps: Receive user requests or raw data, and compile the user requests or raw data into a set of semantic instructions consisting of machine-executable semantic instructions through a task compiler; The semantic instruction set is input into the policy controller, which dynamically generates a model collaboration topology graph. The model collaboration topology graph defines the multiple heterogeneous large model instances that need to be called and the collaboration relationships and data flow between the large model instances. Based on the model collaboration topology graph, multiple heterogeneous large model instances in the large model array are scheduled to perform inference. When passing semantic intermediate results between models, the semantic intermediate results are semantically calibrated to generate aligned semantic data. The semantic data and the attached confidence labels are then passed to downstream model instances for continued execution. The final processing results output after each model instance is executed, as well as the inference chains with confidence labels generated during the execution of each model instance, are input into the evidence network fusion unit. The evidence network fusion device constructs an evidence network graph, and graph computation is used to perform global optimization on the evidence network graph, outputting the final processing result and the corresponding interpretable tracing path.

2. The data processing method based on AI large model according to claim 1, characterized in that, The steps for dynamically generating a model collaboration topology graph by the policy controller include: The policy controller extracts features from the semantic instruction set to obtain the complexity features, domain features, and data dependencies of the current task. The complexity features, domain features, and data dependencies are input into a pre-trained meta-learning prediction model, which then outputs multiple candidate model combinations and the execution confidence of each candidate model combination. Based on the execution confidence of each candidate model combination, and combined with the preset heuristic search algorithm, the calling order and parallel relationship of model instances in each candidate model combination are searched and optimized to generate a model collaboration topology graph containing model instance identifiers, calling order, parallel relationship and data flow path.

3. The data processing method based on a large AI model according to claim 1, characterized in that, Before scheduling multiple heterogeneous large model instances in a large model array to perform inference, the following steps are also included: Pre-deploy a large model array, which contains multiple large model instances that are heterogeneous in terms of architecture type, parameter scale, or professional domain; Configure a unified input / output interface for each model instance in the large model array and assign a unique instance identifier; Establish a mapping table between model instances and executable semantic instruction types, and store the mapping table in the association database of the policy controller.

4. The data processing method based on a large AI model according to claim 1, characterized in that, The steps of semantically calibrating the semantic intermediate results to generate aligned semantic data, and passing the semantic data and accompanying confidence labels to downstream model instances for further execution include: The semantic calibration module intercepts the semantic intermediate results and the accompanying confidence labels; The semantic intermediate result is mapped to a preset public semantic space benchmark. By comparing it with the corresponding standard semantic representation in the public semantic space benchmark, the offset between the representation vector of the semantic intermediate result and the standard semantic representation is calculated. If the offset exceeds a preset threshold, it is determined that there is semantic drift, and the representation vector of the semantic intermediate result is adjusted according to the public semantic space benchmark to generate aligned semantic data. Extract the reasoning path corresponding to the semantic intermediate result, perform consistency verification between the reasoning path and the preset standard reasoning logic in the public semantic space benchmark, and identify logical conflict nodes in the reasoning path that conflict with the standard reasoning logic; The logical conflict nodes are marked and corrected, and the corrected reasoning path is associated with the aligned semantic data and then passed to the downstream model instance.

5. The data processing method based on a large AI model according to claim 1, characterized in that, The confidence label includes at least one of the following: the original confidence before semantic calibration, the calibration offset during the semantic calibration process, the corrected confidence after calibration, the logical strength of each node in the inference path, and the identifier and version information of the upstream model instance.

6. The data processing method based on a large AI model according to claim 1, characterized in that, The steps of constructing an evidence network graph using the evidence network fusion device include: The final processing results output by each model instance are extracted as evidence nodes through the evidence network fusion device. Extract the logical causal relationships that exist in the reasoning chain of each model instance, and use the logical causal relationships as directed edges between evidence nodes; After normalizing the confidence labels attached to the inference chain of each model instance, they are used as the node weights of the corresponding evidence nodes, and the logical strengths corresponding to the directed edges are used as the edge weights. The evidence nodes, directed edges, node weights, and edge weights form a weighted directed graph, which serves as the evidence network graph.

7. The data processing method based on a large AI model according to claim 6, characterized in that, The steps of performing global optimization on the evidence network graph through graph computation include: In the evidence network diagram, the evidence node corresponding to the initial data is taken as the starting point, and each end evidence node is taken as the ending point. Execute a preset graph search algorithm to traverse all paths from the starting point to each ending point, and calculate the comprehensive confidence of each path based on the weighted accumulation of node weights and edge weights on each path; The final processing result corresponding to the endpoint on the path with the highest overall confidence is selected as the output result, and all evidence nodes and corresponding reasoning chains on the path with the highest overall confidence are output as interpretable tracing paths.

8. The data processing method based on a large AI model according to claim 6, characterized in that, After the steps of constructing the evidence network diagram, the following are also included: Conflict detection is performed on evidence nodes in the aforementioned evidence network graph that exhibit logical contradictions; If the reasoning chains of two or more evidence nodes are detected to point to mutually exclusive conclusions, the evidence node with the highest confidence is selected as the dominant node based on the confidence label and node weight of each evidence node, and the remaining conflicting nodes are marked as nodes to be verified. The information of the node to be verified is fed back to the policy controller, which then triggers the rescheduling of the relevant model instances for secondary verification.

9. The data processing method based on a large AI model according to claim 1, characterized in that, After outputting the final processing result and the corresponding interpretable tracing path, the following steps are also included: The final processing result and the corresponding interpretable tracing path are input into the performance feedback loop; The performance feedback loop verifies the correctness of the final processing result, and performs backpropagation optimization on the model collaboration topology graph, the calibration parameters of the semantic calibration module, and the graph calculation weights of the evidence network fusion device involved in the interpretability tracing path based on the verification result. The optimized parameters are updated in the policy controller, semantic calibration module, and evidence network fusion unit.

10. A data processing system based on a large AI model, characterized in that, include: The task compilation unit is used to receive user requests or raw data, and compile the user requests or raw data into a semantic instruction set consisting of machine-executable semantic instructions through the task compiler. The policy control unit is used to input the semantic instruction set into the policy controller, which then dynamically generates the model collaboration topology graph. The large model array unit is used to schedule multiple heterogeneous large model instances in the large model array to perform inference according to the model collaboration topology graph, and when passing semantic intermediate results between models, perform semantic calibration on the semantic intermediate results, generate aligned semantic data, and pass the semantic data and the attached confidence label to the downstream model instance to continue execution. The semantic calibration unit is used to input the final processing results output by each model instance after execution, as well as the inference chain with confidence labels generated by each model instance during execution, into the evidence network fusion unit. The network fusion unit is used to construct an evidence network graph through the evidence network fusion device, perform global optimization on the evidence network graph through graph computation, and output the final processing result and the corresponding interpretable tracing path.

Images