Large model dynamic optimization method and system based on multi-modal knowledge graph

Abstract

The invention relates to the field of neural network models, in particular to a large model dynamic optimization method and system based on a multi-modal knowledge graph, and the method comprises the steps: extracting the semantic information of each modal; carrying out uncertainty quantification to obtain uncertainty measurement of each mode; a plurality of candidate interpretation hypotheses are inferred and verified by using semantic association information stored in the multi-modal knowledge graph, and the establishment probability of each hypothesis is evaluated; a multi-level uncertainty evaluation framework is constructed, uncertainty is divided according to levels of data, semantics, reasoning and decision making, and the propagation process of the uncertainty between the levels is modeled through a multi-modal knowledge graph to obtain the overall uncertainty measurement of the system; and calculating the overall confidence coefficient of the system, and dynamically selecting and executing a preset corresponding output strategy according to the overall confidence coefficient of the system and the uncertainty measurement of each level. The method has the effect of improving decision reliability and system efficiency.

Description

Technical Field

[0001] This application relates to the field of neural network models, and in particular to a method and system for dynamic optimization of large models based on multimodal knowledge graphs. Background Technology

[0002] With the increasing demand for multimodal data processing, large-scale pre-trained models have demonstrated outstanding capabilities in fusing various types of data (such as images, text, code, and logs). However, in complex scenarios, the decision reliability and confidence assessment capabilities of large models still face challenges, especially when dealing with multi-source heterogeneous data. How to effectively quantify uncertainty and achieve dynamic optimization has become a key issue.

[0003] Existing technologies typically analyze different modalities of data in isolation, extracting their feature information. Some methods assess uncertainty based on the feature extraction results of a single modality, such as by using probability distributions or error statistics to measure the quality of a particular modality's data. While these methods can reflect the characteristics of a single modality to some extent, they fail to fully consider the correlation between multimodal data and its impact on overall decision-making, and they also lack the ability to model the uncertainty propagation process across modalities. When processing multimodal data, existing technologies lack the ability to perform fine-grained uncertainty quantification, multi-level uncertainty propagation modeling, and dynamic output strategy adjustment. This can lead to unreliable results in high-risk scenarios, while in low-risk scenarios, there may be excessive reliance on human intervention, affecting system efficiency and reliability. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a method and system for dynamic optimization of large models based on multimodal knowledge graphs.

[0005] Firstly, this application provides a dynamic optimization method for large models based on multimodal knowledge graphs, employing the following technical solution: A dynamic optimization method for large models based on multimodal knowledge graphs includes the following steps: parsing input multimodal data to extract semantic information of each modality; quantifying uncertainty based on the semantic information of each modality and combining it with entities and relationships in the multimodal knowledge graph to obtain uncertainty measures for each modality; inferring and verifying multiple candidate explanatory hypotheses based on the uncertainty measures of each modality using semantic association information stored in the multimodal knowledge graph, and evaluating the probability of each hypothesis being true; constructing a multi-level uncertainty assessment framework, dividing uncertainty into data, semantic, reasoning, and decision-making levels, and modeling the propagation process of uncertainty between levels through the multimodal knowledge graph to obtain the overall uncertainty measure of the system; the levels include a data layer, a semantic layer, a reasoning layer, and a decision-making layer, each level having its own independent uncertainty measure; calculating the overall confidence of the system, and dynamically selecting and executing preset corresponding output strategies based on the overall confidence of the system and the uncertainty measures of each level.

[0006] Optionally, the method for inferring and verifying multiple candidate explanatory hypotheses is as follows: based on the overall uncertainty reflected by each modal uncertainty measure, the number of candidate explanatory hypotheses generated is adaptively determined; multimodal evidence is used to verify the candidate explanatory hypotheses, and the probability of each hypothesis being true is evaluated by fusing the consistency between each modal evidence and the hypothesis.

[0007] Optionally, in the process of verifying candidate explanatory hypotheses using multimodal evidence, the degree of consistency between the multimodal evidence and the candidate explanatory hypotheses is calculated by weighted fusion of consistency measures of each modality, and the weight of each modality is dynamically adjusted based on its uncertainty measure.

[0008] Optionally, the process of modeling uncertainty propagation between layers includes: calculating the confidence level of the i-th layer, and adjusting it by attenuation based on the uncertainty level of the (i-1)-th layer and using a preset weighting coefficient.

[0009] Optionally, a preset output strategy can be dynamically selected and executed, including the steps of: comparing the overall confidence level of the system, the uncertainty measure of the decision level, and the minimum value of the confidence level at each level with multiple preset thresholds; and selecting one of the preset strategies to execute based on the comparison results.

[0010] Optional preset strategies include: a high-confidence direct output strategy, an output strategy with uncertainty warnings, and a manual intervention strategy.

[0011] Optionally, based on the comparison results, one of the preset strategies is selected for execution, including: executing a high-confidence direct output strategy when the overall system confidence is not lower than a first threshold, the decision-level uncertainty measure is not higher than a second threshold, and the confidence of all levels is not lower than a third threshold; executing an output strategy with uncertainty warning when the high-confidence output conditions are not met but the overall system confidence is not lower than a fourth threshold; and executing a manual intervention strategy when the overall system confidence is lower than a fourth threshold.

[0012] Optionally, the multimodal data includes visual data and code dataset log data.

[0013] Optionally, the confidence scores for each modality are calculated, including: for visual data, the confidence scores are calculated based on one or more of the probability distribution of behavior recognition, the degree of occlusion in the video, and the similarity of behavior actions; for code data, the confidence scores are calculated based on one or more of the probability of correct code execution and code complexity; for dataset log data, the confidence scores are calculated based on one or more of the specificity of log information description, the internal consistency of log sequences, and the generalization degree of log templates.

[0014] Secondly, this application provides a large-scale dynamic optimization system based on multimodal knowledge graphs, employing the following technical solution: A large-scale dynamic optimization system based on a multimodal knowledge graph includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the aforementioned large-scale dynamic optimization method based on a multimodal knowledge graph.

[0015] The beneficial effect is that the above-mentioned dynamic optimization method for large models based on multimodal knowledge graphs is generated into a computer program and stored in memory so that it can be loaded and executed by the processor. Thus, the system can be made according to the memory and processor, which is convenient to use.

[0016] This application has the following technical advantages: 1. To address the issue of one-sidedness in single-modal confidence assessment, this invention integrates multi-source information such as vision, code, and logs for joint uncertainty quantification, effectively overcoming the limitations of inaccurate single data sources and significantly improving the system's reliability in perceiving complex scenarios.

[0017] 2. To address the lack of flexibility in static reasoning mechanisms, this invention can dynamically generate and verify multiple candidate hypotheses based on uncertainty, realizing a shift from passive acceptance to active exploration, and enhancing the robustness of the model's reasoning in the event of conflicting or missing evidence.

[0018] 3. To overcome the shortcomings of local errors directly affecting the final decision, this invention achieves quantitative management and precise positioning of the cumulative effect of errors by modeling the propagation process of uncertainty from data to decision level, providing a clear path for system optimization. By modeling the propagation of uncertainty from data to decision, the source of error can be accurately located.

[0019] 4. By intelligently linking internal confidence states with external output behaviors, a leap from "one-size-fits-all" to on-demand, tiered responses is achieved, balancing automation efficiency with security in high-risk scenarios. By intelligently linking internal confidence states with teaching actions, a balance is achieved between automated feedback and human intervention, improving teaching efficiency while ensuring the security of high-risk decisions. Attached Figure Description

[0020] Figure 1 This is a flowchart of steps S1-S5 in a large model dynamic optimization method based on multimodal knowledge graphs according to an embodiment of this application. Detailed Implementation

[0021] This application discloses a dynamic optimization method for large models based on multimodal knowledge graphs, referring to... Figure 1 The steps include: S1: Parse the input multimodal data to extract semantic information of each modality.

[0022] Multimodal data includes visual data, code dataset log data, and more. This application uses an online teaching scenario as an example, but it can also be applied to other scenarios where this application is applicable. For instance, the video data in the multimodal data can be videos of learning activities, the code data can be programming assignment code, and the dataset log data can be logs from a learning platform.

[0023] In one embodiment, the method for parsing multimodal data is as follows: for learning behavior video data, it is converted into a structured representation through instance segmentation and feature extraction; for student programming assignment code data, its operational logic is understood through syntax analysis and semantic parsing; and for learning platform log data, key information is extracted through template matching and event recognition.

[0024] Specifically, for learning behavior video data, a pre-trained instance segmentation model, such as Mask R-CNN, is first used to segment the input image into multiple semantically meaningful image patches. This step transforms the entire image into a set of image patches, thus achieving an initial conversion from pixel-level information to object-level concepts. Next, a deep convolutional neural network, such as ResNet, is used to extract deep features from each segmented image patch, obtaining a high-dimensional feature vector. Technical features are introduced during this process. The probability distribution used to characterize the identification of a specific visual concept from an image patch is calculated using the following formula: ,in Representing the Image blocks, This represents a specific visual concept object. This probability distribution directly reflects the model's confidence in recognizing the content of that image patch.

[0025] For student programming assignment code data, the input code snippets are parsed into an abstract syntax tree (AST), whose tree structure accurately reflects the syntactic hierarchy and nesting relationships of the code. Subsequently, this AST is traversed, and key node information is extracted, including but not limited to operator types, operands, and control flow nodes. Based on this, these discrete syntactic elements are mapped into continuous vector representations, i.e., code embedding. Technical features are introduced during this process. This can be used to quantify the complexity of code structure; for example, it can be defined as a function of the total number of nodes in an abstract syntax tree and the number of branch nodes. Complex code structures typically imply greater difficulty in understanding and potential uncertainty in execution paths. This feature provides crucial input for subsequent code confidence calculations and can be used to assess students' programming skills.

[0026] For learning platform log data, the key to parsing lies in extracting structured event information from unstructured text streams. In practice, a pre-defined event template library covering known log patterns is used. Each template contains a variable part for matching dynamic parameters in the log text and a static part for identifying the event type. The parsing process employs a fuzzy matching algorithm based on regular expressions, calculating the similarity between each input log message and all templates in the library, and assigning it to the event type with the highest matching score. In this process, we introduce technical features. This value quantifies the consistency between a single log entry and the matched template. The higher the value, the more standardized the log entry is and the more reliable the information it carries.

[0027] S2: Based on the semantic information of each modality, and combined with the entities and relationships in the multimodal knowledge graph, uncertainty quantification is performed to obtain the uncertainty measure of each modality.

[0028] The system verifies whether visual concepts extracted from image patches conform to the expected scenario by using relevant entities (such as "student behavior" and "classroom activities") stored in the multimodal knowledge graph. For example, if the knowledge graph records that "classroom activities typically include student hand-raising and teacher explanations," the system can compare these entities to determine if the visual concepts extracted from the image patches are reasonable. The system also utilizes relational information from the knowledge graph (such as "target A usually appears in scene B") to help determine the degree of influence of occlusion or similarity on the recognition results. If the knowledge graph records that "student hand-raising behavior usually occurs during classroom questioning," the system can increase confidence when it detects a vague "hand-raising" action by using context (such as the current questioning phase). If the knowledge graph records that "certain targets (such as students' hands) are easily occluded in specific scenarios," the system can be more cautious about recognizing these targets.

[0029] In one embodiment, the quantification of visual modal uncertainty in learning behavior integrates three factors: concept recognition, physical occlusion, and appearance confusion, and introduces technical features. As the final measure of visual confidence, the calculation formula can be: In this formula: Represents the probability distribution The maximum value taken from the sample directly reflects the model's confidence in the concept to which the image patch most likely belongs. The higher the value, the higher the confidence in the recognition. This represents an occlusion coefficient ranging from 0 to 1, calculated by the overlap ratio of the bounding boxes in the instance segmentation results (e.g., ...). The value is used to quantify the severity of target occlusion; a higher value indicates more severe occlusion. Representing appearance similarity, it is obtained by calculating the mean cosine similarity between the feature vector of the target image patch and the feature vectors of non-target categories in a standard feature library. It is used to evaluate recognition ambiguity caused by appearance confusion; a higher value indicates a greater likelihood of confusion. Hyperparameter and These are coefficients used to adjust the weights of occlusion and similarity. Empirical values ​​can be set to 0.3 and 0.2 respectively to ensure that neither factor excessively dominates the confidence score calculation.

[0030] The formula uses a multiplicative model, emphasizing that any factor (low recognition probability, high occlusion, or high similarity) will significantly reduce the confidence level, ensuring that the weakest link dominates the final confidence level.

[0031] Regarding the quantification of uncertainty in coding tasks, the specific implementation process focuses on the impact of code behavior on the system state and its logical complexity, introducing technical features. This is used to characterize code confidence. Its calculation is based on two core dimensions: first, the uncertainty of the impact of operations. It assesses the impact of code operations (such as function calls and variable assignments) by statistically analyzing the range of subsequent system state changes. For example, if code operations lead to diverse or unpredictable system state changes, then... The second is the high level of structural complexity. This metric can be calculated by combining metrics such as the depth of the code abstract syntax tree, the number of nodes, and the number of nested loops (e.g., This indicator can be obtained from the analysis steps. The features are obtained through feature quantization.

[0032] In one embodiment, code confidence The calculation formula is: This is a function that maps uncertainty to a confidence level between 0 and 1, where, and This is a weighted hyperparameter, empirically set to 0.5, used to balance the relative importance of operational impact and structural complexity in the final confidence assessment. The formula uses an inverse proportional form: when uncertainty and complexity are both 0, the confidence level is 1; as uncertainty increases, the confidence level monotonically decreases.

[0033] To quantify the uncertainty of learning log confidence, technical features are introduced. As a log confidence level. Describing specificity. This is measured by calculating the ratio of the length of dynamic parameters in the log message to the length of the static template text (e.g., The higher the value, the more detailed the log description. (Setting sequence consistency) The evaluation is performed by analyzing the degree of similarity between the current log and the logs before and after it in terms of event sequence patterns, for example, by using a sequence alignment algorithm to calculate a similarity score. Template generalization is set. This is an indicator calculated based on the reciprocal of the frequency of the log template in historical data (e.g., Templates that appear too frequently have low information specificity; a higher value indicates a more specific template. Log confidence. It is calculated from the weighted geometric mean of the above indicators.

[0034] In one embodiment, The expression can be, ;in The weighting indexes for each indicator can be set to 0.4, 0.4, and 0.2 respectively, based on experience, to emphasize the importance of descriptive quality and consistency. This formula emphasizes that low values ​​for any sub-indicator will significantly lower the overall confidence level.

[0035] S3: Based on the uncertainty measure of each modality, use the semantic association information stored in the multimodal knowledge graph to infer and verify multiple candidate explanation hypotheses, and evaluate the probability of each hypothesis being true.

[0036] In one embodiment, the visual uncertainty measure can be defined as Code uncertainty measurement Log uncertainty measurement .

[0037] In one embodiment, the method for inferring and verifying multiple candidate explanatory hypotheses is as follows: Based on the overall degree of uncertainty reflected by each modal uncertainty measure, the number of candidate explanatory hypotheses is adaptively determined. Multimodal evidence is used to verify these candidate hypotheses, and the probability of each hypothesis's validity is assessed by fusing the consistency between the evidence from each modality and the hypothesis. In verifying candidate explanatory hypotheses using multimodal evidence, the consistency between the evidence and the hypothesis is calculated by weighted fusion of consistency measures from each modality, with the weight of each modality dynamically adjusted based on its uncertainty measure.

[0038] Specifically, the multimodal uncertainty obtained from the initial quantification is used as a key regulatory signal to dynamically guide the breadth and depth of subsequent reasoning processes. The higher the overall uncertainty of the system, the more complex and ambiguous the current situation is. Therefore, it is necessary to generate more diverse candidate hypotheses and conduct more rigorous verification to avoid decision-making errors caused by premature convergence to a local optimum.

[0039] In the candidate explanation hypothesis generation stage, the specific implementation process is as follows: The system first retrieves entities and relations related to the current semantic information from the multimodal knowledge graph as seeds. Subsequently, technical features are introduced. The number of candidate hypotheses generated from these seeds is determined by the following formula: In this formula, This is the preset base sampling number, used to ensure a minimum level of exploration. An empirical value can be set to 5. It is the normalized overall system uncertainty, obtained by aggregating various modal uncertainty measures (e.g., Or use a weighted average); This is a preset maximum additional sampling limit to prevent unlimited expansion of computing resources; an empirical value of 15 is recommended. This formula ensures that the number of samples increases linearly with the level of uncertainty, achieving an intelligent match between computing resources and problem difficulty, and embodying the adaptive principle of "the higher the uncertainty, the broader the exploration."

[0040] In the stage of validating candidate explanatory hypotheses using multimodal evidence, a probabilistic evaluation framework based on Bayesian principles was adopted. Multimodal evidence Structured outputs from the parsing step, such as visual concept probability distributions, code abstraction syntax tree node information, and log event template matching results. Introducing technical features. This indicates that after observing all the evidence Then, assuming The posterior probability of the statement being true is the core output of Bayesian verification. Its posterior probability is expressed by Bayes' theorem. Perform calculations, where It is an assumption The prior probability can be learned from the knowledge graph. It is the likelihood of multimodal evidence, and its calculation formula is: ;in, It is a function used to compute the evidence for the m-th mode. With assumption The degree of logical consistency, for example, using semantic similarity or logical matching algorithms, has a value range of [0,1]. It is the first The modal weights are not fixed, but rather vary with the uncertainty measure of that mode. Negative correlation, meaning that the higher the uncertainty of a mode, the higher its weight. It will be dynamically lowered (e.g.) The weights of all modes need to be normalized to ensure that the sum of the weights is 1. This Bayesian framework updates the hypothesis probabilities by fusing weighted evidence, ensuring that the contribution of high-uncertainty modes is reduced.

[0041] S4: Construct a multi-level uncertainty assessment framework, which divides uncertainty into levels of data, semantics, reasoning, and decision-making, and models the propagation process of uncertainty between levels through multimodal knowledge graphs to obtain a measure of the overall uncertainty of the system.

[0042] In one embodiment, the hierarchy includes a data layer, a semantic layer, a reasoning layer, and a decision layer, each with its own independent uncertainty metric. Modeling the propagation of uncertainty between layers includes: calculating the confidence level of the i-th layer and adjusting it by attenuation based on the uncertainty level of the (i-1)-th layer using a preset weighting coefficient.

[0043] Existing technologies for assessing uncertainty often remain at a single level or the final output level, lacking fine-grained tracing of the sources of uncertainty. This approach cannot distinguish whether the difficulty in final decision-making stems from noise in the original data, semantic misunderstanding errors, or flaws in logical reasoning. This results in a lack of targeted system optimization and improvement, and may even lead to catastrophic decision failures due to the implicit accumulation and amplification of underlying uncertainties within the system. To address these issues, this invention constructs a hierarchical uncertainty assessment framework, decomposing overall uncertainty into four progressive levels: data, semantics, reasoning, and decision-making. Its core innovation lies not only in conducting independent confidence assessments for each level, but more importantly, in explicitly modeling the propagation and amplification effects of uncertainty from the bottom to the top. This enables the system to accurately pinpoint the root causes of uncertainty and understand how they affect the final judgment, providing a crucial theoretical foundation for achieving precise fault-tolerant control based on the sources of uncertainty.

[0044] In practical implementation, the first step is to define an independent initial uncertainty measure for each level. (Data layer uncertainty) : Measured by the uncertainty of each mode (e.g. It is calculated from the average or weighted average of ( ). Semantic layer uncertainty Evaluation is based on the strength and clarity of semantic relationships extracted from the knowledge graph, for example, using relation confidence or entity link scores for quantification. Inference layer uncertainty. Quantified by the posterior probability distribution entropy during the candidate hypothesis testing phase, for example... Higher entropy indicates higher uncertainty. Uncertainty measurement at the decision-making level. It is directly determined by the strength of the supporting evidence for the final selected decision, for example... That is, the complementary value of the posterior probability of the best hypothesis.

[0045] In one embodiment, technical features are introduced during the uncertainty propagation process between modeling levels. Let represent the "conditional uncertainty" of layer i after considering the uncertainty of layer i-1. Its propagation model is achieved through the formula: In this formula, The initial uncertainty of the i-th layer (e.g.) wait), It is the uncertainty of the (i-1)th layer immediately below it. This is a key preset uncertainty propagation weighting coefficient, which quantifies the degree of influence of lower-level uncertainty on the upper level. An empirical value can be set between 0.1 and 0.3 (e.g., 0.2) depending on the specific application scenario. This additive model clearly depicts how lower-level errors accumulate upwards as additional risk.

[0046] System overall uncertainty measurement By analyzing all levels (propagation-adjusted conditional uncertainty) The aggregation is performed to obtain the result. A preferred implementation method is to use a weighted geometric average method for calculation, that is: in It represents the conditional uncertainty of the i-th layer. These are the weights assigned to the i-th layer, and satisfy... The weight allocation can be adjusted according to application needs, for example, for higher inference layers. and decision-making level Assign greater weight. This aggregation process ensures “chain vulnerability” by calculating the overall confidence level (product term) and then taking its complementary value, meaning that high uncertainty at any level will be sensitively reflected in the overall metric.

[0047] S5: Calculate the overall confidence level of the system, and dynamically select and execute the corresponding preset output strategy based on the overall confidence level of the system and the uncertainty measure of each level.

[0048] In one embodiment, the overall system confidence level Defined as It is used for dynamic output strategy decisions.

[0049] In one embodiment, the method for dynamically selecting and executing a preset corresponding output strategy is as follows: comparing the overall confidence level of the system, the uncertainty measure of the decision level, and the minimum value of the confidence level of each level with multiple preset thresholds; and selecting one of the preset strategies to execute based on the comparison results.

[0050] For visual data, the confidence level is calculated based on one or more of the following: the probability distribution of behavior recognition, the degree of occlusion in the video, and the similarity of behavior actions. For code data, the confidence level is calculated based on one or more of the following: the probability of correct code execution and the code complexity. For dataset log data, the confidence level is calculated based on one or more of the following: the level of detail in the log information description, the internal consistency of the log sequence, and the generalization degree of the log template.

[0051] In one embodiment, the preset strategy includes: a high-confidence direct output strategy, an output strategy with uncertainty warnings, and a manual intervention strategy.

[0052] Based on the comparison results, one of the preset strategies is selected for execution, including: responding to the overall system confidence level not being lower than a first threshold. The uncertainty measure at the decision-making level is not higher than the second threshold. Furthermore, the confidence level at all levels is not lower than the third threshold. The high-confidence direct output strategy is executed when the high-confidence output condition is not met, but the overall system confidence is not lower than the fourth threshold. The system executes an output strategy with uncertainty warnings when the overall system confidence level falls below the fourth threshold; and executes a manual intervention strategy when the overall system confidence level falls below the fourth threshold.

[0053] Specifically, firstly, the confidence level at each level is defined as the complementary value of the corresponding conditional uncertainty measure, i.e. ; ; Set a threshold for policy distribution and introduce technical features. , , and Among the four key thresholds, The overall system confidence required for a high-confidence direct output strategy. The lower limit, based on experience, can be set to 0.85; It is the uncertainty of the decision-making level The upper limit, experience points can be set to 0.15; It consists of four levels of confidence: data, semantics, reasoning, and decision. The experience value can be set to 0.7; It is the overall system confidence level for initiating both the warning strategy and the manual intervention strategy. The dividing point, empirically, can be set to 0.6. These thresholds together constitute a rigorous, multi-condition decision network.

[0054] First, the policy conditions are determined. The system calculates and monitors the overall confidence level in real time. Decision-making level uncertainty measurement and the minimum value among the confidence levels. The comparison of these three core metrics with preset thresholds follows a multi-branch conditional statement: If and only if , and The system determines that it is in a high-reliability state only when all three conditions are met simultaneously. Based on the above judgment, the system executes the corresponding output strategy. If all high-confidence conditions are met, the system executes the direct output strategy, presenting the best hypothesis and its answer derived from the reasoning to the user in their entirety.

[0055] If the high confidence condition is not met, but If the output strategy includes an uncertainty warning, the system will output the answer while prominently displaying "The system has low confidence in this result" and may briefly indicate the level of the main source of uncertainty.

[0056] like If this is not the case, the manual intervention process will be triggered. The system will pause automatic output and instead submit all current multimodal evidence, candidate hypotheses and their probability assessments, and uncertainty analysis reports at each level to the human operator, requesting them to make a final decision.

[0057] The system's complex, quantified confidence states are mapped to a series of gradient external actions. This not only considers the overall confidence level of the system but also deeply analyzes the local uncertainties at the decision-making level and the fundamental reliability at all levels, ensuring that the output behavior matches the actual risk of the current situation.

[0058] This application also discloses a large model dynamic optimization system based on a multimodal knowledge graph, including a processor and a memory. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, the large model dynamic optimization method based on a multimodal knowledge graph according to this application is implemented.

[0059] The system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces, the settings and functions of which are known in the art and will not be described in detail here.

[0060] In this application, the aforementioned memory can be any tangible medium that contains or stores a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (DRAM), dynamic random access memory (DRAM), static random access memory (SRAM), etc., or any other medium that can be used to store desired information and can be accessed by an application program, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device.

[0061] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A dynamic optimization method for large models based on multimodal knowledge graphs, characterized in that, Including the following steps: The input multimodal data is parsed to extract semantic information for each modality; Based on the semantic information of each modality, and combined with the entities and relationships in the multimodal knowledge graph, uncertainty quantification is performed to obtain the uncertainty measure of each modality; Based on the uncertainty measure of each modality, the semantic association information stored in the multimodal knowledge graph is used to infer and verify multiple candidate explanatory hypotheses, and the probability of each hypothesis being true is evaluated. A multi-level uncertainty assessment framework is constructed, which divides uncertainty into data, semantics, reasoning and decision-making levels, and models the propagation process of uncertainty between levels through multimodal knowledge graphs to obtain the overall uncertainty measure of the system. The levels include data layer, semantic layer, reasoning layer and decision-making layer, and each level has its own independent uncertainty measure. Calculate the overall confidence level of the system, and dynamically select and execute the corresponding preset output strategies based on the overall confidence level and the uncertainty measures at each level.

2. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 1, characterized in that, The method for inferring and verifying multiple candidate explanatory hypotheses is as follows: Based on the overall degree of uncertainty reflected by each modal uncertainty measure, the number of candidate explanatory hypotheses generated is adaptively determined; We use multimodal evidence to verify candidate explanatory hypotheses and assess the probability of each hypothesis being true by fusing the consistency between the evidence from each modality and the hypothesis.

3. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 2, characterized in that, In the process of verifying candidate explanatory hypotheses using multimodal evidence, the degree of consistency between the multimodal evidence and the candidate explanatory hypotheses is calculated by weighted fusion of consistency measures of each modality, and the weight of each modality is dynamically adjusted based on its uncertainty measure.

4. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 3, characterized in that, The process of modeling uncertainty propagation between layers includes: calculating the confidence level of the i-th layer, and adjusting the uncertainty level of the (i-1)-th layer by using a preset weighting coefficient.

5. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 1, characterized in that, Dynamically select and execute the corresponding preset output strategy, including the following steps: The overall confidence level of the system, the uncertainty measure of the decision level, and the minimum confidence level of each level are compared with multiple preset thresholds. Based on the comparison results, one of the preset strategies is selected and executed.

6. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 5, characterized in that, The preset strategies include: a high-confidence direct output strategy, an output strategy with uncertainty warnings, and a manual intervention strategy.

7. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 6, characterized in that, Based on the comparison results, one of the preset strategies is selected for execution, including: When the overall confidence level of the system is not lower than the first threshold, the uncertainty measure of the decision level is not higher than the second threshold, and the confidence level of all levels is not lower than the third threshold, the high confidence level direct output strategy is executed. When the high confidence output condition is not met but the overall system confidence is not lower than the fourth threshold, an output strategy with an uncertainty warning is executed. When the overall confidence level of the system falls below the fourth threshold, a manual intervention strategy is implemented.

8. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 1, characterized in that, The multimodal data includes visual data, code dataset log data, and more.

9. The large-scale dynamic optimization method based on multimodal knowledge graphs according to claim 8, characterized in that, Calculate the confidence level for each mode, including: For visual data, the confidence level of the visual data is calculated based on one or more of the following: the probability distribution of behavior recognition, the degree of occlusion in the video, and the similarity of the behavior actions. For code data, the confidence level of the code data is calculated based on one or more of the probability of correct code execution and code complexity. For dataset log data, the confidence level of the dataset log data is calculated based on one or more of the following: the level of detail in the log information description, the internal consistency of the log sequence, and the generalization degree of the log template.

10. A large-scale dynamic optimization system based on multimodal knowledge graphs, characterized in that, include: A processor and a memory, wherein the memory stores computer program instructions that, when executed by the processor, implement the large model dynamic optimization method based on multimodal knowledge graphs according to any one of claims 1-9.

Images