Multimodal fusion analysis method and system for experimental data

Abstract

The present application belongs to the field of multi-modal data processing, and particularly relates to a multi-modal fusion analysis method and system for test data. The method realizes intelligent analysis of real-time multi-dimensional demand data by constructing a bidirectional multi-modal constraint generation model, fusing a multi-modal reasoning generation, a forward logical constraint and a reverse causal constraint sub-model. The forward logical constraint sub-model is based on test standard criteria and historical constraint sequences, constructs a forward constraint hierarchical reasoning correlation node network through entity-relation extraction and graph algorithm, and generates a forward logical constraint set in combination with a deep index algorithm. The reverse causal constraint sub-model constructs a combination conflict value through a modal reasoning accuracy, performs conflict tracing and adjustment in combination with a cross-modal conflict threshold, and generates a reverse causal constraint set. The two constraint sets are fused through a multi-modal reasoning generation sub-model, and finally a target generation demand text meeting a confidence requirement is generated, realizing intelligent analysis of clinical test data.

Description

A method and system for multimodal fusion analysis of experimental data Technical Field

[0001] This invention belongs to the field of multimodal data processing, and in particular relates to a method and system for multimodal fusion analysis of experimental data. Background Technology

[0002] Existing technologies primarily improve the standardization of clinical trial data through structured data storage and access control. For example, Chinese patent application CN105678084A discloses a data processing method and device based on Good Clinical Practice (GCP) for drug clinical trials, a partitioned storage mechanism based on trial item identification, and data security achieved by combining electronic signatures, encryption processing, and dynamic access control. Additionally, it discloses specific physiological parameter analysis based on statistical methods. For instance, Chinese patent CN119272211B discloses a clinical trial data analysis method and system that detects abnormal data by constructing a time-series mutation factor model of blood oxygen saturation and respiratory rate, combined with Chebyshev's inequality. However, existing technologies still have shortcomings: First, anomaly detection only targets local physiological parameters and cannot integrate multi-dimensional clinical indicators to achieve real-time prediction of patient dropout risk and dynamic optimization of enrollment criteria; second, there are many fine-grained feature perspectives corresponding to high-precision data in the clinical field. If only static analysis constraints are applied from one aspect, it is impossible to extract the local time-varying fine-grained features of clinical precision data. Therefore, how to effectively integrate multi-source heterogeneous clinical trial data, dynamically coordinate the conflict between positive logical constraints and reverse causal constraints, and achieve intelligent analysis and decision-making with high confidence and high timeliness based on multi-level indexing strategies and cross-modal resolution mechanisms, in order to solve the problems of data silos, model staticity, low efficiency in constraint conflict resolution, and insufficient interpretability of results in traditional methods, has become one of the key issues in existing research. To this end, this invention provides a multi-modal fusion analysis method and system for experimental data. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a multimodal fusion analysis method and system for experimental data. This method constructs a bidirectional multimodal constraint generation model, integrating multimodal reasoning generation, forward logical constraints, and reverse causal constraint sub-models to achieve intelligent analysis of real-time multidimensional demand data. Specifically, the forward logical constraint sub-model, based on experimental standard criteria and historical constraint sequences, constructs a hierarchical inference network of related nodes using entity-relationship extraction and graph algorithms, and generates a set of forward logical constraints using a deep indexing algorithm. The reverse causal constraint sub-model constructs a combined conflict value based on modal reasoning accuracy, and performs conflict tracing and adjustment using cross-modal conflict thresholds to generate a set of reverse causal constraints. The two constraint sets are fused through the multimodal reasoning generation sub-model to ultimately generate target demand text that meets confidence requirements. This invention effectively improves the accuracy and logical consistency of clinical trial data generation through dynamic conflict resolution strategies, multi-level indexing mechanisms, and cross-modal constraint integration, making it suitable for intelligent multimodal data analysis in complex scenarios.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] The method for multimodal fusion analysis of experimental data includes the following steps:

[0006] Acquire real-time multidimensional demand generation data and a bidirectional multimodal constraint generation model; the bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model;

[0007] The real-time multidimensional demand generation data is input into the bidirectional multimodal constraint generation model. Based on the positive logical constraint set generated by the positive logical constraint sub-model and the negative causal constraint set generated by the negative causal constraint sub-model, the target demand generation text is obtained through the multimodal reasoning generation sub-model.

[0008] The set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence; the set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence with the number of input data modes, respectively.

[0009] Specifically, the construction process of the positive logic constraint sub-model includes:

[0010] Obtain experimental standard criteria information, positive initial logical constraints in the historical generation process, and positive logical constraint information updated in each step of training constraint process, and construct a positive logical constraint sequence;

[0011] Based on the positive logical constraint sequence, the entity-relationship extraction algorithm is used to obtain the constraint keywords and logical constraint relationships between constraint keywords under different modal data, the hierarchical inheritance relationship of constraints under the same modal data, the relationship of frequent interaction constraints between different modal data, the confidence of interaction constraints and the conflict relationship of interaction constraints, and to construct the constraint condition triplet corresponding to each modal data based on the obtained keywords and relationships.

[0012] Based on the constraint triplet graph algorithm corresponding to each modal data, a positive constraint hierarchical reasoning associated node network is obtained;

[0013] A positive logical constraint sub-model is constructed based on a positive constraint hierarchical reasoning associated node network combined with a deep indexing algorithm.

[0014] Specifically, the forward constraint hierarchical reasoning associated node network includes modality parsing nodes, atomic constraint variable node sets, constraint combination nodes, and conflict resolution nodes; each mode corresponds to a constraint combination node and each atomic constraint variable node set in a one-to-one correspondence.

[0015] The modal parsing node calls the corresponding constraint combination node based on the modal vectors in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current primary modal data and secondary modal data;

[0016] The constraint combination node is used to generate the target requirement feature space of each long input in the text generation process according to the target requirements. The atomic constraint variables stored in the atomic constraint variable node set corresponding to the constraint combination node index are indexed and combined.

[0017] Each atomic constraint variable sub-node in the atomic constraint variable node set is used to store a single constraint variable and the historical index frequency of each modality, and each atomic constraint variable sub-node is configured with a version traceability chain to record the change history of the corresponding atomic constraint variable in the test standard criterion information.

[0018] The internal child nodes of the atomic constraint variable node set and the atomic constraint variable node sets corresponding to each mode are connected by a graph attention algorithm, which combines the variable connection relationship constructed by the frequency of two constraint variables simultaneously indexed by the same mode, to obtain a constraint child node network;

[0019] The conflict resolution node is used to resolve intramodal and intermodal constraint conflicts that occur in each constraint combination process of the corresponding constraint combination node using preset resolution rules, so that the constraints after each step of long input resolution meet the preset confidence threshold.

[0020] The preset resolution rules are constructed by a resolution path decision tree built using a preset hierarchical resolution strategy, the priority of constraint variables, preset time sensitivity, and a decision tree algorithm.

[0021] The preset hierarchical resolution strategy includes a first layer: intramodal resolution strategy, a second layer: intermodal resolution strategy, and a third layer: manual intervention.

[0022] Specifically, the construction process of the aforementioned positive constraint hierarchical reasoning associated node network includes:

[0023] Based on the fact that each modal parsing node is a first-level node of the positive constraint hierarchical reasoning association node network, the constraint combination node corresponding one-to-one with the modal parsing node is a second-level node of the positive constraint hierarchical reasoning association node network, the set of atomic constraint variable nodes corresponding to the constraint combination node is a third-level node, and the conflict resolution node is an intermediate node between the second-level node and the set of atomic constraint variable nodes.

[0024] Based on the first-level nodes, second-level nodes, third-level nodes, and intermediate nodes corresponding to each mode, an initial positive constraint hierarchical reasoning associated node network is obtained through graph algorithms;

[0025] Modal mapping connections are constructed based on the correspondence between first-level nodes and second-level nodes, and constraint call connections are constructed based on the call relationship between second-level nodes and third-level nodes.

[0026] Based on the corresponding atomic constraint variable node set pre-stored in the secondary node, the preset index label of each atomic constraint variable sub-node, the frequency of each modal historical index stored in the atomic constraint variable sub-node, and the combined non-conflict confidence of each atomic constraint variable, an index connection is constructed between the secondary node and each atomic constraint variable sub-node.

[0027] Based on the historical constraint conflict resolution information and the frequency of successful conflict resolution between the secondary nodes and intermediate nodes, intramodal conflict resolution connections are constructed. At the same time, based on the combined constraint set information between modes, constraint connections between different modes and intermodal conflict resolution connections are constructed.

[0028] The modal mapping connection, constraint call connection, index connection, intramodal resolution connection, combined constraint connection, and intermodal resolution connection are fed back to the initial positive constraint hierarchical reasoning association node network to obtain the positive constraint hierarchical reasoning association node network;

[0029] Based on the positive constraint hierarchical reasoning associated node network combined with the multi-level indexing strategy and the preset index label of each atomic constraint variable sub-node in the corresponding atomic constraint variable node set, in the pre-training process of the bidirectional multimodal constraint generation model, the positive index training is carried out by the deep indexing algorithm so that the combined constraints obtained for each mode meet the preset forward confidence threshold.

[0030] The multi-level indexing strategy includes a first level: modal index, a second level: semantic index, and a third level: time-sensitive index.

[0031] Specifically, the process of generating a set of positive logic constraints from the positive logic constraint sub-model includes:

[0032] Based on the target requirement feature space and the first-level node, the corresponding second-level node is obtained, and the main modality space and auxiliary modality space corresponding to the data generated at each time point are determined.

[0033] Based on the determined primary modal space and secondary modal space corresponding to each time point, the corresponding primary secondary node, primary intermediate node, secondary secondary node set, and secondary intermediate node set are determined in the called secondary nodes.

[0034] Based on the determined primary secondary nodes and their corresponding primary intermediate nodes and secondary secondary nodes and their corresponding secondary intermediate nodes, and combined with the demand features input at each window step size in the target demand feature space, the corresponding constraint variables are queried through the index connection of each secondary node to obtain the constraint variable set corresponding to each constraint combination node.

[0035] Specifically, the process of generating the positive logic constraint set from the positive logic constraint sub-model also includes:

[0036] Based on the set of constraint variables corresponding to each constraint combination node, combined with the resolution rules pre-stored in the intermediate node corresponding to each calling second-level node and the combination non-conflict confidence obtained by the reverse causal constraint sub-model reasoning, the preset forward confidence threshold and the logical operator confidence, the constraint combination that satisfies the forward confidence threshold corresponding to each window step size requirement feature is obtained, and the applicable scenario is marked by the semantic association features in the input requirement features, and pre-stored in the corresponding second-level node;

[0037] Simultaneously, based on the combined constraint connection between the primary secondary node and the secondary secondary node set, and the intermodal resolution connection between the primary intermediate node and the corresponding secondary intermediate node set, combined with the constraint combination corresponding to each secondary node, the combined constraint set that satisfies the intermodal confidence level corresponding to each window step size requirement feature is obtained.

[0038] Specifically, the process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to each window step size requirement feature includes:

[0039] Set an initial forward confidence threshold. When the combination non-conflict confidence of the constraint combination generated by the second-level node corresponding to the initial input demand feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is determined to be valid, and the combination non-conflict confidence of the initial input demand feature is used as the forward confidence threshold for the constraint combination generated by the demand feature corresponding to the next window step.

[0040] Repeat the above process. When the confidence of the combination of constraints corresponding to the demand feature input at any window step size is less than the corresponding forward confidence threshold, the corresponding constraint combination is determined to be invalid.

[0041] When invalid, the reverse causal constraint sub-model is invoked. Based on the input demand features and the corresponding constraint combination, forward reasoning is performed through the forward constraint hierarchical reasoning association node network to obtain the invalid causal constraint variables.

[0042] Update the current invalid constraint combination based on the invalid causal constraint variables, and obtain the updated valid constraint combination;

[0043] Repeat the above steps to obtain the effective constraint combination corresponding to the step size requirement feature of each modal input window during the target requirement text generation process.

[0044] Specifically, the process of obtaining the set of combined constraints that satisfy the inter-modal confidence level corresponding to each window step size requirement feature includes:

[0045] Based on the requirement features of each window step, all constraint combinations that satisfy the forward confidence threshold are generated. Through the combination constraint connection and intermodal resolution connection, cross-modal conflict resolution is performed on the constraint combination corresponding to the secondary second-level node set and the constraint combination corresponding to the primary second-level node. If the combination conflict value corresponding to the combination constraint set obtained after conflict resolution satisfies the preset cross-modal conflict threshold, then the combination constraint set corresponding to the current window step step requirement feature is determined to be valid.

[0046] If the combined conflict value corresponding to the combined constraint set obtained after conflict resolution does not meet the preset cross-modal conflict threshold, then the constraint variables of the auxiliary second-level node set corresponding to the constraint combination that conflicts with the constraint combination corresponding to the main second-level node are adjusted through the constraint sub-node network until the cross-modal conflict threshold is met.

[0047] Specifically, the construction process of the reverse causal constraint sub-model includes:

[0048] The accuracy of the output requirement text generated by the multimodal reasoning generation sub-model at each step length is used to construct the combined conflict value corresponding to the combined constraint set under the corresponding input window step length.

[0049] Based on the obtained combined conflict value and the cross-modal conflict threshold, cross-modal conflict is determined. If a conflict is found, the process of conflict adjustment between the constraint combination corresponding to the primary second-level node and the constraint combination corresponding to the secondary second-level node set is repeated. The conflict source is traced for the secondary second-level node set that conflicts with the primary second-level node through the forward constraint hierarchical reasoning association node network, based on the corresponding intermediate node. Based on the source tracing result and the constraint sub-node network, the constraint variables that conflict in the source tracing result are adjusted to obtain the combined constraint set after reverse causal adjustment.

[0050] A multimodal fusion analysis system for experimental data includes: a requirements acquisition module and a bidirectional reasoning generation module;

[0051] The demand acquisition module is used to acquire real-time multidimensional demand generation data and bidirectional multimodal constraint generation model;

[0052] The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model.

[0053] The bidirectional reasoning generation module is used to input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model, and obtain the target demand generation text through the multimodal reasoning generation sub-model based on the positive logical constraint set generated by the positive logical constraint sub-model and the negative causal constraint set generated by the negative causal constraint sub-model.

[0054] The set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence; the set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence with the number of input data modes, respectively.

[0055] Compared with the prior art, the beneficial effects of the present invention are:

[0056] This invention addresses the shortcomings of existing technologies by constructing a bidirectional collaborative mechanism between a forward logical constraint sub-model and a reverse causal constraint sub-model. Combined with the dynamic indexing and conflict resolution strategies of the forward constraint hierarchical reasoning association node network, it achieves precise mapping and real-time adjustment of multimodal data constraints. Specifically, the forward constraint hierarchical reasoning association node network, through the synergistic effect of modality parsing nodes, atomic constraint variable node sets, and conflict resolution nodes, can dynamically capture the logical relationships, hierarchical inheritance relationships, and interactive constraint rules among multimodal data. Combined with a multi-level indexing strategy and combined non-conflict confidence assessment, it ensures the forward validity of each step in generating constraints. The reverse causal constraint sub-model, based on cross-modal conflict thresholds, performs conflict tracing and constraint variable adjustment, forming a closed-loop feedback mechanism that effectively solves the dynamic conflict problem among multimodal constraints. This bidirectional multimodal constraint generation model, through the deep integration of forward logical reasoning and reverse causal adjustment, significantly improves the logical consistency, cross-modal collaboration, and dynamic adaptability of the generated text in clinical trial scenarios, providing an efficient and reliable solution for intelligent analysis under complex multimodal data constraints. Attached Figure Description

[0057] Figure 1 is a flowchart of the multimodal fusion analysis method for experimental data in Embodiment 1 of the present invention;

[0058] Figure 2 is a block diagram of the multimodal fusion analysis system for experimental data in Embodiment 2 of the present invention. Detailed Implementation

[0059] Example 1

[0060] Please refer to Figure 1. One embodiment of the present invention provides a multimodal fusion analysis method for experimental data, comprising the following steps:

[0061] S1. Obtain real-time multidimensional demand generation data and a bidirectional multimodal constraint generation model; the bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model;

[0062] S2. Input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model. Based on the positive logical constraint set generated by the positive logical constraint sub-model and the negative causal constraint set generated by the negative causal constraint sub-model, obtain the target demand generation text through the multimodal reasoning generation sub-model.

[0063] The set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence; the set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence with the number of input data modes, respectively.

[0064] Furthermore, the multimodal fusion analysis method for experimental data described in this embodiment is used for data analysis and analysis report generation during clinical trials.

[0065] Furthermore, the construction process of the positive logic constraint sub-model in this embodiment includes:

[0066] Obtain experimental standard criteria information, positive initial logical constraints in the historical generation process, and positive logical constraint information updated in each step of training constraint process, and construct a positive logical constraint sequence;

[0067] Based on the positive logical constraint sequence, the entity-relationship extraction algorithm is used to obtain the constraint keywords and logical constraint relationships between constraint keywords under different modal data, the hierarchical inheritance relationship of constraints under the same modal data, the relationship of frequent interaction constraints between different modal data, the confidence of interaction constraints and the conflict relationship of interaction constraints, and to construct the constraint condition triplet corresponding to each modal data based on the obtained keywords and relationships.

[0068] Furthermore, this embodiment constructs a medical entity relationship system by combining rule-driven and statistical learning. In the entity-relationship extraction stage, based on predefined logical templates, such as condition-action structure parsing medical rules, natural language descriptions are transformed into structured semantic pairs. At the same time, a frequent pattern mining algorithm is used to discover potential association rules from historical data, revealing implicit medical knowledge associations.

[0069] In constructing hierarchical inheritance relationships, this embodiment establishes a multi-level constraint classification tree based on the standard medical terminology ontology, achieving knowledge systematization through parent-child inheritance relationships. Simultaneously, a dynamic merging mechanism is introduced to automatically optimize the hierarchical structure based on the actual application frequency of subclass constraints. When the trigger rate of a subclass rule falls below a set threshold, its logical attributes are moved up to the parent class, thus maintaining the simplicity and adaptability of the classification system. This scheme ensures the rigor of medical logic through rule templates, utilizes statistical methods to uncover implicit relationships, and combines the dynamic optimization mechanism of the ontology hierarchy to form a medical knowledge system that possesses systematicity, interpretability, and self-evolutionary capabilities, providing a structured rule foundation for clinical decision support.

[0070] Based on the constraint triplet graph algorithm corresponding to each modal data, a positive constraint hierarchical reasoning associated node network is obtained;

[0071] A positive logical constraint sub-model is constructed based on a positive constraint hierarchical reasoning associated node network combined with a deep indexing algorithm.

[0072] Furthermore, in this embodiment, the forward constraint hierarchical reasoning associated node network includes modal parsing nodes, atomic constraint variable node sets, constraint combination nodes, and conflict resolution nodes;

[0073] Furthermore, in this embodiment, each mode corresponds to a constraint combination node and a set of atomic constraint variable nodes.

[0074] The modal parsing node calls the corresponding constraint combination node based on the modal vectors in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current primary modal data and secondary modal data;

[0075] Furthermore, the exemplary functional descriptions corresponding to the modal parsing nodes in this embodiment include:

[0076] Based on feature vector identification of input data modality types, a unique modality identifier is generated. Further, in this embodiment, modality types include text, image, and time series. By calculating modality feature similarity and confidence, the primary and secondary modality levels are determined, where the primary modality carries core decision-making information, and secondary modalities provide supplementary verification. Quality assessment analyzes the reliability of the data source in real time. Further, in this embodiment, quality assessment evaluates image clarity, text integrity, and sampling compliance, dynamically adjusting the contribution weight of each modality. Further, taking a clinical diagnostic scenario as an example, when the input includes gene testing reports, medical images, and laboratory time series data, the parsing node prioritizes complete gene text as the primary modality, clear images as auxiliary verification, and incomplete time series data as a secondary reference. This mechanism, through a three-level processing flow of modality identification, primary and secondary decision-making, and quality assessment, constructs a hierarchical data fusion framework, ensuring efficient integration and credibility optimization of multi-source heterogeneous information, and providing an adaptive weighted multimodal analysis foundation for precision medicine decision-making.

[0077] The constraint combination node is used to generate the target requirement feature space of each long input in the text generation process according to the target requirements. The atomic constraint variables stored in the atomic constraint variable node set corresponding to the constraint combination node index are indexed and combined.

[0078] Furthermore, the exemplary functional descriptions corresponding to the constraint combination nodes in this embodiment include:

[0079] Based on the target requirement characteristics, relevant rule nodes are retrieved from the atomic constraint library, and composite clinical pathways are generated by combining them through logical operators;

[0080] It should be noted that the implementation of the constraint combination node function in this embodiment is divided into three stages:

[0081] First, the logic combination generation module filters matching atomic constraints based on semantic relevance, such as gene mutation indicators, imaging risk characteristics, and laboratory indicator thresholds; it constructs an "AND / OR" combination rule chain according to clinical logical relationships. Second, the dynamic confidence calculation module comprehensively considers the credibility of atomic rules, the clinical effectiveness of logical operators, and the historical application success rate, and generates an overall confidence assessment of the combination rules through a multi-factor fusion algorithm. Third, the context adaptation module annotates the rule combinations with scenarios based on medical knowledge ontology, such as emergency care, pediatric medication, and postoperative management, clarifying their applicable scope and execution priority.

[0082] For example, in this embodiment, taking the generation of a tumor chemotherapy regimen as an example, when the treatment needs of an EGFR-mutant patient are input, the system extracts targeted drug recommendation clauses from text-based guidelines, analyzes surgical risk assessment results from imaging data, and generates a composite logical expression by combining blood count indicators as constraints. Finally, it outputs a personalized plan that includes dose adjustment and adjuvant therapy suggestions. This technology achieves intelligent transformation from discrete medical evidence to structured clinical pathways through the synergistic effect of atomic rule retrieval, confidence fusion, and scenario adaptation. It not only ensures the scientific rigor of the plan but also improves the decision-making adaptability in complex medical scenarios through a dynamic weighting mechanism, providing a scalable technical framework for balancing standardization and personalization in clinical diagnosis and treatment.

[0083] Each atomic constraint variable sub-node in the atomic constraint variable node set is used to store a single constraint variable and the historical index frequency of each modality, and each atomic constraint variable sub-node is configured with a version traceability chain to record the change history of the corresponding atomic constraint variable in the test standard criterion information.

[0084] Furthermore, the exemplary functional descriptions corresponding to the atomic constraint variable node set in this embodiment include:

[0085] A dynamic and traceable medical rule knowledge base is constructed by using a set of atomic constraint variable nodes, providing a modular and evolvable logical foundation for clinical decision-making;

[0086] Furthermore, in this embodiment, the core functionality of the atomic constraint variable node set covers four dimensions:

[0087] First, a single constraint storage module solidifies indivisible medical rules (such as drug contraindications and treatment recommendations) in the form of the smallest logical unit, forming structured rule atoms;

[0088] Second, the historical index tracking function records the frequency of rule calls, time distribution, and contextual relationships in clinical scenarios, quantifying the practical application value of the rules;

[0089] Third, the version traceability management module fully records the rule iteration process (such as guideline version updates and institutional standard adjustments) through the change chain tracking mechanism, ensuring that the application of rules is synchronized with the latest medical evidence;

[0090] Fourth, the cross-modal association engine constructs a semantic association network based on graph neural networks, which automatically identifies and quantifies the logical synergistic relationship between different modal rules (such as the joint triggering probability of gene mutation indicators and imaging features).

[0091] In this embodiment, taking targeted therapy rules as an example, the node not only stores the core clause of "recommending specific drugs for EGFR mutations," but also records the high-frequency triggering scenarios of this rule in historical diagnosis and treatment, tracks the adjustment trajectory of drug dosage as safety evidence is updated, and establishes a dynamic weight association with the tumor volume rule of the imaging modality. This design achieves flexible combination of rules through atomic storage, ensures clinical compliance through historical traceability, and enhances the three-dimensionality of decision-making logic through cross-modal association, forming a medical rule management system that combines stability and adaptability, providing interpretable and auditable rule application support for precision medicine.

[0092] The internal child nodes of the atomic constraint variable node set and the atomic constraint variable node sets corresponding to each mode are connected by a graph attention algorithm, which combines the variable connection relationship constructed by the frequency of two constraint variables simultaneously indexed by the same mode, to obtain a constraint child node network.

[0093] Furthermore, in this embodiment, the interaction logic between the modal parsing node and the constraint combination node includes:

[0094] In this embodiment, collaborative decision-making based on multimodal medical data is achieved through the dynamic interaction between modal parsing nodes and constraint combination nodes, specifically as follows:

[0095] First, the modality parsing node identifies the data modality type based on feature vector matching, determines the primary and secondary modality levels and generates quality assessment parameters, and transmits the modality identifier and quality score to the downstream constraint combination node;

[0096] Second, the constraint combination node dynamically adjusts the rule indexing strategy according to the received modality priority. The core treatment rule node triggered by the main modality is loaded first, followed by the verification or supplementary rule nodes associated with the auxiliary modality, forming a hierarchical decision logic flow. For example, when gene testing data is used as the main modality to trigger targeted therapy combination, the synchronously associated imaging data is used as the auxiliary modality to activate the surgical risk assessment node. The two achieve rule collaboration through a semantic association network. In summary, this interaction mechanism ensures that the high-confidence modality dominates the decision direction through priority adjustment driven by quality scores, while the auxiliary modality rules provide verification and correction support. This constructs a multimodal medical decision system with clear primary and secondary elements and dynamic adaptation, effectively improving the accuracy and robustness of solutions in complex clinical scenarios.

[0097] The conflict resolution node is used to resolve intramodal and intermodal constraint conflicts that occur in each constraint combination process of the corresponding constraint combination node using preset resolution rules, so that the constraints after each step of long input resolution meet the preset confidence threshold.

[0098] The preset resolution rules are constructed by a resolution path decision tree built using a preset hierarchical resolution strategy, the priority of constraint variables, preset time sensitivity, and a decision tree algorithm.

[0099] The preset hierarchical resolution strategy includes a first layer: intramodal resolution strategy, a second layer: intermodal resolution strategy, and a third layer: manual intervention.

[0100] Furthermore, the exemplary functional descriptions corresponding to the conflict resolution nodes in this embodiment include:

[0101] The conflict resolution node in this embodiment adopts a three-level progressive resolution architecture, specifically:

[0102] The first layer prioritizes decisions based on the authority of rules within a single modality, ensuring that the latest clinical guidelines and advanced medical standards guide decision-making.

[0103] The second layer achieves cross-modal opinion fusion through a primary-secondary modality weighted voting mechanism. The primary modality weight reflects the advantages of data quality and confidence, while the secondary modality provides key supplementary verification.

[0104] The third layer establishes a manual review channel to initiate expert intervention for high-risk conflicts.

[0105] In terms of conflict tracing, the system uses graph backtracking algorithm to locate the root cause of rule contradictions, and combines reverse causal reasoning to automatically detect missing data or logical loopholes, forming a closed-loop correction mechanism.

[0106] Furthermore, taking the generation of lung cancer chemotherapy regimens as an example in this embodiment, when the text modality recommends targeted drugs based on gene mutation indicators, the imaging modality suggests surgical contraindications, and the time-series modality warns of abnormal drug metabolism, the system first optimizes the initial recommendation in the text modality according to the dosage adjustment rules, then implements dynamic replacement of treatment components through the allocation of primary and secondary modal weights (such as adjusting the chemotherapy regimen based on drug toxicity data), and finally combines reverse analysis to supplement monitoring constraints, outputting a personalized regimen that takes into account both efficacy and safety.

[0107] In summary, this process achieves four core breakthroughs through a three-tiered resolution logic of intramodal rule optimization, cross-modal collaborative voting, and manual review intervention, combined with closed-loop control of conflict tracing and dynamic adjustment: First, it establishes an intramodal self-consistent mechanism driven by rule authority to ensure that single-modal decisions conform to the latest medical evidence; second, it designs a quality-weighted intermodal collaborative model to balance data credibility and clinical comprehensiveness; third, it develops conflict root cause tracing technology based on medical knowledge graphs to improve the interpretability of the decision-making process; and fourth, it constructs a bidirectional decision flow that includes positive solution generation and reverse vulnerability patching to enhance the system's anti-interference capability. This multi-level, adaptive, and closed-loop conflict resolution framework fully leverages the complementary advantages of multimodal data while avoiding mechanical rule stacking through rigorous medical logic constraints. It provides a technical paradigm for intelligent medical decision-making systems that balances scientific rigor, safety, and flexibility, marking a significant evolution of AI-assisted diagnosis and treatment from single rule execution to complex clinical reasoning.

[0108] Furthermore, the construction process of the forward constraint hierarchical reasoning associated node network in this embodiment includes:

[0109] Based on the fact that each modal parsing node is a first-level node of the positive constraint hierarchical reasoning association node network, the constraint combination node corresponding one-to-one with the modal parsing node is a second-level node of the positive constraint hierarchical reasoning association node network, the set of atomic constraint variable nodes corresponding to the constraint combination node is a third-level node, and the conflict resolution node is an intermediate node between the second-level node and the set of atomic constraint variable nodes.

[0110] Based on the first-level nodes, second-level nodes, third-level nodes, and intermediate nodes corresponding to each mode, an initial positive constraint hierarchical reasoning associated node network is obtained through graph algorithms;

[0111] Modal mapping connections are constructed based on the correspondence between first-level nodes and second-level nodes, and constraint call connections are constructed based on the call relationship between second-level nodes and third-level nodes.

[0112] Based on the corresponding atomic constraint variable node set pre-stored in the secondary node, the preset index label of each atomic constraint variable sub-node, the frequency of each modal historical index stored in the atomic constraint variable sub-node, and the combined non-conflict confidence of each atomic constraint variable, an index connection is constructed between the secondary node and each atomic constraint variable sub-node.

[0113] Based on the historical constraint conflict resolution information and the frequency of successful conflict resolution between the secondary nodes and intermediate nodes, intramodal conflict resolution connections are constructed. At the same time, based on the combined constraint set information between modes, combined constraint connections between different modes and intermodal conflict resolution connections are constructed.

[0114] The modal mapping connection, constraint call connection, index connection, intramodal resolution connection, combined constraint connection, and intermodal resolution connection are fed back to the initial positive constraint hierarchical reasoning association node network to obtain the positive constraint hierarchical reasoning association node network;

[0115] Based on the positive constraint hierarchical reasoning associated node network combined with the multi-level indexing strategy and the preset index label of each atomic constraint variable sub-node in the corresponding atomic constraint variable node set, in the pre-training process of the bidirectional multimodal constraint generation model, the positive index training is carried out by the deep indexing algorithm so that the combined constraints obtained for each mode meet the preset forward confidence threshold.

[0116] The multi-level indexing strategy includes a first level: modal index, a second level: semantic index, and a third level: time-sensitive index.

[0117] Furthermore, the positive constraint hierarchical reasoning associated node network in this embodiment achieves precise rule management and conflict resolution in complex clinical scenarios through multi-level node definition and dynamic connection optimization;

[0118] The positive constraint hierarchical reasoning association node network adopts a three-level node system and an intermediate conflict resolution node. Specifically, the first-level modality parsing node is responsible for feature extraction and quality assessment of multi-source data; the second-level constraint combination node realizes the semantic assembly of logical rules; the third-level atomic constraint node stores the minimized medical rule unit and its version traceability information; and the intermediate conflict resolution node carries the hierarchical resolution strategy and historical performance data.

[0119] Furthermore, the construction of the forward constraint hierarchical reasoning associated node network in this embodiment is divided into three stages, specifically:

[0120] First, define node attributes and establish initial connections. Modal parsing nodes combine nodes through feature similarity association constraints, and the latter indexes atomic nodes based on logical dependencies.

[0121] Second, a dynamic optimization mechanism is implemented, which resolves conflicts through authoritative ranking within a modality, weighted voting between modalities, and cross-modal tracing, and adjusts node weights based on historical success rates; finally, a multi-level indexing strategy is deployed, integrating modal tag filtering, medical terminology ontology mapping, and timeliness priority ranking to form an efficient rule-based retrieval system.

[0122] Third, data features are dynamically matched with logical rules through modal mapping connections. For example, text modal nodes are embedded based on gene mutation keywords to trigger targeted therapy combination nodes, and then related atomic constraints are indexed.

[0123] Fourth, the conflict resolution in the hierarchical reasoning network of related nodes under positive constraints adopts a three-level progressive strategy: within a modality, higher-level clinical guidelines are prioritized; between modalities, multi-source evidence is integrated through primary and secondary weight allocation; and complex unresolved conflicts are subject to manual review. A dynamic connection optimization mechanism endows the network with self-learning capabilities, with node weights dynamically adjusted based on the resolution success rate. Simultaneously, graph traversal technology is used to locate the root causes of rule contradictions, achieving closed-loop correction of data sources and constraint logic. A multi-level indexing strategy balances efficiency and accuracy: modality labels are quickly filtered to reduce computational load; semantic similarity expansion enhances rule coverage; and timeliness sorting ensures the priority application of the latest evidence.

[0124] Furthermore, in this embodiment, the process of generating the positive logic constraint set from the positive logic constraint sub-model includes:

[0125] Based on the target requirement feature space and the first-level node, the corresponding second-level node is obtained, and the main modality space and auxiliary modality space corresponding to the data generated at each time point are determined.

[0126] Based on the determined primary modal space and secondary modal space corresponding to each time point, the corresponding primary secondary node, primary intermediate node, secondary secondary node set, and secondary intermediate node set are determined in the called secondary nodes.

[0127] Based on the determined primary secondary nodes and their corresponding primary intermediate nodes and secondary secondary nodes and their corresponding secondary intermediate nodes, and combined with the demand features input at each window step size in the target demand feature space, the corresponding constraint variables are queried through the index connection of each secondary node to obtain the constraint variable set corresponding to each constraint combination node.

[0128] Based on the set of constraint variables corresponding to each constraint combination node, combined with the resolution rules pre-stored in the intermediate node corresponding to each calling second-level node and the combination non-conflict confidence obtained by the reverse causal constraint sub-model reasoning, the preset forward confidence threshold and the logical operator confidence, the constraint combination that satisfies the forward confidence threshold corresponding to each window step size requirement feature is obtained, and the applicable scenario is marked by the semantic association features in the input requirement features, and pre-stored in the corresponding second-level node;

[0129] Simultaneously, based on the combined constraint connection between the primary secondary node and the secondary secondary node set, and the intermodal resolution connection between the primary intermediate node and the corresponding secondary intermediate node set, combined with the constraint combination corresponding to each secondary node, the combined constraint set that satisfies the intermodal confidence level corresponding to each window step size requirement feature is obtained.

[0130] Furthermore, the process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to each window step size requirement feature in this embodiment includes:

[0131] Set an initial forward confidence threshold. When the combination non-conflict confidence of the constraint combination generated by the second-level node corresponding to the initial input demand feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is determined to be valid, and the combination non-conflict confidence of the initial input demand feature is used as the forward confidence threshold for the constraint combination generated by the demand feature corresponding to the next window step.

[0132] Repeat the above process. When the confidence of the combination of constraints corresponding to the demand feature input at any window step size is less than the corresponding forward confidence threshold, the corresponding constraint combination is determined to be invalid.

[0133] When invalid, the reverse causal constraint sub-model is invoked. Based on the input demand features and the corresponding constraint combination, forward reasoning is performed through the forward constraint hierarchical reasoning association node network to obtain the invalid causal constraint variables.

[0134] Update the current invalid constraint combination based on the invalid causal constraint variables, and obtain the updated valid constraint combination;

[0135] Repeat the above steps to obtain the effective constraint combination corresponding to the step size requirement feature of each modal input window during the target requirement text generation process.

[0136] Furthermore, the process of obtaining the set of combined constraints satisfying the inter-modal confidence level corresponding to each window step size requirement feature in this embodiment includes:

[0137] Based on the requirement features of each window step, all constraint combinations that satisfy the forward confidence threshold are generated. Through the combination constraint connection and intermodal resolution connection, cross-modal conflict resolution is performed on the constraint combination corresponding to the secondary second-level node set and the constraint combination corresponding to the primary second-level node. If the combination conflict value corresponding to the combination constraint set obtained after conflict resolution satisfies the preset cross-modal conflict threshold, then the combination constraint set corresponding to the current window step step requirement feature is determined to be valid.

[0138] If the combined conflict value corresponding to the combined constraint set obtained after conflict resolution does not meet the preset cross-modal conflict threshold, then the constraint variables of the auxiliary second-level node set corresponding to the constraint combination that conflicts with the constraint combination corresponding to the main second-level node are adjusted through the constraint sub-node network until the cross-modal conflict threshold is met.

[0139] Furthermore, the construction process of the reverse causal constraint sub-model in this embodiment includes:

[0140] The accuracy of the output requirement text generated by the multimodal reasoning generation sub-model at each step length is used to construct the combined conflict value corresponding to the combined constraint set under the corresponding input window step length.

[0141] Based on the obtained combined conflict value and the cross-modal conflict threshold, cross-modal conflict is determined. If a conflict is found, the process of conflict adjustment between the constraint combination corresponding to the primary second-level node and the constraint combination corresponding to the secondary second-level node set is repeated. The conflict source is traced for the secondary second-level node set that conflicts with the primary second-level node through the forward constraint hierarchical reasoning association node network, based on the corresponding intermediate node. Based on the source tracing result and the constraint sub-node network, the constraint variables that conflict in the source tracing result are adjusted to obtain the combined constraint set after reverse causal adjustment.

[0142] Furthermore, the specific implementation details and examples of the positive logic constraint set generated by the positive logic constraint sub-model in this embodiment include:

[0143] Acquire heterogeneous data such as text, images, and time series data, and deploy specialized feature extraction models to extract features from each to obtain the corresponding modal feature spaces:

[0144] Furthermore, in this embodiment, for the text modality: medical terminology entities and semantic associations are extracted based on a medical pre-trained language model to generate high-dimensional semantic vectors. In this embodiment, the pre-trained language model is BioBERT.

[0145] Furthermore, in this embodiment, for image modalities: 3D convolutional neural networks are used to extract spatial features of lesions and encode them into structured feature vectors.

[0146] Furthermore, in this embodiment, for temporal modalities: dynamic change patterns are captured through a long short-term memory network and transformed into temporal feature vectors.

[0147] Based on semantic weight and confidence evaluation of text, image, and time series feature vectors, the core modality is selected from text, image, and time series as the main decision basis, while the feature importance of secondary modalities is identified to trigger auxiliary constraint nodes, such as anatomical risks in images.

[0148] By connecting a pre-trained medical knowledge graph with an index, the smallest logical unit matching the main-secondary mode is retrieved from the atomic constraint library;

[0149] Extending synonym constraints based on semantic similarity enhances rule coverage, such as the SNOMED CT terminology system;

[0150] A composite rule chain is generated by concatenating atomic constraint variables based on logical operators, such as "gene mutation AND no contraindications → recommended targeted therapy". The logical operators include, but are not limited to, IF-THEN and AND / OR.

[0151] The confidence fusion model is used to evaluate the global credibility of constraint combinations, support dynamic threshold determination, form a hierarchical and interpretable decision rule chain, and ensure the rigor and traceability of medical logic.

[0152] Initialize the confidence threshold for scene adaptation, and dynamically adjust the subsequent threshold standard based on the combined confidence generated within the window step size;

[0153] If the confidence level of the constraint combination meets the standard, the threshold is raised to enhance the rigor of subsequent decisions; if it does not meet the standard, reverse causal analysis is triggered to backtrack along the forward reasoning path to locate the invalid constraint variables, such as the failure to include the effect of abnormal liver function on dosage.

[0154] By replacing constraints, such as replacing prohibited drugs, or supplementing rules, such as adding monitoring indicators, we can iteratively optimize the combination to obtain a combination of constraints that meets the conditions.

[0155] Furthermore, in this embodiment, intramodal resolution resolves single-modal intra-conflicts according to rule authority. Furthermore, the hierarchy of rule authority in this embodiment includes: guidelines > specifications > historical data.

[0156] Furthermore, in this embodiment, the process of tracing back along the forward reasoning path to locate the failed constraint variable includes:

[0157] Based on graph networks, backtracking can locate the root cause of conflicts, such as expired data sources or logical vulnerabilities.

[0158] Dynamically update the data source or rule base to regenerate conflict-free constraint combinations.

[0159] Based on keywords in the input features, scene labels are added to the constraint combination to support personalized solution matching;

[0160] Valid combinations are pre-stored in the constraint combination node, and the trigger frequency and confidence evolution trajectory are recorded.

[0161] The rules are dynamically optimized based on historical usage data, and inefficient or outdated constraints are eliminated.

[0162] In summary, this embodiment constructs a multimodal fusion intelligent medical decision-making system. Through multi-level technical collaboration, it achieves accurate constraint modeling and dynamic optimization in complex scenarios. Specifically: First, the system dynamically allocates weights across modal data based on feature vector matching algorithms. Combining text semantic density, image spatial features, and temporal fluctuation patterns, it constructs a modal priority evaluation model to ensure that high-confidence medical evidence dominates the decision-making direction. Second, it establishes cross-modal semantic associations through graph attention networks, breaking through the limitations of traditional single-modal parsing and achieving deep collaboration of multi-source data.

[0163] Second, the positive constraint reasoning system adopts a three-level node system and a three-stage architecture. Through hierarchical connections between modal parsing nodes, constraint combination nodes, and atomic constraint nodes, combined with a deep indexing algorithm, it achieves efficient rule retrieval and combination. Pre-trained modal mapping connections and a historical trigger weighting mechanism accelerate constraint localization, while a dynamic subclass merging strategy significantly reduces the impact of redundant rules on reasoning efficiency. The conflict resolution module deploys a hierarchical and progressive strategy: single-source conflicts within a modality are resolved based on authority ranking; multi-source evidence is fused between modalities through weighted voting; and a manual review channel is triggered in high-risk scenarios, forming a closed-loop processing flow.

[0164] Third, the system introduces a dynamic confidence threshold mechanism, using continuous evaluation of the window step size to drive adaptive optimization of decision criteria. The reverse causal analysis module traces the source of failure constraints along the forward inference path and dynamically reconstructs rules by combining cross-modal semantic association. The dual drive of timeliness index and semantic index ensures that the rule base is synchronized with the latest medical advancements, while terminology ontology expansion enhances the breadth of constraint retrieval coverage.

[0165] Fourth, the scenario adaptation mechanism dynamically adjusts the application strategy of constraint combinations through semantic tag recognition and contextual feature extraction. In chemotherapy regimen generation, the system matches targeted treatment rules based on individual patient characteristics and provides early warnings of potential complication risks through cross-modal association. This system achieves systematic integration of medical evidence, interpretable and traceable decision-making logic, and intelligent resolution of complex conflicts, providing highly robust and scalable technical support for precision medicine.

[0166] Example 2

[0167] Please refer to Figure 2. Another embodiment of the present invention: a multimodal fusion analysis system for experimental data, comprising: a requirement acquisition module and a bidirectional reasoning generation module;

[0168] The demand acquisition module is used to acquire real-time multidimensional demand generation data and bidirectional multimodal constraint generation model;

[0169] The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model.

[0170] The bidirectional reasoning generation module is used to input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model, and obtain the target demand generation text through the multimodal reasoning generation sub-model based on the positive logical constraint set generated by the positive logical constraint sub-model and the negative causal constraint set generated by the negative causal constraint sub-model.

[0171] The set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence; the set of forward logical constraints and the set of reverse causal constraints are in one-to-one correspondence with the number of input data modes, respectively.

[0172] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the claims. All of these variations are within the protection scope of the present invention.

Claims

1. A multimodal fusion analysis method for experimental data, characterized in that, The steps include: The process involves acquiring real-time multidimensional demand generation data and a bidirectional multimodal constraint generation model. This model includes a multimodal reasoning generation sub-model, a forward logical constraint sub-model, and a reverse causal constraint sub-model. The real-time multidimensional demand generation data is input into the bidirectional multimodal constraint generation model. Based on the forward logical constraint set generated by the forward logical constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model, the target demand generation text is obtained through the multimodal reasoning generation sub-model. The forward logical constraint set and the reverse causal constraint set are in one-to-one correspondence. Each set corresponds to a modality of the input data. The construction process of the forward logical constraint sub-model... The process includes: acquiring experimental standard criteria information, positive initial logical constraints in the historical generation process, and positive logical constraint information updated in each step of training constraint process, and constructing a positive logical constraint sequence; based on the positive logical constraint sequence, using an entity-relationship extraction algorithm, obtaining the constraint keywords corresponding to different modal data and the logical constraint relationships between constraint keywords, the hierarchical inheritance relationship of constraints under the same modal data, the relationship of frequent interaction constraints between different modal data, the confidence of interaction constraints, and the conflict relationship of interaction constraints, and constructing constraint condition triples corresponding to each modal data based on the acquired keywords and relationships; and based on the constraint condition triples corresponding to each modal data combined with a graph algorithm, obtaining the positive... A hierarchical reasoning network with forward constraints is constructed. Based on this network and a deep indexing algorithm, a forward logical constraint sub-model is generated. The network includes modality parsing nodes, atomic constraint variable node sets, constraint combination nodes, and conflict resolution nodes. Each mode's constraint combination node corresponds one-to-one with each atomic constraint variable node set. The modality parsing node calls the corresponding constraint combination node based on the modality vectors in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current primary modality data and secondary modality data. The constraint combination node is used to generate target requirement features at each step of the text generation process based on the target requirements. The constraint space is indexed and combined by calling the atomic constraint variable nodes stored in the atomic constraint variable node set corresponding to the constraint combination node index. Each atomic constraint variable child node in the atomic constraint variable node set is used to store a single constraint variable and the historical index frequency of each modality. Each atomic constraint variable child node is configured with a version traceability chain to record the change history of the corresponding atomic constraint variable in the test standard criterion information. The internal child nodes of the atomic constraint variable node set and the atomic constraint variable node sets corresponding to each modality are connected by a variable connection relationship constructed by combining the frequency of two constraint variables simultaneously indexed by the same modality through a graph attention algorithm to obtain a constraint child node network.The conflict resolution node is used to resolve intra-modal and inter-modal constraint conflicts that occur during each constraint combination process of the corresponding constraint combination node, using preset resolution rules to ensure that the constraints after resolution at each long input step meet a preset confidence threshold. The preset resolution rules are constructed using a preset hierarchical resolution strategy, constraint variable priorities, preset time sensitivity, and a decision tree constructed using a decision tree algorithm to build the resolution path. The preset hierarchical resolution strategy includes a first layer: intra-modal resolution strategy; a second layer: inter-modal resolution strategy; and a third layer: manual intervention.

2. The multimodal fusion analysis method for experimental data as described in claim 1, characterized in that, The construction process of the forward constraint hierarchical reasoning associated node network includes: each modality parsing node is a first-level node of the forward constraint hierarchical reasoning associated node network; the constraint combination node corresponding one-to-one with the modality parsing node is a second-level node of the forward constraint hierarchical reasoning associated node network; the set of atomic constraint variable nodes corresponding to the constraint combination node is a third-level node; and the conflict resolution node is an intermediate node between the second-level node and the set of atomic constraint variable nodes. Based on the first-level node, second-level node, third-level node, and intermediate node corresponding to each modality, an initial forward constraint hierarchical reasoning associated node network is obtained through a graph algorithm. Modality mapping connections are constructed based on the correspondence between first-level nodes and second-level nodes, and constraint call connections are constructed based on the call relationship between second-level nodes and third-level nodes. Based on the pre-stored corresponding atomic constraint variable node set of the second-level node, the preset index label of each atomic constraint variable sub-node, the frequency of each modality historical index stored in the atomic constraint variable sub-node, and the combination non-conflict confidence corresponding to each atomic constraint variable, an index connection is constructed between the second-level node and each atomic constraint variable sub-node. Based on the pre-stored corresponding atomic constraint variable node set of the second-level node... The system establishes a pre-defined index label and historical index frequency for each atomic constraint variable child node, along with the combined non-conflict confidence score corresponding to each atomic constraint variable, and constructs index connections between the secondary nodes and each atomic constraint variable child node. Based on the historical constraint conflict resolution information and successful conflict resolution frequency between the secondary nodes and intermediate nodes, it constructs intra-modal conflict resolution connections. Simultaneously, based on the combined constraint set information between modes, it constructs combined constraint connections and inter-modal conflict resolution connections. Finally, it integrates modal mapping connections, constraint invocation connections, index connections, intra-modal conflict resolution connections, combined constraint connections, and inter-modal conflict resolution connections. The connection is fed back to the initial positive constraint hierarchical inference association node network to obtain the positive constraint hierarchical inference association node network; based on the positive constraint hierarchical inference association node network, combined with the multi-level indexing strategy and the preset index label of each atomic constraint variable child node in the corresponding atomic constraint variable node set, in the pre-training process of the bidirectional multimodal constraint generation model, the forward index training is performed by the deep indexing algorithm so that the combined constraints obtained for each modality meet the preset forward confidence threshold; the multi-level indexing strategy includes the first level: modality index, the second level: semantic index and the third level: time-sensitive index.

3. The multimodal fusion analysis method for experimental data as described in claim 2, characterized in that, The process of generating the positive logical constraint set by the positive logical constraint sub-model includes: obtaining the corresponding secondary nodes based on the target requirement feature space and the first-level nodes, and determining the primary modal space and secondary modal space corresponding to the generated data at each time point; determining the corresponding primary secondary nodes, primary intermediate nodes, secondary secondary node sets, and secondary intermediate node sets in the called secondary nodes based on the determined primary and secondary modal spaces corresponding to the generated data at each time point; and obtaining the constraint variable set corresponding to each constraint combination node by querying the corresponding constraint variables through the index connection of each secondary node, based on the determined primary secondary nodes, primary intermediate nodes, secondary secondary node sets, and secondary intermediate node sets, combined with the requirement features input by each window step size in the target requirement feature space.

4. The multimodal fusion analysis method for experimental data as described in claim 3, characterized in that, The process of generating a positive logic constraint set by the positive logic constraint sub-model further includes: based on the constraint variable set corresponding to each constraint combination node, combined with the resolution rules pre-stored in the intermediate node corresponding to each calling second-level node and the combination non-conflict confidence obtained by the reverse causal constraint sub-model, the preset forward confidence threshold and the logic operator confidence, to obtain the constraint combination that satisfies the forward confidence threshold corresponding to each window step size requirement feature, and to mark the applicable scenarios of the obtained constraint combination by the semantic association features in the input requirement feature, and pre-store it in the corresponding second-level node; at the same time, based on the combination constraint connection between the main second-level node and the auxiliary second-level node set, the inter-modal resolution connection between the main intermediate node and the corresponding auxiliary intermediate node set, combined with the constraint combination corresponding to each second-level node, to obtain the combination constraint set that satisfies the inter-modal confidence corresponding to each window step size requirement feature.

5. The multimodal fusion analysis method for experimental data as described in claim 4, characterized in that, The process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to the requirement feature of each window step includes: setting an initial forward confidence threshold; when the combination non-conflict confidence of the constraint combination generated by the secondary node corresponding to the initial input requirement feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is determined to be valid, and the combination non-conflict confidence of the initial input requirement feature is used as the forward confidence threshold for the constraint combination generated by the requirement feature corresponding to the next window step; repeating the above process, when the combination non-conflict confidence of the constraint combination corresponding to the requirement feature input at any window step is less than the corresponding forward confidence threshold, the corresponding constraint combination is determined to be invalid; when invalid, the reverse causal constraint sub-model is called, and forward inference is performed through the forward constraint hierarchical inference association node network based on the input requirement feature and the corresponding constraint combination to obtain invalid causal constraint variables; the current invalid constraint combination is updated based on the invalid causal constraint variables, and the updated valid constraint combination is obtained; repeating the above steps to obtain the valid constraint combination corresponding to the requirement feature of each modal input window step in the target requirement generation text process.

6. The multimodal fusion analysis method for experimental data as described in claim 5, characterized in that, The process of obtaining the combined constraint set that satisfies the intermodal confidence level corresponding to each window step size requirement feature includes: generating all constraint combinations that satisfy the forward confidence threshold based on each window step size requirement feature; performing cross-modal conflict resolution on the constraint combinations corresponding to the secondary second-level node set and the constraint combinations corresponding to the primary second-level node set through the combined constraint connection and intermodal resolution connection; if the combined conflict value corresponding to the combined constraint set obtained after conflict resolution satisfies the preset cross-modal conflict threshold, then the combined constraint set corresponding to the current window step size requirement feature is determined to be valid; if the combined conflict value corresponding to the combined constraint set obtained after conflict resolution does not satisfy the preset cross-modal conflict threshold, then adjusting the constraint variables of the constraint combinations corresponding to the secondary second-level node set that conflict with the constraint combinations corresponding to the primary second-level node through the constraint sub-node network until the cross-modal conflict threshold is satisfied.

7. The multimodal fusion analysis method for experimental data as described in claim 6, characterized in that, The construction process of the reverse causal constraint sub-model includes: constructing a combined conflict value corresponding to the combined constraint set under the input window step size based on the accuracy of the output requirement text generated by the multimodal reasoning generation sub-model at each step length; determining cross-modal conflict based on the obtained combined conflict value and the cross-modal conflict threshold; if there is a conflict, repeating the process of conflict adjustment between the constraint combination corresponding to the main secondary node and the constraint combination corresponding to the secondary secondary node set; and using the forward constraint hierarchical reasoning associated node network to trace the conflict source of the secondary secondary node set that conflicts with the main secondary node, and adjusting the constraint variables that conflict in the traceability result based on the traceability result and the constraint sub-node network to obtain the combined constraint set after reverse causal adjustment.

8. A multimodal fusion analysis system for experimental data, implemented based on the multimodal fusion analysis method for experimental data according to any one of claims 1-7, characterized in that, include: Requirements gathering module and bidirectional reasoning generation module; The requirement acquisition module is used to acquire real-time multidimensional requirement generation data and a bidirectional multimodal constraint generation model. The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logical constraint sub-model, and a reverse causal constraint sub-model. The bidirectional reasoning generation module is used to input the real-time multidimensional requirement generation data into the bidirectional multimodal constraint generation model, and obtain the target requirement generation text through the multimodal reasoning generation sub-model based on the forward logical constraint set generated by the forward logical constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model. The forward logical constraint set and the reverse causal constraint set correspond one-to-one. The forward logical constraint set and the reverse causal constraint set correspond one-to-one with the number of modalities of the input data.

Images