Multi-modal fusion analysis method and system for test data

Abstract

The invention belongs to the field of multi-modal data processing, and particularly relates to a multi-modal fusion analysis method and system for test data, and the method achieves the intelligent analysis of real-time multi-dimensional demand data through constructing a bidirectional multi-modal constraint generation model and fusing multi-modal reasoning generation, forward logic constraint and reverse causal constraint sub-models. Wherein the forward logic constraint sub-model constructs a forward constraint hierarchical reasoning association node network through entity-relation extraction and a graph algorithm based on a test standard criterion and a historical constraint sequence, and generates a forward logic constraint set in combination with a depth index algorithm; the reverse causal constraint sub-model constructs a combined conflict value through modal reasoning accuracy, performs conflict tracing and adjustment in combination with a cross-modal conflict threshold, and generates a reverse causal constraint set; the two constraint sets are fused through a multi-modal reasoning generation sub-model, a target generation demand text meeting the confidence coefficient requirement is finally generated, and intelligent analysis of clinical test data is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal data processing, and in particular relates to a multimodal fusion analysis method and system for experimental data. Background Art

[0002] Existing technologies mainly improve the standardization of clinical trial data through structured data storage and authority management. For example, the Chinese patent application with publication number CN105678084A discloses a data processing method and equipment based on the Good Concepts for Drug Clinical Trials (GCP), a partitioned storage mechanism based on the test project identification, combined with electronic signatures, encryption processing and dynamic access control to achieve data security, and specific physiological parameter analysis based on statistical methods. For example, the Chinese patent with authorization announcement number CN119272211B discloses a clinical trial data analysis method and system, which realizes abnormal data detection by constructing a time series mutation factor model of blood oxygen saturation and respiratory rate and combining it with Chebyshev inequality. However, the existing technology still has defects: first, anomaly detection only targets local physiological parameters and cannot integrate multi-dimensional clinical indicators to achieve real-time prediction of patient dropout risk and dynamic optimization of inclusion criteria; second, in the clinical field, there are many fine-grained feature angles corresponding to high-precision data. If static analysis constraints are only performed from one aspect, the fine-grained features of clinical precision data in local time-varying cannot be extracted. Therefore, how to effectively integrate multi-source heterogeneous clinical trial data, dynamically coordinate the conflicts between forward logical constraints and reverse causal constraints, and based on multi-level indexing strategies and cross-modal resolution mechanisms, realize high-confidence and high-timeliness intelligent analysis and decision-making to solve the problems of data silos, model staticity, low efficiency of constraint conflict resolution and insufficient interpretability of results in traditional methods has become one of the key issues in existing research. To this end, the present invention provides a multimodal fusion analysis method and system for experimental data. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention proposes a multimodal fusion analysis method and system for test data. The method realizes intelligent analysis of real-time multidimensional demand data by constructing a bidirectional multimodal constraint generation model and integrating multimodal reasoning generation, forward logic constraint and reverse causal constraint sub-models. Among them, the forward logic constraint sub-model is based on the test standard criteria and historical constraint sequence, and constructs a forward constraint hierarchical reasoning association node network through entity-relationship extraction and graph algorithm, and generates a forward logic constraint set in combination with a deep indexing algorithm. The reverse causal constraint sub-model constructs a combined conflict value through modal reasoning accuracy, and traces and adjusts the conflict in combination with the cross-modal conflict threshold to generate a reverse causal constraint set. The two constraint sets are fused through the multimodal reasoning generation sub-model to finally generate a target demand generation text that meets the confidence requirements. The present invention effectively improves the accuracy and logical consistency of clinical trial data generation through dynamic conflict resolution strategy, multi-level indexing mechanism and cross-modal constraint integration, and is suitable for intelligent analysis of multimodal data in complex scenarios.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The multimodal fusion analysis method for experimental data includes the following steps:

[0006] Acquire real-time multi-dimensional demand generation data and a bidirectional multimodal constraint generation model; the bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model;

[0007] Inputting the real-time multi-dimensional demand generation data into a bidirectional multimodal constraint generation model, generating a sub-model through multimodal reasoning based on a forward logic constraint set generated by a forward logic constraint sub-model and a reverse causal constraint set generated by a reverse causal constraint sub-model, and obtaining a target demand generation text;

[0008] The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

[0009] Specifically, the construction process of the forward logic constraint sub-model includes:

[0010] Acquire test standard criteria information, forward initial logic constraint conditions in the history generation process, and forward logic constraint information updated in each step of training constraint process to construct a forward logic constraint sequence;

[0011] Based on the forward logical constraint sequence, the entity-relationship extraction algorithm is used to obtain the corresponding constraint keywords under different modal data and the logical constraint relationship between the constraint keywords, the hierarchical inheritance relationship of the constraints under the same modal data, the frequent item relationship of the interactive constraints between different modal data, the interactive constraint confidence and the interactive constraint conflict relationship, and the constraint condition triple corresponding to each modal data is constructed based on the obtained keywords and relationships;

[0012] Based on the constraint triples corresponding to each modal data combined with the graph algorithm, the forward constraint hierarchical reasoning associated node network is obtained;

[0013] Based on the forward constraint hierarchical reasoning associated node network combined with the deep indexing algorithm, a forward logic constraint sub-model is constructed.

[0014] Specifically, the forward constraint hierarchical reasoning associated node network includes modal analysis nodes, atomic constraint variable node sets, constraint combination nodes, and conflict resolution nodes; the constraint combination node corresponding to each modal corresponds one-to-one to each atomic constraint variable node set;

[0015] The modal analysis node calls the corresponding constraint combination node according to the modal vector in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current main modal data and auxiliary modal data;

[0016] The constraint combination node is used to generate the target requirement feature space input at each step in the text generation process according to the target requirement, and index combination is performed by calling the atomic constraint variable stored in the atomic constraint variable node corresponding to the constraint combination node index;

[0017] Each atomic constraint variable sub-node in the atomic constraint variable node set is used to store a single constraint variable and each modal history index frequency, and each atomic constraint variable sub-node is configured with a version traceability chain for recording the change history of the corresponding atomic constraint variable in the test standard criterion information;

[0018] The internal child nodes of the atomic constraint variable node set are connected to the atomic constraint variable node set corresponding to each modality by using a graph attention algorithm combined with variable connection relationships constructed based on the frequencies corresponding to two constraint variables simultaneously indexed by the same modality to obtain a constraint child node network;

[0019] The conflict resolution node is used to resolve the combination conflicts that occur during each constraint combination process of the corresponding constraint combination node, using preset resolution rules to resolve intra-modal constraint conflicts and inter-modal constraint conflicts, so that the constraints after each step input resolution meet the preset confidence threshold;

[0020] The preset resolution rule is obtained by constructing a resolution path decision tree constructed by a preset hierarchical resolution strategy, the priority of the constraint variables and the preset time sensitivity and a decision tree algorithm;

[0021] The preset hierarchical resolution strategy includes a first layer: an intra-modality resolution strategy, a second layer: an inter-modality resolution strategy, and a third layer: manual intervention.

[0022] Specifically, the process of constructing the above-mentioned forward constrained hierarchical reasoning associated node network includes:

[0023] Based on the fact that each modal analysis node is a first-level node, the constraint combination node corresponding to the constraint combination node is used as a second-level node, the atomic constraint variable node set corresponding to the constraint combination node is used as a third-level node, and the conflict resolution node is used as an intermediate node between the second-level node and the atomic constraint variable node set;

[0024] Based on the first-level nodes, second-level nodes, third-level nodes and intermediate nodes corresponding to each mode, the initial forward constraint hierarchical reasoning associated node network is obtained through graph algorithm;

[0025] A modal mapping connection is constructed based on the corresponding relationship between the first-level nodes and the second-level nodes, and a constraint call connection is constructed based on the calling relationship between the second-level nodes and the third-level nodes;

[0026] Based on the preset index label and historical index frequency of each atomic constraint variable child node in the corresponding atomic constraint variable node set pre-stored in the secondary node and the combined non-conflict confidence corresponding to each atomic constraint variable, an index connection between the secondary node and each atomic constraint variable child node is constructed;

[0027] Based on the historical constraint conflict resolution information and the frequency of successful conflict resolution between the secondary nodes and the intermediate nodes, an intra-modal resolution connection is constructed, and based on the combined constraint set information between the modes, a combined constraint connection between different modes and an inter-modal resolution connection are constructed;

[0028] Feeding back the modal mapping connection, the constraint call connection, the index connection, the intra-modal resolution connection, the combined constraint connection, and the inter-modal resolution connection to the initial forward constraint hierarchical reasoning associated node network to obtain a forward constraint hierarchical reasoning associated node network;

[0029] Based on the forward constraint hierarchical reasoning associated node network combined with a multi-level indexing strategy and a preset index label for each atomic constraint variable subnode in the corresponding atomic constraint variable node set, during the pre-training process of the bidirectional multimodal constraint generation model, forward index training is performed using a deep indexing algorithm so that the combined constraints obtained corresponding to each modality meet a preset forward confidence threshold;

[0030] The multi-level indexing strategy includes the first level: modal indexing, the second level: semantic indexing and the third level: temporal indexing.

[0031] Specifically, the process of generating the forward logic constraint set by the forward logic constraint sub-model includes:

[0032] Based on the target demand feature space and the first-level node, the corresponding second-level node is obtained and the primary modal space and the auxiliary modal space corresponding to the generated data at each time point are determined;

[0033] According to the primary modal space and the secondary modal space corresponding to the data generated at each determined time point, the corresponding primary secondary node and the corresponding primary intermediate node and the secondary secondary node set and the corresponding secondary intermediate node set are determined in the called secondary nodes;

[0034] Based on the determined main secondary nodes and the corresponding main intermediate nodes and auxiliary secondary node sets and the corresponding auxiliary intermediate node sets, combined with the demand characteristics of each window step input in the target demand feature space, the corresponding constraint variables are queried through the index connection corresponding to each secondary node to obtain the constraint variable set corresponding to each constraint combination node.

[0035] Specifically, the process of generating the forward logic constraint set by the forward logic constraint sub-model also includes:

[0036] Based on the constraint variable set corresponding to each constraint combination node, combined with the pre-stored resolution rule of the intermediate node corresponding to each calling secondary node and the combined non-conflict confidence, preset forward confidence threshold and logical operator confidence obtained by inference of the reverse causal constraint sub-model, a constraint combination that meets the forward confidence threshold corresponding to each window step requirement feature is obtained, and the obtained constraint combination is marked for applicable scenarios through the semantic association features in the input requirement features, and pre-stored in the corresponding secondary node;

[0037] At the same time, based on the combined constraint connection between the main secondary node and the auxiliary secondary node set, the inter-modal resolution connection between the main intermediate node and the corresponding auxiliary intermediate node set, and the constraint combination corresponding to each secondary node, a combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature is obtained.

[0038] Specifically, the process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to each window step requirement feature includes:

[0039] Set the initial forward confidence threshold. When the combined non-conflict confidence corresponding to the constraint combination generated by the secondary node corresponding to the initially input demand feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is considered valid, and the combined non-conflict confidence corresponding to the initially input demand feature is used as the forward confidence threshold for the constraint combination generated by the demand feature corresponding to the next window step size.

[0040] Repeat the above process. When the combined non-conflict confidence of the constraint combination corresponding to the demand feature input by any window step is less than the corresponding forward confidence threshold, the corresponding constraint combination is judged to be invalid.

[0041] When it is determined to be invalid, the reverse causal constraint sub-model is called to perform forward reasoning through the forward constraint hierarchical reasoning associated node network according to the input demand characteristics and the corresponding constraint combination to obtain invalid causal constraint variables;

[0042] Update the current invalid constraint combination based on the invalid causal constraint variable and obtain the updated valid constraint combination;

[0043] Repeat the above steps to obtain the effective constraint combination corresponding to the step requirement feature of each modal input window in the target requirement generation text generation process.

[0044] Specifically, the process of obtaining the combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature includes:

[0045] Based on all constraint combinations that meet the forward confidence threshold generated by each window step requirement feature, cross-modal conflict resolution is performed on the constraint combination corresponding to the auxiliary secondary node set and the constraint combination corresponding to the primary secondary node through the combined constraint connection and the inter-modal resolution connection. If the combined conflict value corresponding to the combined constraint set obtained after the conflict resolution meets the preset cross-modal conflict threshold, the combined constraint set corresponding to the current window step requirement feature is determined to be valid;

[0046] If the combined conflict value corresponding to the combined constraint set obtained after the conflict is resolved does not meet the preset cross-modal conflict threshold, the constraint variables of the constraint combination corresponding to the auxiliary secondary node set that conflicts with the constraint combination corresponding to the main secondary node are adjusted through the constraint sub-node network until the cross-modal conflict threshold is met.

[0047] Specifically, the construction process of the reverse causal constraint sub-model includes:

[0048] Constructing a combination conflict value corresponding to the combination constraint set under the corresponding input window step size based on the accuracy rate corresponding to the output requirement text generated by each step size input of the multimodal reasoning generation sub-model;

[0049] Based on the obtained combined conflict value and the cross-modal conflict threshold, a cross-modal conflict judgment is performed. If there is a conflict, the conflict adjustment process between the constraint combination corresponding to the main secondary node and the constraint combination corresponding to the auxiliary secondary node set is repeated, and the auxiliary secondary node set that conflicts with the main secondary node is associated with the node network through the forward constraint hierarchical reasoning. The conflict is traced according to the corresponding intermediate nodes, and the constraint variables in conflict in the tracing result are adjusted according to the tracing result in combination with the constraint sub-node network to obtain the combined constraint set after reverse causal adjustment.

[0050] A multimodal fusion analysis system for test data, including: a demand acquisition module and a bidirectional reasoning generation module;

[0051] The demand acquisition module is used to obtain real-time multi-dimensional demand generation data and a bidirectional multi-modal constraint generation model;

[0052] The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model and a reverse causal constraint sub-model;

[0053] The bidirectional reasoning generation module is used to input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model, and obtain the target demand generation text through the multimodal reasoning generation sub-model based on the forward logic constraint set generated by the forward logic constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model;

[0054] The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] In response to the deficiencies of the prior art, the present invention achieves precise mapping and real-time adjustment of multimodal data constraints by constructing a bidirectional collaborative mechanism between a forward logic constraint sub-model and a reverse causal constraint sub-model, and combining the dynamic indexing and conflict resolution strategy of the forward constraint hierarchical reasoning associated node network. Specifically, the forward constraint hierarchical reasoning associated node network can dynamically capture the logical relationships, hierarchical inheritance relationships and interactive constraint laws between multimodal data through the collaborative action of modal parsing nodes, atomic constraint variable node sets and conflict resolution nodes, and ensure the forward validity of each step of constraint generation by combining multi-level indexing strategy and combined non-conflict confidence assessment. The reverse causal constraint sub-model performs conflict tracing and constraint variable adjustment based on cross-modal conflict thresholds, forming a closed-loop feedback mechanism, which effectively resolves the dynamic conflict problem between multimodal constraints. The bidirectional multimodal constraint generation model significantly improves the logical consistency, cross-modal collaboration and dynamic adaptability of demand generation text in clinical trial scenarios through the deep integration of forward logic reasoning and reverse causal adjustment, providing an efficient and reliable solution for intelligent analysis under complex multimodal data constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a multimodal fusion analysis method for test data according to Example 1 of the present invention;

[0058] Figure 2 This is a module diagram of the multimodal fusion analysis system for experimental data in Example 2 of the present invention. DETAILED DESCRIPTION

[0059] Example 1

[0060] See also Figure 1 The present invention provides an embodiment of a multimodal fusion analysis method for test data, comprising the following steps:

[0061] S1. Acquire real-time multi-dimensional demand generation data and a bidirectional multimodal constraint generation model; the bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model;

[0062] S2. Input the real-time multi-dimensional demand generation data into the bidirectional multimodal constraint generation model, generate a sub-model based on the forward logic constraint set generated by the forward logic constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model through multimodal reasoning, and obtain the target demand generation text.

[0063] The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

[0064] Furthermore, the multimodal fusion analysis method for experimental data in this embodiment is used for data analysis and analysis report generation during clinical trials.

[0065] Furthermore, the process of constructing the forward logic constraint sub-model in this embodiment includes:

[0066] Acquire test standard criteria information, forward initial logic constraint conditions in the history generation process, and forward logic constraint information updated in each step of training constraint process to construct a forward logic constraint sequence;

[0067] Based on the forward logical constraint sequence, the entity-relationship extraction algorithm is used to obtain the corresponding constraint keywords under different modal data and the logical constraint relationship between the constraint keywords, the hierarchical inheritance relationship of the constraints under the same modal data, the frequent item relationship of the interactive constraints between different modal data, the interactive constraint confidence and the interactive constraint conflict relationship, and the constraint condition triple corresponding to each modal data is constructed based on the obtained keywords and relationships;

[0068] Furthermore, this embodiment constructs a medical entity relationship system by combining rule-driven and statistical learning. In the entity-relationship extraction phase, medical rules are parsed based on predefined logical templates, such as condition-action structures, to convert natural language descriptions into structured semantic pairs. At the same time, a frequent pattern mining algorithm is used to discover potential association rules from historical data, revealing implicit medical knowledge associations.

[0069] In the construction of hierarchical inheritance relationships, this embodiment relies on the standard medical terminology ontology to establish a multi-level constraint classification tree, and realizes the systematization of knowledge through the inheritance relationship between parent and subclass. At the same time, a dynamic merging mechanism is introduced to automatically optimize the hierarchical structure according to the actual application frequency of subclass constraints. When the subclass rule trigger rate is lower than the set threshold, its logical attributes are moved up to the upper parent class, thereby maintaining the simplicity and adaptability of the classification system. This solution ensures the rigor of medical logic through rule templates, uses statistical methods to mine implicit associations, and combines the dynamic optimization mechanism of the ontology level to form a medical knowledge system that is both systematic, explainable and self-evolving, providing a structured rule basis for clinical decision support.

[0070] Based on the constraint triples corresponding to each modal data combined with the graph algorithm, the forward constraint hierarchical reasoning associated node network is obtained;

[0071] Based on the forward constraint hierarchical reasoning associated node network combined with the deep indexing algorithm, a forward logic constraint sub-model is constructed.

[0072] Furthermore, in this embodiment, the forward constraint hierarchical reasoning associated node network includes a modal analysis node, an atomic constraint variable node set, a constraint combination node, and a conflict resolution node;

[0073] Furthermore, in this embodiment, the constraint combination node corresponding to each mode corresponds one-to-one to each atomic constraint variable node set.

[0074] The modal analysis node calls the corresponding constraint combination node according to the modal vector in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current main modal data and auxiliary modal data;

[0075] Furthermore, exemplary functional descriptions corresponding to the modal analysis nodes in this embodiment include:

[0076] Based on the feature vector, the modality type of the input data is identified and a unique modality identifier is generated. Furthermore, in this embodiment, the modality types include text, image, and time series; by calculating the modality feature similarity and confidence, the primary and secondary modal hierarchy is determined, wherein the primary modality carries the core decision information and the secondary modality provides supplementary verification; the quality assessment analyzes the reliability of the data source in real time. Furthermore, in this embodiment, the quality assessment is evaluated from three aspects: image clarity, text completeness, and sampling compliance, and the contribution weight of each modality is dynamically adjusted; further, in this embodiment, taking the clinical diagnosis scenario as an example, when the input includes genetic test reports, medical images and laboratory time series data, the parsing node preferentially selects the genetic text with complete information as the primary modality and the clear image as the auxiliary verification, and the time series data with incomplete sampling is downgraded to a secondary reference; this mechanism constructs a hierarchical data fusion framework through a three-level processing flow of modality identification, primary and secondary decision-making, and quality assessment, to ensure the efficient integration and credibility optimization of multi-source heterogeneous information, and provide a multi-modal analysis basis with adaptive weights for precision medical decision-making.

[0077] The constraint combination node is used to generate the target requirement feature space input at each step in the text generation process according to the target requirement, and index combination is performed by calling the atomic constraint variable stored in the atomic constraint variable node corresponding to the constraint combination node index;

[0078] Furthermore, exemplary functional descriptions corresponding to the constraint combination node in this embodiment include:

[0079] Relevant rule nodes are retrieved from the atomic constraint library based on target requirement characteristics, and composite clinical pathways are generated through logical operator combination.

[0080] It should be noted that the implementation of the constraint combination node function in this embodiment is divided into three stages:

[0081] First, the logical combination generation module screens matching atomic constraints based on semantic relevance, such as gene mutation indicators, imaging risk characteristics, laboratory indicator thresholds, etc.; and constructs an "and / or" combination rule chain according to clinical logical relationships; second, the dynamic confidence calculation module comprehensively considers the credibility of atomic rules, the clinical effectiveness of logical operators, and the historical application success rate, and generates an overall confidence assessment of the combination rules through a multi-factor fusion algorithm; third, the context adaptation module annotates the rule combinations based on medical knowledge ontology, such as emergency first aid, pediatric medication, postoperative management, etc., to clarify their scope of application and execution priority.

[0082] For example, in this embodiment, taking the generation of tumor chemotherapy regimen as an example, when the treatment needs of patients with EGFR mutations are input, the system extracts targeted drug recommendation clauses from the text-based guideline rules, parses the surgical risk assessment results from the imaging data, and generates a composite logical expression in combination with the blood index constraints, and finally outputs a personalized regimen including dose adjustment and adjuvant treatment recommendations; this technology realizes the intelligent transformation from discrete medical evidence to structured clinical pathways through the synergy of atomic rule retrieval, confidence fusion and scenario adaptation, which not only ensures the scientific rigor of the regimen, but also improves the decision-making adaptability of complex medical scenarios through a dynamic weight mechanism, providing a scalable technical framework for the balance between standardization and personalization of clinical diagnosis and treatment.

[0083] Each atomic constraint variable sub-node in the atomic constraint variable node set is used to store a single constraint variable and each modal history index frequency, and each atomic constraint variable sub-node is configured with a version traceability chain for recording the change history of the corresponding atomic constraint variable in the test standard criterion information;

[0084] Furthermore, in this embodiment, exemplary functional descriptions corresponding to the atomic constraint variable node set include:

[0085] A dynamic and traceable medical rule knowledge base is constructed through a set of atomic constraint variable nodes, providing a modular and evolvable logical basis for clinical decision-making;

[0086] Furthermore, in this embodiment, the core functions of the atomic constraint variable node set cover four dimensions:

[0087] First, a single constraint storage module solidifies inseparable medical rules (such as drug contraindications and treatment recommendations) in the form of minimum logical units, forming structured rule atoms.

[0088] Second, the historical index tracking function records the frequency, time distribution, and contextual association of rule calls in clinical scenarios, quantifying the actual application value of the rules;

[0089] Third, the version traceability management module fully records the rule iteration process (such as guideline version updates and institutional specification adjustments) through the change chain tracking mechanism, ensuring that the application of rules is synchronized with the latest medical evidence;

[0090] Fourth, the cross-modal association engine builds a semantic association network based on graph neural networks, automatically identifying and quantifying the logical synergistic relationship between different modal rules (such as the joint triggering probability of gene mutation indicators and imaging features).

[0091] In this embodiment, taking the targeted therapy rule as an example, the node not only stores the core clause of "recommending specific drugs for EGFR mutations", but also records the high-frequency triggering scenarios of this rule in historical diagnosis and treatment, tracks the adjustment trajectory of drug dosage as safety evidence is updated, and establishes a dynamic weight association with the tumor volume rule of the imaging modality; this design achieves flexible combination of rules through atomic storage, ensures clinical compliance with historical traceability, and enhances the three-dimensionality of decision-making logic through cross-modal association, forming a medical rule management system that is both stable and adaptable, providing explainable and auditable rule application support for precision medicine.

[0092] The internal child nodes of the atomic constraint variable node set are connected to the atomic constraint variable node set corresponding to each mode by using a graph attention algorithm combined with a variable connection relationship constructed by the frequencies corresponding to two constraint variables simultaneously indexed by the same mode to obtain a constraint child node network;

[0093] Furthermore, in this embodiment, the interaction logic of the modal analysis node → constraint combination node includes:

[0094] In this embodiment, collaborative decision-making of multimodal medical data is achieved through dynamic interaction between modal analysis nodes and constraint combination nodes, specifically:

[0095] First, the modal analysis node identifies the data modal type based on feature vector matching, determines the primary and secondary modal hierarchy, generates quality assessment parameters, and passes the modal identification and quality score to the downstream constraint combination node;

[0096] Second, the constraint combination node dynamically adjusts the rule indexing strategy according to the received modality priority. The core treatment rule nodes triggered by the main modality are loaded first, followed by the verification or supplementary rule nodes associated with the auxiliary modality, forming a hierarchical decision-making logic flow. For example, when genetic testing data is used as the main modality to trigger the targeted treatment combination, the synchronously associated imaging data is used as the auxiliary modality to activate the surgical risk assessment node. The two realize rule collaboration through the semantic association network. In summary, this interactive mechanism ensures that high-credibility modalities dominate the decision-making direction through quality score-driven priority adjustment, and the auxiliary modality rules provide verification and correction support, constructing a multi-modal medical decision-making system with clear priorities and dynamic adaptation, effectively improving the accuracy and robustness of solutions in complex clinical scenarios.

[0097] The conflict resolution node is used to resolve the combination conflicts that occur during each constraint combination process of the corresponding constraint combination node, using preset resolution rules to resolve intra-modal constraint conflicts and inter-modal constraint conflicts, so that the constraints after each step input resolution meet the preset confidence threshold;

[0098] The preset resolution rule is obtained by constructing a resolution path decision tree constructed by a preset hierarchical resolution strategy, the priority of the constraint variables and the preset time sensitivity and a decision tree algorithm;

[0099] The preset hierarchical resolution strategy includes a first layer: an intra-modality resolution strategy, a second layer: an inter-modality resolution strategy, and a third layer: manual intervention.

[0100] Furthermore, exemplary functional descriptions of the conflict resolution nodes in this embodiment include:

[0101] The conflict resolution node in this embodiment adopts a three-level progressive resolution architecture, specifically:

[0102] First, prioritize within a single modality based on the authority of rules, ensuring that the latest clinical guidelines and advanced medical standards guide decision-making.

[0103] The second layer achieves cross-modal opinion fusion through a primary-secondary modality weighted voting mechanism. The primary modality weight reflects the data quality and confidence advantage, and the secondary modality provides key supplementary verification.

[0104] The third level is to establish a manual review channel to initiate expert intervention in conflicts involving high risks.

[0105] In terms of conflict tracing, the system uses a graph backtracking algorithm to locate the root cause of rule contradictions, and combines reverse causal reasoning to automatically detect data missing or logical loopholes, forming a closed-loop correction mechanism.

[0106] Furthermore, in this embodiment, taking the generation of chemotherapy regimens for lung cancer as an example, when the gene mutation indication in the text modality recommends a targeted drug, the imaging modality indicates surgical contraindications, and the time series modality warns of abnormal drug metabolism, the system first optimizes the initial recommendation within the text modality based on the dose adjustment rules. It then implements dynamic replacement of treatment components (e.g., adjusting chemotherapy regimens based on drug toxicity data) through the weight distribution of primary and secondary modalities. Finally, it combines reverse analysis with supplementary monitoring constraints to output a personalized regimen that balances efficacy and safety.

[0107] In summary, this process achieved four core breakthroughs through a three-level conflict resolution logic consisting of intra-modal rule optimization, cross-modal collaborative voting, and manual review intervention, combined with closed-loop control of conflict tracing and dynamic adjustment. First, a self-consistent intra-modal mechanism driven by rule authority was established to ensure that single-modal decisions were consistent with the latest medical evidence. Second, a quality-weighted inter-modal collaboration model was designed to balance data credibility and clinical comprehensiveness. Third, a conflict root cause tracing technology based on a medical knowledge graph was developed to improve the interpretability of the decision-making process. Fourth, a two-way decision flow consisting of forward solution generation and reverse vulnerability patching was constructed to enhance the system's anti-interference ability. This multi-level, adaptive, closed-loop conflict resolution framework fully leverages the complementary advantages of multimodal data and avoids mechanical rule stacking through rigorous medical logic constraints. It provides a technical paradigm for intelligent medical decision-making systems that balances scientificity, security, and flexibility, marking an important evolution of artificial intelligence-assisted diagnosis and treatment from single rule execution to complex clinical reasoning.

[0108] Furthermore, the process of constructing the forward constrained hierarchical reasoning associated node network in this embodiment includes:

[0109] Based on the fact that each modal analysis node is a first-level node, the constraint combination node corresponding to the constraint combination node is used as a second-level node, the atomic constraint variable node set corresponding to the constraint combination node is used as a third-level node, and the conflict resolution node is used as an intermediate node between the second-level node and the atomic constraint variable node set;

[0110] Based on the first-level nodes, second-level nodes, third-level nodes and intermediate nodes corresponding to each mode, the initial forward constraint hierarchical reasoning associated node network is obtained through graph algorithm;

[0111] A modal mapping connection is constructed based on the corresponding relationship between the first-level nodes and the second-level nodes, and a constraint call connection is constructed based on the calling relationship between the second-level nodes and the third-level nodes;

[0112] Based on the preset index label and historical index frequency of each atomic constraint variable child node in the corresponding atomic constraint variable node set pre-stored in the secondary node and the combined non-conflict confidence corresponding to each atomic constraint variable, an index connection between the secondary node and each atomic constraint variable child node is constructed;

[0113] Based on the historical constraint conflict resolution information and the frequency of successful conflict resolution between the secondary nodes and the intermediate nodes, an intra-modal resolution connection is constructed, and based on the combined constraint set information between the modes, a combined constraint connection and an inter-modal resolution connection between different modes are constructed;

[0114] Feeding back the modal mapping connection, the constraint call connection, the index connection, the intra-modal resolution connection, the combined constraint connection, and the inter-modal resolution connection to the initial forward constraint hierarchical reasoning associated node network to obtain a forward constraint hierarchical reasoning associated node network;

[0115] Based on the forward constraint hierarchical reasoning associated node network combined with a multi-level indexing strategy and a preset index label for each atomic constraint variable subnode in the corresponding atomic constraint variable node set, during the pre-training process of the bidirectional multimodal constraint generation model, forward index training is performed using a deep indexing algorithm so that the combined constraints obtained corresponding to each modality meet a preset forward confidence threshold;

[0116] The multi-level indexing strategy includes the first level: modal indexing, the second level: semantic indexing and the third level: temporal indexing.

[0117] Furthermore, the forward-constrained hierarchical reasoning associated node network in this embodiment achieves precise rule management and conflict resolution in complex clinical scenarios through multi-level node definition and dynamic connection optimization;

[0118] The forward constraint hierarchical reasoning association node network adopts a three-level node system and intermediate conflict resolution nodes. Specifically, the first-level modal analysis node is responsible for feature extraction and quality assessment of multi-source data; the second-level constraint combination node realizes the semantic assembly of logical rules; the third-level atomic constraint node stores the minimized medical rule units and their version traceability information; the intermediate conflict resolution node carries the hierarchical resolution strategy and historical performance data;

[0119] Furthermore, the construction of the forward constrained hierarchical reasoning associated node network in this embodiment is divided into three stages, specifically:

[0120] First, define node attributes and establish initial connections. Modal parsing nodes combine nodes through feature similarity association constraints, which index atomic nodes based on logical dependencies.

[0121] Second, a dynamic optimization mechanism was implemented to resolve conflicts through intra-modality authority ranking, inter-modality weighted voting, and cross-modality traceability, and node weights were adjusted based on historical success rates. Finally, a multi-level indexing strategy was deployed, integrating modality label filtering, medical term ontology mapping, and timeliness priority sorting to form an efficient rule-based retrieval system.

[0122] Third, data features are dynamically matched with logical rules through modal mapping connections. For example, a text modal node triggers a targeted therapy combination node based on gene mutation keyword embedding, and then indexes related atomic constraints.

[0123] Fourth, conflict resolution within the forward-constrained hierarchical reasoning associated node network utilizes a three-level progressive strategy: prioritizing high-level clinical guidelines within modalities, integrating multi-source evidence across modalities through primary and secondary weight allocation, and initiating manual review for complex unresolved conflicts. A dynamic connection optimization mechanism empowers the network with self-learning capabilities, dynamically adjusting node weights based on the resolution success rate. Graph traversal techniques simultaneously locate the root causes of rule conflicts, enabling closed-loop correction of data sources and constraint logic. A multi-level indexing strategy balances efficiency and accuracy, with rapid filtering of modality labels to reduce computational load, semantic similarity-based expansion to enhance rule coverage, and timeliness-based sorting to prioritize the latest evidence.

[0124] Furthermore, the process of generating the forward logic constraint set by the forward logic constraint sub-model in this embodiment includes:

[0125] Based on the target demand feature space and the first-level node, the corresponding second-level node is obtained and the primary modal space and the auxiliary modal space corresponding to the generated data at each time point are determined;

[0126] According to the primary modal space and the secondary modal space corresponding to the data generated at each determined time point, the corresponding primary secondary node and the corresponding primary intermediate node and the secondary secondary node set and the corresponding secondary intermediate node set are determined in the called secondary nodes;

[0127] Based on the determined main secondary nodes and the corresponding main intermediate nodes and auxiliary secondary node sets and the corresponding auxiliary intermediate node sets, combined with the demand characteristics of each window step input in the target demand feature space, the corresponding constraint variables are queried through the index connection corresponding to each secondary node to obtain the constraint variable set corresponding to each constraint combination node.

[0128] Based on the constraint variable set corresponding to each constraint combination node, combined with the pre-stored resolution rule of the intermediate node corresponding to each calling secondary node and the combined non-conflict confidence, preset forward confidence threshold and logical operator confidence obtained by inference of the reverse causal constraint sub-model, a constraint combination that meets the forward confidence threshold corresponding to each window step requirement feature is obtained, and the obtained constraint combination is marked for applicable scenarios through the semantic association features in the input requirement features, and pre-stored in the corresponding secondary node;

[0129] At the same time, based on the combined constraint connection between the main secondary node and the auxiliary secondary node set, the inter-modal resolution connection between the main intermediate node and the corresponding auxiliary intermediate node set, and the constraint combination corresponding to each secondary node, a combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature is obtained.

[0130] Furthermore, in this embodiment, the process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to each window step requirement feature includes:

[0131] Set the initial forward confidence threshold. When the combined non-conflict confidence corresponding to the constraint combination generated by the secondary node corresponding to the initially input demand feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is considered valid, and the combined non-conflict confidence corresponding to the initially input demand feature is used as the forward confidence threshold for the constraint combination generated by the demand feature corresponding to the next window step size.

[0132] Repeat the above process. When the combined non-conflict confidence of the constraint combination corresponding to the demand feature input by any window step is less than the corresponding forward confidence threshold, the corresponding constraint combination is judged to be invalid.

[0133] When it is determined to be invalid, the reverse causal constraint sub-model is called to perform forward reasoning through the forward constraint hierarchical reasoning associated node network according to the input demand characteristics and the corresponding constraint combination to obtain invalid causal constraint variables;

[0134] Update the current invalid constraint combination based on the invalid causal constraint variable and obtain the updated valid constraint combination;

[0135] Repeat the above steps to obtain the effective constraint combination corresponding to the step requirement feature of each modal input window in the target requirement generation text generation process.

[0136] Furthermore, in this embodiment, the process of obtaining the combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature includes:

[0137] Based on all constraint combinations that meet the forward confidence threshold generated by each window step requirement feature, cross-modal conflict resolution is performed on the constraint combination corresponding to the auxiliary secondary node set and the constraint combination corresponding to the primary secondary node through the combined constraint connection and the inter-modal resolution connection. If the combined conflict value corresponding to the combined constraint set obtained after the conflict resolution meets the preset cross-modal conflict threshold, the combined constraint set corresponding to the current window step requirement feature is determined to be valid;

[0138] If the combined conflict value corresponding to the combined constraint set obtained after the conflict is resolved does not meet the preset cross-modal conflict threshold, the constraint variables of the constraint combination corresponding to the auxiliary secondary node set that conflicts with the constraint combination corresponding to the main secondary node are adjusted through the constraint sub-node network until the cross-modal conflict threshold is met.

[0139] Furthermore, the construction process of the reverse causal constraint sub-model in this embodiment includes:

[0140] Constructing a combination conflict value corresponding to the combination constraint set under the corresponding input window step size based on the accuracy rate corresponding to the output requirement text generated by each step size input of the multimodal reasoning generation sub-model;

[0141] Based on the obtained combined conflict value and the cross-modal conflict threshold, a cross-modal conflict judgment is performed. If there is a conflict, the conflict adjustment process between the constraint combination corresponding to the main secondary node and the constraint combination corresponding to the auxiliary secondary node set is repeated, and the auxiliary secondary node set that conflicts with the main secondary node is associated with the node network through the forward constraint hierarchical reasoning. The conflict is traced according to the corresponding intermediate nodes, and the constraint variables in conflict in the tracing result are adjusted according to the tracing result in combination with the constraint sub-node network to obtain the combined constraint set after reverse causal adjustment.

[0142] Furthermore, the specific implementation details and examples of the forward logic constraint set generated by the forward logic constraint sub-model in this embodiment include:

[0143] Obtain heterogeneous data such as text, images, and time series data, deploy specialized feature extraction models for each, and obtain the corresponding modal feature space:

[0144] Furthermore, in this embodiment, for text modality: medical term entities and semantic associations are extracted based on a pre-trained medical language model to generate a high-dimensional semantic vector. The pre-trained language model in this embodiment is BioBERT;

[0145] Furthermore, in this embodiment, for the imaging modality: a 3D convolutional neural network is used to extract the spatial features of the lesion and encode them into a structured feature vector.

[0146] Furthermore, in this embodiment, for the time series mode: the dynamic change rules are captured through the long short-term memory network and converted into a time series feature vector.

[0147] Based on the semantic weight and confidence evaluation of text, image, and time series feature vectors, the core modality is selected from text, image, and time series as the main decision basis. At the same time, the feature importance of secondary modalities is identified to trigger auxiliary constraint nodes, such as anatomical risks in images;

[0148] By connecting the pre-trained medical knowledge graph with the index, the smallest logical unit matching the primary and secondary modalities is retrieved from the atomic constraint library;

[0149] Expand synonym constraints based on semantic similarity to enhance rule coverage, such as the SNOMED CT terminology system;

[0150] Combine atomic constraint variables based on logical operators to generate a composite rule chain, such as "gene mutation AND no contraindications → recommended targeted therapy." Logical operators include but are not limited to IF-THEN, AND / OR.

[0151] The global credibility of the constraint combination is evaluated through the confidence fusion model, dynamic threshold judgment is supported, and a hierarchical and explainable decision rule chain is formed to ensure the rigor and traceability of medical logic.

[0152] Initialize the confidence threshold for scene adaptation and dynamically adjust subsequent threshold standards based on the combined confidence generated within the window step;

[0153] If the confidence level of the constraint combination meets the standard, the threshold is raised to enhance the rigor of subsequent decisions; if it does not meet the standard, reverse causal analysis is triggered, and the failed constraint variables are traced back along the forward reasoning path, such as not including the impact of abnormal liver function on dosage.

[0154] By replacing constraints, such as replacing contraindicated drugs, or supplementing rules, such as adding new monitoring indicators, iteratively optimizing the combination to obtain a constraint combination that meets the conditions;

[0155] Furthermore, in this embodiment, intra-modal resolution is to resolve conflicts within a single modality according to the authority of rules. Furthermore, the hierarchy of the authority of rules in this embodiment includes: guidelines > specifications > historical data.

[0156] Furthermore, in this embodiment, the process of backtracking and locating the invalid constraint variable along the forward reasoning path includes:

[0157] Locate the root cause of conflicts based on graph network backtracking, such as outdated data sources or logic vulnerabilities;

[0158] Dynamically update the data source or rule base to regenerate conflict-free constraint combinations.

[0159] Based on the keywords in the input features, scenario tags are added to the constraint combinations to support personalized solution matching;

[0160] Pre-store the valid combination to the constraint combination node and record the trigger frequency and confidence evolution trajectory;

[0161] Dynamically optimize rule weights based on historical usage data to eliminate inefficient or outdated constraints.

[0162] In summary, this embodiment constructs a multimodal fusion intelligent medical decision-making system, and realizes precise constraint modeling and dynamic optimization in complex scenarios through the collaboration of multi-level technologies. Specifically: First, the system realizes the dynamic weight allocation of cross-modal data based on the feature vector matching algorithm, and combines the text semantic density, image spatial characteristics and time series fluctuation laws to construct a modality priority evaluation model to ensure that high-confidence medical evidence dominates the decision-making direction; through the graph attention network, cross-modal semantic associations are established, breaking through the limitations of traditional single-modal analysis and realizing deep collaboration of multi-source data.

[0163] Second, the forward constraint reasoning system employs a three-level node system and a three-stage architecture. Through hierarchical connections between modal parsing nodes, constraint combination nodes, and atomic constraint nodes, combined with a deep indexing algorithm, efficient rule retrieval and combination are achieved. Pre-trained modal mapping connections and a historical trigger weighting mechanism accelerate constraint localization, while a dynamic subclass merging strategy significantly reduces the impact of redundant rules on reasoning efficiency. The conflict resolution module deploys a hierarchical and progressive strategy: single-source conflicts are resolved within modalities based on authoritative ranking, multi-source evidence is integrated across modalities through weighted voting, and high-risk scenarios trigger a manual review channel, forming a closed-loop processing process.

[0164] Third, the system introduces a dynamic confidence threshold mechanism, which drives adaptive optimization of decision criteria through continuous window step evaluation. A reverse causal analysis module traces the source of invalid constraints along the forward reasoning path, integrating cross-modal semantic associations to achieve dynamic rule reconstruction. The dual drive of time-sensitive and semantic indexing ensures that the rule base is synchronized with the latest medical advances, and the expansion of the term ontology enhances the coverage of constraint retrieval.

[0165] Fourth, the scenario adaptation mechanism dynamically adjusts the application strategy of constraint combinations through semantic tag recognition and contextual feature extraction. When generating chemotherapy regimens, the system matches targeted treatment rules based on individual patient characteristics and uses cross-modal correlation to warn of potential complication risks. This system achieves systematic integration of medical evidence, explainable traceability of decision-making logic, and intelligent resolution of complex conflicts, providing highly robust and scalable technical support for precision medicine.

[0166] Example 2

[0167] See also Figure 2 ,Another embodiment provided by the present invention: a multimodal fusion analysis system for test data, comprising: a demand acquisition module and a bidirectional reasoning generation module;

[0168] The demand acquisition module is used to obtain real-time multi-dimensional demand generation data and a bidirectional multi-modal constraint generation model;

[0169] The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model and a reverse causal constraint sub-model;

[0170] The bidirectional reasoning generation module is used to input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model, and obtain the target demand generation text through the multimodal reasoning generation sub-model based on the forward logic constraint set generated by the forward logic constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model;

[0171] The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

[0172] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are protected by the present invention.

Claims

1. A multimodal fusion analysis method for experimental data, characterized in that the steps include: Acquire real-time multi-dimensional demand generation data and a bidirectional multimodal constraint generation model; the bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model, and a reverse causal constraint sub-model; Inputting the real-time multi-dimensional demand generation data into a bidirectional multimodal constraint generation model, generating a sub-model through multimodal reasoning based on a forward logic constraint set generated by a forward logic constraint sub-model and a reverse causal constraint set generated by a reverse causal constraint sub-model, and obtaining a target demand generation text; The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

2. The multimodal fusion analysis method for test data according to claim 1, characterized in that: The construction process of the forward logic constraint sub-model includes: Acquire test standard criteria information, the forward initial logic constraint conditions in the history generation process, and the forward logic constraint information updated in each step of the training constraint process to construct a forward logic constraint sequence; Based on the forward logical constraint sequence, the entity-relationship extraction algorithm is used to obtain the corresponding constraint keywords under different modal data and the logical constraint relationship between the constraint keywords, the hierarchical inheritance relationship of the constraints under the same modal data, the frequent item relationship of the interactive constraints between different modal data, the interactive constraint confidence and the interactive constraint conflict relationship, and the constraint condition triple corresponding to each modal data is constructed based on the obtained keywords and relationships; Based on the constraint triples corresponding to each modal data combined with the graph algorithm, the forward constraint hierarchical reasoning associated node network is obtained; Based on the forward constraint hierarchical reasoning associated node network combined with the deep indexing algorithm, a forward logic constraint sub-model is constructed.

3. The multimodal fusion analysis method for test data according to claim 2, characterized in that: The forward constraint hierarchical reasoning associated node network includes a modal analysis node, an atomic constraint variable node set, a constraint combination node, and a conflict resolution node; the constraint combination node corresponding to each modal corresponds to each atomic constraint variable node set one by one; The modal analysis node calls the corresponding constraint combination node according to the modal vector in the target requirement feature space extracted by the multimodal reasoning generation sub-model, and determines the current main modal data and auxiliary modal data; The constraint combination node is used to generate the target requirement feature space input at each step in the text generation process according to the target requirement, and index combination is performed by calling the atomic constraint variable stored in the atomic constraint variable node corresponding to the constraint combination node index; Each atomic constraint variable sub-node in the atomic constraint variable node set is used to store a single constraint variable and each modal history index frequency, and each atomic constraint variable sub-node is configured with a version traceability chain for recording the change history of the corresponding atomic constraint variable in the test standard criterion information; The internal child nodes of the atomic constraint variable node set are connected to the atomic constraint variable node set corresponding to each mode by using a graph attention algorithm combined with a variable connection relationship constructed by the frequencies corresponding to two constraint variables simultaneously indexed by the same mode to obtain a constraint child node network; The conflict resolution node is used to resolve the combination conflicts that occur during each constraint combination process of the corresponding constraint combination node, using preset resolution rules to resolve intra-modal constraint conflicts and inter-modal constraint conflicts, so that the constraints after each step input resolution meet the preset confidence threshold; The preset resolution rule is obtained by constructing a resolution path decision tree constructed by a preset hierarchical resolution strategy, the priority of the constraint variables and the preset time sensitivity and a decision tree algorithm; The preset hierarchical resolution strategy includes a first layer: an intra-modality resolution strategy, a second layer: an inter-modality resolution strategy, and a third layer: manual intervention.

4. The multimodal fusion analysis method for test data according to claim 3, characterized in that: The process of constructing the forward constrained hierarchical reasoning associated node network includes: Based on the fact that each modal analysis node is a first-level node, the constraint combination node corresponding to the constraint combination node is used as a second-level node, the atomic constraint variable node set corresponding to the constraint combination node is used as a third-level node, and the conflict resolution node is used as an intermediate node between the second-level node and the atomic constraint variable node set; Based on the first-level nodes, second-level nodes, third-level nodes and intermediate nodes corresponding to each mode, the initial forward constraint hierarchical reasoning associated node network is obtained through graph algorithm; A modal mapping connection is constructed based on the corresponding relationship between the first-level nodes and the second-level nodes, and a constraint call connection is constructed based on the call relationship between the second-level nodes and the third-level nodes; Based on the preset index label and historical index frequency of each atomic constraint variable child node in the corresponding atomic constraint variable node set pre-stored in the secondary node and the combined non-conflict confidence corresponding to each atomic constraint variable, an index connection between the secondary node and each atomic constraint variable child node is constructed; Based on the historical constraint conflict resolution information and the frequency of successful conflict resolution between the secondary nodes and the intermediate nodes, an intra-modal resolution connection is constructed, and based on the combined constraint set information between the modes, a combined constraint connection and an inter-modal resolution connection between different modes are constructed; Feeding back the modal mapping connection, the constraint call connection, the index connection, the intra-modal resolution connection, the combined constraint connection, and the inter-modal resolution connection to the initial forward constraint hierarchical reasoning associated node network to obtain a forward constraint hierarchical reasoning associated node network; Based on the forward constraint hierarchical reasoning associated node network combined with a multi-level indexing strategy and a preset index label for each atomic constraint variable subnode in the corresponding atomic constraint variable node set, during the pre-training process of the bidirectional multimodal constraint generation model, forward index training is performed using a deep indexing algorithm so that the combined constraints obtained corresponding to each modality meet a preset forward confidence threshold; The multi-level indexing strategy includes the first level: modal indexing, the second level: semantic indexing and the third level: temporal indexing.

5. The multimodal fusion analysis method for test data according to claim 4, characterized in that: The process of generating the forward logic constraint set by the forward logic constraint sub-model includes: Based on the target demand feature space and the first-level node, the corresponding second-level node is obtained and the primary modal space and the auxiliary modal space corresponding to the generated data at each time point are determined; According to the primary modal space and the secondary modal space corresponding to the data generated at each determined time point, the corresponding primary secondary node and the corresponding primary intermediate node and the secondary secondary node set and the corresponding secondary intermediate node set are determined in the called secondary nodes; Based on the determined main secondary nodes and the corresponding main intermediate nodes and auxiliary secondary node sets and the corresponding auxiliary intermediate node sets, combined with the demand characteristics of each window step input in the target demand feature space, the corresponding constraint variables are queried through the index connection corresponding to each secondary node to obtain the constraint variable set corresponding to each constraint combination node.

6. The multimodal fusion analysis method for test data according to claim 5, characterized in that: The process of generating the forward logic constraint set by the forward logic constraint sub-model further includes: Based on the constraint variable set corresponding to each constraint combination node, combined with the pre-stored resolution rule of the intermediate node corresponding to each calling secondary node and the combined non-conflict confidence, preset forward confidence threshold and logical operator confidence obtained by inference of the reverse causal constraint sub-model, a constraint combination that meets the forward confidence threshold corresponding to each window step requirement feature is obtained, and the obtained constraint combination is marked for applicable scenarios through the semantic association features in the input requirement features, and pre-stored in the corresponding secondary node; At the same time, based on the combined constraint connection between the main secondary node and the auxiliary secondary node set, the inter-modal resolution connection between the main intermediate node and the corresponding auxiliary intermediate node set, and the constraint combination corresponding to each secondary node, a combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature is obtained.

7. The multimodal fusion analysis method for test data according to claim 6, characterized in that: The process of obtaining the constraint combination that satisfies the forward confidence threshold corresponding to each window step requirement feature includes: Set the initial forward confidence threshold. When the combined non-conflict confidence corresponding to the constraint combination generated by the secondary node corresponding to the initially input demand feature is greater than or equal to the preset initial forward confidence threshold, the current constraint combination is considered valid, and the combined non-conflict confidence corresponding to the initially input demand feature is used as the forward confidence threshold for the constraint combination generated by the demand feature corresponding to the next window step size. Repeat the above process. When the combined non-conflict confidence of the constraint combination corresponding to the demand feature input by any window step is less than the corresponding forward confidence threshold, the corresponding constraint combination is judged to be invalid. When it is determined to be invalid, the reverse causal constraint sub-model is called to perform forward reasoning through the forward constraint hierarchical reasoning associated node network according to the input demand characteristics and the corresponding constraint combination to obtain invalid causal constraint variables; Update the current invalid constraint combination based on the invalid causal constraint variable and obtain the updated valid constraint combination; Repeat the above steps to obtain the effective constraint combination corresponding to the step requirement feature of each modal input window in the target requirement generation text generation process.

8. The multimodal fusion analysis method for test data according to claim 7, characterized in that: The process of obtaining the combined constraint set that satisfies the inter-modal confidence level corresponding to each window step requirement feature includes: Based on all constraint combinations that meet the forward confidence threshold generated by each window step requirement feature, cross-modal conflict resolution is performed on the constraint combination corresponding to the auxiliary secondary node set and the constraint combination corresponding to the primary secondary node through the combined constraint connection and the inter-modal resolution connection. If the combined conflict value corresponding to the combined constraint set obtained after the conflict resolution meets the preset cross-modal conflict threshold, the combined constraint set corresponding to the current window step requirement feature is determined to be valid; If the combined conflict value corresponding to the combined constraint set obtained after the conflict is resolved does not meet the preset cross-modal conflict threshold, the constraint variables of the constraint combination corresponding to the auxiliary secondary node set that conflicts with the constraint combination corresponding to the main secondary node are adjusted through the constraint sub-node network until the cross-modal conflict threshold is met.

9. The multimodal fusion analysis method for test data according to claim 8, characterized in that: The construction process of the reverse causal constraint sub-model includes: Constructing a combination conflict value corresponding to the combination constraint set under the corresponding input window step size based on the accuracy rate corresponding to the output requirement text generated by each step size input of the multimodal reasoning generation sub-model; Based on the obtained combined conflict value and the cross-modal conflict threshold, a cross-modal conflict judgment is performed. If there is a conflict, the conflict adjustment process between the constraint combination corresponding to the main secondary node and the constraint combination corresponding to the auxiliary secondary node set is repeated, and the auxiliary secondary node set that conflicts with the main secondary node is associated with the node network through the forward constraint hierarchical reasoning. The conflict is traced according to the corresponding intermediate nodes, and the constraint variables in conflict in the tracing result are adjusted according to the tracing result in combination with the constraint sub-node network to obtain the combined constraint set after reverse causal adjustment.

10. A multimodal fusion analysis system for test data, which is implemented based on the multimodal fusion analysis method for test data according to any one of claims 1 to 9, characterized in that: include: Requirements acquisition module and bidirectional reasoning generation module; The demand acquisition module is used to obtain real-time multi-dimensional demand generation data and a bidirectional multi-modal constraint generation model; The bidirectional multimodal constraint generation model includes a multimodal reasoning generation sub-model, a forward logic constraint sub-model and a reverse causal constraint sub-model; The bidirectional reasoning generation module is used to input the real-time multidimensional demand generation data into the bidirectional multimodal constraint generation model, and obtain the target demand generation text through the multimodal reasoning generation sub-model based on the forward logic constraint set generated by the forward logic constraint sub-model and the reverse causal constraint set generated by the reverse causal constraint sub-model; The forward logic constraint set and the reverse causal constraint set correspond one-to-one; the forward logic constraint set and the reverse causal constraint set correspond one-to-one to the input data modal number respectively.

Images