A multi-center clinical data management system and method based on multi-modal acquisition

Abstract

The application discloses a multi-center clinical data management system and method based on multi-modal acquisition, relates to the field of clinical data management, and comprises the following steps: constructing a disease modal atlas, wherein the disease modal atlas comprises a plurality of disease nodes and a plurality of modal type nodes, an edge between the disease nodes and the modal type nodes represents a dependence relationship of a disease on a corresponding modal type in a historical case; receiving a preliminary diagnosis label of a current case; based on the disease modal atlas, starting from a disease node related to the current case, searching for connected modal type nodes, and generating a modal fusion path of the case according to modal fusion sequence information and modal statistical weights; and performing a feature-level fusion operation based on the modal fusion path. The application can automatically generate a personalized modal fusion path according to different diseases, and realize more accurate, efficient and medically logical multi-modal data processing and fusion.

Description

Technical Field

[0001] This invention relates to the field of clinical data management, and more specifically, to a multi-center clinical data management system and method based on multimodal acquisition. Background Technology

[0002] With the development of medical informatization, the types of data generated during clinical diagnosis and treatment are becoming increasingly diverse, encompassing multiple modalities such as medical images (e.g., CT, MRI), structured or unstructured text (e.g., electronic medical records, doctor's notes), audio (e.g., patient complaint recordings), and physiological parameters (e.g., heart rate, body temperature, blood pressure). Especially in multi-center collaborative healthcare systems, multimodal clinical data from different hospitals and equipment are gradually becoming an important foundation for decision support.

[0003] However, existing multi-center clinical data management systems still rely primarily on the standardization and management of single modalities, offering limited support for the fusion and utilization of multimodal data, particularly in the following aspects:

[0004] 1. Current multimodal analysis often adopts a fixed fusion process, such as inputting all modalities into the model simultaneously for early or late fusion without differentiation. This ignores the degree of dependence of different diseases on different modalities and the order of diagnosis, which can easily lead to a decrease in the model's generalization ability or information interference.

[0005] 2. Current clinical data systems lack a structural characterization of the "disease-modality dependency pattern," making it impossible to derive a better modality fusion path based on the disease, or to optimize the data processing order based on the statistical patterns of historical cases.

[0006] 3. Existing systems use a uniform data fusion strategy for each case, which lacks flexibility and cannot customize modal processing procedures for the diagnostic characteristics of individual cases, thus limiting the practical value of multimodal AI systems in clinical decision support.

[0007] Therefore, there is a need for a multimodal data management mechanism that can establish modality dependency maps based on different diseases and generate personalized fusion paths accordingly, thereby improving the intelligence and adaptability of multi-center clinical data in diagnostic support, disease analysis, and other aspects. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a multi-center clinical data management system and method based on multimodal acquisition, so as to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A multi-center clinical data management method based on multimodal acquisition includes the following steps:

[0011] A disease modality graph is constructed, which includes multiple disease nodes and multiple modality type nodes. The edges between the disease nodes and the modality type nodes represent the dependency relationship of the disease on the corresponding modality type in historical cases. Each edge is accompanied by the following attribute information: modality fusion order information, which is used to indicate the typical order of use of the modality in the diagnosis process of the disease; and modality statistical weight, which is used to indicate the frequency of occurrence or diagnostic contribution of the modality in historical cases of the disease.

[0012] Receive the preliminary diagnostic label of the current case, or the candidate disease and its probability distribution;

[0013] Based on the disease modality map, starting from the disease node related to the current case, retrieve the connected modality type nodes, and generate the modality fusion path for the case according to the modality fusion order information and modality statistical weight.

[0014] Based on the modality fusion path, feature extraction is performed on the multimodal raw data collected from the current case to obtain feature representations for each modality, and feature-level fusion operations are performed according to the modality fusion path.

[0015] In some embodiments, the modality type includes at least two of the following: medical image modality, text modality, speech modality, and physiological parameter modality.

[0016] In some embodiments, the modal fusion path includes setting the modal fusion method.

[0017] In some embodiments, the modality fusion path includes controlling the modality input order during the fusion process and skipping redundant modalities that are not significantly related to the current disease.

[0018] In some embodiments, the search for the modality fusion path employs a greedy search algorithm based on edge weight sorting.

[0019] In some embodiments, the search for the modality fusion path employs a Bayesian search strategy based on fusion accuracy feedback optimization.

[0020] In some embodiments, the method further includes inputting the fusion results of the current medical records into the hospital's EDC system for storage.

[0021] This invention also discloses a multi-center clinical data management system based on multimodal acquisition, comprising:

[0022] The disease atlas construction module is used to construct a disease modality atlas. The disease modality atlas includes multiple disease nodes and multiple modality type nodes. The edges between the disease nodes and the modality type nodes represent the dependency relationship of the disease on the corresponding modality type in historical cases. Each edge is accompanied by the following attribute information: modality fusion order information, which is used to indicate the typical order of use of the modality in the diagnosis process of the disease; and modality statistical weight, which is used to indicate the frequency of occurrence or diagnostic contribution of the modality in historical cases of the disease.

[0023] The case receiving module is used to receive the preliminary diagnostic label of the current case, or the candidate disease and its probability distribution;

[0024] The modal path generation module is used to retrieve the modal type nodes connected to the disease nodes related to the current case based on the disease modal map, and generate the modal fusion path of the case according to the modal fusion order information and modal statistical weights.

[0025] The feature extraction and fusion module is used to extract features from the multimodal raw data collected from the current case based on the modality fusion path, obtain feature representations of each modality, and perform feature-level fusion operations according to the modality fusion path.

[0026] In some embodiments, the search for the modality fusion path employs a greedy search algorithm based on edge weight sorting.

[0027] In some embodiments, the search for the modality fusion path employs a Bayesian search strategy based on fusion accuracy feedback optimization.

[0028] The advantages of this invention over existing technologies lie in its construction of a disease modality atlas containing disease nodes and modality type nodes. This atlas characterizes the dependencies of different diseases on multiple modality types in historical cases. Furthermore, it utilizes fusion order information and statistical weights to describe the modality usage patterns in the diagnostic process. This allows the system to dynamically generate the optimal modality fusion path based on the preliminary diagnostic results of the current case, avoiding information interference and performance degradation caused by fixed fusion strategies. Through the dual guidance of fusion order and weights, this invention clearly defines the processing priority and fusion strength of different modalities during feature extraction and fusion, enhancing the structural rationality and contextual consistency of multimodal fusion, thereby improving the fusion model's ability to understand complex cases and its diagnostic accuracy. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention;

[0030] Figure 2 This is a schematic diagram of the modal fusion path generation of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0032] In clinical data management, especially when multiple medical centers and diverse data types are involved, efficient integration and analysis of this data is crucial for improving diagnostic accuracy. Traditional multimodal data processing methods often rely on fixed fusion processes, such as indiscriminately inputting all data into a model or uniformly splicing features at a certain stage. This approach easily overlooks the degree to which different diseases depend on specific data types and the order in which data is processed during diagnosis, leading to inaccurate analysis results and even information redundancy or interference.

[0033] The order of modal fusion has a significant impact on the final diagnostic results. It not only determines the processing sequence of various modal information during the fusion process but also directly affects the model's focus on information, feature interaction structure, and decision path. In multimodal neural networks, the modalities processed first typically form the dominant feature representation in the fusion layer, while the modalities processed later participate in the decision-making process as supplementary or corrective signals. Therefore, different fusion orders will result in significant differences in the feature space learned by the model. Furthermore, different modalities may have information redundancy, semantic conflicts, or temporal differences. A reasonable fusion order can effectively alleviate these problems and improve information synergy. For example, in pneumonia cases, prioritizing the processing of imaging modalities before introducing textual modalities helps the model establish a structural judgment first, and then refine the diagnosis through textual analysis; conversely, this may lead to the model over-relying on the patient's complaints and ignoring imaging evidence. The fusion order also affects the direction of information flow in attention or gating mechanisms, thereby controlling the information transmission path and activation intensity. More importantly, the design of the fusion order should align with the diagnostic habits and processes of clinicians, such as "reviewing images first, then laboratory tests, and finally listening to the patient's complaints."

[0034] Based on the above reasons, this invention proposes a novel method that optimizes the processing flow of multimodal data collected from multiple centers by constructing disease modality maps and dynamically generating data fusion paths.

[0035] More specifically, such as Figure 1 The above is a general flowchart of the method of the present invention, including:

[0036] A disease modality graph is constructed, which includes multiple disease nodes and multiple modality type nodes. The edges between the disease nodes and the modality type nodes represent the dependency relationship of the disease on the corresponding modality type in historical cases. Each edge is accompanied by the following attribute information: modality fusion order information, which is used to indicate the typical order of use of the modality in the diagnosis process of the disease; and modality statistical weight, which is used to indicate the frequency of occurrence or diagnostic contribution of the modality in historical cases of the disease.

[0037] Receive the preliminary diagnostic label of the current case, or the candidate disease and its probability distribution;

[0038] Based on the disease modality map, starting from the disease node related to the current case, retrieve the connected modality type nodes, and generate the modality fusion path for the case according to the modality fusion order information and modality statistical weight.

[0039] Based on the modality fusion path, feature extraction is performed on the multimodal raw data collected from the current case to obtain feature representations for each modality, and feature-level fusion operations are performed according to the modality fusion path.

[0040] The disease modality map of this invention is essentially a network structure containing two types of nodes: disease nodes, representing common clinical diseases such as pneumonia, diabetes, or heart disease; and modality type nodes, representing data types that may be used in diagnosis, including medical images (such as X-rays and CT scans), text (such as medical records and examination reports), speech (such as patients' verbal descriptions of symptoms), and physiological parameters (such as heart rate and blood oxygen saturation). Disease nodes and modality type nodes are connected by directed edges, which reflect the dependencies of a particular disease on specific data types in historical cases.

[0041] Each side is labeled with two important pieces of information. First is the modal fusion order information, which indicates the typical order in which various data types are used when diagnosing a disease. For example, in diagnosing pneumonia, a doctor might first review a chest X-ray, then refer to the medical record, and finally combine this with blood test results for a final judgment. Second is the modal statistical weight, which reflects the frequency of a particular data type in historical cases of the disease or its impact on the diagnostic outcome. The weights can be calculated based on statistical analysis of historical data. For instance, analyzing 10,000 past pneumonia cases might reveal that chest X-rays were used in 90% of cases and had a significant impact on diagnosis, while voice data might only be used in 5% of cases and contribute less to diagnosis.

[0042] Constructing this graph relies on multi-center clinical data. In practice, historical data can be extracted from the electronic medical record systems of multiple hospitals to ensure the sample covers diverse geographical and demographic characteristics. Next, for each disease, the types of data involved in its diagnosis are statistically analyzed, the frequency of use of each type of data and its contribution to the diagnosis are calculated, and dependencies and weights are determined. Simultaneously, the chronological order of data use is extracted by analyzing the timestamps of diagnostic reports or medical order records. This information is then organized to form a graph structure containing nodes and attributed edges.

[0043] To maintain the atlas's usability, it needs to be updated regularly. As new cases accumulate and diagnostic technologies advance, the disease's dependence on data types may change. For example, the advent of a new imaging device may increase the importance of medical images in the diagnosis of certain diseases. Therefore, every six months or a year, the latest historical data can be collected, dependencies and weights recalculated, and edges and attributes in the atlas adjusted.

[0044] When a new case enters the system, its preliminary diagnostic information is first obtained. This could be a doctor's initial judgment based on the patient's chief complaint and physical signs, such as "suspected pneumonia," or a list of candidate diseases output by a pre-diagnostic model based on symptom data, such as "pneumonia 70%, bronchitis 20%, tuberculosis 10%." This preliminary information will serve as the starting point for subsequent analysis, guiding the system to find relevant disease nodes in the atlas. For example, the most suspicious disease nodes may be prioritized for fusion path generation.

[0045] like Figure 2 As shown, after a preliminary diagnosis, the system, based on the disease modality graph, starts from the disease nodes related to the current case, finds the connected data type nodes, and generates a modality fusion path specifically tailored for that case based on the order information and statistical weights of the edges. This path not only determines which data types will be used, but also clarifies their fusion order and method.

[0046] Different search strategies can be used when generating paths. One approach is a greedy search based on edge weight ranking. The core idea is to select the modal node with the highest edge weight score at each step to construct a fusion path with optimal information priority.

[0047] In the specific implementation process, the system first takes the disease node corresponding to the current case as the starting node, and extracts the attribute information of all modality type nodes connected to this node and their corresponding edges. Each edge contains two key attributes: modality fusion order information and modality statistical weight. The system standardizes these two attributes and performs a weighted combination to generate a fusion score for the modality node. The higher the fusion score, the greater the diagnostic contribution of the modality under the current disease and the higher the priority of its use.

[0048] Then, the system sorts all candidate modality type nodes from highest to lowest according to their fusion scores. Starting with the modality with the highest score, the system adds modality nodes sequentially to the fusion path, determining whether the number of modalities already added to the path reaches the preset maximum number of fusion modalities or whether it covers at least one key modality type (e.g., image, text, parameter). If the conditions are met, the search terminates and the fusion path is output; otherwise, the system continues to select the next highest-scoring modality node to add to the path until the stopping condition is met.

[0049] The greedy strategy does not exhaustively search all modality combinations, but instead locally selects the optimal modality node at each step, ensuring that the generated fusion path achieves a good balance between information coverage and execution efficiency. The final output fusion path not only includes a list of selected modality types, but also the fusion order index and recommended fusion method for each modality, which can be used for subsequent feature extraction and fusion control of multimodal data.

[0050] In another embodiment of the invention, the system may employ a Bayesian search algorithm to obtain the modality fusion path with optimal fusion performance in historical diagnostic performance or model training feedback. This method is suitable for scenarios with a large modality combination space, complex fusion strategy dimensions, and cross-influence, and can quickly find the fusion path with the best global performance with a limited number of evaluations.

[0051] First, the system defines the modality fusion path as an adjustable parameter structure within a search space. This structure includes multiple dimensions such as the combination and selection of modality types, the order of fusion, and the configuration of fusion methods (e.g., weighted fusion, attention fusion, or gating mechanisms). Each fusion path is a point in the search space, and its performance evaluation metrics can include the diagnostic accuracy of historical models along that path, the loss function, and the stability of the confidence distribution.

[0052] During the Bayesian search process, the system first randomly samples several fusion paths and sequentially inputs them into the historical training model or cross-validation system for performance evaluation, obtaining performance metrics for the initial path set. Next, the system uses a Gaussian process or tree-structured Parzen estimator as a surrogate model to fit the functional relationship between fusion path configuration and performance, and calculates sampling strategies such as expected improvement or confidence upper limits to predict the potential performance of unevaluated paths in the current search space.

[0053] In each iteration, the system selects the fusion path most likely to improve performance based on the sampling strategy, conducts experimental evaluation, and feeds the results back to update the surrogate model. This iteration continues until the maximum number of evaluations or the convergence condition is reached, and finally, the globally optimal fusion path is output as the recommended path for that case.

[0054] Compared to greedy search methods, Bayesian search can fully consider the nonlinear impact of different modal combinations and their interactions on overall diagnostic performance. It is suitable for high-dimensional combinatorial optimization problems, has stronger global search capabilities and convergence efficiency, and is especially suitable for multi-center clinical data environments that require long-term maintenance and dynamic updates.

[0055] When generating the path, redundant data can be skipped based on dependencies. For example, if the weight of a certain data type is below a certain threshold (e.g., 0.1), it can be considered to have little diagnostic significance for the current case and directly excluded. This skipping mechanism reduces computational burden and avoids irrelevant information interfering with the fusion results.

[0056] After generating the fusion path, this invention requires processing the multimodal data of the current case. The first step is feature extraction, selecting an appropriate method based on the characteristics of the data type. For medical images, convolutional neural networks (CNNs), such as ResNet50 or VGG16, can be used to extract spatial features from X-ray or CT images; for text, natural language processing techniques, such as the BERT model, can be used to extract semantic features from medical records; for speech, it can first be converted to text using speech recognition technology and then processed using similar methods, or acoustic features can be directly extracted; for physiological parameters, time series analysis can be used, such as calculating the fluctuation characteristics of heart rate or the trend characteristics of blood oxygen.

[0057] After feature extraction, the features are integrated according to the fusion path. Specifically, following the fusion order specified by the modal fusion path, the features of each modality are sequentially input into the fusion module. The fusion module performs feature-level fusion operations using methods such as weighted fusion, attention fusion, or gating mechanisms, based on the fusion order and fusion strength of each modality.

[0058] After fusion, the results are entered into the hospital's Electronic Data Capture (EDC) system for storage. The EDC system is a platform specifically designed to manage clinical data, storing patient characteristic representations or diagnostic results for use in subsequent research or clinical decision-making. During storage, the fused vector can be recorded along with the patient's basic information (such as ID and date) to ensure data traceability.

[0059] To better understand the entire process, let's look at a concrete example. Suppose a patient comes to the clinic complaining of cough and fever, and the doctor initially suspects pneumonia. After receiving this information, the system finds the "pneumonia" node in the disease modality graph. It discovers that this node is connected to chest X-ray (weight 0.9, order 1), medical record text (weight 0.6, order 2), and blood routine data (weight 0.4, order 3), while the voice data has a weight of only 0.05, which is below the threshold of 0.1, and is therefore skipped.

[0060] Next, the system generates a fusion path: first, CNN is used to extract features from X-ray images, then BERT is used to extract features from medical record text, and finally time series analysis is used to extract features from routine blood data. The fusion is performed in the order described above, and the results are stored in the EDC system.

[0061] Given that the data comes from multiple medical centers, there may be differences in equipment or recording standards, requiring additional processing. For example, X-ray images from different hospitals may have different resolutions; standardization can be performed before feature extraction, scaling the images to a uniform size and normalizing pixel values. Text data may also have format differences, which can be preprocessed using uniform encoding rules. During model training, center-specific parameters can also be incorporated, such as learning an independent bias term for each center to adapt to data heterogeneity.

[0062] In summary, this invention introduces disease modality maps to structurally model the dependencies of different diseases on multimodal data in clinical practice, thereby achieving personalized data fusion path generation oriented towards diagnostic goals. Compared to the traditional approach of processing all modalities in a fixed order, this invention can intelligently select the most relevant data types based on the diagnostic background of each case and rationally arrange their fusion order and method, thus avoiding information redundancy and interference, and improving the model's focus on key modalities and fusion effect. This disease-oriented fusion strategy not only improves the accuracy and adaptability of the diagnostic model but also enhances the system's robustness when facing modality missing or heterogeneous data, demonstrating stronger medical rationality and engineering feasibility.

[0063] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-center clinical data management method based on multi-modal acquisition, characterized in that, The method comprises the following steps: constructing a disease modality graph, the disease modality graph comprising a plurality of disease nodes and a plurality of modality type nodes, an edge between the disease node and the modality type node representing a dependence relationship of the disease on the corresponding modality type in historical cases, each edge being attached with the following attribute information: modality fusion order information, used to represent a typical use sequence of the modality in the diagnosis process of the disease, and modality statistical weight, used to represent a frequency or diagnosis contribution degree of the modality in the historical cases of the disease; receiving a preliminary diagnosis label of a current case, or a candidate disease and a probability distribution thereof; based on the disease modality graph, retrieving the modality type nodes connected to the disease node related to the current case, and generating a modality fusion path of the current case according to the modality fusion order information and the modality statistical weight; wherein the method for generating the modality fusion path of the current case is any one of the following: 1) standardizing and weighting combining the modality fusion order information and the modality statistical weight of each edge connecting the disease node and the modality type node to obtain a fusion score of the corresponding modality type, sorting the candidate modality type nodes according to the fusion score, and under the conditions of meeting a preset maximum number of fusion modalities and covering at least a preset key modality type, sequentially selecting the modality type nodes with higher fusion scores to join the modality fusion path by using a greedy strategy; when generating the modality fusion path, the modality type nodes with a modality statistical weight lower than a preset threshold are excluded from the candidate set and are not included in the modality fusion path; 2) using a Bayesian search strategy based on fusion accuracy feedback optimization; the system defines the modality fusion path as an adjustable parameter structure in a search space, including the combination selection, fusion order arrangement and fusion mode configuration of the modality type; each fusion path is a point in the search space, and the performance evaluation index is the diagnosis accuracy, loss function and confidence distribution stability of the historical model under the path; in the Bayesian search process, the system first randomly samples several fusion paths, sequentially inputs the historical training model or the cross-validation system for performance evaluation to obtain the performance indicators of the initial path set; then a Gaussian process or a tree structure Parzen estimator is used as a proxy model to fit the functional relationship between the fusion path configuration and the performance, and based on this, the expected improvement or confidence upper limit and other sampling strategies are calculated to predict the potential performance of the paths in the current search space that have not been evaluated; in each iteration, the system selects a fusion path that is most likely to bring performance improvement according to the sampling strategy for trial evaluation, and updates the proxy model with the results; the iteration is continued until the maximum evaluation number or the convergence condition is reached, and finally the modality fusion path with the best global performance is output as the recommended path of the current case; based on the modality fusion path, performing feature extraction on the multi-modality original data collected for the current case to obtain feature representations of each modality, and performing a feature-level fusion operation according to the modality fusion path; In the process of feature extraction and fusion of the multi-modal raw data, independent center-specific parameters are set for each medical center to correct the data distribution deviation caused by differences in acquisition equipment and recording standards.

2. The multi-center clinical data management method based on multi-modal acquisition according to claim 1, characterized in that, The modality types include at least two of the following: medical image modality, text modality, voice modality, and physiological parameter modality. 3.The multi-center clinical data management method based on multi-modal acquisition of claim 1, wherein, The method further comprises inputting the fusion result of the current medical record into an EDC system of the hospital for storage.

4. A multi-center clinical data management system based on multi-modal acquisition, characterized in that, Comprise: A disease type graph construction module is configured to construct a disease type modality graph, the disease type modality graph comprising a plurality of disease type nodes and a plurality of modality type nodes, an edge between the disease type node and the modality type node representing a dependence relationship of the disease type on the corresponding modality type in historical cases, each edge being attached with the following attribute information: modality fusion order information, used to represent a typical use order of the modality in the diagnosis process of the disease type, and modality statistical weight, used to represent a frequency of occurrence or a diagnosis contribution degree of the modality in historical cases of the disease type; A case receiving module is configured to receive a preliminary diagnosis label of a current case, or a candidate disease type and its probability distribution; A modality path generation module is configured to generate a modality fusion path of the case based on the disease type modality graph, starting from a disease type node related to the current case, retrieving the connected modality type nodes, and generating the modality fusion path of the case according to the modality fusion order information and the modality statistical weight; wherein the method of generating the modality fusion path of the case is any of the following: 1) standardizing and weightedly combining the modality fusion order information and the modality statistical weight of each edge connecting the disease type node and the modality type node to obtain a fusion score of the corresponding modality type, sorting the candidate modality type nodes according to the fusion score, and under the conditions of meeting a preset maximum fusion modality number and covering at least a preset key modality type, using a greedy strategy to sequentially select the modality type nodes with higher fusion scores to join the modality fusion path; when generating the modality fusion path, for the modality type nodes with a modality statistical weight lower than a preset threshold, the modality type nodes are excluded from the candidate set and not included in the modality fusion path. 2) Bayesian search strategy based on fusion accuracy feedback optimization; the system defines the modal fusion path as a tunable parameter structure in the search space, including the combination selection of modal types, the arrangement of fusion order, and the configuration of fusion mode; each fusion path is a point in the search space, and the performance evaluation index is the diagnostic accuracy, loss function, and confidence distribution stability of the historical model under the path; in the Bayesian search process, the system first randomly samples several fusion paths, inputs them into the historical training model or cross-validation system in turn for performance evaluation, and obtains the performance indicators of the initial path set; then, a Gaussian process or tree structure Parzen estimator is used as a surrogate model to fit the functional relationship between the fusion path configuration and performance, and based on this, the expected improvement or confidence upper limit and other sampling strategies are calculated to predict the potential performance of the paths in the current search space that have not been evaluated; in each iteration, the system selects a fusion path that is most likely to improve performance according to the sampling strategy for experimental evaluation, and updates the surrogate model with the results; this iteration continues until the maximum evaluation number or convergence condition is reached, and the final output is the globally optimal modal fusion path as the recommended path for the case; The feature extraction and fusion module is configured to perform feature extraction on the multi-modal raw data collected for the current case based on the modal fusion path, obtain feature representations of each modality, and perform a feature-level fusion operation according to the modal fusion path. In the process of feature extraction and fusion of the multi-modal raw data, for data from different medical centers, independent center-specific parameters are set for each medical center to correct the data distribution deviation caused by differences in acquisition equipment and recording standards.

Images