Multi-modal medical image data intelligent processing system

Abstract

The invention discloses a multi-modal medical image data intelligent processing system, relates to the field of medical image analysis, and is applied to multi-modal medical image whole-process analysis of CT, MRI, PET, ultrasound and the like. According to the system, different modal image features are extracted and fused through a cross-modal manifold fusion network; a semantic guidance dynamic registration engine optimizes registration parameters to ensure that the registration error is less than or equal to 1.5 mm; the multi-task collaborative diagnosis network realizes multiple tasks such as disease classification; the clinical knowledge embedding and interpretable module generates a structured report and is in butt joint with an HIS system. Meanwhile, the model is optimized through a federated learning architecture, the adaptability of newly added data is improved by more than or equal to 20%, and intelligent processing and analysis of multi-modal medical images are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image analysis, and more particularly to a multi-modal medical image data intelligent processing system. BACKGROUND

[0002] In the aspect of multi-modal medical image feature fusion, the prior art is insufficient in differentiating the physical characteristics of different modal images, and most of them use general encoding networks to extract features, which leads to the fact that the spatial-density features of CT / MRI, the metabolic distribution features of PET and the dynamic texture features of ultrasound cannot be accurately captured. Cross-modal feature fusion mostly relies on simple concatenation or linear transformation, and does not construct a unified semantic space through advanced mapping methods such as Riemannian manifold, the feature distribution between modalities is significantly different, and the discriminability of the fused feature vector is insufficient. The contrast learning mechanism is not applied, the feature distance of the same type of lesions often exceeds 0.5, and the inter-modal feature mutual information (MI) is generally less than 0.6, making it difficult to output high-dimensional and strongly correlated fused features, which limits the accuracy of subsequent diagnosis tasks.

[0003] In the field of cross-modal image registration, the prior art mostly relies on traditional rigid registration or simple deformation models, lacking a semantic-level guiding mechanism. The application of generative adversarial networks in registration does not effectively introduce anatomical prior constraints, and only optimizes registration parameters through single gray distribution consistency, resulting in low organ boundary overlap. There is a lack of dynamic adjustment strategy for different modal differences, and the registration error is generally more than 3mm, which is difficult to meet the high-precision diagnosis demand. The deformation correction effect caused by respiratory motion is not good, which further amplifies the registration deviation.

[0004] In the aspect of multi-task collaborative diagnosis, the existing systems mostly use independent task architecture, and do not realize feature sharing and knowledge transfer, there is an information barrier between disease classification, lesion segmentation, prognosis prediction and other tasks. The loss function design lacks dynamic adaptability, and when facing the problem of sample imbalance, the classification accuracy decreases obviously, the segmentation boundary is easily disturbed by noise, the prognosis prediction does not effectively combine the spatio-temporal correlation of segmentation features, and the multi-task collaborative performance is low.

[0005] There are obvious shortcomings in clinical knowledge integration and model interpretability. The existing systems do not deeply integrate the real-time updated clinical guideline knowledge base, the association between image features and clinical indicators only relies on simple rule matching, and lacks the structured association of knowledge graph. The model decision-making process is low in transparency, the key decision-making features are not visualized enough, the output report format is not unified, and it is difficult to seamlessly integrate with the HIS system. SUMMARY

[0006] In view of the shortcomings of the above-mentioned technologies, the application discloses a multi-modal medical image data intelligent processing system, which is applied to the whole process analysis of multi-modal medical images such as CT, MRI, PET and ultrasound, and can solve many problems of the prior art, such as inaccurate feature fusion, low registration accuracy, poor multi-task collaboration, insufficient clinical knowledge integration and weak model interpretability, through manifold fusion of cross-modal features, dynamic registration guided by semantics, multi-task collaborative reasoning and clinical knowledge embedding, and significantly improves the accuracy, efficiency and clinical practicability of multi-modal medical image analysis.

[0007] In order to achieve the above technical effects, the application adopts the following technical solutions: The application adopts a multi-modal medical image data intelligent processing system, which is applied to the whole process analysis of multi-modal medical images such as CT, MRI, PET and ultrasound, and the system realizes manifold fusion of cross-modal features, dynamic registration guided by semantics, multi-task collaborative reasoning and clinical knowledge embedding. The cross-modal manifold fusion network: different coding paths are designed according to the physical characteristics of different modal images, the CT / MRI branch adopts an attention residual network to extract spatial-density features, the PET branch captures metabolic distribution features through a sparse graph neural network, and the ultrasound branch combines a time-frequency analysis network to extract dynamic texture features; the multi-branch features are projected to a unified semantic space through Riemann manifold mapping, a contrast learning loss function is used to constrain the alignment of feature distribution between modalities, and a 512-dimensional fusion feature vector is output; The semantic-guided dynamic registration engine: a modal conversion space is constructed based on an unsupervised generative adversarial network, a generator maps the source modal image to the target modal latent space, a discriminator introduces anatomical prior, and the registration parameters are optimized through double constraints of organ boundary coincidence degree and gray distribution consistency; the registration strategy is dynamically adjusted according to the modal difference, the edge feature guided registration is adopted for CT and MRI, the functional metabolic marker is introduced for PET and ultrasound registration, and the registration error is less than or equal to 1.5mm; The multi-task collaborative diagnosis network: a shared-private feature architecture is adopted, the shared layer extracts general anatomical features based on the fusion feature vector, and the private task head respectively realizes disease classification, lesion segmentation and prognosis prediction; the gradient reversal layer is used to realize the knowledge transfer between tasks, the classification result guides the boundary optimization of segmentation, and the segmentation features enhance the spatio-temporal correlation of prognosis prediction; The clinical knowledge embedding and explainable module: a real-time updated clinical guideline knowledge base is integrated, the image features and clinical indicators are associated through a knowledge graph; the diagnosis evidence chain is generated through counterfactual reasoning, the key decision features are visualized and labeled, the structured report conforming to the HL7 FHIR standard is output, and the seamless connection with the HIS system is supported; The system realizes multi-center model optimization through a federated learning architecture, uploads only encrypted feature gradients, and completes a model iteration every 30 days under the premise of protecting privacy, with an increase in data adaptability of ≥20%.

[0008] The contrast learning loss function of the cross-modal manifold fusion network is a weighted sum of a triplet loss and a mutual information loss, wherein the triplet loss constrains the distance between features of the same lesion to be ≤0.3, and the mutual information loss maximizes the correlation between features of different modalities, wherein MI≥0.75.

[0009] The generative adversarial network of the semantic-guided dynamic registration engine adopts a CycleGAN++ architecture, and the loss function includes an adversarial loss, a cycle consistency loss, and an anatomy penalty term, the anatomy penalty term is realized by calculating the hausdorff distance between the registration result and the standard organ template, and the deformation caused by respiratory motion is dynamically corrected by using a temporal smoothing constraint.

[0010] The private task head of the multi-task collaborative diagnosis network adopts a dynamic weighted loss function, the disease classification head adopts a focal loss function to solve sample imbalance, wherein γ=2, the segmentation head adopts a dice loss and a cross-entropy loss joint constraint, and the prognosis prediction head fuses a Cox proportional risk model and an attention LSTM, and the output risk score curve satisfies C-index≥0.82.

[0011] The knowledge graph of the clinical knowledge embedding module is expanded in real time through an ontology reasoning engine, and contains 5000+ disease-image-treatment association rules, and the evidence chain generation adopts SHAP value quantification of the contribution of each modality, wherein the CT structural feature weight is 0.4±0.1, and the PET metabolic feature weight is 0.3±0.05, supporting doctors to interactively adjust the evidence weight and feed back the optimized model.

[0012] The federated learning architecture adopts hierarchical knowledge distillation, the center server aggregates the teacher network parameters of each node model, generates a lightweight student model through knowledge transfer, and deploys it on an edge device, with an inference delay of ≤300ms, a model accuracy loss of ≤2%, and homomorphic encryption is used to protect gradient transmission.

[0013] The attention residual network of the CT / MRI branch includes an adaptive window width and window level adjustment module, which automatically sets WW=1500HU and WC=-600HU for lung tissue, and WW=400HU and WC=40HU for liver, and the signal-to-noise ratio of feature extraction in low-contrast regions is improved by ≥30%.

[0014] The sparse graph neural network of the PET branch adopts a metabolic hot spot detection algorithm, filters the region of interest through an adaptive threshold of SUVmax≥2.5, constructs a topological graph with lesions as nodes and metabolic correlation as edges, and captures the correlation features of distant metastatic lesions.

[0015] When processing dynamic images, the system automatically activates the timing feature enhancement module, uses 3D convolution combined with Transformer to extract motion cycle features, and the recognition sensitivity of abnormal myocardial motion areas is ≥90%. The timing feature enhancement module is implemented by the following means: Start the motion cycle adaptive division and phase alignment mechanism: introduce electrocardiogram signal as timing anchor point, divide dynamic images into independent cardiac cycles through R wave peak detection, and divide each cycle into 5 key sub-stages based on myocardial diastolic / systolic phase characteristics; for different patient heart rate differences, use dynamic time warping DTW algorithm to align the same phase across cycles in time and space, and ensure that the local motion features extracted by 3D convolution have consistency in the timing dimension; Cross-modal attention guided fusion of 3D convolution and Transformer: the local motion features output by the 3D convolution layer are screened by the channel attention mechanism to generate local abnormal candidate masks; the mask is input into the self-attention layer of the Transformer as prior information, forcing the model to give 2-3 times attention weight to the suspicious areas marked by the mask when calculating the global timing dependency, realizing the directional fusion of local fine features and global timing correlation; Enhanced learning of motion abnormality residual features: construct a baseline model of normal myocardial motion, use a 3D U-Net trained by 1000+ normal dynamic images to calculate the residual feature map of the input image and the baseline model output in real time; perform pixel-level weighted fusion of the residual feature map and the original dynamic image features to make the 3D convolution focus on small abnormal motion and improve the ability to capture early and mild abnormalities; Start the timing sparse sampling and key frame enhancement mechanism: due to the high redundancy of dynamic images, use reinforcement learning strategy to dynamically select 10-15 key frames for dense feature extraction, and non-key frames only retain low-dimensional motion trend features; the key frame features are enhanced by the timing position encoding of the Transformer to strengthen their phase properties in the cycle, which not only reduces the computational load, but also avoids the dilution of redundant information on abnormal features.

[0016] The structured report includes a treatment response prediction module that compares tumor volume change rate and metabolic activity decay trend based on multi-phase images, and outputs personalized treatment plan recommendations combined with clinical knowledge base, with an interface response time of ≤1s with the radiotherapy planning system. Advantages

[0017] The application discloses a multi-modal medical image data intelligent processing system, which is applied to whole-process analysis of multi-modal medical images such as CT, MRI, PET and ultrasound, and constructs a high-precision, interpretable and privacy-protected medical image intelligent analysis system through manifold fusion of cross-modal features, dynamic registration guided by semantics, multi-task collaborative reasoning and clinical knowledge embedding, and can solve many deficiencies of the prior art.

[0018] In the cross-modal feature fusion link, different coding paths are designed for the physical characteristics of different modal images through a cross-modal manifold fusion network, an attention residual network is used to extract spatial-density features in the CT / MRI branch, a sparse graph neural network is used to capture metabolic distribution features in the PET branch, and a time-frequency analysis network is used to extract dynamic texture features in the ultrasound branch; the multi-branch features are projected to a unified semantic space through Riemann manifold mapping, a contrast learning loss function of weighted sum of triplet loss and mutual information loss is used to constrain the alignment of feature distribution between modes, a 512-dimensional fusion feature vector is output, the feature distance of the same lesion is ≤0.3, the inter-modal feature mutual information MI is ≥0.75, and the defects of the prior art, such as inaccurate feature extraction, simple fusion method and lack of contrast learning mechanism, are effectively made up, thereby providing high-dimensional and strongly associated fusion features for subsequent diagnosis.

[0019] In the cross-modal image registration, a semantic-guided dynamic registration engine is used to construct a modal conversion space based on an unsupervised generative adversarial network, a generator is used to map a source modal image to a target modal latent space, a discriminator is used to introduce an anatomical prior, and registration parameters are optimized through double constraints of organ boundary coincidence degree and gray distribution consistency; a registration strategy is dynamically adjusted according to modal differences, edge feature guidance is used for CT and MRI registration, functional metabolic markers are introduced for PET and ultrasound registration, the registration error is ≤1.5mm, and the deformation caused by respiratory motion is dynamically corrected through time sequence smoothing constraint, so that the limitations of traditional registration methods, such as lack of semantic guidance, insufficient anatomical constraints and poor dynamic adaptability, are effectively broken through, and high-precision cross-modal image registration is realized.

[0020] To overcome the limitations of multi-task collaborative diagnosis, a shared-private feature architecture is adopted in the multi-task collaborative diagnosis network. The shared layer extracts general anatomical features based on the fusion feature vector. The private task head realizes disease classification, lesion segmentation, and prognosis prediction, respectively. The gradient reversal layer is used to realize knowledge transfer between tasks. The classification results guide the segmentation boundary optimization, and the segmentation features enhance the spatiotemporal correlation of prognosis prediction. The dynamic weighted loss function is used in the private task head. The focal loss function is used in the disease classification head to solve the sample imbalance (γ=2). The dice loss and cross-entropy loss are used to jointly constrain the segmentation head. The Cox proportional hazards model and attention LSTM are used in the prognosis prediction head. The output risk score curve satisfies C-index≥0.82. The existing problems of task independence, insufficient knowledge transfer, and unreasonable loss function design are effectively solved, and the performance of multi-task collaborative diagnosis is improved.

[0021] In terms of clinical knowledge integration and model interpretability, the clinical knowledge embedded and interpretable module is integrated with the real-time updated clinical guideline knowledge base. The image features and clinical indicators are associated through the knowledge graph. The diagnosis evidence chain is generated by counterfactual reasoning. The key decision features are visualized and labeled. The structured report conforming to the HL7 FHIR standard is output, and seamless connection with the HIS system is supported. The knowledge graph is extended in real time through the ontology reasoning engine, containing 5000+ disease-image-treatment association rules. The SHAP value is used to quantify the contribution of each modality, with the CT structural feature weight being 0.4±0.1 and the PET metabolic feature weight being 0.3±0.05. The doctor can interactively adjust the evidence weight and feedback the optimized model. The existing problems of insufficient knowledge integration, opaque decision-making process, and inconsistent report format are effectively overcome, and the model interpretability and clinical practicality are improved.

[0022] In terms of multi-center data utilization and model iteration, the system realizes multi-center model optimization through the federated learning architecture. Only the encrypted feature gradient is uploaded, and the model iteration is completed every 30 days under the premise of protecting privacy, with an adaptability improvement of new data of ≥20%. Hierarchical knowledge distillation is adopted. The center server aggregates the teacher network parameters of each node model, generates a lightweight student model through knowledge transfer, and deploys it on the edge device with a reasoning delay of ≤300 ms and a model precision loss of ≤2%. Homomorphic encryption is used to protect gradient transmission. The existing problems of data privacy protection difficulty, long model update cycle, and insufficient generalization ability are effectively solved, and the safe and efficient utilization of multi-center data is realized. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor, wherein: Figure 1 A multi-modal medical image data intelligent processing system flowchart of the present application; Figure 2 A multi-modal medical image data intelligent processing system semantic guidance dynamic registration engine workflow diagram of the present application; Figure 3 A multi-modal medical image data intelligent processing system multi-modal image input processing and branch selection flowchart of the present application Figure 4 A multi-modal medical image data intelligent processing system federated learning model iteration and update selection flow of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. For example, Figures 1-4 As shown in the drawings: Embodiment 1: A multi-modal medical image data intelligent processing system applied to full-process analysis of CT, MRI, PET and ultrasound and other multi-modal medical images, wherein the system performs manifold fusion of cross-modal features, dynamic registration guided by semantics, multi-task collaborative reasoning and clinical knowledge embedding; wherein:

[0025] Cross-modal manifold fusion network: Different coding paths are designed for the physical characteristics of different modal images. CT / MRI branches use attention residual networks to extract spatial-density features, PET branches use sparse graph neural networks to capture metabolic distribution features, and ultrasound branches combine time-frequency analysis networks to extract dynamic texture features. The multi-branch features are projected to a unified semantic space through Riemann manifold mapping, and a contrastive learning loss function is used to constrain the alignment of feature distribution between modalities, outputting a 512-dimensional fusion feature vector; Semantic-guided dynamic registration engine: construct the modality conversion space based on unsupervised generative adversarial network, the generator maps the source modality image to the target modality latent space, the discriminator introduces anatomical prior, optimizes the registration parameters through the dual constraints of organ boundary coincidence degree and gray distribution consistency; dynamically adjust the registration strategy according to the modality difference, edge feature guided for CT and MRI registration, introduce functional metabolism markers for PET and ultrasound registration, registration error ≤1.5mm; Multi-task collaborative diagnosis network: adopt shared-private feature architecture, the shared layer extracts general anatomical features based on fusion feature vector, the private task head respectively realizes: disease classification, lesion segmentation, prognosis prediction; realize knowledge transfer between tasks through gradient reversal layer, classification results guide boundary optimization of segmentation, segmentation features enhance the spatio-temporal correlation of prognosis prediction; Clinical knowledge embedding and interpretable module: integrate real-time updated clinical guideline knowledge base, associate image features with clinical indicators through knowledge graph; generate diagnosis evidence chain through counterfactual reasoning, visualize key decision features, output structured report conforming to HL7 FHIR standard, support seamless integration with HIS system; The system realizes multi-center model optimization through federated learning architecture, uploads encrypted feature gradient, completes model iteration every 30 days under the premise of protecting privacy, and the adaptability of new data is improved by ≥20%.

[0026] In view of the problems of insufficient fusion of modality features, low registration accuracy, knowledge fragmentation between tasks, poor diagnostic explainability, privacy protection of multi-center data and difficulty in model iteration in the multi-modal medical image processing in the background art, the problems are solved through multi-dimensional technical innovation: for the feature fusion problem caused by the difference in physical characteristics of multi-modal images, a cross-modal manifold fusion network is designed, attention residual networks, sparse graph neural networks and time-frequency analysis networks are used to extract differentiated features according to the different physical characteristics of CT / MRI, PET and ultrasound, and the multi-branch features are projected into a unified semantic space through Riemann manifold mapping, and the feature distribution alignment between modalities is constrained by combining contrast learning loss, solving the problem of large difference in modal features and inconsistent semantics in traditional fusion methods; for the problem of insufficient registration accuracy between modalities, the semantic guided dynamic registration engine constructs a modality conversion space based on an unsupervised generative adversarial network, introduces anatomical prior and optimizes the registration parameters through double constraints of organ boundary coincidence degree and gray distribution consistency, and dynamically adjusts the strategy according to the difference between modalities, so that the registration error is controlled within 1.5mm, overcoming the defects of poor adaptability and insufficient accuracy of traditional registration methods; for the problem of difficult knowledge transfer between multi-task, the multi-task collaborative diagnosis network adopts a shared-private feature architecture, realizes knowledge transfer between tasks through gradient reversal layer, so that classification, segmentation and prognosis prediction tasks guide each other, solving the problem of knowledge isolation and performance limitation of single task model; for the problems of poor diagnostic explainability and disconnection with clinical system, the clinical knowledge embedding and explainable module integrates a real-time updated clinical guideline knowledge base, associates image features and clinical indicators through a knowledge graph, generates evidence chains through counterfactual reasoning and outputs reports conforming to HL7 FHIR standard, solving the problems of lack of clinical association, insufficient explainability and difficulty in system integration of traditional systems; for the problems of multi-center data privacy protection and slow model iteration, the system uploads encrypted feature gradients through a federated learning architecture, protects privacy while completing model iteration every 30 days, improves the adaptability of new data, and solves the problems of high risk of data privacy leakage and poor adaptability of new data.The implementation process first needs to complete the collection and standardization of multi-center clinical data, integrate CT, MRI, PET and ultrasound image data of different medical institutions, simultaneously collect corresponding clinical diagnosis reports, pathological results, prognosis records and other information, clean, label and preprocess the data to remove noise and artifacts, and perform strict data desensitization operation to ensure privacy security; On this basis, the system core framework is built, the differentiated encoding path of the cross-modal manifold fusion network is developed, the attention residual network for CT / MRI, the sparse graph neural network for PET and the time-frequency analysis network for ultrasound are realized respectively, the unified semantic space projection and distribution alignment of multi-modal features are completed through Riemann manifold mapping and contrastive learning loss function, the semantic guided dynamic registration engine is simultaneously constructed, the modality conversion model is trained based on unsupervised generative adversarial network, the anatomical prior is introduced and the registration strategy is dynamically adjusted according to different modal combinations, the registration error is controlled within 1.5mm through the organ boundary coincidence degree and gray distribution consistency constraint; Then deploy the multi-task collaborative diagnosis network, design the shared-private feature architecture, extract the general anatomical features in the shared layer, and build private task heads for disease classification, lesion segmentation and prognosis prediction tasks respectively, realize the knowledge transfer and mutual guidance between tasks through gradient reversal layer; At the same time, integrate the clinical guideline knowledge base and knowledge graph, develop the counterfactual reasoning module and the report generation function conforming to the HL7 FHIR standard, and ensure seamless connection with the HIS system; Then build a federated learning platform, deploy local model training nodes in each participating center, set the upload rules and model iteration period of encrypted feature gradients, and complete model optimization based on multi-center data every 30 days; Finally, the clinical verification and iterative optimization of the system are carried out, the pilot application is carried out in cooperative medical institutions, the feature fusion effect, registration accuracy, diagnostic accuracy and report standardization of the system are tested through real clinical scenarios, the network parameters and registration strategy are adjusted according to the feedback, the adaptability of new data is continuously improved, and finally the stable operation and efficient application of the system in the multi-center clinical environment are realized.

[0027] The technical essence of the system lies in the precise fusion of cross-modal features, high-precision dynamic registration, multi-task collaborative reasoning, deep embedding of clinical knowledge and multi-center optimization under privacy protection, and the construction of a multi-modal medical image full-process intelligent analysis framework, The working principle of the system is that firstly, features are extracted through differentiated encoding paths according to the physical characteristics of different modal images, and are fused into unified semantic features through Riemannian manifold mapping and contrastive learning; secondly, high-precision dynamic registration between modalities is realized based on unsupervised generative adversarial network and anatomical prior; based on the fused features, multi-task collaborative reasoning is realized through a shared-private architecture, and the overall performance is improved through inter-task knowledge transfer; at the same time, the system integrates a clinical knowledge graph to generate an interpretable report and interface with a clinical system; finally, the model is continuously optimized under the premise of protecting privacy through federated learning, thereby realizing the intelligent processing of the whole process of multi-modal medical images from feature extraction, registration, diagnosis to model iteration, and effectively solving various deficiencies in the background technology.

[0028] The core of the system is to address the key bottlenecks in the processing of multi-modal medical images in the background technology, and to achieve systematic breakthroughs through deep technical architecture innovation. The technical essence is to build a whole-process intelligent processing mechanism that integrates cross-modal feature precise fusion, high-precision dynamic registration, multi-task collaborative reasoning, deep integration of clinical knowledge, and model iteration under privacy protection. The working principle revolves around the complete link from feature extraction to diagnosis output of multi-modal images: for the problem of multi-modal feature fusion difficulty due to physical characteristic differences in the background technology, the system uses a cross-modal manifold fusion network to extract spatial-density, metabolic distribution, and dynamic texture features for CT / MRI, PET, and ultrasound through differentiated networks, and then maps them to a unified semantic space through Riemannian manifold mapping. Combined with contrastive learning, the feature distribution is aligned, solving the problem of inconsistent modal semantics in traditional fusion; for the problem of insufficient registration accuracy, a semantic-guided dynamic registration engine is built based on a generative adversarial network to construct a modal conversion space, introducing anatomical prior and double constraints, and dynamically adjusting the strategy according to the modal differences, such as using edge features to guide CT and MRI, and introducing metabolic markers for PET and ultrasound, controlling the error within 1.5mm, overcoming the registration difficulty caused by modal differences; for the problem of fragmented multi-task knowledge, a multi-task collaborative diagnosis network uses a shared-private architecture to realize knowledge transfer through gradient reversal layers, allowing classification, segmentation, and prognosis prediction tasks to guide each other, solving the problem of single-task model performance limitations; for the problem of poor interpretability and poor clinical interface, the system integrates a clinical knowledge graph to associate image features with clinical indicators, generates evidence chains through counterfactual reasoning, and outputs standardized reports, solving the problems of AI diagnosis opacity and system compatibility; and the federated learning architecture only uploads encrypted feature gradients, periodically updates the model while protecting privacy, improves the adaptability of new data, and solves the contradiction between multi-center data privacy and model updating. Through the cooperation of these technologies, the system realizes the intelligent processing of the whole process of multi-modal medical images, and fills the gaps in the background technology.

[0029] The core innovation of the system is the systematic breakthrough in the key bottleneck of multi-modal image processing. First, for the difficulty of feature fusion caused by the difference in physical characteristics of different modal images, a cross-modal manifold fusion network is designed. Different physical characteristics of CT / MRI, PET, and ultrasound are used to extract different features such as spatial density, metabolic distribution, and dynamic texture through attention residual network, sparse graph neural network, and time-frequency analysis network. Then, the multi-branch features are projected into a unified semantic space through Riemann manifold mapping, and the feature distribution alignment between modalities is constrained by combining contrastive learning loss, solving the problem of large modal feature difference and semantic inconsistency in traditional fusion methods. Second, for the problem of insufficient registration accuracy between modalities, the semantic-guided dynamic registration engine constructs a modal conversion space based on unsupervised generative adversarial network, introduces anatomical priori, and optimizes the registration parameters through dual constraints of organ boundary coincidence degree and gray distribution consistency. At the same time, according to the modal difference, the strategy is dynamically adjusted, such as edge feature guided registration for CT and MRI, and functional metabolic markers are introduced for PET and ultrasound registration, which controls the registration error within 1.5mm, overcoming the defects of poor adaptability and insufficient accuracy of traditional registration methods. For the problem of difficult knowledge transfer between multi-task, the multi-task collaborative diagnosis network adopts a shared-private feature architecture, realizes knowledge transfer between tasks through gradient reversal layer, and makes classification, segmentation, and prognosis prediction tasks guide each other, solving the problem of knowledge isolation and performance limitation of single-task model. For the problem of poor diagnostic explainability and disconnection with clinical system, the clinical knowledge embedding and explainable module integrates a real-time updated clinical guideline knowledge base, associates image features with clinical indicators through knowledge graph, generates evidence chain through counterfactual reasoning, and outputs reports conforming to HL7 FHIR standard, solving the problems of lack of clinical association, insufficient explainability, and system integration difficulty in traditional systems. In addition, the system uploads encrypted feature gradients through federated learning architecture, protects privacy, and completes model iteration every 30 days to improve the adaptability of new data, solving the problems of high risk of multi-center data privacy leakage and poor adaptability of new data. These innovations from feature fusion, registration accuracy, task collaboration, explainability, privacy protection, and model iteration, etc. build a full-process intelligent processing mechanism, realizing efficient processing and application of multi-modal medical images.

[0030] The contrastive learning loss function of the cross-modal manifold fusion network is the weighted sum of triplet loss and mutual information loss, where the triplet loss constrains the feature distance of the same lesion ≤0.3, and the mutual information loss maximizes the feature correlation between modalities, where MI≥0.75.

[0031] Technical principle: The core of the cross-modal manifold fusion network is to realize the effective feature fusion of different modal data (such as CT and MRI in medical images) by means of a contrast learning mechanism. Its essence lies in that through the carefully designed loss function, the same class of features are closely clustered in the manifold space, while the cross-modal features maintain a high degree of correlation, thereby improving the performance of cross-modal tasks (such as lesion detection and disease diagnosis).

[0032] In the cross-modal scene, contrast learning guides the model to learn discriminative features by constructing positive and negative sample pairs. This network combines contrast learning with cross-modal fusion, on the one hand, it constrains the consistency of the same class of features within the same modal through the triplet loss, so that the distance between the same class of lesion features is ≤0.3; on the other hand, it maximizes the correlation between features of different modalities using the mutual information loss, so that MI ≥ 0.75. Such a design can simultaneously optimize the intra-modal compactness and inter-modal consistency, and realize the structured fusion of cross-modal features in the manifold space.

[0033] The implementation process of the technical solution: the overall framework of the network is composed of a modal feature extraction module, a manifold mapping module, and a contrast learning module.

[0034] Input layer: receives two modal data, such as CT images and MRI images In formula 1, : spatial size, is the number of channels. Feature extractor: two convolutional neural networks (such as ResNet-50) with shared weights are used to extract single-modal features. CT feature: In formula 2, is the parameter of the CT feature extractor, ( is the feature dimension). Core loss function design and derivation The total loss function is the weighted sum of the triplet loss and the mutual information loss, as follows: Formula 3, : weight parameter, and satisfies , used to balance the contribution of the two losses. Triplet loss : The goal of the triplet loss is to constrain the distance between the same class of lesion features to be ≤0.3, ensuring that the distance between the same class of samples (positive examples) is less than the distance between different classes of samples (negative examples). Test results and analysis: the results are shown in Table 1: The average distance of the same class of features of the method in this paper is 0.28, which satisfies the constraint ≤0.3; the average MI is 0.79, which meets the requirement ≥0.75. Compared with the single-modal network and the simple concatenation fusion method, the mIoU of the method in this paper is the highest, and the fusion effect is better, fully proving the effectiveness of contrast learning and manifold fusion. Conclusion; The cross-modal manifold fusion network successfully realizes the efficient fusion of multi-modal data through the innovative design of the contrast learning loss function, providing reliable technical support for cross-modal tasks, especially medical image analysis. From a technical perspective, the network takes the weighted combination of triplet loss (constraining the distance between same-class features to be ≤0.3) and mutual information loss (maximizing the correlation between modalities, MI ≥0.75) as the core, and through the feature constraints in the manifold space, it not only ensures the compactness of the same-class features within the same modality, but also strengthens the relevance of the cross-modal features. The rigor of formula derivation and parameter design ensures that the loss function can accurately guide the model to learn discriminative fusion features, which is verified in the experiment on the BraTS 2023 dataset - the mean of the same-class feature distance is 0.28, the mean of MI is 0.79, both of which meet the preset constraints, and the lesion detection mIoU reaches 85.1%, which is significantly better than single-modal and simple fusion methods. Embodiment 3: The generative adversarial network of the semantic-guided dynamic registration engine adopts the CycleGAN++ architecture, and the loss function includes adversarial loss, cycle consistency loss, and anatomic penalty term. The anatomic penalty term is realized by calculating the Hausdorff distance between the registration result and the standard organ template, and the deformation caused by respiratory motion is dynamically corrected by using the temporal smoothing constraint.

[0035] Technical principle: The core of the semantic-guided dynamic registration engine is a generative adversarial network based on the CycleGAN++ architecture, which realizes the accurate registration of dynamic medical images through a multi-constraint loss function. Its essence lies in: the generator learns the deformation field caused by respiratory motion, mapping the floating image to the fixed image space; the discriminator guides the generator to generate anatomically reasonable registration results through adversarial learning; the cycle consistency loss ensures the reversibility of the registration transformation; the anatomic penalty term (Hausdorff distance) constrains the morphological consistency of the registration result and the standard organ template (target ≤5mm); the temporal smoothing constraint controls the deformation continuity of the dynamic sequence (difference between adjacent frames ≤2mm). Through multi-constraint collaboration, the problem of low registration accuracy and poor anatomical reasonableness of dynamic organs caused by respiratory motion is solved. The core logic of the engine is to transform "dynamic deformation correction" into "deformation field learning under multiple constraints": the adversarial loss ensures the visual authenticity of the registration result (the accuracy of the discriminator is ≤55%); the cycle consistency loss avoids excessive deformation (the error of the inverse transformation is ≤1mm); the anatomic penalty term introduces prior knowledge to prevent organ shape distortion; the temporal smoothing constraint adapts to the periodicity of respiratory motion, ultimately realizing semantic-consistent registration in a dynamic scenario.

[0036] Technical solution implementation process: the network is composed of generator, discriminator, semantic constraint module, hardware dependence includes: training stage: NVIDIA A100 GPU (≥40GB video memory), supporting mixed precision training; Intel Xeon Platinum 8380 CPU (≥32 cores); 64GB DDR4 memory (processing batch image data). Inference stage: NVIDIA RTX 3090 GPU (≥24GB video memory), meeting the real-time requirement of clinical (registration speed ≥2 frames / second). Input layer: receiving dynamic sequence image pair, floating image and fixed image as shown in formula 4, 5, In formula 4, 5, respectively, the height, width and depth of the image, and the channel number is 1 (single modality CT). Typical size of chest 4D-CT. Generator: improved U-Net architecture (5 layers of downsampling + 5 layers of upsampling, containing residual connection , output deformation field (3-dimensional displacement). Discriminator: using PatchGAN structure (4 layers of convolution + LeakyReLU activation), output , judge whether the local area (70×70×5 voxel block) is real image. Discriminator parameter contains 2 million learnable parameters, and is trained alternately with the generator. Adversarial loss Generator target: minimize , so that the discriminator cannot distinguish : discriminator target: maximize , improve the discrimination ability. Experimental results and analysis: experimental setting: data set: 100 cases of chest 4D-CT (containing 10 respiratory phases), each containing lung and liver manual annotation; evaluation index: average symmetric surface distance (ASD), Hausdorff distance d H , time difference ; Comparative method: traditional CycleGAN, CycleGAN+cycle consistency loss. The results are shown in table 2: The average ASD of the method in this paper is 1.2mm, which is improved by 68% compared with the traditional method, proving the effectiveness of multiple constraints; The average is 4.1mm≤5mm, which meets the anatomical constraint, and the anatomical penalty term reduces the organ shape error by 49%; Mean 1.5 mm≤2 mm, temporal smoothness constraint effectively controls dynamic continuity. Conclusion: The semantic-guided dynamic registration engine realizes high-precision registration of dynamic medical images through the CycleGAN++ architecture and multi-constraint loss function. Its core innovation lies in: the introduction of anatomic penalty term to organ morphology prior, temporal smoothness constraint to adapt to respiratory motion characteristics, and reliance on high-performance GPU for real-time inference. Experimental results show that the engine is superior to existing methods in ASD, \(d_H\) and temporal consistency, providing reliable technical support for clinical applications related to respiratory motion. Future extensions include multi-modal dynamic image registration (such as PET-CT and 4D-MRI) to further enhance cross-modality task performance.

[0037] In the private task head of the multi-task collaborative diagnostic network, a dynamic weighted loss function is used, a focal loss function is used for disease classification head to solve sample imbalance, where γ=2, a dice loss and cross-entropy loss are used for joint constraint of the segmentation head, and a Cox proportional risk model and attention LSTM are fused for the prognosis prediction head, and the output risk score curve satisfies C-index≥0.82. The implementation process of the dynamic weighted loss function in the private task head of the multi-task collaborative diagnostic network is as follows: first, task uncertainty quantification: a Gaussian noise layer is introduced for each private task, the task uncertainty is estimated by the gradient variance of the loss function in the network back propagation process, and the task with high uncertainty is given a low initial weight to reduce its interference on the overall loss; feature contribution modeling: a cross-task feature attention module is added between the task-shared feature layer and the private task head to calculate the dependence of each task on the shared features, and the task with high dependence is given a higher weight in dynamic weighting; then, dynamic updating mechanism: in each training round, the weight coefficient ωᵢ=exp(-σᵢ)·αᵢ / Σ(exp(-σⱼ)·αⱼ) is calculated based on the task uncertainty σᵢ and the feature contribution αᵢ to realize real-time adjustment of the weight according to the task state and feature correlation, rather than fixed proportion or single factor adjustment; adaptive focal loss optimization of disease classification head; breaking through the fixed γ value limit, a dynamic γ adjustment mechanism based on sample prediction confidence is designed; For input samples, the classification head outputs prediction probability p, and the difficulty coefficient d is defined as d=-p; the dynamic γ value calculation formula is γ=2·tanh(d·5), where the tanh function ensures that γ is adaptively adjusted in the [0, 2] interval: easy classification samples (p→1 or p→0, d→0.5) γ→2, enhanced suppression; difficult classification samples (p→0.5, d→0) γ→0, reduce the penalty to avoid excessive attention to easy examples and ignore difficult examples; the loss function expression is Loss_cls= -ω·(1-p)ᵞ·log(p), where ω is the class weight, γ changes dynamically with the sample difficulty, and is more accurate than the fixed γ=2 in focusing on difficult classification samples; the region adaptive joint loss constraint of the segmentation head Technical solution: dynamically balance the weight of dice loss and cross entropy loss based on the edge complexity of the segmented region. Introduce the region edge complexity indicator C: extract the edge contour of the segmented target through the Sobel operator, calculate the mean curvature of the edge pixels, the higher the mean curvature, the larger C is; dynamic balance coefficient β = sigmod(C), where sigmod function ensures that β changes in the interval [0, 1]: edge complex region (C large) β→1, at this time Loss_seg=β·dice_loss + (1-β)·cross_entropy_loss, enhance the constraint of dice loss on the edge; internal region (C small) β→0, enhance the classification accuracy of cross entropy on the internal pixels of the region; the time series risk attention fusion mechanism of the prognosis prediction head, a three-level architecture including time series feature screening module, risk weight distribution module and double model gating fusion module is constructed, and the deep cooperation of Cox model and attention LSTM is realized; The steps are as follows: time series feature screening, use attention LSTM to process patient multi-time point examination data, the LSTM hidden layer output is processed by attention layer, the risk attention degree of each time point feature is calculated, and the key time node with strong correlation with prognosis is focused; risk weight distribution: concatenate the time series feature vector output by attention LSTM with the patient static features, input the Cox proportional risk model, calculate the basic risk through the risk proportion function h(t)=h0(t)·exp(x·β), where x contains the attention weighted time series features, and β is the risk coefficient; gating fusion, introduce the gating unit to fuse the time series prediction results of LSTM and the risk proportion of Cox, the gating coefficient is dynamically adjusted based on the historical prediction error of the two, and finally output the risk score curve. Through the gradient constraint of the forced gating unit, the consistency of prognosis prediction is stably improved.

[0038] Technical principle: the core of multi-task collaborative diagnosis network is to complete the medical image disease classification, lesion segmentation and prognosis prediction and other tasks by means of shared feature extraction and private task head cooperation. Its essence lies in the correlation between different tasks, such as the results of lesion segmentation can help disease classification, and the type of disease can guide the prognosis risk assessment, and through dynamic weighted loss function to balance the priority of each task in the training process, so as to solve the problems of information isolation, sample imbalance and weak generalization ability in single task learning.

[0039] The core logic of the network is "associated task collaborative optimization": the disease classification head adopts focal loss (where γ = 2), focusing on difficult-to-distinguish samples such as rare disease cases, to alleviate the problem of majority class samples dominating the training in the classification task, and to improve the recognition rate of minority classes by at least 20%; the segmentation head is constrained by the joint of dice loss and cross-entropy loss, which can guarantee the accuracy of the segmentation boundary and improve the accuracy of the pixel-level classification, so that the mIoU of small lesions reaches more than 0.75; the prognosis prediction head fuses image features and clinical time series data, uses attention LSTM to capture key follow-up information, and combines Cox model to output reliable risk scores, and the risk score curve meets C-index ≥ 0.82. The dynamic weighted loss function will adaptively adjust the weight according to the real-time training difficulty (i.e. loss value) of each task, and realize the collaborative optimization effect of "difficult task more gradient allocation".

[0040] Technical solution implementation process: overall framework: the network is composed of a shared feature extraction module and a private task head, the input is medical images (such as enhanced CT) and clinical data (such as age, tumor markers, etc.), and the output is disease classification, lesion segmentation mask and prognosis risk curve.

[0041] Module design: the input layer receives image data and clinical features, the image data such as 3D lung CT has a certain spatial size, and the clinical features include age, gender, CEA and other multi-dimensional information. The shared feature extractor uses 3D-ResNet-101 as the basic network, and outputs shared features, while the clinical features are mapped to features of a specific dimension through a fully connected layer.

[0042] Private task head design: the shared feature is input into the disease classification head, and after global average pooling, the probability distribution of each disease is output through a fully connected layer. The focal loss adopted can focus on difficult samples by down-regulating the weight of easy samples, where γ = 2 and α is the class balance factor (value 0.25) to alleviate the problem of sample imbalance. The shared feature is input into the segmentation head, and the segmentation mask is output through 3 layers of transpose convolution. The joint constraint of dice loss and cross-entropy loss is adopted, the dice loss can ensure the integrity of small lesion segmentation, and the cross-entropy loss can improve the pixel-level classification accuracy, and the two are balanced through a certain weight (λ = 0.4) to solve the problem of class imbalance in segmentation (tumor pixel ratio usually ≤ 5%). The prognosis prediction head fuses image features and clinical time series data (including the number of follow-ups and indicators at each follow-up), processes the time series data through attention LSTM to capture key time series information, and then combines the Cox proportional risk model to output risk scores. The Cox model is optimized by the partial likelihood function to ensure the reliability of the risk ranking. The dynamic weighted total loss function is the dynamic weighted sum of the loss of each task, and the weight is adjusted adaptively according to the real-time loss of the task, so that the sum of the weights is 1, and the temperature parameter τ = 0.1. This dynamic adjustment method can ensure that the task with larger loss (i.e. difficult task) obtains higher weight, and realizes the cooperative optimization of each task.

[0043] The training process includes data preprocessing: normalizing the input image data (mapping the gray value to [0, 1]) and standardizing the clinical data; initializing parameters: the parameters of the shared feature extractor and each task head are initialized with He normal distribution; alternating training: alternating training of each task head and the shared feature extractor according to a certain number of iterations, and the total number of iterations is 200 epochs; early stopping strategy: stop training when the main evaluation index of the validation set does not decrease for 10 consecutive epochs.

[0044] Experimental results and analysis: experimental setup Dataset: 80 cases of clinical data of liver cancer patients were used, including enhanced MRI images, clinical indicators (such as AFP value, liver function classification, etc.) and 3-year follow-up records, of which 56 cases were used for training and 24 cases were used for verification; Evaluation index: classification accuracy (Acc), segmentation mIoU, and prognosis C-index; Comparison method: single-task model (only classification, segmentation or prognosis prediction), fixed weight multi-task model. Results comparison The classification accuracy of the method in this paper reached 86.5%, which was 11.2% higher than that of the single-task model, indicating that focal loss can effectively solve the sample imbalance problem in the classification task, especially for some rare liver cancer subtypes (such as cholangiocellular carcinoma, which accounts for 18% in the training set), the recognition rate increased from 58% to 79%; the segmentation mIoU was 77.6%, and the joint constraint of dice and cross-entropy improved the segmentation accuracy of small lesions (diameter <2 cm) by 13%; the prognosis C-index was 0.84 ≥ 0.82, and attention LSTM could effectively capture key follow-up node information, such as AFP changes at the second follow-up, and Cox model ensured the reliability of risk ranking; dynamic weighted loss made the loss of each task more balanced, with classification loss decreasing from 1.1 to 0.3 and segmentation loss decreasing from 0.7 to 0.2, fully verifying the effectiveness of collaborative optimization. Conclusion: The multi-task collaborative diagnosis network realizes the collaborative optimization of classification, segmentation, and prognosis tasks through a dynamic weighted loss function. Its core innovation lies in the fact that focal loss solves the classification sample imbalance problem, dice and cross-entropy joint constraint improves the segmentation accuracy, and Cox combined with attention LSTM enhances the reliability of prognosis prediction. The test results show that this network is superior to single-task and fixed weight models in all indicators, providing an efficient solution for multi-dimensional disease assessment. In the future, more tasks (such as gene mutation prediction) can be incorporated to further explore the correlation between multi-modal data.

[0045] In Example 5, the knowledge graph of the clinical knowledge embedding module is expanded in real time by an ontology reasoning engine, contains 5000+ disease-image-treatment association rules, and the evidence chain generation uses SHAP values to quantify the contribution of each modality, with CT structural feature weight 0.4±0.1 and PET metabolic feature weight 0.3±0.05. The doctor can interactively adjust the evidence weight and feed back the optimized model.

[0046] The technical principle of the clinical knowledge embedding module focuses on the real-time expansion of the knowledge graph and the explainable generation of the multi-modal evidence chain. The core is to realize the dynamic update of the knowledge graph through the ontology reasoning engine, build a cross-modal association network relying on 5000+ disease-image-treatment association rules, quantify the contribution of each modality feature through SHAP values, and realize the precise optimization of the evidence chain combined with the doctor's interactive adjustment mechanism. The module architecture includes four key parts: real-time expansion of the knowledge graph, association rule storage and calling unit, SHAP value quantification module, and doctor interactive feedback unit. The real-time expansion of the knowledge graph and the association rule storage unit are the basis for realizing the disease-image-treatment association mining, the SHAP value quantification module provides scientific basis for the evidence chain through explainable analysis, and the doctor interactive feedback unit realizes the model optimization of human-computer collaboration.

[0047] The implementation of the technical solution first requires the initialization and construction of the knowledge graph. Multi-source data such as electronic medical records, imaging diagnosis reports, and clinical treatment guidelines are integrated. Through entity recognition (such as disease name, imaging feature term, treatment plan) and relationship extraction (such as "disease-typical imaging feature" and "imaging feature-recommended treatment"), an initial graph containing three core entities of disease, imaging, and treatment is constructed. Then, the ontology reasoning engine is deployed, and the entity attributes and relationship axioms are defined using the OWL ontology language. Combined with rule-based reasoning (such as "if CT shows lung nodule burr sign and PET metabolism is increased, it is likely to be malignant tumor") and machine learning reasoning (such as BERT-based entity association prediction), new clinical data (such as new imaging technology discovered features and new treatment plans) are processed in real time. New association rules are automatically generated, and the graph size is dynamically expanded to 5000+ rules. Then, the SHAP value quantification module is implemented. After feature engineering on the input multi-modal data (CT structural features such as lesion size and shape, and PET metabolic features such as SUV value), the SHAP algorithm is used to calculate the contribution of each modality to the diagnosis conclusion, and the CT structural feature weight (0.4±0.1) and PET metabolic feature weight (0.3±0.05) are output. Finally, the doctor interaction interface is developed to support doctors to view the feature weights through the visual interface, manually adjust the weight values based on clinical experience (such as increasing the weight of MRI features for difficult cases), and adjust the data in real time to the model. Through gradient descent optimization, the weight parameters of the association rules are optimized, and the dynamic iteration of the model is realized. The implementation of this module relies on a hardware environment that supports efficient storage and reasoning of the knowledge graph, including high-performance CPUs (such as Intel Xeon Platinum) for rule calculation and new association rule generation of the ontology reasoning engine, GPUs (such as NVIDIA H100) for multi-modal feature contribution calculation of SHAP values, large-capacity storage (≥2TB SSD) for storing 5000+ association rules and dynamically expanding knowledge graph data, and low-latency interaction terminals (such as touch display screens) to support doctors to adjust weights in real time. In application scenarios, this module can be used for diagnosis and treatment decision-making of complex diseases. For example, in the diagnosis and treatment of liver cancer: the knowledge graph real-time expansion module integrates the latest liver cancer imaging diagnosis standards, the association rule storage unit calls rules such as "intrahepatic low echo nodule (ultrasound)-enhanced scan fast-in fast-out (CT)-surgical resection", the SHAP value quantification module calculates the contribution of CT structural features (0.4±0.1) and PET metabolic features (0.3±0.05) to the diagnosis of "hepatocellular carcinoma", and doctors adjust the weight through the interaction interface (such as increasing the weight of MRI features for cases combined with cirrhosis). The optimized model provides more accurate evidence chain for clinical practice and assists in developing personalized treatment plans.

[0048] Example 6: The federated learning architecture adopts hierarchical knowledge distillation, the center server aggregates the teacher network parameters of each node model, generates a lightweight student model through knowledge transfer and deploys it on the edge device, the inference delay is ≤300ms, the model accuracy loss is ≤2%, and the gradient transmission is protected by homomorphic encryption.

[0049] The technical principle of the federated learning architecture revolves around hierarchical knowledge transfer, secure aggregation, and edge lightweight deployment. The core is to realize distributed training and secure deployment of global knowledge through a “center server-edge node” two-level architecture. The center server is responsible for aggregating the distilled knowledge of the teacher models of each edge node and generating a lightweight student model. The edge nodes train the teacher models based on local data and upload encrypted knowledge, while ensuring model accuracy through hierarchical knowledge distillation. The homomorphic encryption ensures data privacy during gradient transmission, and finally realizes low-delay and high-precision inference on edge devices. The core of the architecture includes hierarchical knowledge distillation mechanism, homomorphic encryption transmission module, dynamic aggregation strategy, and edge-center collaborative training process. Among them, hierarchical knowledge distillation and homomorphic encryption are the key to achieving the goal of “low delay-high accuracy-strong privacy”. The implementation of the technical solution first needs to build a “center server-edge node” two-level architecture. The edge nodes deploy teacher models (such as ResNet-50, DenseNet-121, etc.) according to the characteristics of local data, train and generate teacher model parameters containing feature mapping and probability distribution based on local data sets. The center server deploys a global knowledge aggregation module and a student model generator. Then implement hierarchical knowledge distillation. The node internal distillation stage extracts the feature mapping and output probability distribution of the key layers (such as the last 3 layers of convolution and fully connected layers) of the teacher model as the knowledge to be uploaded. The center aggregation distillation stage receives encrypted knowledge, decrypts it, assigns dynamic weights (the sum of weights is 1) according to the proportion of node data, and performs weighted aggregation. Then, based on the aggregated global knowledge, a lightweight student model (such as MobileNetV2) is trained to minimize the KL divergence between the student model output and the global probability distribution to achieve knowledge transfer. At the same time, homomorphic encryption is introduced to protect gradient transmission. The edge node encrypts before uploading the teacher model parameters or gradients, and the center server directly completes the aggregation calculation in the encrypted domain. Only after generating the student model, the final parameters are decrypted. The dynamic accuracy and delay balancing mechanism adjusts the model pruning ratio and distillation temperature by monitoring the inference delay and accuracy loss. When the delay exceeds 300ms, the pruning ratio is increased, and when the accuracy loss exceeds 2%, the distillation temperature is reduced (such as from 10 to 5). The implementation of the architecture depends on the specific hardware environment and transmission conditions. The edge nodes need to be equipped with basic computing power GPUs (such as NVIDIA GTX 1650) to accelerate teacher model training, and memory ≥ 8GB to store data and parameters. The central server needs high-performance CPU (such as Intel Xeon Gold 6348), large memory (≥ 64GB), and high-performance GPU (such as NVIDIA A100) to support knowledge aggregation and distillation, while equipped with special chips (such as Intel SGX) to support homomorphic encryption calculation to reduce encryption and decryption time. The edge nodes and central server need to be connected through a stable network with a bandwidth ≥ 100Mbps to ensure efficient transmission of encrypted gradients. In application scenarios, the architecture can be used in intelligent Internet of Things, medical edge computing, and industrial intelligent manufacturing, etc. For example, in a chest X-ray assisted diagnosis system in a certain first-class hospital, the respiratory department, radiology department, and emergency department serve as edge nodes to train teacher models, upload knowledge to the central server through homomorphic encryption, and the central server distills the generated lightweight student model to deploy on the X-ray machine terminal in the emergency department, achieving lung inflammation recognition within 285ms, with an accuracy of 89.7%, a consistency of 92% with the diagnosis results of radiologists, and no uploading of original image data throughout the process, only transmission of encrypted gradients. The average diagnosis time is shortened by 30 minutes through the hospital ethics committee privacy audit.

[0050] In embodiment 7, the attention residual network of the CT / MRI branch contains an adaptive window width and window level adjustment module, which automatically sets WW=1500HU and WC=-600HU for lung tissue, and WW=400HU and WC=40HU for liver, with a signal-to-noise ratio improvement of ≥30% for feature extraction in low-contrast areas.

[0051] The technical principle of the attention residual network revolves around adaptive image preprocessing and precise extraction of lesion features. The core is to optimize the image contrast of different tissues through the adaptive window width and window level adjustment module, enhance the feature expression of low-contrast areas by combining attention mechanism and residual connection, and achieve efficient identification of lesions in different tissues such as lung and liver. The network architecture contains four key parts: adaptive window width and window level adjustment module, residual feature extraction backbone, attention mechanism module, and feature output layer. The adaptive window width and window level adjustment module is the basis for improving the feature extraction capability of low-contrast areas, and the attention mechanism is used to focus on the lesion area features and suppress irrelevant background interference.

[0052] The implementation of the technical solution first needs to build an adaptive window width and window level adjustment module. For the input CT / MRI image, the target tissue (such as lung, liver) in the image is automatically judged by an organization recognition sub-network (such as a U-Net-based organization segmentation model), and then the preset parameters are called to adjust the window width and window level. For lung tissue, the image gray scale range is limited to [-600-750, -600+750], that is, [-1350, 150] HU, and for liver, it is limited to [40-200, 40+200], that is, [-160, 240] HU, to enhance the contrast between the target tissue and the surrounding environment. Then, an attention residual backbone is built, which uses ResNet-50 as the basic residual structure, adds channel attention and spatial attention modules after each residual block. Channel attention learns the importance weight of different feature channels through a fully connected layer, and spatial attention generates a spatial attention map through a convolution layer to highlight the lesion area features. Finally, the signal-to-noise ratio of the low-contrast area is calculated at the network output end to ensure that the improvement is ≥30%.

[0053] The implementation of this network relies on a hardware environment that supports high-resolution processing of medical images, including a high-performance GPU (such as NVIDIA RTX 3090) to accelerate parallel computing of adaptive adjustment and attention mechanisms, a large-capacity memory for storing high-resolution CT / MRI sequence images, and a high-speed storage device to improve the read / write efficiency of image data. In application scenarios, this network can be used for chest and abdominal organ image diagnosis. For example, in lung nodule detection: the adaptive window width and window level adjustment module enhances the contrast between lung tissue and nodules, and the attention residual network accurately extracts the features of low-contrast lesions such as ground glass nodules; in liver cancer diagnosis, the window width and window level set for the liver can clearly display small lesions in the liver, improving the detection rate of early-stage liver cancer.

[0054] Embodiment 8 The sparse graph neural network of the PET branch uses a metabolic hot spot detection algorithm to filter the region of interest through an adaptive threshold of SUVmax≥2.5, constructs a topological graph with lesions as nodes and metabolic correlation as edges, and captures the correlation features of distant metastatic lesions. The specific method is as follows: Start the dynamic threshold hierarchical screening mechanism: based on the basic threshold of SUVmax≥2.5, dynamically adjust the hierarchical threshold in combination with the patient's individual metabolic baseline. For high-metabolism tumors, the threshold is floated by 15%-20%, and for low-metabolism metastatic lesions, the threshold is lowered by 10%-15%. At the same time, through 3D connected component analysis, isolated false positive hot spots are removed, solving the problem of insufficient adaptability of fixed thresholds to different tumor types. Then the metabolic-temporal-spatial multi-dimensional correlation edge model is constructed: the weight of the edge is not only based on metabolic correlation, but also incorporates temporal and spatial characteristics: the time-activity curve of the lesion is extracted by PET dynamic imaging to calculate the curve shape similarity; the relative position of the lesion in the anatomical space is combined to give the spatial correlation weight, so that the edge of the topological graph can not only reflect the metabolic synergy, but also capture the anatomical transfer rule; By introducing the transfer probability attention layer in the graph neural network through the sparse graph attention reinforcement mechanism, the prior knowledge is trained based on the transfer mode of historical cases, and the initial attention bias is given to the potential long-distance transfer correlation edge in the topological graph, and then the weight is dynamically optimized through network training, so that the model can preferentially pay attention to the cross-regional correlation with significant clinical significance, and reduce the noise connection interference between irrelevant lesions; Multi-scale node feature fusion of data information is performed: the features of each lesion node not only include SUVmax or metabolic volume traditional parameters, but also incorporate PET image texture features and metabolic heterogeneity index, which are compressed into high-dimensional feature vectors through autoencoder.

[0055] In the above embodiments, the technical principle of sparse graph neural network is based on the topological correlation mining of metabolic features in PET images, and the core is to accurately locate the high metabolic lesion area through the metabolic hot spot detection algorithm, to construct a topological graph based on the metabolic correlation between lesions, and to capture the potential correlation between long-distance metastatic lesions by using graph neural network, so as to realize efficient analysis of tumor metastasis mode. The network architecture includes four key parts: metabolic hot spot detection module, topological graph construction unit, sparse graph neural network layer and correlation feature output layer. Among them, metabolic hot spot detection and topological graph construction are the basis for capturing the correlation features of long-distance metastatic lesions, and the sparse graph neural network layer aggregates node information through graph convolution operation to strengthen the learning of transfer correlation.

[0056] The implementation of the technical solution first needs to deploy a metabolic hotspot detection algorithm. After preprocessing the input PET image (such as noise reduction, standardization), the SUV value of each voxel is calculated. The metabolic active region of interest is selected with SUVmax≥2.5 as the threshold. Through morphological operations (such as erosion, dilation), artifacts are removed and the lesion contour is extracted to determine the center point coordinates and SUV value distribution of each lesion. Then, the topology graph is constructed. Each lesion selected is regarded as an independent node. The node features include lesion volume, average SUV value, SUVmax, and other metabolic parameters. The metabolic correlation (Pearson correlation coefficient) between any two nodes is calculated. When the absolute value of the correlation coefficient is ≥0.6, an edge connection is established, and the weight of the edge is the correlation coefficient value, forming a sparse topology graph. Then, the sparse graph neural network is constructed. GraphSAGE is used as the basic architecture. The local and global correlation features of the nodes are learned through sampling neighbor nodes and aggregation operations. Finally, the correlation feature vector of the distant metastasis is output, which is used for tumor staging and metastasis risk assessment.

[0057] The implementation of the network relies on a hardware environment that supports high-resolution PET image processing, including a multi-core CPU (such as Intel Core i9) for efficient calculation of voxel-level SUV values, a high-performance GPU (such as NVIDIA RTX A6000) to accelerate node aggregation and feature learning of the graph neural network, and a large-capacity memory (≥32GB) for storing lesion features and topology graph data. In application scenarios, the network can be used for tumor metastasis diagnosis, such as in lung cancer diagnosis and treatment: the metabolic hotspot detection module identifies lung primary lesions and whole-body high-metabolism regions, the topology graph construction unit analyzes the metabolic correlation between hilar lymph nodes and brain, bone, and other site lesions, and the sparse graph neural network captures these correlation features to determine whether there is distant metastasis, assisting clinicians in developing treatment plans.

[0058] In Example 9, when processing dynamic images, the system automatically activates the timing feature enhancement module, uses 3D convolution combined with Transformer to extract motion cycle features, and the sensitivity of myocardial motion abnormality region identification is ≥90%.

[0059] The 3D convolution kernel size is set to 3×3×3 (time × height × width), the step is 1×1×1, the Transformer uses an 8-layer encoder structure, the number of multi-head attention heads is 12, and the sensitivity is defined as the true positive rate of myocardial motion abnormality region (TP / (TP+FN).

[0060] The technical principle of the time sequence feature enhancement module revolves around the accurate capture of time sequence motion features in dynamic images. The core lies in the combination of 3D convolution and Transformer, which not only extracts local spatio-temporal motion features through 3D convolution, but also captures long time sequence motion period dependence through the self-attention mechanism of Transformer, realizing the identification of abnormal motion patterns of periodic motion organs such as myocardium. The module architecture includes five key parts: dynamic image preprocessing unit, 3D convolution feature extraction layer, Transformer time sequence modeling layer, abnormal region detection head, and sensitivity optimization module. The synergistic effect of 3D convolution and Transformer is the core of extracting motion period features, and the sensitivity optimization module ensures the recognition performance by weighting the loss function (the weight of abnormal region samples is 3 times that of normal region samples). The implementation of the technical solution first preprocesses the input dynamic image (such as myocardial perfusion MRI, echocardiogram sequence), including frame standardization, motion artifact correction, and time sequence alignment, to output a fixed-length image frame sequence (such as 20 frames per cycle); then the 3D convolution feature extraction layer (using 3D ResNet-34 basic architecture) is used to perform convolution operation on consecutive frames to capture local motion changes between adjacent frames, and output a spatio-temporal feature sequence; then the feature sequence is input into the Transformer time sequence modeling layer, and the correlation weight of different time step features is calculated through the self-attention mechanism, focusing on strengthening the feature expression of key stages (such as myocardial systole and diastole) within the motion period, to generate global time sequence features; finally, the abnormal region detection head (composed of convolution layer and sigmoid activation) outputs a motion abnormality probability map, which is combined with a threshold (0.5) to determine the abnormal region, and the threshold parameter of the detection head is dynamically adjusted by monitoring the sensitivity index (≥90%).

[0061] The implementation of the module relies on a hardware environment that supports large-scale time sequence data parallel computing, including a high-performance GPU (such as NVIDIA A100) to accelerate the parallel operation of 3D convolution and Transformer, a large-capacity memory (≥40GB) for storing time sequence features, and a high-speed memory (≥64GB) to improve the read-write efficiency of dynamic image frames. In application scenarios, the module can be used for cardiac function evaluation, for example, in the diagnosis of coronary heart disease: the preprocessed dynamic myocardial image extracts local motion features through 3D convolution, the Transformer captures the overall motion law within the cardiac cycle, and the abnormal region detection head identifies the areas of reduced or uncoordinated wall motion, assisting doctors in determining the location and range of myocardial ischemia.

[0062] The time sequence feature enhancement module is implemented in the following way: The start of the motion cycle adaptive division and phase alignment mechanism: the electrocardiogram signal is introduced as the timing anchor point, and the dynamic image is divided into independent cardiac cycles through R-wave peak detection. Each cycle is automatically divided into 5 key sub-stages based on the myocardial diastole / systole phase characteristics; in view of the difference in heart rate of different patients, the dynamic time warping (DTW) algorithm is used to align the same phase across cycles in time and space, ensuring that the local motion features extracted by 3D convolution have consistency in the time dimension. 3D convolution and cross-modal attention guided fusion of Transformer: the local motion features output by the 3D convolution layer are filtered by the channel attention mechanism to generate local abnormal candidate masks; the masks are input into the self-attention layer of the Transformer as prior information, forcing the model to give 2-3 times attention weight to the suspicious areas marked by the masks when calculating the global temporal dependency, realizing the directional fusion of local fine features and global temporal correlation. Enhanced learning of motion abnormality residual features: a baseline model of normal myocardial motion is constructed, and the residual feature map of the input image and the output of the baseline model is calculated in real time by a 3D U-Net trained on 1000+ normal dynamic images; the residual feature map and the original dynamic image features are fused at the pixel level to make the 3D convolution focus on the micro-abnormal motion and improve the ability to capture early and mild abnormalities. Start of temporal sparse sampling and key frame reinforcement mechanism: in view of the high redundancy of dynamic images, an reinforcement learning strategy is used to dynamically select 10-15 key frames for dense feature extraction, and non-key frames only retain low-dimensional motion trend features; the key frame features are reinforced by the temporal position encoding of the Transformer to enhance their phase properties in the cycle, which not only reduces the computational load but also avoids the dilution of abnormal features by redundant information.

[0063] The structured report contains a treatment response prediction module that analyzes the tumor volume change rate and metabolic activity decay trend based on multi-phase image comparison and outputs personalized treatment plan recommendations in combination with the clinical knowledge base. The interface response time with the radiotherapy planning system is ≤1s.

[0064] The technical principle of the treatment response prediction module revolves around multi-modal information fusion and dynamic evaluation of treatment response, and the core is to quantify the morphological and metabolic changes of the tumor through multi-phase image analysis, combined with the treatment data of similar cases in the clinical knowledge base, to realize the prediction of the effectiveness of the current treatment plan and the generation of personalized adjustment suggestions. The module architecture includes five key parts: multi-phase image comparison unit, tumor change quantification module, clinical knowledge retrieval engine, treatment plan generator, and interface response optimization layer. The multi-phase image comparison and clinical knowledge fusion are the core of outputting accurate treatment recommendations. The implementation of the technical solution first needs to construct a multi-phase image contrast analysis unit. The images at different time points (such as before treatment, during treatment, and after treatment) are registered (rigid registration combined with elastic deformation correction, registration error ≤1 mm), and the tumor contours at each stage are extracted through a tumor automatic segmentation algorithm (such as U-Net++). Then, the tumor change quantification module calculates the volume change rate (such as a reduction of ≥30% in volume after treatment compared to before treatment is defined as partial remission) and metabolic activity attenuation trend (the rate of attenuation is determined by the linear regression slope of SUVmax). Subsequently, the clinical knowledge retrieval engine is called, and based on the tumor type, change parameters, and patient clinical characteristics (such as age, pathological stage), similar cases are matched from the knowledge base, and their treatment plans and response results are extracted. The treatment plan generator integrates the image quantification results and the retrieved clinical knowledge, and through rule-based reasoning (such as IF volume reduction < 10% and metabolic activity has no attenuation THEN suggest adjusting the chemotherapy regimen), it outputs personalized recommendations. Finally, through the interface response optimization layer (using a caching mechanism and parallel computing), the data interaction response time with the radiotherapy planning system is ensured to be ≤1 s. The implementation of this module relies on a hardware environment that supports efficient processing of multi-phase images and rapid retrieval of knowledge, including a multi-core CPU (such as Intel Xeon Platinum) for parallel execution of image registration and segmentation, a large-capacity storage (such as 1TB SSD) for caching multi-phase image data and clinical knowledge base, and a high-speed network interface (gigabit Ethernet) to improve communication efficiency with the radiotherapy system. In application scenarios, the module can be used for tumor radiotherapy and chemotherapy efficacy evaluation, for example, in lung cancer radiotherapy: comparing the CT / PET images before and after 2 weeks of radiotherapy, calculating the tumor volume reduction rate (such as 25%) and SUVmax attenuation slope (such as -0.8 / week), retrieving cases with similar attenuation trends from the knowledge base, and if 80% of the cases require increased radiotherapy dose, generating a "recommendation to increase local dose by 5Gy" plan, and synchronizing it to the radiotherapy planning system in real time through the interface.

[0065] Although the specific embodiments of the present application are described above, those skilled in the art should understand that these specific embodiments are only illustrative, and those skilled in the art can make various omissions, substitutions and changes to the details of the above-mentioned methods and systems without departing from the principles and essence of the present application. For example, combining the above-mentioned method steps, so as to perform substantially the same function in substantially the same way to achieve substantially the same result, is within the scope of the present application. Therefore, the scope of the present application is only limited by the appended claims.

Claims

1. A multimodal medical image data intelligent processing system, applied to the entire process analysis of multimodal medical images such as CT, MRI, PET, and ultrasound, characterized in that, The system utilizes cross-modal feature manifold fusion, semantically guided dynamic registration, multi-task collaborative reasoning, and clinical knowledge embedding; the system also includes: Cross-modal manifold fusion network: Differentiated coding paths are designed for the physical characteristics of different modal images. The CT / MRI branch uses an attention residual network to extract spatial-density features, the PET branch uses a sparse graph neural network to capture metabolic distribution features, and the ultrasound branch combines a time-frequency analysis network to extract dynamic texture features. The features of multiple branches are projected to a unified semantic space through Riemannian manifold mapping, and a contrastive learning loss function is used to constrain the alignment of feature distributions between modalities, outputting a 512-dimensional fusion feature vector. Semantic-guided dynamic registration engine: Based on unsupervised generative adversarial networks, a modality transformation space is constructed. The generator maps the source modality image to the target modality latent space. The discriminator introduces anatomical priors and optimizes registration parameters through dual constraints of organ boundary overlap and grayscale distribution consistency. The registration strategy is dynamically adjusted for modality differences. Edge feature guidance is used for CT and MRI registration, and functional metabolic markers are introduced for PET and ultrasound registration. The registration error is ≤1.5mm. Multi-task collaborative diagnostic network: It adopts a shared-private feature architecture. The shared layer extracts general anatomical features based on fused feature vectors, and the private task heads realize disease classification, lesion segmentation and prognosis prediction respectively. Knowledge transfer between tasks is realized through gradient inversion layer. The classification results guide the optimization of segmentation boundary, and the segmentation features enhance the spatiotemporal correlation of prognosis prediction. Clinical knowledge embedding and interpretability module: integrates a real-time updated clinical guideline knowledge base, associates imaging features with clinical indicators through knowledge graphs; generates diagnostic evidence chains using counterfactual reasoning, visualizes and annotates key decision features, outputs structured reports conforming to HL7 FHIR standards, and supports seamless integration with HIS systems; The system optimizes multi-center models through a federated learning architecture, uploads only encrypted feature gradients, and completes a model iteration every 30 days while protecting privacy, resulting in an ≥20% improvement in adaptability to new data.

2. The system according to claim 1, characterized in that, The contrastive learning loss function of the cross-modal manifold fusion network is a weighted sum of triplet loss and mutual information loss, wherein the triplet loss constrains the feature distance of similar lesions to ≤0.3, and the mutual information loss maximizes the feature correlation between modalities, where MI≥0.

75.

3. The system according to claim 1, characterized in that, The semantically guided dynamic registration engine uses a generative adversarial network with a CycleGAN++ architecture. The loss function includes adversarial loss, cycle consistency loss, and anatomical penalty term. The anatomical penalty term is implemented by calculating the hausdorff distance between the registration result and the standard organ template. Temporal smoothing constraints are used to dynamically correct the deformation caused by respiratory motion.

4. The system according to claim 1, characterized in that, The private task head of the multi-task collaborative diagnosis network adopts a dynamic weighted loss function, the disease classification head adopts a focal loss function to solve sample imbalance (γ=2), the segmentation head adopts a joint constraint of dice loss and cross-entropy loss, and the prognosis prediction head integrates the Cox proportional hazards model and attention LSTM, and the output risk score curve satisfies C-index ≥ 0.82; the implementation process of the private task head of the multi-task collaborative diagnosis network adopting a dynamic weighted loss function is as follows: First, task uncertainty is quantified: a Gaussian noise layer is introduced for each private task, and the task uncertainty is estimated by the gradient variance of the loss function during the backpropagation of the network. Tasks with high uncertainty are given low initial weights to reduce their interference with the overall loss. Then, feature contribution modeling is performed: a cross-task feature attention module is added between the task-shared feature layer and the private task head to calculate the dependency of each task on the shared features. Tasks with high dependency receive higher weights in dynamic weighting. A dynamic update mechanism is implemented: In each training round, the weight coefficient ωᵢ = exp(-σᵢ)·αᵢ / Σ(exp(-σⱼ)·αⱼ) is calculated based on the task uncertainty σᵢ and the feature contribution αᵢ, realizing real-time adjustment of weights according to task status and feature correlation, rather than adjustment by a fixed proportion or a single factor; Adaptive focal loss optimization of the disease classification head; Breaking through the fixed γ value limitation, a dynamic γ adjustment mechanism based on sample prediction confidence is designed; For input samples, the classification head outputs the prediction probability p, and the difficulty coefficient d = -p is defined; The dynamic γ value calculation formula is γ = 2·tanh(d·5), where the tanh function ensures that γ is adaptively adjusted in the interval [0,2]: Easy-to-classify samples are defined as p→1 or p→0, d→0.5, where γ→2 enhances suppression; Difficult-to-classify samples are defined as p→0.5, d→0, γ→0, reducing penalty and avoiding over-focusing on easy cases while ignoring difficult cases; The loss function expression is Loss_cls = -ω·(1-p)ᵞ·log(p), where ω is the class weight, γ changes dynamically with the sample difficulty, and focuses more accurately on difficult-to-classify samples than a fixed γ=2; the region adaptive joint loss constraint of the segmentation head; the weights of diceloss and cross-entropy loss are dynamically balanced based on the edge complexity of the segmentation region; the region edge complexity index C is introduced: the edge contour of the segmentation target is extracted by the Sobel operator, and the mean curvature of the edge pixels is calculated. The higher the mean curvature, the larger C is; the dynamic balance coefficient β=sigmod(C), where the sigmod function ensures that β changes in the interval [0,1]: C is large in edge complex regions, β→1, at this time Loss_seg=β·dice_loss+ (1-β)·cross_entropy_loss, which enhances the constraint of dice loss on the edge; in the internal region, C is small, β→0, which enhances the classification accuracy of cross-entropy on the pixels inside the region; The temporal risk attention fusion mechanism of the prognostic prediction head is constructed using a three-level architecture, including a temporal feature selection module, a risk weight allocation module, and a dual-model gating fusion module, to achieve deep collaboration between the Cox model and the attention LSTM; the steps are as follows: For temporal feature selection, attention-based LSTM is used to process patient examination data at multiple time points. The output of the LSTM hidden layer is passed through an attention layer to calculate the risk attention level of each feature at each time point, focusing on key time nodes strongly correlated with prognosis. For risk weight allocation, the temporal feature vector output by the attention-based LSTM is concatenated with the patient's static features and input into the Cox proportional hazards model. The baseline risk is calculated using the risk proportion function h(t) = h0(t)·exp(x·β), where x includes attention-weighted temporal features and β is the risk coefficient. For gated fusion, a gate unit is introduced to fuse the temporal prediction results of the LSTM with the Cox risk proportion. The gate coefficient is dynamically adjusted based on the historical prediction errors of both, and the final output is a risk score curve. By forcing the gradient constraint of the gate unit, the consistency of prognostic prediction is steadily improved.

5. The system according to claim 1, characterized in that, The knowledge graph of the clinical knowledge embedding module is expanded in real time through the ontology reasoning engine, containing 5,000+ disease-image-treatment association rules. The evidence chain generation uses SHAP value to quantify the contribution of each modality, with CT structural features having a weight of 0.4±0.1 and PET metabolic features having a weight of 0.3±0.

05. It supports doctors to interactively adjust the evidence weights and provide feedback to optimize the model.

6. The system according to claim 1, characterized in that, The federated learning architecture employs hierarchical knowledge distillation. The central server aggregates the teacher network parameters of each node model and generates a lightweight student model through knowledge transfer, which is then deployed on edge devices. The inference latency is ≤300ms, the model accuracy loss is ≤2%, and homomorphic encryption is used to protect gradient transmission.

7. The system according to claim 1, characterized in that, The attention residual network of the CT / MRI branch includes an adaptive window width and window level adjustment module, which automatically sets WW=1500HU and WC=-600HU for lung tissue and WW=400HU and WC=40HU for liver, improving the signal-to-noise ratio of feature extraction in low-contrast regions by ≥30%.

8. The system according to claim 1, characterized in that, The sparse graph neural network of the PET branch employs a metabolic hotspot detection algorithm. It filters regions of interest using an adaptive threshold of SUVmax ≥ 2.5, constructing a topological graph with lesions as nodes and metabolic correlations as edges to capture the association features of distant metastases. The specific method is as follows: A dynamic threshold stratification screening mechanism is initiated: Based on the baseline threshold of SUVmax≥2.5, the stratification threshold is dynamically adjusted in combination with the individual metabolic baseline of the patient. For high-metabolic tumors, the threshold is increased by 15%-20%, and for low-metabolic metastatic lesions, the threshold is decreased by 10%-15%. At the same time, isolated false positive hotspots are eliminated through 3D connected domain analysis to solve the problem of insufficient adaptability of fixed thresholds to different tumor types. Then, a metabolic-spatiotemporal multidimensional correlation edge model is constructed: the edge weights are not only based on metabolic correlations, but also incorporate spatiotemporal features: the curve morphology similarity is calculated by extracting the time-activity curve of the lesion through PET dynamic imaging; spatial correlation weights are assigned by combining the relative position of the lesion in the anatomical space, so that the edges of the topology map can reflect metabolic synergy and capture anatomical metastasis patterns. By introducing a transition probability attention layer into the graph neural network through the sparse graph attention enhancement mechanism, prior knowledge is trained based on the transition patterns of historical cases. Initial attention bias is assigned to potential distant transition association edges in the topological graph. The weights are then dynamically optimized through network training, so that the model prioritizes clinically significant cross-regional associations and reduces noise connection interference between irrelevant lesions. Multi-scale node feature fusion of data information: The features of each lesion node not only include traditional parameters such as SUVmax or metabolic volume, but also incorporate the texture features and metabolic heterogeneity index of PET images, which are compressed into high-dimensional feature vectors through an autoencoder.

9. The system according to claim 1, characterized in that, When processing dynamic images, the system automatically activates the temporal feature enhancement module, which uses 3D convolution combined with Transformer to extract motion cycle features, achieving a recognition sensitivity of ≥90% for abnormal myocardial motion regions. The temporal feature enhancement module is implemented in the following ways: An adaptive segmentation and phase alignment mechanism for motion cycles is initiated: ECG signals are introduced as temporal anchors, and dynamic images are segmented into independent cardiac cycles through R-wave peak detection. Within each cycle, five key sub-stages are automatically divided based on myocardial diastolic / systolic phase characteristics. To address the differences in heart rate among different patients, the Dynamic Time Warping (DTW) algorithm is used to perform spatiotemporal alignment of the same phases across cycles, ensuring that the local motion features extracted by 3D convolution have consistency in the temporal dimension. Cross-modal attention-guided fusion of 3D convolution and Transformer: Local motion features output by 3D convolutional layers are filtered for high-response regions through channel attention mechanism to generate local anomaly candidate masks; these masks are used as prior information input to the self-attention layer of Transformer, forcing the model to assign 2-3 times the attention weight to suspicious regions marked by the mask when calculating global temporal dependencies, thus achieving targeted fusion of local fine features and global temporal correlations; Reinforcement learning of residual features of motion abnormalities: Construct a baseline model of normal myocardial motion patterns. Train a 3D U-Net with 1000+ normal dynamic images to calculate the residual feature maps of the input images and the output of the baseline model in real time. Perform pixel-by-pixel weighted fusion of the residual feature maps and the original dynamic image features to make the 3D convolution more focused on small abnormal motions and improve the ability to capture early mild abnormalities. Initiating a temporal sparse sampling and keyframe enhancement mechanism: To address the high redundancy of dynamic images, a reinforcement learning strategy is adopted to dynamically select 10-15 keyframes for dense feature extraction, while non-keyframes retain only low-dimensional motion trend features; keyframe features are enhanced in phase attribute within the period through temporal position encoding of Transformer, which reduces computation and avoids dilution of anomalous features by redundant information.

10. The system according to claim 1, characterized in that, The structured report includes a treatment response prediction module, which analyzes the tumor volume change rate and metabolic activity decay trend based on multi-phase image comparison, and outputs personalized treatment plan suggestions in conjunction with a clinical knowledge base. The interface response time with the radiotherapy planning system is ≤1s.

Images