A precise segmentation and intelligent diagnosis method for cervical spinal cord injury multi-modal images
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2026-04-17
- Publication Date
- 2026-06-26
Smart Images
Figure CN122289240A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of medical image processing, deep learning and intelligent diagnosis technology. Specifically, it relates to a method and system for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury. It is applicable to the early diagnosis, detection of small lesions, assessment of disease severity, formulation of individualized treatment plans and prognosis prediction of cervical spinal cord injury. It can be applied to scenarios such as clinical diagnosis and treatment of spinal surgery, medical image-assisted diagnosis, and multi-center medical collaboration. Background Technology
[0002] Cervical spinal cord injury is a leading cause of disability following violent neck trauma, imposing a heavy burden on individuals, families, and society. Current diagnosis, treatment, and prognosis still rely on traditional methods, requiring a comprehensive assessment combining imaging, physical examination, and medical history. However, individual patient differences and variations in physician skill levels lead to inaccurate and personalized diagnostic results, resulting in practical problems such as insufficient early intervention and imprecise treatment strategies.
[0003] With the widespread application of artificial intelligence (AI) and deep learning (DL) technologies in the medical field, they have demonstrated strong potential in spinal surgery, especially in assisted diagnosis. Internationally, numerous research teams have applied DL technology to imaging diagnosis, and various basic AI models and innovative algorithms are constantly emerging, providing valuable tools for assisted diagnostic research and development. In prognostic assessment and treatment planning, AI technology can automatically uncover potential relationships between multidimensional data such as imaging and clinical findings, providing a more objective and accurate basis for prognostic judgment and helping physicians formulate scientifically sound treatment strategies.
[0004] However, existing technologies still face many challenges in diagnosis and treatment:
[0005] 1. Insufficient accuracy in detecting minute lesions: Existing models mostly focus on detecting obvious lesions, and their prediction accuracy for minute lesions (≤3mm²) in areas with small fields of view is poor. The edges of the lesions are easily blurred, resulting in a high rate of missed diagnoses of minute lesions (the detection rate of existing methods is only about 70%). 2. Data adaptability issues: Existing relevant datasets suffer from imbalanced data distribution, with significant differences between samples, affecting the model's generalization performance; furthermore, there are challenges in aligning text and image data in subsequent predictions and treatment plan development, requiring the rational use of various data types while minimizing errors. 3. Poor performance of multimodal data fusion: Diagnosis requires the combination of multiple sequences of magnetic resonance imaging such as T1, T2, and short-time inversion recovery, as well as clinical data such as the American Spinal Cord Injury Association score and medical history. However, existing methods lack efficient multimodal fusion mechanisms, making it difficult to fully explore cross-modal complementary information, which limits the accuracy of diagnosis and prognosis prediction. 4. Insufficient model practicality and generalization: Existing segmentation models mostly rely on manual interactive prompts and cannot achieve fully automatic end-to-end segmentation; moreover, the models are large in size, have slow inference speed, are easily affected by differences in data distribution when applied across institutions, have poor generalization performance, and are difficult to meet the clinical application needs of primary hospitals. 5. Lack of personalized treatment plans: Different patients respond differently to the same treatment plan. Existing methods are unable to predict individual treatment effects by comprehensively considering multiple factors such as patient age, medical history, and imaging data, resulting in insufficient targeting of treatment plans and a high clinical decision-making error rate (15-20%).
[0006] To address these issues, existing research has attempted optimization through data augmentation and model structure adjustments, but a complete solution has yet to be found. For example, some studies use traditional segmentation models such as unified networks and neural network unified networks to process images, but their sensitivity to small lesions is limited; some studies explore multimodal data fusion, but the fusion strategies are simple and fail to fully utilize the intermodal correlation information; in addition, most existing models do not consider cross-institutional data privacy and distribution differences, making it difficult to achieve widespread clinical application.
[0007] In view of this, the present invention proposes a method and system for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury. By optimizing scanning strategies, designing efficient multimodal fusion mechanisms, improving model architecture, and introducing lightweight federated learning, it solves the problems of low detection rate of small lesions, poor multimodal data fusion effect, insufficient model generalization, and lack of personalized treatment plans in existing technologies, and provides efficient, accurate and objective decision support for clinical diagnosis and treatment. Summary of the Invention
[0008] (a) The technical problem to be solved by the present invention: The purpose of this invention is to address the technical problems in existing diagnosis and treatment, such as high missed rates of small lesions, poor multimodal data fusion, insufficient model generalization and practicality, and lack of personalized treatment plans. This invention provides a method and system for precise multimodal image segmentation and intelligent diagnosis of cervical spinal cord injury. This method achieves fully automatic lesion segmentation by optimizing image feature extraction strategies, designing a lightweight and efficient multimodal fusion mechanism, improving arbitrary segmentation models, fusing image and clinical data for diagnostic prediction, and combining dynamic causal reinforcement learning to generate individualized treatment plans. Furthermore, it achieves cross-institutional generalization through lightweight federated learning, ultimately improving early diagnostic accuracy, the detection rate of small lesions, and the targeting of treatment plans, while reducing clinical decision-making errors.
[0009] (II) To achieve the above objectives, the present invention adopts the following technical solution: A method for precise image segmentation and intelligent diagnosis of cervical spinal cord injury, comprising the following steps: S1. Multimodal Data Acquisition and Preprocessing: T1, T2, and short-time inversion recovery sequence magnetic resonance imaging data of patients with cervical spinal cord injury were acquired, along with clinical data including Spinal Cord Injury Association (SCIA) scores, medical history records, and specialist physical examinations. The imaging data underwent denoising, standardization, and data augmentation (random rotation, uniform scaling, mirror flipping, and Gaussian noise addition). The clinical structured data was standardized using standard scores, and the unstructured clinical text was encoded into a 768-dimensional semantic feature vector using a bidirectional bio-clinical encoder representation model. Key semantic features were extracted using attention pooling.
[0010] S2. Image Feature Extraction Based on Local Scan Markov Monte Carlo Model [Huang T, Pei XH, You S, et al. LocalMamba: Visual State Space Model with Windowed Selective Scan [C] / / Computer Vision – ECCV 2024 Workshops. Cham: Springer Nature Switzerland, 2024: 113-130.]: Based on Improved Segmentation Arbitrary Model Architecture [Kirillov A, Mintun E, Ravi N, et al. Segment Anything [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 113-130.] [3992-4003.] The cue encoder of the original arbitrary segmentation model is removed, and a local scanning Markov Monte Carlo scanning module is embedded in the transformer layer; the linear coding layer in the Markov Monte Carlo block is replaced with a multilayer perceptron to improve the feature depth representation capability; the magnetic resonance image is divided into 81 information feature blocks according to the size of the magnetic resonance image, and the central feature block and the surrounding related feature blocks containing the core information of the lesion are selected; a weighted scanning strategy is adopted, with the central feature block scanned 4 times and the surrounding low foreground feature blocks scanned once each, to enhance the extraction of small lesion features and suppress background interference; the scanned feature sequence is input into the improved Markov Monte Carlo module, and the spatiotemporal correlation model of the feature is completed by combining the state space model (SSM) to output the depth features of the single-modal magnetic resonance image.
[0011] S3, Weighted Lightweight Cross-Attention Multimodal Image Fusion: We design a weighted lightweight cross-attention mechanism to achieve low-parameter adaptive fusion of T1, T2 and short-time inversion recovery (STMI) image features; The three-modal basic features are extracted by a shared segmentation arbitrary model encoder to obtain modal features. , , Where N is the number of features, c is the feature dimension, and X Q X is the target query modal feature. A X is a T2-weighted modal feature. B For T1 weighted modal features, ∈ represents the membership symbol. It is the set of real numbers; Constructing a set of trimodal learnable agents: T2 agents T1 Agent Short-term reversal recovery agent ,and , where n A Let n be the number of weighted proxy features for T2. B Let n be the number of weighted proxy features for T1. S To recover the number of proxy features in a short time reversal; Refined features are generated through intramodal attention interactions: ,
[0012] Among them O A For T2 weighted modal refinement features, O B For T1 weighted modal refinement features, Softmax is the normalized exponential function, and the superscript... This is the matrix transpose symbol; Cross-modal fusion was completed using a short-time inversion recovery agent as a guide, resulting in:
[0013] Among them O S For cross-modal fusion features, [;] represents the feature concatenation symbol; Finally, by querying the attention interaction between modalities and fused features, a trimodal aligned image feature map is generated. Output a three-modal aligned image feature map, where O final This is the final three-modal alignment feature.
[0014] S4. Fully automated lesion segmentation via progressive upsampling: A progressive upsampling decoder is built, which combines transposed convolution and attention mechanisms to fuse multi-scale image features; The trimodal aligned feature map is input into the decoder, and the feature map is upsampled at multiple scales through 2D transposed convolution to capture lesion features at different resolutions. An attention mechanism is introduced to fuse mask prior information with multi-scale image features, forming a unified feature representation that includes lesion location and morphological information; Multi-scale fusion features are enhanced to highlight key signals of small lesions, and all features are aligned to a uniform spatial resolution through bilinear interpolation. The aligned multi-scale features are fused, and a pixel-level lesion segmentation mask is output through a convolutional layer to achieve fully automatic end-to-end segmentation of cervical spinal cord injury lesions.
[0015] S5. Imaging-Clinical Multimodal Data Fusion Diagnosis: Construct a hierarchical graph attention fusion network, using the lesion segmentation features obtained in step S4 and the clinical features processed in step S1 as heterogeneous nodes; Based on the anatomical prior knowledge of cervical vertebral segment functional zoning, an initial adjacency matrix is constructed and dynamically optimized through learnable weights; The importance of nodes within the same modality is calculated using an in-layer attention mechanism, as shown in the formula:
[0016] Where a ij Here, σt represents the in-layer attention weights, exp is the exponential function, LeakyReLU is the linear unit activation function with leakage correction, W is the learnable weight matrix, and ht is the in-layer attention weights. i h j h k N represents the feature vectors of different nodes, | represents the feature concatenation symbol, and N represents the feature vectors of different nodes. i Let Σ be the set of adjacent nodes of node i, and let Σ be the summation symbol to achieve weighted fusion of features of the same modality. A bilinear attention mechanism is introduced to capture cross-modal nonlinear correlations. The weight calculation formula is as follows:
[0017] Where βpq is the inter-layer cross-modal attention weight, σ is the sigmoid activation function, U, V, W are learnable parameter matrices, tanh is the hyperbolic tangent activation function, and h p h is the image feature node vector. q This is a vector of clinical feature nodes, capturing the non-linear relationship between images and clinical features; The features fused within and between layers are concatenated to output a 512-dimensional multimodal fused feature vector. H fusion For multimodal fusion features, ∈ represents the membership symbol. Let be the set of real numbers, and 512 be the dimension of the feature vectors; Design a comparative learning objective function (s( () represents the cosine similarity. (where N is the batch number of samples), and force the multimodal features of similar samples to align in the latent space; using the multimodal fusion features as input, train a multi-task model including a classification head and a regression head. The classification head implements the American Spinal Cord Injury Association (ASPA) five-level classification, and the regression head completes the quantitative assessment of the degree of injury, outputting the diagnostic results.
[0018] S6. Personalized treatment suggestion generation and clinical decision support: Based on the diagnosis and prognostic prediction results of step S5, and combined with a dynamic causal reinforcement learning model, the causal effect of treatment intervention on prognosis is quantified. The formula for constructing a dynamic structural causal model (DSCM) is as follows:
[0019] Where Y(t) is the prognostic result at time t, α, β, and γ are learnable coefficients, T is the treatment variable, X is the observed confounding factor, and U is the latent variable (its distribution is estimated by variational autoencoder VAE). This is the error term; Using patient baseline characteristics (age, injury grade, etc.) and real-time treatment feedback (drug dosage, rehabilitation frequency, etc.) as state inputs, a reinforcement learning agent is constructed, and a proximal policy optimization algorithm is used to optimize the policy network. A multidimensional reward function was designed, using the improvement rate of the American Spinal Cord Injury Association score (weight 0.6), the incidence of complications (weight 0.3), and medical costs (weight 0.1) as core indicators to calculate the real-time reward value. Through the strategy iteration of the reinforcement learning agent, the optimal individualized treatment plan (including intervention methods and parameters for drug treatment, surgical treatment, and rehabilitation training) is output. The treatment plan and diagnosis results are input into a lightweight federated learning system. A neural architecture search algorithm without agents is used to compress the model, reducing the number of model parameters to 30% of the original. Tensor RT 8-bit integer quantization is used to improve inference speed by 3 times. Gaussian noise is added during the model parameter aggregation stage. ),satisfy With a privacy budget, we achieve cross-agency data privacy protection and rapid model inference, with a system response time of ≤10 seconds per case.
[0020] Preferably, the data augmentation strategies in S1 include random rotation, uniform scaling, mirror flipping, and Gaussian noise addition to improve the model's generalization ability. In clinical data preprocessing, structured data is standardized by standard scores and then input into a multilayer perceptron (MLP). Unstructured text is encoded by a biological clinical bidirectional encoder representation model and then key semantic features are extracted through attention pooling to ensure the effective use of clinical data.
[0021] Preferably, the core of the local scanning Markov Monte Carlo scanning module in S2 lies in strengthening the feature extraction of the central lesion region and suppressing background noise interference through a partitioned weighted scanning strategy, thus solving the problem that small lesions are easily overlooked due to traditional Markov Monte Carlo and global scanning with transformers; the state-space model (SSM) uses ordinary differential equations , ( , , To model the spatiotemporal correlation of features (system evolution and projection parameters), the model is discretized using zero-order preserve (ZOH) and then converted into a convolutional neural network. (M is the convolution kernel), ensuring efficient feature extraction.
[0022] Preferably, the weighted lightweight cross-attention mechanism in S3 achieves low-parameter adaptive fusion through a learnable proxy set. Only the proxy set parameters (A, B, S) need to be trained to complete the progressive semantic integration of trimodal features, avoiding the problems of large number of parameters and computational complexity in traditional fusion methods, while ensuring effective interaction of cross-modal information.
[0023] Preferably, the progressive upsampling decoder in S4 improves the boundary clarity and structural coherence of small lesion segmentation by introducing multi-scale feature fusion and mask prior information, thus solving the problem of low segmentation accuracy caused by blurred boundaries and strong heterogeneity of small lesions in the image; the segmentation mask output by the decoder can be directly used for quantitative analysis of lesion areas (such as volume and area calculation), providing quantitative basis for subsequent diagnosis.
[0024] Preferably, the multi-task model training in S5 uses weighted cross-entropy loss (the class weight is inversely proportional to the sample size) to optimize the hierarchical task, and uses a composite loss function of Dess loss + focus loss to optimize the pixel-level localization task of small lesions, thus solving the problems of sample imbalance and low segmentation accuracy of small lesions. The model adopts a transfer learning strategy and is pre-trained on the publicly available spine dataset (54,618 cases) of the North American Spinal Cord Injury Clinical Data Center. The clinical text is pre-trained on the clinical notes dataset of the Intensive Care Medical Information Database to improve the convergence speed and generalization performance of the model.
[0025] Preferably, the lightweight federated learning system in S6 ensures the model's cross-institutional generalization (area under curve fluctuation ≤ 0.05, F1 score decay ≤ 5%) through techniques such as neural architecture search, weight quantization, and adversarial domain adaptation (gradient inversion layer aligns with data distribution across different institutions), while also meeting the hardware requirements of primary hospitals (model size ≤ 50MB, GPU memory usage ≤ 2GB); dynamic causal reinforcement learning quantifies the potential causal effects of different treatment interventions through a counterfactual reasoning framework, ensuring the scientific validity and targeted nature of individualized treatment plans.
[0026] This invention also provides a multimodal image precision segmentation and intelligent diagnosis system for cervical spinal cord injury, used to implement the above method, the system comprising: Data acquisition module: Used to acquire T1, T2, short-time inversion recovery sequence magnetic resonance imaging data and clinical data (Spinal Cord Injury Association score, medical history, specialist physical examination) of patients with cervical spinal cord injury. It supports the access and storage of data from multiple devices and in multiple formats, ensuring data integrity and availability.
[0027] Image preprocessing module: Connected to the data acquisition module, it is used to denoise, standardize, and augment magnetic resonance images, standardize clinical structured data with standard scores, encode unstructured clinical text using a bio-clinical bidirectional encoder representation model and attention pooling, and output the processed image data and clinical feature vectors.
[0028] Local scanning Markov Monte Carlo feature extraction module: Connected to the image preprocessing module, it is used to embed the local scanning Markov Monte Carlo scanning module. Through the partitioned weighted scanning strategy and state space model, it completes the deep feature extraction of multimodal magnetic resonance images, enhances the features of small lesions, and suppresses background interference.
[0029] Weighted Lightweight Multimodal Fusion Module: Connected to the local scanning Markov Monte Carlo feature extraction module, it is used to achieve low-parameter adaptive alignment and fusion of T1, T2, and short-time inversion recovery three-modal image features through a weighted lightweight cross-attention mechanism, and output a three-modal fusion feature map.
[0030] Progressive segmentation module: Connected to the weighted lightweight multimodal fusion module, it is used to achieve fully automatic pixel-level segmentation of cervical spinal cord injury lesions through progressive upsampling decoder, and output lesion segmentation mask and quantitative analysis results (volume, area).
[0031] Multimodal diagnostic module: Connected to the progressive segmentation module and the image preprocessing module respectively, it is used to achieve deep fusion of image segmentation features and clinical features through a hierarchical graph attention fusion network. Based on the multi-task model, it outputs the grade (American Spinal Cord Injury Association AE grade), quantitative assessment results of injury degree and prognosis prediction (motor function recovery within 3 months).
[0032] Intelligent Decision Module: Connected to the multimodal diagnostic module, it generates personalized treatment recommendations based on dynamic causal reinforcement learning. It achieves cross-institutional generalization and rapid inference through a lightweight federated learning system, and outputs a decision report containing treatment intervention methods, parameters, and expected effects.
[0033] Human-computer interaction module: Connected to the multimodal diagnosis module and intelligent decision-making module respectively, it is used to visualize the lesion segmentation results (mask overlay display), browse multimodal images in linkage, display diagnostic reports and treatment suggestions, support one-click export of diagnostic reports and adjustment of treatment plan parameters, and adapt to the operating habits of clinicians.
[0034] Preferably, the lightweight federated learning system employs an agentless neural architecture search algorithm for model compression, reducing the number of model parameters to 30% of the original. It also improves inference speed by 3 times through tensor RT octet integer quantization, and adds Gaussian noise during the model parameter aggregation stage. ),satisfy The privacy budget enables privacy protection and model generalization of cross-institutional data; the human-computer interaction module supports the overlay display of lesion segmentation results, the linked browsing of multimodal images, and the one-click export of diagnostic reports. It also provides a parameter adjustment interface for treatment plans, allowing clinicians to modify intervention strategies according to actual conditions.
[0035] (III) The beneficial effects of the present invention include the following: (1) The detection rate of small lesions has been significantly improved: This invention utilizes a partitioned weighted scanning strategy in a localized Markov Monte Carlo scanning module to enhance feature extraction of the central lesion region, suppress background interference, and, in conjunction with a progressive upsampling decoder and multi-scale feature fusion, effectively target small lesions of the cervical spinal cord (≤3 mm). 2 The Dessian segmentation coefficient reached 63.87%, and the detection rate was increased to 80%, solving the problem of high missed diagnosis rate of small lesions in existing methods.
[0036] (2) Excellent multimodal data fusion effect: We designed a weighted lightweight cross-attention mechanism to achieve low-parameter adaptive fusion of trimodal image features, and constructed a hierarchical graph attention network to fuse images and clinical data, fully exploring cross-modal complementary information, achieving a diagnostic accuracy of ≥90% and an area under the curve minus receiver operating characteristic of ≥0.91, which is significantly better than the diagnostic performance of existing methods.
[0037] (3) Achieve fully automated end-to-end segmentation: The model architecture for segmenting arbitrary lesions has been improved by removing the prompt encoder and combining local scanning Markov Monte Carlo feature extraction with progressive upsampling decoding to achieve fully automatic lesion segmentation without the need for manual interaction prompts, significantly reducing the operating costs for physicians and improving diagnostic efficiency.
[0038] (4) Individualized treatment plans are highly targeted: Based on dynamic causal reinforcement learning and dynamic structural causal model, the causal effect of treatment intervention on prognosis is quantified. Personalized treatment plans are generated by combining the patient's multidimensional characteristics. The error rate of treatment effect prediction is ≤8%, which is significantly lower than the existing clinical decision error rate (15-20%), which helps to improve treatment effect and shorten the rehabilitation cycle.
[0039] (5) The model has high generalization and practicality: The lightweight federated learning system achieves model compression (reducing the number of parameters by 70%) and cross-institutional generalization. The system response time is ≤10 seconds per case, and the GPU memory usage is ≤2GB. It is suitable for the hardware conditions of primary hospitals and can be stably applied in multiple centers of more than three tertiary hospitals, which has broad clinical application value.
[0040] (6) Comprehensive clinical decision support: The system integrates functions such as lesion segmentation, diagnostic grading, prognosis prediction, and generation of individualized treatment suggestions. It provides a visual interactive interface, supports one-click export of diagnostic reports and adjustment of treatment plan parameters, provides comprehensive decision support for clinicians, reduces human error, and ensures the consistency and accuracy of diagnostic results. Attached Figure Description
[0041] Figure 1 This is an overall flowchart of the multimodal image precise segmentation and intelligent diagnosis method for cervical spinal cord injury proposed in this invention; Figure 2 This is a schematic diagram of the structure of the local scanning Markov Monte Carlo scanning module in this invention, illustrating image partitioning, weighted scanning strategy, and integration with the Markov Monte Carlo module; Figure 3 This is a schematic diagram of the framework of the weighted lightweight cross-attention mechanism in this invention, illustrating the intra-modal interaction and cross-modal fusion process of trimodal features; Figure 4 This is a diagram of the overall architecture of the local scanning Markov Monte Carlo model, which shows the cervical spinal cord injury lesion segmentation framework based on the encoder-decoder structure of the segmentation arbitrary model. The architecture includes three core components: a local scanning Markov Monte Carlo scanning module, a weighted lightweight cross-attention fusion module, and a progressive upsampling decoder. Figure 5 To demonstrate the Markov Monte Carlo local perception scanning paradigm based on Hilbert curves, we achieve dense aggregation and dependency enhancement of spatiotemporal neighborhood features by traversing all elements in the space using curves and strengthening the preservation of locality during high-dimensional mapping. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] This invention proposes a method for precise segmentation and intelligent diagnosis of multimodal images in cervical spinal cord injury. First, it acquires and preprocesses multiple sequences of MRI images and clinically relevant data from the patient, performing denoising, standardization, and data augmentation. By optimizing the feature scanning method, dividing information blocks, and weighting scans, it enhances the features of small lesions and reduces background interference. A lightweight cross-attention mechanism is employed to adaptively fuse multimodal image features, fully utilizing complementary information from different images. The segmentation model structure is improved to achieve fully automated end-to-end segmentation of cervical spinal cord injury lesions. A hierarchical graph attention fusion network is constructed, combining lesion segmentation features with multidimensional clinical data to train diagnostic and prognostic prediction models, outputting intelligent diagnostic results and individualized treatment recommendations. This invention also provides a supporting diagnostic system, comprising four modules: data acquisition, image segmentation, multimodal fusion diagnosis, and clinical decision support.
[0044] This invention can automatically and accurately segment lesions, improve the detection rate and diagnostic accuracy of small lesions, and complete intelligent diagnosis and prognostic assessment based on multimodal data, providing efficient and objective decision-making basis for clinical practice. It is suitable for early screening, disease assessment and individualized treatment plan formulation of cervical spinal cord injury, and has high clinical application value.
[0045] The following description, in conjunction with relevant accompanying drawings and specific examples, illustrates the method and system for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury proposed in this invention.
[0046] Example 1: Please see Figure 1-5 This invention proposes a method for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury, comprising the following steps: S1. Multimodal data acquisition and preprocessing: S101. Data Collection: Multimodal data of 590 patients from Tianjin University Tianjin Hospital and the First Affiliated Hospital of Hebei North University were collected from July 2019 to June 2024, including T1, T2, and short-time inversion recovery sequence magnetic resonance imaging data, as well as clinical data such as the American Spinal Cord Injury Association score, medical history records, and specialist physical examinations; the resolution of the imaging data was uniformly adjusted to 512×512, and the clinical data were classified and organized into structured and unstructured categories.
[0047] S102. Image Preprocessing: Gaussian filtering is used to denoise the magnetic resonance images, and standard score normalization is used to map the image pixel values to a uniform distribution. Data augmentation strategies such as random rotation (0-360°), uniform scaling (0.8-1.2 times), mirror flipping (horizontal / vertical), and Gaussian noise addition (signal-to-noise ratio = 15dB) are used to expand the diversity of training samples. Black borders are filled for images smaller than 512×512, and images larger than this size are adjusted to the target resolution through bilinear interpolation.
[0048] S103. Clinical Data Preprocessing: Structured data such as Spinal Cord Injury Association (SCIA) scores, age, and disease duration are standardized using standard scores. The formula is as follows:
[0049] in The mean, Standard deviation; After standardization, the data is input into a multilayer perceptron (MLP). For unstructured texts such as medical history records and specialist physical examinations, a pre-trained bio-clinical bidirectional encoder representation model is used to encode the data into a 768-dimensional vector. Key semantic features are then extracted using attention pooling, as shown in the formula:
[0050] Where h_CLIN is the clinical semantic feature vector, and x_text is the input clinical text.
[0051] S2. Image feature extraction based on local scanning Markov Monte Carlo: S201, Improved architecture of arbitrary segmentation model: Based on the original arbitrary segmentation model framework, retain its encoder and decoder structure, remove the cue encoder to adapt it to the requirements of fully automatic segmentation; only fine-tune the top-level parameters of the base model of arbitrary segmentation model (arbitrary segmentation model-B), freeze the first 3 convolutional kernels, and reduce the number of training parameters.
[0052] S202. Construction of Local Scanning Markov Monte Carlo Module: The linear coding layer in the Markov Monte Carlo block is replaced with a 3-layer multilayer perceptron (hidden layer dimension is 1024) to improve the feature depth representation capability; based on the 512×512 image size, it is divided into 9×9=81 information feature blocks (each feature block size is 56×56); one central feature block and three surrounding related feature blocks are selected as the key scanning area, and the remaining feature blocks are non-key areas.
[0053] S203, Weighted Scanning Strategy Execution: Four scans are performed on the central key feature blocks, one scan is performed on each of the surrounding related feature blocks, and one scan is performed on non-key areas. This strategy enhances the capture of lesion area features and suppresses background interference. The scanned feature sequences are input into the Markov Monte Carlo module and combined with the State Space Model (SSM) for spatiotemporal correlation modeling. The SSM is discretized by zero-order hold (ZOH) and converted into a convolutional form to achieve parallel computing and improve feature extraction efficiency.
[0054] S204. Feature Output: The feature dimension output by the Markov Monte Carlo module is C×T×H / 4×W / 4 (C is the number of feature channels, T is the sequence length), which serves as the depth feature of the single-modal image for subsequent multimodal fusion.
[0055] S3, Weighted Lightweight Cross-Attention Multimodal Image Fusion: S301. Trimodal Basic Feature Extraction: Basic features of the T1, T2, and short-time inversion recovery sequences are extracted using a shared segmentation arbitrary model encoder. , , (N=512×512=262144, c=256).
[0056] S302, Construction of Learnable Agent Set: Initialize the trimodal learnable agent set, T2 agent ( T1 Agent ( ), short-term reversal recovery agent ( The proxy set parameters are optimized along with the model training.
[0057] S303, Intramodal Feature Interaction: Refinement of single-modal features through an attention mechanism, T1 modality refinement features. T2 modal refinement features The refinement process enhances semantic consistency within modalities.
[0058] S304, Cross-modal feature fusion: and By splicing along the channel dimension, we obtain ;Reverse the agent in a short time To guide the computation of cross-modal fusion features This achieves complementary enhancement of T1 and T2 modal features.
[0059] S305, Feature Fusion Alignment: Modal Features Recovered by Short-Time Inversion To query the modality, calculate and The attention weights are used to obtain the final trimodal aligned feature map. The feature map dimension is Softmax is the normalized exponential function, and the superscript... This is the matrix transpose symbol. Let N be the set of real numbers, N be the number of features, and c be the feature dimension.
[0060] S4. Fully automated lesion segmentation via progressive upsampling: S401. Decoder Construction: Construct a progressive upsampling decoder consisting of 3 upsampling blocks and 3 convolutional blocks. The upsampling blocks use 2D transposed convolution (stride = 2, kernel size = 3×3), and the convolutional blocks use 3×3 convolution + batch normalization + modified linear unit activation function.
[0061] S402, Multi-scale Feature Fusion: [This section appears to be incomplete and requires further context.] The input to the decoder is upsampled to H / 2×W / 2 by the first transposed convolution, and then concatenated with the corresponding scale features output by the encoder of any segmentation model. The features are then fused through convolutional blocks. The above process is repeated, and the input is upsampled to H×W for the second time, and then fused with the encoder features again to enhance the detailed information.
[0062] S403, Mask Prior Fusion: An attention mechanism is introduced to combine lesion mask prior information (a coarse mask generated based on anatomical priors) with multi-scale fusion features, as shown in the formula: , where h fusion For the final fusion feature, h multi scale For multi-scale fusion features, h mask The mask uses prior features, and Attention is the attention calculation function, which improves the accuracy of lesion boundary localization.
[0063] S404, Segmentation Mask Generation: The fused features are mapped to a single channel through a 1×1 convolution, and a pixel-level segmentation mask is output through a sigmoid activation function (threshold=0.5). In the mask, a pixel value of 1 represents a lesion area, and a pixel value of 0 represents a normal area. Quantitative indicators such as lesion volume and area are calculated to provide a basis for subsequent diagnosis.
[0064] S5. Imaging-Clinical Multimodal Data Fusion Diagnosis: S501, Hierarchical Graph Attention Network Construction: The lesion segmentation features output by S4 (flattened into a 512-dimensional vector) and the clinical features output by S1 (512-dimensional vector) are used as heterogeneous nodes; based on cervical segment functional zoning knowledge (such as the correlation between C3-C5 segment injury and upper limb motor function), an initial adjacency matrix is constructed. Matrix elements This represents the initial association strength between nodes i and j.
[0065] S502. Intra-layer attention calculation: Within the image modality node set and the clinical modality node set, the node importance weight is calculated separately, using the following formula:
[0066] in Let W be the set of neighboring nodes of node i, and W be a learnable weight matrix (dimension = 1024 × 2048). The intra-modal fusion features are obtained by weighted summation of the features of nodes of the same modality according to the weights. , .
[0067] S503, Inter-layer attention calculation: A bilinear attention mechanism is introduced to capture the non-linear correlation between images and clinical features. The weight calculation formula is as follows:
[0068] in For image modal node features, For clinical modality node features, U, V, and W are learnable parameter matrices; based on Weighted fusion of the two modal features yields cross-modal fused features. .
[0069] S504, Contrastive Learning Optimization: Designing the Contrastive Learning Objective Function:
[0070] in The loss function is cosine similarity, and N is the number of samples in the batch (N=32). This loss function forces the multimodal features of similar samples to align in the latent space, thereby improving the discriminative power of the fused features.
[0071] S505, Multi-task Model Training: Using 512-dimensional fused features The model is trained using the following inputs: the primary task is the American Spinal Cord Injury Association (ASPA) AE grade 5 classification, employing weighted cross-entropy loss (class weights are inversely proportional to sample size); the auxiliary task is pixel-level localization of small lesions, using a composite loss function of Descein loss and focal loss; the model optimizer is Adam, with an initial learning rate of 1 × 10⁻⁶. -4 A multi-step learning rate scheduler is used, and the learning rate is multiplied by 0.5 in the 7th and 12th training iterations, with a total of 200 training iterations.
[0072] S506. Diagnostic Results Output: The model outputs the grading results (AE level), the quantitative score of the degree of injury (0-100 points), and the prediction of motor function recovery within 3 months (the probability of improvement of ≥1 grade in the American Spinal Cord Injury Association score).
[0073] S6. Personalized treatment suggestion generation and clinical decision support: S601. Construction of Dynamic Structural Causal Model (DSCM): Constructing a DSCM model to quantify the causal relationship between treatment intervention and prognosis, using the following formula:
[0074] Where T is the treatment variable (0 = non-surgical treatment, 1 = surgical treatment, 2 = rehabilitation treatment), X is the observed confounding factor (age, injury grade, disease duration, etc.), and U is the latent variable (its distribution is estimated through a VAE model). , , For regression coefficients, The error term (following a normal distribution) ).
[0075] S602. Construction of a reinforcement learning agent: using patient baseline characteristics (age, injury grade, imaging features, clinical features) and real-time treatment feedback (drug dosage, rehabilitation frequency, occurrence of complications) as state inputs. Adjustments to the treatment plan (such as the timing of surgery and the intensity of rehabilitation) are the main actions. The policy network is optimized using a near-end policy optimization algorithm. .
[0076] S603, Multidimensional Reward Function Design: Reward Function:
[0077] in Improvement rate of the American Spinal Cord Injury Association score (improvement ≥ Grade 1) (otherwise it is 0). The incidence of complications (in the absence of complications) (otherwise it is 0). Normalized value of medical costs (the lower the cost, the higher the normalized value). The closer to 1); by maximizing the expected cumulative reward. ( (As a discount factor), optimize treatment strategies.
[0078] S604, Personalized Treatment Plan Generation: The reinforcement learning agent outputs the optimal treatment plan, including parameters such as treatment method selection (non-surgical / surgical / rehabilitation / drug therapy), surgical timing (12h / 24h / 72h after injury), rehabilitation training frequency (1 / 2 / 3 times / week), and drug dosage, and predicts the treatment effect (such as the probability of improvement in the American Spinal Cord Injury Association score within 3 months and the rehabilitation cycle).
[0079] S605, Lightweight Federated Learning Deployment: Employs an agentless neural architecture search algorithm to automatically search for the optimal model structure, compressing the number of model parameters to 30% of the original; uses a Tensor RT inference engine to quantize model weights with 8-bit integers, improving inference speed by 3 times and ensuring system response time ≤ 10 seconds / case; adds Gaussian noise during the model parameter aggregation stage. ),satisfy Privacy budget to protect cross-institutional data privacy; improve the model's cross-institutional generalization by aligning the differences in data distribution across different hospitals through gradient inversion layers (area under the receiver operating characteristic curve ≤ 0.05).
[0080] S606, Clinical Decision Support Output: Displays diagnostic results, prognostic predictions, and individualized treatment plans through a human-computer interaction module. It supports the overlay display of lesion segmentation masks and original images, multimodal image linkage browsing, one-click export of diagnostic reports (in PDF / Word format), and provides an interface for adjusting treatment plan parameters, allowing physicians to modify intervention strategies according to actual clinical conditions.
[0081] Example 2: This embodiment is a multimodal image precise segmentation and intelligent diagnosis system for cervical spinal cord injury based on the method of Embodiment 1. The system includes: Data acquisition module: It uses medical digital imaging and communication standard protocols to access the hospital's medical image archiving and communication system, and acquires T1, T2 and short-time inversion recovery sequence magnetic resonance imaging data (supports .dcm format). It collects clinical data such as American Spinal Cord Injury Association scores, medical history records, and specialist physical examinations through structured forms, and supports batch data import and storage (clinical data is stored in a MySQL database, and image data is stored in a file server).
[0082] Image preprocessing module: Implemented based on Python OpenCV and the open-source deep learning framework (PyTorch), it provides functions such as image denoising, standardization, and data augmentation, and supports both batch processing and real-time processing modes. The batch processing mode is suitable for model training data preprocessing, while the real-time processing mode is suitable for clinical online diagnostic data preprocessing, with a processing latency of ≤1 second / image.
[0083] Local scanning Markov Monte Carlo feature extraction module: Implemented based on the PyTorch framework, it integrates an improved segmentation arbitrary model and a local scanning Markov Monte Carlo module, supports GPU acceleration (adapted to NVIDIA A100 / V100 GPU), and has a feature extraction speed of ≥10 images / second (512×512 resolution).
[0084] Weighted lightweight multimodal fusion module: The weighted lightweight cross-attention mechanism is implemented using PyTorch. The number of module parameters is ≤1 million, and the fusion process latency is ≤0.5 seconds / case, ensuring the real-time performance of the system.
[0085] Progressive segmentation module: integrates a progressive upsampling decoder, supports lesion segmentation mask generation and quantitative analysis (volume and area calculation), and the segmentation results can be exported to medical digital imaging and communication standard annotation format, which is convenient for interfacing with hospital medical image archiving and communication systems.
[0086] Multimodal diagnostic module: Based on the PyTorch Lightning deep learning training framework, it realizes multi-task model training and inference, and supports functions such as 5-fold cross-validation, ablation experiment, and generalization test; during inference, it takes preprocessed images and clinical data as input and outputs diagnostic results and prognostic prediction probabilities, with an inference latency of ≤3 seconds / case.
[0087] Intelligent decision-making module: integrates dynamic structural causal model, proximal policy optimization algorithm and lightweight federated learning system, supports functions such as treatment plan generation, cross-institution model deployment and privacy-preserving data training; system size ≤50MB, GPU memory usage ≤2GB, adapted to the hardware conditions of primary hospitals.
[0088] Human-computer interaction module: The web interface is developed using the Vue.js front-end development framework and the Element UI front-end component library. It supports functions such as image browsing (zooming, panning, rotating), overlay display of segmentation results, preview and export of diagnostic reports, and adjustment of treatment plan parameters. The interface is simple and intuitive, adapted to the operating habits of clinicians, and supports simultaneous access from 200 terminals (system crash rate <1%).
[0089] Experimental verification and result analysis 1. Dataset and Evaluation Metrics: - Dataset: Multimodal data from 590 patients (472 in the training set and 118 in the test set) were used, and the generalization of the model was validated on the public dataset (Spine Generic). The test set included 35 cases of small lesions (≤3mm²) and 83 cases of conventional lesions.
[0090] - Evaluation metrics: For segmentation tasks, Descein coefficient, intersection-over-union ratio (IoU), precision, and sensitivity were used; for diagnostic tasks, ACC (accuracy), area under the receiver operating characteristic (AUC) curve, and Kappa coefficient (consistency with senior physician diagnoses) were used; for prognostic prediction tasks, ACC and F1 score were used; system performance was measured by response time, model size, and GPU memory usage. 2. Experimental Results: - Segmentation performance: On the test set, the method of this invention achieved a Dyss coefficient of 63.87%, an intersection-over-union ratio of 53.09%, a precision of 67.79%, and a sensitivity of 69.07%. Among these, the Dyss coefficient for small lesions was 58.32%, which is significantly higher than existing methods (SAMMed2D: 50.12%, MedSAM: 52.45%). The Kappa coefficient of agreement with the diagnostic results of three senior spinal surgeons (with more than 10 years of experience) was 0.82, indicating that the segmentation results have high clinical acceptance.
[0091] - Diagnostic performance: Grading accuracy = 91.53%, area under the receiver operating characteristic curve = 0.93, detection rate of small lesions = 82.86% (average detection rate of existing methods = 70.14%); prognostic prediction accuracy (improvement of ≥1 grade in the American Spinal Cord Injury Association score within 3 months) = 89.83%, F1 score = 0.87, significantly better than existing clinical decision-making methods (accuracy = 75.32%, F1 score = 0.73).
[0092] - System performance: System response time = 8.7 seconds / case, model size = 42MB, GPU memory usage = 1.8GB; During cross-institutional validation, the AUC fluctuation in 3 tertiary hospitals was 0.03, and the F1-score decay was 3.2%, which meets the generalization requirements.
[0093] 2. Ablation experiment: - Removing the local scanning Markov Monte Carlo module: The Dess coefficient decreased to 56.47%, and the detection rate of small lesions decreased to 71.43%, indicating that the module can effectively enhance the extraction of small lesion features.
[0094] - Removing the weighted lightweight multimodal fusion module: the diagnostic accuracy decreased to 83.05%, and the area under the receiver operating characteristic curve decreased to 0.86, indicating that the module could improve the multimodal information fusion effect.
[0095] - Removing the dynamic causal reinforcement learning module: The treatment plan prediction error rate increased to 14.2%, indicating that the module could improve the targeting of treatment plans.
[0096] 3. Comparison with existing methods: - Segmentation task comparison: The Dessian coefficient of the method of this invention is improved by 8.36% compared with the medical SAM2D model (55.51%), by 4.65% compared with the arbitrary medical segmentation model (59.22%), and by 23.02% compared with the neural network U-Net model (40.85%), with particularly obvious advantages in the segmentation of small lesions.
[0097] - Diagnostic task comparison: The diagnostic accuracy of this invention is 26.72% higher than the X-ray large model (64.81%), 18.90% higher than the medical multimodal large model (72.63%), and 24.22% higher than the Qianwen visual large model (67.31%).
[0098] - System performance comparison: The model volume of the present invention is reduced by an average of 75% compared with the existing methods, and the response time is shortened by an average of 60%, making it more suitable for clinical applications.
[0099] The experimental results above demonstrate that the multimodal image segmentation and intelligent diagnosis method and system for cervical spinal cord injury proposed in this invention outperforms existing technologies in terms of detecting minute lesions, multimodal data fusion, diagnostic accuracy, personalized treatment plans, and system practicality. It can provide efficient, accurate, and objective decision support for clinical diagnosis and treatment and has significant clinical application value.
[0100] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.
Claims
1. A method for precise segmentation and intelligent diagnosis of cervical spinal cord injury multi-modal images, characterized in that, Includes the following steps: S1. Multimodal Data Acquisition and Preprocessing: Acquire T1-weighted, T2-weighted, and short-time inversion recovery sequence magnetic resonance imaging data of patients with cervical spinal cord injury, as well as clinical data including American Spinal Cord Injury Association scores, medical history records, and specialist physical examinations; perform denoising, standardization, and data augmentation on the imaging data; standardize the clinical structured data using standard scores; and encode the clinical unstructured text into semantic feature vectors using a biological clinical bidirectional encoder representation model. S2. Image feature extraction based on local scanning Markov Monte Carlo model: Based on the improved arbitrary segmentation model architecture, the prompt encoder of the original arbitrary segmentation model is removed, and the local scanning Markov Monte Carlo model scanning module is embedded in the transformer layer to extract features from the preprocessed multimodal magnetic resonance images, enhance feature mining of small lesion areas and suppress background interference. S3, Weighted Lightweight Cross-Attention Multimodal Image Fusion: A weighted lightweight cross-attention mechanism is designed to construct a learnable proxy set corresponding to the three modalities of T1, T2, and short-time inversion recovery. Through intramodal feature interaction and cross-modal feature fusion, a three-modal aligned image feature map is generated. S4. Progressive upsampling fully automatic lesion segmentation: A progressive upsampling decoder is built, which combines transposed convolution and attention mechanism to fuse multi-scale image features, performs pixel-level segmentation on the three-modal aligned feature map, and outputs the segmentation mask of cervical spinal cord injury lesions to achieve end-to-end fully automatic lesion segmentation. S5. Imaging-Clinical Multimodal Data Fusion Diagnosis: Construct a hierarchical graph attention fusion network, using the lesion segmentation features obtained in step S4 and the clinical features processed in step S1 as heterogeneous nodes. Construct an adjacency matrix based on prior knowledge of spinal cord anatomy and dynamically optimize it. Achieve deep fusion of cross-modal features through intra-layer and inter-layer attention mechanisms, train the injury diagnosis and prognosis prediction model, and output the cervical spinal cord injury grading and condition assessment results. S6. Personalized Treatment Recommendation Generation and Clinical Decision Support: Based on the diagnosis and prognosis prediction results of step S5, combined with a dynamic causal reinforcement learning model, the causal effect of treatment intervention on prognosis is quantified. A multidimensional reward function is designed with the improvement rate of the American Spinal Cord Injury Association score, the incidence of complications, and medical costs as the core to generate personalized treatment plans. The diagnosis results and treatment recommendations are quickly output through a lightweight federated learning system.
2. The method according to claim 1, wherein, The specific implementation process of feature extraction in the local scanning Markov Monte Carlo model scanning module described in S2 is as follows: S201. Replace the linear coding layer in the Markov Monte Carlo block with a multilayer perceptron to improve the deep representation capability of image features. S202. Divide the magnetic resonance image into 81 information feature blocks according to the size of the image, and select the central feature block and surrounding related feature blocks that contain the core information of the lesion. S203. A weighted scanning strategy is used to scan the feature blocks. The central feature block is scanned 4 times, and the peripheral feature blocks with low foreground ratio are scanned once each, which enhances the capture of lesion area features and reduces background noise interference. S204. Input the scanned feature sequence into the improved Markov Monte Carlo module, combine it with the state space model to complete the spatiotemporal correlation modeling of the features, and output the depth features of the single-mode magnetic resonance image. 3.The method of precise segmentation and intelligent diagnosis of cervical spinal cord injury multi-modal images according to claim 1, characterized in that, The specific implementation process of the weighted lightweight cross-attention mechanism described in S3 to achieve multimodal image fusion is as follows: S301. Extract the basic features of T1, T2, and short-time inversion recovery trimodal magnetic resonance images through a shared segmentation arbitrary model encoder to obtain modal features. , , Where N is the number of features; c is the feature dimension; X Q Query modal features for the target; X A T2 weighted modal features; X B For T1 weighted modal features; ∈ represents the membership symbol; It is the set of real numbers; S302. Constructing a trimodal learnable agent set: T2 agent T1 Agent Short-term reversal recovery agent ,and , where n A The number of weighted proxy features for T2; n B The number of weighted proxy features for T1; n S To recover the number of proxy features in a short time reversal; S303. Perform intra-modal feature interaction and refine the single-modal features through an attention mechanism to obtain: 、 Among them, O A For T2 weighted modal refinement features; O B T1 weighted modal refinement features; Softmax is the normalized exponential function; superscript This is the matrix transpose symbol; S304. Using a short-time inversion recovery proxy as a guide, cross-modal fusion is completed. The refined features of T1 and T2 are concatenated and feature interaction is achieved through the short-time inversion recovery proxy, resulting in: Among them, O S [;] represents cross-modal fusion features; [;] represents feature concatenation symbols. S305. Using the target query modal features as a benchmark, attention weighting is applied to the cross-modal fusion features to obtain: Output a trimodal aligned image feature map, where O final This is the final three-modal alignment feature.
4. The method for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury according to claim 1, characterized in that, The specific implementation process of the progressive upsampling decoder described in S4 to achieve fully automatic lesion segmentation is as follows: S401. Input the trimodal aligned feature map into the progressive upsampling decoder, which consists of multiple upsampling blocks and convolutional blocks. Perform multi-scale upsampling on the feature map through 2D transposed convolution to capture lesion features at different resolutions. S402. An attention mechanism is introduced to fuse mask prior information with multi-scale image features to form a unified feature representation that includes lesion location and morphological information. S403. Enhance the multi-scale fusion features to highlight the key signals of small lesions, and align all features to a uniform spatial resolution through bilinear interpolation. S404. The aligned multi-scale features are fused, and a pixel-level lesion segmentation mask is output through a convolutional layer to achieve fully automatic segmentation of cervical spinal cord injury lesions.
5. The method of claim 1, wherein, The specific implementation process of the hierarchical graph attention fusion network described in S5 to achieve image-clinical multimodal data fusion is as follows: S501. Using lesion segmentation features and clinical semantic features as heterogeneous nodes, an initial adjacency matrix is constructed based on the anatomical prior knowledge of cervical spine segmental functional zoning. S502. Calculate the importance weights of nodes within the same modality using an intra-layer attention mechanism, with the following formula: Among them, a ij Here, is the in-layer attention weight; exp is the exponential function; LeakyReLU is the linear unit activation function with leakage correction; W is the learnable weight matrix; h i h j h k N represents the feature vectors of different nodes; | represents the feature concatenation symbol; N i Let be the set of adjacent nodes of node i; Σ is the summation symbol, which realizes the weighted fusion of features of the same modality; S503. A bilinear attention mechanism is introduced to complete cross-modal feature interaction between layers. The weight calculation formula is as follows: in, σ represents the inter-layer cross-modal attention weights; σ is the sigmoid activation function; U, V, and W are learnable parameter matrices; tanh is the hyperbolic tangent activation function; h p h is the image feature node vector. q This is a vector of clinical feature nodes, capturing the non-linear relationship between images and clinical features; S504. Concatenate the features fused within and between layers to output a 512-dimensional multimodal fusion feature vector. , where H fusion This represents a multimodal fusion feature; ∈ represents the membership symbol. 512 is the set of real numbers; 512 is the dimension of the feature vectors; S505: Using multimodal fusion features as input, train a multi-task model including a classification head and a regression head. The classification head realizes the American Spinal Cord Injury Association (ASPA) five-level classification of cervical spinal cord injury, and the regression head completes the quantitative assessment of the degree of injury and outputs the diagnostic results. 6.The method of precise segmentation and intelligent diagnosis for cervical spinal cord injury multi-modal images according to claim 1, characterized in that, The specific implementation process of generating personalized treatment suggestions based on dynamic causal reinforcement learning, as described in S6, is as follows: S601. Construct a dynamic structural causal model (DSCM) to quantify the causal effect of treatment intervention on prognosis. The model formula is as follows: Where Y(t) represents the prognostic outcome at time t; α, β, and γ are learnable coefficients; T is the treatment variable; X is the observed confounding factor; and U is the latent variable. The error term is used to estimate the distribution of the latent variable U using a variational autoencoder (VAE). S602. Using patient baseline characteristics and real-time treatment feedback as state inputs, construct a reinforcement learning agent and optimize the policy network using a proximal policy optimization algorithm. S603. Design a multidimensional reward function, using the American Spinal Cord Injury Association score improvement rate, complication rate, and medical cost as core indicators to calculate the real-time reward value. S604. Through strategy iteration of the reinforcement learning agent, the optimal individualized treatment plan is output, including intervention methods and parameters such as drug treatment, surgical treatment, and rehabilitation training. S605. Input the treatment plan and diagnosis results into the lightweight federated learning system, complete the model compression and privacy protection processing, and realize the rapid output of diagnosis and treatment suggestions. The system response time is ≤10 seconds / case.
7. The method for precise segmentation and intelligent diagnosis of multimodal images of cervical spinal cord injury according to any one of claims 1-6, characterized in that, The data augmentation described in S1 employs strategies such as random rotation, uniform scaling, mirror flipping, and Gaussian noise addition; the multi-task model training described in S5 uses a weighted cross-entropy loss and a composite loss function of Dess loss + focus loss to address the problems of sample imbalance and low segmentation accuracy of small lesions, respectively.
8. A multimodal image precision segmentation and intelligent diagnostic system for cervical spinal cord injury, characterized in that, The system for implementing the method according to any one of claims 1-7 comprises: Data acquisition module: used to acquire T1, T2, short-time inversion recovery sequence magnetic resonance imaging data and clinical data of patients with cervical spinal cord injury, and supports the access and storage of data from multiple devices and in multiple formats; Image preprocessing module: Connected to the data acquisition module, it is used to denoise, standardize, and augment magnetic resonance images, and to perform structured and semantic coding on clinical data; Local scanning Markov Monte Carlo feature extraction module: connected to the image preprocessing module, used to embed the local scanning Markov Monte Carlo scanning module to complete the depth feature extraction of multimodal magnetic resonance images; Weighted Lightweight Multimodal Fusion Module: Connected to the local scanning Markov Monte Carlo feature extraction module, it is used to align and fuse three-modal image features through a weighted lightweight cross-attention mechanism; Progressive segmentation module: Connected to the weighted lightweight multimodal fusion module, it is used to achieve fully automatic pixel-level segmentation of cervical spinal cord injury lesions through a progressive upsampling decoder; Multimodal diagnostic module: Connected to the progressive segmentation module and the image preprocessing module respectively, it is used to achieve image-clinical feature fusion through a hierarchical graph attention fusion network and output damage diagnosis and prognostic assessment results; Intelligent decision-making module: Connected to the multimodal diagnostic module, it is used to generate personalized treatment suggestions based on dynamic causal reinforcement learning and achieve rapid output through a lightweight federated learning system; Human-computer interaction module: Connected to the multimodal diagnosis module and intelligent decision-making module respectively, it is used to visualize lesion segmentation results, diagnostic reports and treatment suggestions, and supports clinicians' interactive operation and parameter adjustment.
9. The multimodal image precise segmentation and intelligent diagnosis system for cervical spinal cord injury according to claim 8, characterized in that, The lightweight federated learning system employs an agentless neural architecture search algorithm for model compression, reducing the number of model parameters to 30% of the original. It also achieves a 3x speedup inference through tensor RT 8-bit integer quantization and adds Gaussian noise during the model parameter aggregation stage to meet certain requirements. A privacy budget to achieve privacy protection and model generalization across organizations, including This represents the privacy budget under the differential privacy framework.
10. The multimodal image precise segmentation and intelligent diagnosis system for cervical spinal cord injury according to claim 8, characterized in that, The human-computer interaction module supports the overlay display of lesion segmentation masks, the linked browsing of multimodal images, and the one-click export of diagnostic reports. It also provides an interface for adjusting treatment plan parameters, allowing clinicians to modify intervention strategies according to actual conditions.