Tumor radiotherapy system based on multi-modal image and breathing waveform fusion

Abstract

The invention belongs to the technical field of medical equipment, and provides a tumor radiotherapy system based on multi-modal image and breathing waveform fusion, which comprises a data acquisition and processing module, a tumor position prediction module and a radiotherapy determination module, based on an elastic registration method, through feature space alignment, the space-time registration problem of a multi-source image is effectively solved, particularly artifact suppression and small tumor recognition of a rib overlapping region are innovatively optimized, and the target region sketching precision in a complex anatomical environment is remarkably improved; by means of a two-way convolution module, a prediction model with respiratory phase sensing capacity is constructed by analyzing space-time correlation between respiratory signals and image features, tumor motion trail tracking is achieved, tumor centroid coordinates are predicted, and the technical limitation of traditional single-mode tracking is broken through.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical equipment, and in particular relates to a tumor radiotherapy system based on the fusion of multimodal images and respiratory waveforms. Background Art

[0002] In radiotherapy for thoracic and abdominal tumors, respiratory motion can cause target displacement of up to 20-30 mm. Existing radiotherapy techniques rely on pre-set static dose models and are unable to dynamically adjust treatment parameters based on real-time target displacement caused by respiratory motion.

[0003] Traditional respiratory gating technology triggers irradiation only at a specific phase of the respiratory cycle, resulting in prolonged treatment time. Static irradiation methods have to expand the irradiation field to cover the target area's range of motion, resulting in a large amount of healthy tissue being exposed to unnecessary radiation. The single-modality registration method is prone to artifacts in overlapping areas of anatomical structures such as ribs and diaphragm. Summary of the Invention

[0004] In order to solve the above problems, the present invention proposes a tumor radiotherapy system based on the fusion of multimodal images and respiratory waveforms. Based on the elastic registration method, the present invention effectively solves the problem of spatiotemporal registration of multi-source images through feature space alignment. It is particularly innovative and optimized for artifact suppression and small tumor identification in overlapping rib areas, significantly improving the target area delineation accuracy in complex anatomical environments; with the help of a bidirectional convolution module, by analyzing the spatiotemporal correlation between respiratory signals and image features, a prediction model with respiratory phase perception capability is constructed, which realizes tumor motion trajectory tracking and predicts the tumor center of mass coordinates, breaking through the technical limitations of traditional single-modality tracking.

[0005] To achieve the above objectives, the present invention provides a tumor radiotherapy system based on the fusion of multimodal imaging and respiratory waveforms, which adopts the following technical solutions: A tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion, comprising: a data acquisition and processing module, a tumor position prediction module, and a radiotherapy determination module; The data acquisition and processing module is configured to: acquire computed tomography information, magnetic resonance imaging information, fluoroscopic image information, and respiratory waveform information of the patient; and elastically register the computed tomography information, magnetic resonance imaging information, fluoroscopic image information, and respiratory waveform information by minimizing an energy function to align them across modal spaces; and enforce consistency between the computed tomography information and the magnetic resonance imaging information and the fluoroscopic image in feature space using a cross-modal contrast loss function; The tumor location prediction module is configured to obtain a tumor segmentation mask and a tumor motion trajectory based on processed computed tomography information, magnetic resonance imaging information, fluoroscopic imaging information, and respiratory waveform information, as well as a preset prediction model; wherein the prediction model includes a bidirectional convolution module, which is inserted into each layer of skip connections to capture spatiotemporal features; The radiotherapy determination module is configured to perform radiotherapy according to the tumor segmentation mask and tumor motion trajectory obtained by the prediction model.

[0006] Furthermore, when performing elastic registration, the optimal deformation field is found in the displacement vector of each voxel from the floating image to the reference image, so that the floating image under the deformation field is aligned with the reference image; the loss function includes the similarity measure and regularization term , the formula for spatial registration is as follows: ; ; in, is the optimal deformation field; f is the deformation field; is the reference image; It is a floating image under the action of deformation field; are preset parameters; is the regularization term; is the displacement field ϕ At voxel position p The gradient at , ∥∥ 2 is the square of the L2 norm of the gradient; Define the domain for the image; For reference images f Grayscale value at voxel position p.

[0007] Furthermore, cross-modal comparison strategies include static-static comparison and dynamic-static comparison; in static-static comparison, the positive sample pairs are computed tomography tumor region ↔ magnetic resonance imaging tumor region, and computed tomography bone ↔ magnetic resonance imaging bone marrow signal; in static-static comparison, the negative sample pairs include computed tomography tumor region ↔ magnetic resonance imaging normal tissue, and computed tomography bone ↔ magnetic resonance imaging soft tissue; in dynamic-static comparison, the positive sample pairs include tumor features of real-time images ↔ tumor region features of computed tomography / magnetic resonance imaging; in dynamic-static comparison, the negative sample pairs include tumor features of real-time images ↔ non-tumor region features of computed tomography / magnetic resonance imaging.

[0008] Furthermore, the prediction model includes a first-level U-Net, which includes an encoder, a decoder, and a bidirectional convolution; the encoder uses 4 layers of convolution and a LeakyReLU activation function, and the decoder uses 4 layers of deconvolution: ; in, For decoder l The layer outputs feature maps and restores spatial resolution layer by layer; is upsampling; It is a jump connection; and are weights and biases; the feature map at the end of the decoder compresses the number of channels through a 1×1 convolution kernel and is activated as a probability map. The coarse segmentation mask is: ; in, is the sigmoid activation function.

[0009] Furthermore, the tumor motion trajectory in the SI direction is extracted from the fluoroscopic image sequence; the respiratory trajectory is converted into dynamic weights through a convolutional network; and dynamic weights are introduced into the input gate of the bidirectional convolution module to make the activation intensity of the bidirectional convolution module adapt to changes in respiratory amplitude.

[0010] Furthermore, the prediction model includes a second-level U-Net, which is responsible for performing residual correction on the prediction results of the first-level U-Net.

[0011] Furthermore, the rough prediction results of the first-level U-Net are Residual output of the second-level U-Net Overlay: ; Based on segmentation mask , predict the two-dimensional centroid position of the tumor area: ; in, is the physical coordinate of the voxel.

[0012] Furthermore, the knowledge of the complex teacher model is transferred to the lightweight student model, halving the number of parameters. In the teacher-student architecture, the teacher model is a complete dual U-Net, and the student model is a dual U-Net with 50% of the intermediate redundant layers removed, retaining the bidirectional convolution module. The probability distributions of the teacher and student are aligned so that the deformation field output probability distribution of the student model approaches that of the teacher model.

[0013] Furthermore, based on the predicted centroid coordinates of the tumor, centroid displacement calculation is performed to obtain the predicted centroid displacement of the tumor; the predicted centroid displacement of the tumor is converted into a displacement in the multi-leaf collimator plane coordinate system, and radiotherapy is performed according to the obtained displacement in the multi-leaf collimator plane coordinate system.

[0014] Furthermore, through the dynamic dose regulation technology, ensure the execution of radiotherapy under the tumor motion state, and the dynamic dose rate is regulated as: ; where is the planned speed, is the speed compensation coefficient. When the leaf speed lags, the dose rate is automatically increased to compensate for the irradiation time loss; when the current respiratory amplitude A(t) > 1.2×A reference or A(t) < 0.8×A reference, irradiation is paused, where A reference = the average respiratory amplitude trained by the patient before treatment.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention performs elastic registration on the obtained computed tomography information, magnetic resonance imaging information, fluoroscopy image information, and respiratory waveform information through minimizing an energy function for cross-modal spatial alignment; through a cross-modal contrast loss function, enforces the consistent expression of computed tomography information and magnetic resonance imaging information with fluoroscopy images in the feature space; effectively solves the problem of spatio-temporal registration of multi-source images, and particularly innovatively optimizes the artifact suppression in the rib overlap region and the identification of small tumors, significantly improving the target delineation accuracy in complex anatomical environments; on this basis, a prediction model for predicting tumor segmentation masks and tumor motion trajectories, including a bidirectional convolution module, inserts a bidirectional convolution module in each layer of skip connections to capture spatio-temporal features, can construct a prediction model with respiratory phase perception ability by analyzing the spatio-temporal correlation between respiratory signals and image features, realize tumor motion trajectory tracking, predict tumor centroid coordinates, and break through the technical limitations of traditional single-modal tracking; solves the problems of unnecessary radiation of a large amount of healthy tissue caused by expanding the irradiation field and artifacts easily generated in the overlapping regions of anatomical structures such as ribs and diaphragms.

[0016] 2. The present invention is based on the model compression technology of knowledge distillation and the TensorRT quantization implementation plan; through the collaborative design of a residual refinement network and a teacher-student architecture, while maintaining a segmentation accuracy of more than 95%, the inference speed is increased to meet the clinical real-time requirement, providing key technical support for the dynamic control of the MLC.

[0017] 3. This invention derives a dynamic radiation field planning method based on the prediction of tumor centroid displacement. This method uses a dynamic programming algorithm to generate a blade motion path that satisfies mechanical constraints. Combined with a feedback correction mechanism, this method controls the dose deviation caused by mechanical delay to within 2%. 4. This system innovatively couples a deep learning prediction model with a mechanical control system, forming a fully closed-loop control chain from image acquisition and motion prediction to mechanical response, thus overcoming the technical bottlenecks of traditional radiotherapy systems in terms of real-time performance, accuracy, and safety. Precise positioning and dynamic adaptation enable multimodal image fusion, integrating data from different modalities, such as CT, MRI, and real-time fluoroscopic imaging, to provide information on the tumor's anatomical structure, metabolic activity, and dynamic motion, reducing the limitations of single images. Tumor trajectory prediction is achieved by fusing multimodal images with respiratory sensor signals to predict the tumor's periodic motion trajectory with respiration. Through a dynamic compensation mechanism, the multileaf collimator (MLC) dynamically adjusts the radiation field shape and dose distribution based on the predicted tumor location, ensuring that the radiation beam consistently and accurately covers the tumor target and minimizing the risk of off-target effects due to motion. This reduces damage to healthy tissue and the need for a safety margin. Conventional radiotherapy requires expanding the irradiation field to cover the range of tumor motion, while dynamic compensation technology narrows the safety margin, significantly reducing the radiation dose to surrounding normal tissues. It also optimizes dose. Through dynamic adjustment, high-dose areas are more concentrated on the tumor, reducing the incidence of complications such as radiation pneumonitis and cardiotoxicity. Patient comfort and compliance are improved, eliminating the need for forced breath-holding or immobilization. Conventional respiratory gating techniques require patients to hold their breath or use compression devices, while dynamic compensation technology allows patients to receive treatment while breathing naturally, minimizing discomfort during treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0019] Figure 1 This is a schematic diagram of the system structure of Example 1 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0022] Example 1: In radiotherapy for thoracic and abdominal tumors, the target displacement caused by respiratory motion can reach 20-30 mm. Existing radiotherapy technology relies on a preset static dose model and cannot dynamically adjust treatment parameters according to the real-time target displacement caused by respiratory motion. The traditional respiratory gating technology is used to trigger irradiation only at a specific phase of the respiratory cycle, resulting in prolonged treatment time. The traditional static irradiation method has to expand the irradiation field to cover the target motion range, resulting in a large amount of healthy tissue being exposed to unnecessary radiation. The existing single-modality registration method is prone to artifacts in overlapping areas of anatomical structures such as ribs and diaphragm. The existing MLC control has a mechanical delay of more than 150ms. The existing radiotherapy system lacks multi-dimensional abnormality detection capabilities.

[0023] For example, some existing technologies have three technical limitations: First, there is a lack of real-time perception and compensation mechanism for dynamic tumor movement, which increases the risk of normal tissue exposure; second, the dose superposition model relies on static CT images and does not integrate real-time image guidance technology, making it difficult to adapt to anatomical deformation during treatment; third, the adjustment of treatment parameters is based on offline optimization, which cannot achieve sub-second dynamic response, resulting in a significant reduction in the spatial selectivity of dose delivery under motion. Other existing technologies have the following shortcomings: first, cross-modal registration relies on a single V-Net architecture, which does not adequately handle non-rigid deformation caused by respiratory movement; second, the dose calculation module is not linked to the real-time image tracking system, and cannot compensate for the three-dimensional movement trajectory of the tumor during the respiratory cycle; third, the system lacks a dynamic coordination mechanism with MLC, and the dose superposition results are not converted into real-time MLC leaf control instructions, resulting in significant execution delays in clinical implementation.

[0024] In order to solve at least one of the above problems, Figure 1 As shown, this embodiment provides a tumor radiotherapy system based on the fusion of multimodal imaging and respiratory waveforms, including a data acquisition and processing module, a tumor position prediction module, and a radiotherapy determination module. Tumor motion is predicted by fusing multimodal imaging with respiratory physiological signals, and the position of the multileaf grating (MLC) blades is dynamically adjusted to match target displacement for intelligent radiotherapy with compensatory radiation. To address the issue of compensating for tumor displacement, a closed-loop control system combining a tumor motion prediction model and MLC dynamic compensation is established by fusing multimodal imaging and respiratory sensor data, achieving submillimeter motion tracking accuracy and millisecond-level mechanical response. This system focuses on addressing target displacement drift caused by respiratory motion, eliminating the impact of multimodal image registration errors on dose distribution, and ensuring high reliability during treatment through an intelligent safety monitoring mechanism. This system has established an intelligent dynamic radiotherapy system, overcoming the three bottlenecks of existing technologies: image guidance lag, inaccurate motion compensation, and low multi-system coordination efficiency. Ultimately, the system achieves the technical goal of improving tumor dose delivery accuracy by over 40% and reducing normal tissue dose by 30% under complex motion conditions.

[0025] Specifically, a "perception-decision-execution" trinity intelligent radiotherapy system was constructed. First, through a multimodal image fusion engine, CT, MRI and real-time fluoroscopic images are elastically registered to solve the problem of target identification caused by anatomical structure overlap. Then, a bidirectional spatiotemporal convolutional network is used to analyze the intrinsic correlation between respiratory sensor signals and tumor movement, and a submillimeter motion prediction model is established. Finally, a lightweight control algorithm is developed to drive the multi-leaf grating for dynamic compensation, achieving real-time matching of dose distribution and target displacement. The system innovatively deeply couples the deep learning prediction model with the mechanical control system to form a fully closed-loop control chain from image acquisition, motion prediction to mechanical response, breaking through the technical bottlenecks of traditional radiotherapy systems in terms of real-time performance, accuracy and safety. Clinical tests have shown that this solution can still maintain a dose distribution error of less than 3% at the maximum breathing amplitude, significantly improving the overall benefit of radiotherapy.

[0026] In some embodiments, the data acquisition and processing module includes a data preprocessing and input construction unit; the tumor location prediction module includes a generator network setting unit, a discriminator network design unit, and an optimization and training unit, etc.; the radiotherapy determination module includes a real-time tracking and clinical deployment unit, an MLC dynamic control unit, and a real-time treatment execution and verification unit, etc.

[0027] The data preprocessing and input construction unit is configured to: perform step S1 data preprocessing and input, specifically: S1.1. Multimodal data generation and enhancement: S1.1.1. Collect multimodal datasets: Computed tomography (CT) and magnetic resonance imaging (MRI) are static, high-resolution images acquired during the radiotherapy planning phase (before treatment). They serve as a reference for anatomical modeling and radiotherapy planning, but they cannot reflect the dynamic anatomical deformation caused by respiration during treatment. Real-time fluoroscopic images (such as X-rays) are dynamic but lower-resolution images acquired in real time during radiotherapy (during treatment) to capture tumor motion caused by respiration.

[0028] Before treatment, patients undergo CT and MRI scans (covering the complete respiratory cycle), generating data for 10 respiratory phases. During treatment, real-time fluoroscopic imaging (e.g., X-rays) captures multiple frames of 2D images per second. Patients also wear a respiratory sensor that records their respiratory waveforms in real time, which is then used to identify the respiratory phases.

[0029] The dataset should cover a variety of scenarios. The optional datasets used are as follows: Population model: A diverse dataset of CT, MRI, and real-time fluoroscopic images and respiratory sensor signals covering different patient sizes (160-190 cm tall, 50-90 kg weight), tumor diameters (20-40 mm), locations (upper / middle / lower lobes), and respiratory amplitudes (0.8-4 cm in the SI direction, 1.6-2 cm in the AP direction).

[0030] Patient-specific model: Fix the patient's body shape and tumor location, adjust the respiratory amplitude (such as deep breathing, shallow breathing, cough disturbance, etc.), simulate respiratory variation on different treatment days, and collect patient-specific data sets.

[0031] S1.1.2, Data Enhancement: This embodiment randomly crops images to simulate different imaging fields of view to prevent overfitting, enabling the model to learn multi-scale features. Gaussian noise is added to real-time fluoroscopic images to simulate the quantum noise of low-dose X-rays, thereby improving the model's robustness and generalization capabilities for complex clinical scenarios.

[0032] Optional, random cropping: randomly scale the image size to 80%-120%. Gaussian noise: add noise with mean 0 and standard deviation 0.05.

[0033] S1.2, Time Alignment: This invention time-aligns CT, MRI, real-time fluoroscopic imaging (such as X-rays), and respiratory sensor data to ensure data correspondence within the same respiratory phase. This eliminates motion artifacts and timing misalignment, enabling accurate modeling of dynamic processes.

[0034] Optionally, based on the respiratory sensor signal, mark the current time t as respiratory phase n. Select CT or MRI data corresponding to respiratory phase n. Capture three consecutive fluoroscopic images (t-1, t, t+1), covering a time window of approximately 100 ms.

[0035] S1.3, spatial alignment: Optionally, based on the VoxelMorph framework, CT, MRI, and real-time fluoroscopic images are elastically registered by minimizing an energy function, achieving cross-modal spatial alignment. This geometric transformation unifies images from different modalities into the same coordinate system, allowing for precise alignment of identical anatomical structures in physical space, eliminating spatial shifts caused by imaging equipment, patient position, or movement.

[0036] The input of VoxelMorph is a floating image m and reference images f , the output is the deformation field f , represents each voxel p From floating image m To the reference image fThe goal is to find the optimal deformation field f * , making (Deformation Field f Floating images under the influence m ) and the reference image f Alignment. Loss Function L Including similarity measures and regularization term : ; Multimodal commonly used mean square error Perform similarity measurement, introduce the minimization energy function, and the optional spatial registration formula is as follows: ; ; in, ,The purpose is to control the smoothness of the deformation field and ensure the consistency of the anatomical structure; is the optimal deformation field; f is the deformation field; is the reference image; It is a floating image under the action of deformation field; is the regularization term; is the displacement field ϕ At voxel position p The gradient at ∥⋅∥ 2 is the square of the L2 norm of the gradient; Define the domain for the image, i.e. the set of all voxel locations; For reference images f At voxel position p The gray value at .

[0037] S1.4, Feature Alignment: This embodiment uses a cross-modal contrast learning method. By introducing a cross-modal contrast loss function, it enforces consistent representation of CT / MRI and 2D fluoroscopic images in the feature space. The cross-modal contrast strategy is: static-static contrast, the goal is to align similar anatomical structures in static modalities. Positive sample pairs: CT tumor region ↔ MRI tumor region; CT bone ↔ MRI bone marrow signal. Negative sample pairs: CT tumor region ↔ MRI normal tissue; CT bone ↔ MRI soft tissue. Dynamic-static contrast, the goal is to align dynamic real-time image features with static CT / MRI features. Positive sample pairs: real-time image tumor features ↔ CT / MRI tumor region features. Negative sample pairs: real-time image tumor features ↔ CT / MRI non-tumor region features. Here, ↔ represents "and," indicating that a positive sample pair consists of a CT tumor region and an MRI tumor region.

[0038] Through feature space alignment, the problem of spatiotemporal registration of multi-source images is effectively solved. It specifically optimizes artifact suppression and small tumor identification in overlapping rib areas, significantly improving the accuracy of target area delineation in complex anatomical environments.

[0039] S1.4.1. Define the format of input data: The 5D input tensor format of this embodiment is as follows: ; in, B is the number of samples in a single training; N is the number of phases in the respiratory cycle; C is the modal number; H and W is the image resolution.

[0040] Fusion of 3 consecutive frames of real-time perspective images with CT / MRI data to construct input tensors , , , , .in Indicates 32 samples; Represents 10 respiratory phases, covering the complete respiratory cycle, and fills the sequence with zero matrices at the beginning and end to maintain temporal continuity; Represents three consecutive frames of fluoroscopic images, CT, and MRI modalities; invalid areas are shielded by attention masks to reduce redundant calculations.

[0041] S1.4.2. Contrastive loss function: The loss function used in this embodiment is InfoNCE Loss, which is as follows: ; in, is a positive sample pair; is a negative sample pair; is the similarity between a and b; is the temperature coefficient; Represents maximization and Modal feature consistency and cross-modal feature alignment; Indicates the introduction of negative samples By temperature coefficient Adjust the intra-class and inter-class distances to enhance cross-modal discrimination.

[0042] The generator network setting unit is configured to: execute step S2 generator network setting, specifically: In a dynamic radiotherapy system, the generator is the core deep learning model. Its core function is to predict high-precision tumor motion trajectories to compensate for tumor displacement caused by respiratory motion or other physiological activities. In this embodiment, the generator inputs are CT / MRI / real-time fluoroscopic images and respiratory sensor signals, and the output is a tumor segmentation mask and tumor motion trajectory.

[0043] S2.1. First-level U-Net (Convolutional Networks for Biomedical Image Segmentation) (coarse prediction): The U-Net backbone network structure used in this embodiment includes an encoder, a decoder, and a bidirectional convolutional LSTM (Bi-ConvLSTM).

[0044] The encoder uses 4 layers of convolution (kernel 3×3, stride 2) and LeakyReLU activation function to avoid feature discontinuity caused by maximum pooling. Each layer extracts multi-scale features through convolution and activation function: ; in, and are weights and biases; is convolution; For encoder l The layer outputs a feature map, and the resolution is halved layer by layer: .

[0045] The decoder uses 4 layers of deconvolution (kernel 3×3, stride 2): ; in, For decoder l The layer outputs feature maps and restores spatial resolution layer by layer; For upsampling, For jump connection.

[0046] The feature map at the end of the decoder is compressed by a 1×1 convolution kernel to compress the number of channels and activated as a probability map. The coarse segmentation mask is: ; The bidirectional convolutional LSTM (Bi-ConvLSTM) is a convolutional LSTM that inserts a Bi-ConvLSTM module into each layer of skip connections to capture spatiotemporal features. The convolutional LSTM formula is as follows: Input Gate: ; Forget Gate: ; Output gate: ; ; Cell status update: ; in, Jump connection feature map for the current moment; is in hidden state; is the cell state; is the learnable weight; is bias; is element-wise multiplication; is the sigmoid activation function.

[0047] S2.2. Dynamic breathing pattern modeling: Extract the tumor motion trajectory in the SI (Superior-Inferior) direction from the fluoroscopic image sequence: ; in, is the displacement sequence, is the reference position.

[0048] The breathing trajectory is transformed into Generate dynamic weights .

[0049] ; The gating mechanism is extended by introducing a dynamic weight α(t) into the input gate of Bi-ConvLSTM to adapt its activation strength to the changes in breathing amplitude. The specific formula is as follows: ; in, The input gate activation strength is adjusted. When the breathing amplitude is large (α→1), the input gate is fully open, and the model prioritizes learning the current motion features. When the breathing amplitude is small (α→0), the input gate suppresses noise interference and relies on historical state prediction. This operation reduces the COMD error in patients with irregular breathing (<0.08cm in the SI direction).

[0050] S2.3, Second level U-Net (residual refinement): The second-level U-Net is the refinement module of the generator, responsible for performing residual correction on the rough prediction results of the first-level U-Net to improve the accuracy of tumor segmentation and motion trajectory.

[0051] The rough prediction results of the first level U-Net Residual output of the second-level U-Net Overlay: ; Based on segmentation mask , predict the two-dimensional centroid position of the tumor area: ; in, is the physical coordinate of the voxel.

[0052] S2.4. Lightweight knowledge distillation: Migrating the knowledge of a complex teacher model to a lightweight student model reduces the number of parameters by half, speeds up inference, and enables deployment on edge devices.

[0053] Teacher-student architecture: The teacher model is a complete dual U-Net (10.2M parameters), and the student model is a dual U-Net with 50% of the intermediate redundant layers removed (5.1M parameters), retaining the Bi-ConvLSTM core module.

[0054] Distillation loss: Align the probability distributions of the teacher (soft label) and the student (hard prediction) through KL divergence, so that the probability distribution of the deformation field output of the student model is close to that of the teacher model.

[0055] .

[0056] This example utilizes knowledge distillation-based model compression technology and a TensorRT quantization implementation. Through the collaborative design of a residual refinement network and a teacher-student architecture, it achieves inference speeds that meet clinical real-time requirements while maintaining segmentation accuracy exceeding 95%, providing key technical support for MLC dynamic control.

[0057] The discriminator network design unit performs step 3 of discriminator network design, including: S3.1. Multimodal Discriminator Architecture: The main goal of the discriminator is to force the generator (dual U-Net) to output prediction results that are highly consistent with the real clinical data through an adversarial training mechanism.

[0058] Generator's segmentation mask Combined with multimodal fusion features (CT-Xray, MRI-Xray), a multimodal tensor is formed through channel splicing.

[0059] In this example, the multimodal discriminator architecture used is: four convolutional layers (kernel 4×4, stride 2) followed by one fully connected layer to extract multi-scale features. The final layer performs global average pooling followed by a fully connected layer to output true / false probabilities. The LeakyReLU activation function is used to prevent vanishing gradients.

[0060] Convolutional layer: ; Global average pooling: ; Discriminant output, output true or false probability: ; in, and are the weights and biases of the fully connected layer.

[0061] S3.2, Discriminator loss function: Based on Wasserstein GAN-GP, Wasserstein distance and gradient penalty terms are introduced: ; in, The expected value of the discriminator output for generating a sample x (from the generator distribution Pg), The expected value of the discriminator output for a true sample x (from the true distribution Pr). is the gradient penalty coefficient, are linear interpolation samples of real data and generated data. It is a gradient penalty term that forces the training discriminator to converge. The gradient penalty term forces the discriminator to satisfy 1-Lipschitz continuity, improves training stability, and prevents mode collapse.

[0062] The consistency of CT and MRI responses in the discriminator is constrained by L2 distance.

[0063] ; in, and Represent the output of the discriminator for CT and MRI modality features, respectively, is the square of the L2 norm.

[0064] The optimization and training unit performs step S4 optimization and training, including: S4.1. Hybrid loss function: ; in, Controls the weight of the pixel-level L1 loss. A high weight (100) emphasizes the pixel-level absolute error between the prediction and the true label, reduces noise and ensures basic segmentation accuracy. Balanced contribution of structural similarity (SSIM) loss. It optimizes segmentation boundary details by evaluating the structure, brightness, and contrast similarity of local image patches. Adjust the weight of the intersection-over-union (IOU) loss to directly optimize the overlap ratio between the predicted area and the true area, thereby improving the overall segmentation accuracy. Control the adversarial loss weight of the Generative Adversarial Network (GAN) to ensure that the generated results are close to the real data distribution and improve the authenticity of the segmentation boundaries. Adjust the weights of cross-modal contrastive learning (CT-MRI-Xray) to enhance small tumor detection capabilities by maximizing feature consistency between modalities. Balance the weight of knowledge distillation loss, guide the output distribution of the student model (lightweight U-Net) through the teacher model (original dual U-Net), and reduce the accuracy loss after model compression. Constrain the discriminator to ensure consistency in outputs from different modalities (CT and MRI), improve cross-modal feature alignment capabilities, and reduce anatomical structure overlap interference.

[0065] L1 loss: Calculates the pixel-level absolute error between the predicted value and the true value, emphasizes pixel-by-pixel accuracy, reduces noise interference, and helps the model generate smooth and precise segmentation boundaries.

[0066] ; in, is the true label, is the segmentation mask output by the generator.

[0067] SSIM loss: This is to measure the similarity between the predicted image and the real image in terms of structure, brightness, and contrast. It pays special attention to boundary details and avoids blurring.

[0068] ; in, , is the mean brightness of the image block, is the standard deviation, is the covariance, , is a constant.

[0069] IOU loss: It can effectively and directly optimize the overlap ratio between the predicted area and the true area, thereby improving the overall accuracy of segmentation.

[0070] ; in, is the predicted value; is the true value.

[0071] Adversarial loss: The discriminator (D) evaluates the authenticity of the generator (G) output, ensuring that the segmentation results conform to the real data distribution. The gradient penalty term enforces 1-Lipschitz continuity in the discriminator, improving training stability.

[0072] .

[0073] S4.2 Training strategy: S4.2.1. Using the alternating training method, in each round of training, the generator is updated twice and the discriminator is updated once.

[0074] S4.2.2. Dynamic learning rate: Initial learning rate 1e-4, cosine decay to 1e-6.

[0075] The real-time tracking and clinical deployment unit performs step S5 of real-time tracking and clinical deployment, including: S5.1, Lightweight Inference Engine: S5.1.1. Model pruning: Remove redundant convolutional layers from the generator network, reducing the number of parameters by 50%, while retaining the core dual U-Net architecture and Bi-ConvLSTM module.

[0076] S5.1.2, TensorRT quantization: Use the TensorRT framework to convert model weights from FP32 to FP16 precision, and optimize the inference speed to <30ms / frame to meet clinical real-time requirements.

[0077] S5.2, Online Adaptive Module: S5.2.1. Real-time feedback optimization: In clinical deployment, catastrophic forgetting is avoided by fine-tuning model parameters (only adjusting top-level decoder parameters) using a small amount of new data collected in real time.

[0078] ; in, are the updated model parameters, is the old parameter before the update, is the learning rate, is the online loss function pair gradient. , only fine-tuning the top-level parameters and only updating the decoder part parameters.

[0079] S5.3 Key evaluation indicators: Intersection-over-Union (IOU): measures the overlap accuracy of segmented regions.

[0080] ; Dice coefficient (DSC): Similar to IOU, it also measures the overlap accuracy of the segmented area, but is more sensitive to small targets.

[0081] ; Tumor Area Difference (AD): quantifies the percentage difference between the predicted and true tumor areas ; in, is the predicted area, is the actual area.

[0082] Center of mass difference (COMD): measures the spatial shift of the tumor center of mass.

[0083] ; in, To predict the coordinates of the tumor center of mass, is the coordinate of the center of mass of the reference tumor.

[0084] The MLC dynamic control unit performs step S6 MLC dynamic control, including: In this example, a coupling equation for the MLC blade kinematic model and dose rate control was established, and an optimal control strategy based on the predicted trajectory was derived. A dynamic programming algorithm was used to generate a blade motion path that satisfied mechanical constraints. Combined with a PID feedback correction mechanism, dose deviations caused by mechanical delays were controlled to within 2%.

[0085] S6.1. Blade Position Algorithm: The input of the algorithm is the predicted tumor center of mass coordinates .

[0086] S6.1.1. Calculation of center of mass displacement: ; in, are the coordinates of the tumor center of mass in the planning stage, Predicted tumor centroid coordinates.

[0087] S6.1.2 Coordinate system transformation: Tumor centroid displacement will be predicted Converted into its displacement on the MLC plane coordinate system , the x-axis of the MLC plane coordinate system is the blade movement direction, and the y-axis is perpendicular to the x-axis on the MLC plane.

[0088] S6.1.3 Dynamic field planning: The MLC consists of N pairs of blades, and the original planned opening boundary of each pair of blades i is Assuming that a pair of blades a is not closed, the blades that need to be adjusted are a+Δd.

[0089] ; Where h is the blade thickness and [ ] is the rounding function.

[0090] Then the adjustment amount of the blade to the opening boundary of a+Δd is: ; The remaining leaf pairs remain closed. Generate the MLC leaf target opening boundary covering the range of tumor motion .

[0091] Step 6.1.4, Path Constraint Optimization: Ensure blade motion complies with mechanical performance limits with time-stamped blade position command sequences . Dynamic programming is used to generate the optimal trajectory total time ( is the moving distance of each leaf), the constraints are: Maximum speed of a single blade ≥15mm / s; acceleration ≤20mm / s²; spacing between adjacent blades ≥2mm (anti-collision).

[0092] S6.2, Closed-loop feedback control: MLC blade target position With actual location .

[0093] S6.2.2, PID error correction: Dynamically eliminate mechanical execution errors, the calculation formula for the position deviation of blade pair i is: or ; PID control: ; in, is a proportional term that quickly responds to position deviations. is the integral term, eliminating the steady-state error, It is a differential term that suppresses overshoot oscillation.

[0094] This embodiment derives a dynamic field planning method based on the predicted tumor centroid displacement. A dynamic programming algorithm is used to generate a blade motion path that satisfies mechanical constraints. Combined with a PID feedback correction mechanism, the dose deviation caused by mechanical delay is controlled within 2%.

[0095] The real-time treatment execution and verification unit performs step S7 of real-time treatment execution and verification, including: In this embodiment, dynamic dose control technology is used to ensure precise delivery of radiotherapy during tumor motion. Dose delivery accuracy: Dynamically matching MLC blade motion with dose rate eliminates dose unevenness caused by mechanical delay (error <2%). Respiratory synchronization safety: Real-time monitoring of abnormal respiratory amplitude (automatically suspending irradiation if it exceeds the baseline by ±20%) protects normal tissue. Treatment quality verification: Online reconstruction of actual dose distribution, with a gamma pass rate of ≥95% (3mm / 3%), meets clinical quality control requirements.

[0096] S7.1, Multimodal Execution Engine: Dynamically adjust the treatment parameters according to the tumor motion state to achieve precise matching of dose delivery and mechanical response.

[0097] S7.1.1, Dynamic Dose Rate Adjustment: ; Where, is (Actual MLC Leaf Speed - Planned Speed) / Planned Speed, is the speed compensation coefficient (calibrated through MLC mechanical characteristics). When the leaf speed lags , automatically increase the dose rate to compensate for the irradiation time loss. The consistency error of the dose spatial distribution is < 2% (MU / mm standard). The dose rate adjustment range: 50% - 150% of the planned value (to prevent over-irradiation), and the response delay is < 20 ms (synchronized with the MLC control instruction in Step 6).

[0098] S7.1.2, Respiration-Dose Gating: When the current respiration amplitude A(t) > 1.2 × A reference or A(t) < 0.8 × A reference, pause the irradiation, where A reference = the average respiration amplitude during the patient's pre-treatment training (individually calibrated). The continuous monitoring time window: the last 5 respiration cycles (about 10 - 15 seconds). Safety mechanism: The automatic retry interval after pausing: 3 normal respiration cycles (about 6 seconds). The maximum number of pauses: After 3 times, switch to the manual intervention mode.

[0099] S7.2, Online Dose Reconstruction: Use an EPID (Electronic Portal Imaging Device) to collect transmission images with a resolution of 0.2 mm / pixel. The dose back-projection formula: ; Where, is pre-calibrated through phantom irradiation (automatically updated when the machine is turned on daily), is the scattered dose distribution calculated based on the Monte Carlo algorithm. The verification standard: ; Where, is the dose deviation percentage at point , is the spatial distance between point and the corresponding point in the plan.

[0100] The above is only the preferred embodiment of this example and is not used to limit this example. For those skilled in the art, this example can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this example shall be included within the protection scope of this example.

Claims

1. A tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion, characterized by: include: Data acquisition and processing module, tumor location prediction module, and radiotherapy determination module; The data acquisition and processing module is configured to: acquire the patient's computed tomography information, magnetic resonance imaging information, fluoroscopic imaging information, and respiratory waveform information; Furthermore, the computed tomography information, magnetic resonance imaging information, fluoroscopic image information, and respiratory waveform information are elastically registered by minimizing the energy function and aligned across the modal space; through the cross-modal contrast loss function, the computed tomography information and magnetic resonance imaging information are forced to be expressed consistently with the fluoroscopic image in the feature space; The tumor location prediction module is configured to obtain a tumor segmentation mask and a tumor motion trajectory based on processed computed tomography information, magnetic resonance imaging information, fluoroscopic imaging information, and respiratory waveform information, as well as a preset prediction model; wherein the prediction model includes a bidirectional convolution module, which is inserted into each layer of skip connections to capture spatiotemporal features; The radiotherapy determination module is configured to perform radiotherapy according to the tumor segmentation mask and tumor motion trajectory obtained by the prediction model.

2. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 1, characterized in that: When performing elastic registration, the optimal deformation field is found in the displacement vector of each voxel from the floating image to the reference image, so that the floating image under the deformation field is aligned with the reference image; the loss function includes similarity measurement and regularization term , the formula for spatial registration is as follows: ； ； in, is the optimal deformation field; φ is the deformation field; is the reference image; It is a floating image under the action of deformation field; are preset parameters; is the regularization term; is the displacement field ϕ At voxel position p The gradient at , ∥∥ 2 is the square of the L2 norm of the gradient; Define the domain for the image; For reference images f Grayscale value at voxel position p.

3. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 2, characterized in that: Cross-modal comparison strategies include static-static comparison and dynamic-static comparison; in static-static comparison, the positive sample pairs are computed tomography tumor region ↔ magnetic resonance imaging tumor region, and computed tomography bone ↔ magnetic resonance imaging bone marrow signal; in static-static comparison, the negative sample pairs include computed tomography tumor region ↔ magnetic resonance imaging normal tissue, and computed tomography bone ↔ magnetic resonance imaging soft tissue; in dynamic-static comparison, the positive sample pairs include tumor features of real-time images ↔ tumor region features of computed tomography / magnetic resonance imaging; in dynamic-static comparison, the negative sample pairs include tumor features of real-time images ↔ non-tumor region features of computed tomography / magnetic resonance imaging.

4. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 1, characterized in that: The prediction model includes a first-level U-Net, which includes an encoder, a decoder, and bidirectional convolution. The encoder uses 4 layers of convolution and a LeakyReLU activation function, and the decoder uses 4 layers of deconvolution: ； in, For decoder l The layer outputs feature maps and restores spatial resolution layer by layer; is upsampling; It is a jump connection; and are weights and biases; the feature map at the end of the decoder compresses the number of channels through a 1×1 convolution kernel and is activated as a probability map. The coarse segmentation mask is: ； in, is the sigmoid activation function.

5. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 4, characterized in that: The tumor motion trajectory in the SI direction is extracted from the fluoroscopic image sequence; the respiratory trajectory is converted into dynamic weights through a convolutional network; and dynamic weights are introduced into the input gate of the bidirectional convolution module to make the activation intensity of the bidirectional convolution module adapt to changes in respiratory amplitude.

6. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 4, characterized in that: The prediction model includes a second-level U-Net, which is responsible for performing residual correction on the prediction results of the first-level U-Net.

7. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 6, characterized in that: The rough prediction results of the first level U-Net Residual output of the second-level U-Net Overlay: ; Based on segmentation mask , predict the two-dimensional centroid position of the tumor area: ； in, is the physical coordinate of the voxel.

8. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 6, characterized in that: The knowledge of the complex teacher model is transferred to the lightweight student model, halving the number of parameters. In the teacher-student architecture, the teacher model is a complete dual U-Net, and the student model is a dual U-Net with 50% of the intermediate redundant layers removed, retaining the bidirectional convolution module. The probability distributions of the teacher and student are aligned so that the probability distribution of the student model's deformation field output approaches that of the teacher model.

9. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 7, characterized in that: According to the predicted tumor center of mass coordinates, the center of mass displacement is calculated to obtain the predicted tumor center of mass displacement; the predicted tumor center of mass displacement is converted into a displacement on the multileaf grating plane coordinate system, and radiotherapy is performed according to the obtained displacement on the multileaf grating plane coordinate system.

10. The tumor radiotherapy system based on multimodal imaging and respiratory waveform fusion according to claim 1, characterized in that: Dynamic dose control technology is used to ensure that radiotherapy is performed while the tumor is in motion. The dynamic dose rate is adjusted to: ； Among them, is the planned speed, is the speed compensation coefficient. When the blade speed lags, the dose rate is automatically increased to compensate for the irradiation time loss; irradiation is paused when the current respiratory amplitude A(t)>1.2×A reference or A(t)<0.8×A reference, where A reference is the average respiratory amplitude during the patient's pre-treatment training.

Images