Radiotherapy dose prediction system based on multi-modal information

Abstract

The invention belongs to the technical field of radiotherapy dose prediction, and particularly relates to a radiotherapy dose prediction system based on multi-modal information, which comprises a multi-scale cross space attention coding module, a text information representation module, a cross-modal information fusion module and a decoding module. According to the radiotherapy dose prediction system based on the multi-modal information, the model can comprehensively capture key factors influencing radiotherapy dose distribution through a task-adaptive network architecture design, the prediction accuracy and clinical applicability are remarkably improved, and through double-branch coding structure innovation, the prediction efficiency is improved. The perceptual ability of the model to key anatomical structures is greatly enhanced, more targeted feature expression is provided for subsequent dose prediction, text knowledge related to a treatment scheme is deeply fused into an image feature map through a text information and image feature fusion strategy, and the accuracy of the treatment scheme is improved. And a richer feature basis fused with multi-modal semantics is provided for dose prediction in a decoding stage.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of radiotherapy dose prediction, and particularly relates to a radiotherapy dose prediction system based on multi-modal information. BACKGROUND

[0002] At present, radiotherapy, as one of the core means of cancer treatment, realizes precise killing of tumor cells through high-energy rays while minimizing damage to surrounding normal tissues. Compared with surgical treatment, radiotherapy has the advantages of smaller side effects and safer treatment process, and can effectively reduce the risk of damage to normal tissues. In the radiotherapy planning process, a physical therapist needs to construct a target plan list to generate a target function, and obtain a clinically acceptable treatment plan through repeated iterative optimization. This process is highly complex and time-consuming, often requiring thousands of optimization iterations and taking several hours or even several days, which seriously restricts the efficiency of clinical treatment. The quality of a radiotherapy plan depends largely on the professional knowledge and experience of a physical therapist. In clinical practice, a qualified treatment plan requires a physical therapist to continuously adjust the irradiation field and optimize the parameters, and to seek the best treatment plan through repeated trial and error. This process involves numerous adjustable parameters, and different parameter combinations will produce different plan results. In the face of such complex parameter combinations, the quality of radiotherapy plans developed by physical therapists in different regions and hospitals varies greatly. Under this background, accurate prediction of dose distribution has become a key technical link to assist physical therapists in optimizing radiotherapy plans.

[0003] Traditional dose prediction methods rely on physical models or empirical formulas to calculate dose distribution by simulating the interaction process between rays and human tissues. However, due to the differences in complex anatomical structures, dynamic organ motion, and individual physiological characteristics of tumor patients, traditional models gradually fail to meet clinical needs in terms of prediction accuracy and computational efficiency, facing serious technical challenges. In recent years, the rapid development of artificial intelligence technology has brought revolutionary breakthroughs in the field of radiotherapy dose prediction. Deep learning-based dose prediction models can autonomously learn dose distribution patterns from patient CT images and structural labels, significantly reducing manual intervention and improving prediction efficiency. However, existing technical solutions still have two outstanding problems:

[0004] First, the fusion of CT images and structural label information is not sufficient: existing fusion methods mainly fall into two categories. One is to concatenate CT image data and structural labels through channels or perform pixel-level point multiplication operations to achieve information fusion at the input stage of the neural network. However, this method causes the structural label information to be gradually lost as the network depth increases. The second is to extract features from images and structural masks separately, and perform feature-level fusion at the intermediate or subsequent layers of the network. This approach makes the image encoding process lack effective supervision and guidance from structural information.

[0005] II. Missing key information such as prescription dose and radiotherapy irradiation mode: The dose distribution state is closely related to clinical parameters such as prescription dose and radiotherapy mode. Such information usually exists in the form of unstructured text. However, the existing mainstream dose prediction scheme adopts pure visual modeling method, which cannot realize the effective integration of such important clinical information. SUMMARY

[0006] The purpose of the present application is to provide a radiotherapy dose prediction system based on multi-modal information to solve the problems of insufficient fusion of CT images and structural label information and missing key information such as prescription dose and radiotherapy irradiation mode in the background art.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a radiotherapy dose prediction system based on multi-modal information, comprising a multi-scale cross-space attention encoding module, a text information representation module, a cross-modal information fusion module and a decoding module, the multi-scale cross-space attention encoding module innovatively improves the UNet 3D convolutional neural network, and constructs a deep semantic fusion and hierarchical feature abstraction framework of medical image features, the multi-scale cross-space attention encoding module adopts a double-branch parallel encoding design, which expands the traditional single-path encoder into a double-flow processing path of CT image and structure mask.

[0008] Preferably, in the feature fusion stage, the multi-scale cross-space attention encoding module performs element-wise point multiplication operation on the CT image features of the current scale and the spatial attention matrix generated by the mask, realizes the dynamic modulation of structure semantics to image features, and the multi-scale cross-space attention encoding module dynamically adjusts the feature fusion strategy at different levels through the multi-scale cross-space attention mechanism.

[0009] Preferably, the double-branch full-scale correlation design of the multi-scale cross-space attention encoding module fully utilizes the structure information of the organs at risk and the target area to guide the feature extraction of the CT image, and the text information representation module converts the key text information in the radiotherapy dose prediction task into a structured feature vector that can be fused with the image features.

[0010] Preferably, the implementation of the text information representation module includes two core links, one is the semantic alignment stage, which is completed by the text encoder in the CLIP model, and the other is the feature dimension unification link, which makes the dimensions of the text features and the image features output by the multi-scale cross-space attention encoding module consistent through the Embedding operation.

[0011] Preferably, the feature vector generated by the text encoder has a dimension of 512, and the image feature map after four times of downsampling has a corresponding dimension of 1024.

[0012] Preferably, the core function of the cross-modal information fusion module is to organically fuse the structured text feature vector generated by the text information representation module with the deep image feature map output by the multi-scale cross-spatial attention coding module, so as to provide comprehensive feature support rich in multimodal semantics for dose distribution prediction in the subsequent decoding stage.

[0013] Preferably, the core task of the decoding module is to gradually transform the image feature map, which has been fused with textual semantics in the Bottleneck stage, into a prediction result of the radiotherapy dose distribution. After the decoding module completes feature processing to generate the dose prediction map, the final dose prediction head uses MAE regression loss as the optimization objective.

[0014] Compared with the prior art, the beneficial effects of the present invention are:

[0015] 1. Task-Adaptive Network Architecture Design: Based on the UNet 3D basic architecture, and taking into account the specificity of radiotherapy dose prediction tasks, a novel end-to-end network was carefully designed. In the final dose prediction process, the mask structure information of the target area and organs at risk is fully incorporated, along with treatment text information such as the prescribed dose for the target area, the safe dose for organs at risk, and the irradiation method. This multimodal information fusion design enables the model to comprehensively capture key factors affecting radiotherapy dose distribution, significantly improving prediction accuracy and clinical applicability.

[0016] 2. Innovative Dual-Branch Coding Structure: Breaking away from the limitations of traditional single-branch coding, a dual-branch coding structure is developed. One branch focuses on processing image data, while the other processes mask data. In the image feature encoding process, a spatial attention mechanism is introduced. This mechanism dynamically weights image features based on mask structure information, effectively highlighting important features related to the target area and organs at risk, suppressing irrelevant background information, and greatly enhancing the model's ability to perceive key anatomical structures, providing more targeted feature representations for subsequent dose prediction.

[0017] 3. Text Information and Image Feature Fusion Strategy: At the network bottleneck, text information is encoded and semantically aligned. First, text such as prescription dosage values ​​and irradiation methods are converted into structured feature vectors. Then, through a cross-attention mechanism, these text feature vectors are deeply fused with the image feature map output by the encoder. This achieves precise guidance and supplementation of image features by text information, allowing treatment plan-related text knowledge to be deeply integrated into the image feature map, providing a richer feature foundation that incorporates multimodal semantics for dosage prediction in the decoding stage. Attached Figure Description

[0018] Figure 1This is a simplified flowchart of the overall algorithm of the present invention;

[0019] Figure 2 This is a schematic diagram of the multi-scale cross-space attention encoding structure of the present invention;

[0020] Figure 3 This is the text information representation module of the present invention;

[0021] Figure 4 This is the cross-modal information fusion module of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see the appendix Figures 1-4 The present invention provides several embodiments:

[0024] A radiotherapy dose prediction system based on multimodal information includes a multi-scale cross-spatial attention encoding module, a text information representation module, a cross-modal information fusion module, and a decoding module. The multi-scale cross-spatial attention encoding module innovatively improves the UNet 3D convolutional neural network, constructing a deep semantic fusion and hierarchical feature abstraction framework for medical image features. The multi-scale cross-spatial attention encoding module adopts a dual-branch parallel encoding design, extending the traditional single-channel encoder into a dual-stream processing path for CT images and structural masks. The upper branch performs refined feature extraction on the 3D CT image sequence, constructing a hierarchical feature pyramid in each stage through two 3×3×3 three-dimensional convolutions. The lower branch, for sparse structural mask data, uses a lightweight single 3×3×3 convolution combined with a sigmoid activation function to generate a spatial attention matrix, and transforms the spatial distribution of the mask into differentiable attention weights through normalized output.

[0025] Furthermore, in the feature fusion stage, the multi-scale cross-spatial attention coding module performs element-wise multiplication between the CT image features at the current scale and the spatial attention matrix generated by the mask. This achieves dynamic modulation of image features by structural semantics, enhancing CT features in high-attention regions (such as the PTV target area or regions of organs at risk) and suppressing features in low-attention regions. The modulated features are then split into two branches: one branch directly transmits them to the same-scale decoder, preserving high-resolution detail information; the other branch downsamples them to the next scale through 2×2×2 Maxpooling, while the spatial attention matrix is ​​simultaneously downsampled, forming a cross-scale attention propagation link. Through this multi-scale cross-spatial attention mechanism, the multi-scale cross-spatial attention coding module dynamically adjusts the feature fusion strategy at different levels. Specifically, the spatial attention matrix at each scale not only affects the feature fusion at the current level but also participates in the encoding process of subsequent scales through downsampling, forming a full-scale semantic association network.

[0026] Furthermore, the dual-branch full-scale association design of the multi-scale cross-spatial attention coding module fully utilizes the structural information of organs at risk and target areas to guide feature extraction from CT images, avoiding the problem of information loss caused by noise as the network structure of traditional single-branch network with channel stitching deepens. The text information representation module transforms the key text information in the radiotherapy dose prediction task into a structured feature vector that can be fused with image features, providing a foundation for cross-modal information interaction in the Bottleneck stage, and ultimately improving the accuracy of dose distribution prediction.

[0027] Furthermore, the implementation of the text information representation module comprises two core stages: the semantic alignment stage, completed using the text encoder in the CLIP model; and the feature dimension unification stage, which uses embedding operations to ensure consistency between the text features and the image feature dimensions output by the multi-scale cross-space attention encoding module. CLIP, as a cross-modal pre-trained model, trains its text encoder on large-scale image-text pairs, enabling it to map natural language descriptions and visual features to the same semantic space, effectively establishing semantic connections between text and images. Specifically, the Tokenizer component transforms unstructured text into discrete numerical sequences, which are then processed by the Transformer encoder to generate text feature vectors containing contextual semantics, achieving initial semantic alignment between the text semantics and the preceding image features in the semantic space.

[0028] Furthermore, the feature vector generated by the text encoder has a dimension of 512, while the corresponding dimension of the image feature map after four downsampling steps is 1024. To ensure the compatibility of the feature space during cross-modal fusion, the text feature vector needs to be expanded to 1024 dimensions through a linear mapping embedding layer. This operation not only solves the problem of dimensionality differences between features from different modalities, but also creates conditions for subsequent cross-attention mechanisms to mine fine-grained correlations between text and image features through the unification of the feature space, enabling textual information such as treatment parameters to accurately guide the prediction process of dose distribution.

[0029] Furthermore, the core function of the cross-modal information fusion module is to organically fuse the structured text feature vectors generated by the text information representation module with the deep image feature maps output by the multi-scale cross-spatial attention encoding module. This provides comprehensive feature support rich in multimodal semantics for dose distribution prediction in the subsequent decoding stage. The specific implementation process is as follows: First, the image feature map obtained in the Bottleneck stage is transformed into a 1024×1024 sequence vector through a dimension flattening operation, and a sinusoidal positional encoding is added to this sequence vector. This positional encoding mechanism can inject spatial positional information into the image features, ensuring that the model can still perceive the spatial topological relationships in the original image when processing the flattened sequence, providing a spatial context reference for subsequent cross-modal attention calculation. In the cross-attention operation stage, the position-encoded image feature vector and the text feature vector output by the text information representation module are subjected to cross-modal association learning. Specifically, the attention calculation employs text feature vectors as the query and image feature vectors as both the key and value. First, an attention weight matrix is ​​generated based on the similarity between the text query and the image key. This matrix quantifies the association strength between each text token and each spatial location in the image. Then, the image value is weighted and summed based on this weight matrix, allowing the image feature vector to be dynamically adjusted according to the semantic meaning of the text information. For example, when the text contains information such as "prescription dosage 60Gy" or "PTV target area," the attention mechanism automatically enhances the feature weights of the target area in the image while weakening features of areas unrelated to the text semantics. After the cross-attention operation, the image feature vector, incorporating the text semantics, undergoes a dimensionality reshaping operation, restoring the spatial feature map structure from a 1024×1024 sequence to a 16×16×4×1024 structure to match the input requirements of the subsequent decoder.

[0030] Furthermore, the core task of the decoding module is to gradually transform the image feature map, which has been fused with textual semantics in the Bottleneck stage, into a predicted result for the radiotherapy dose distribution. This stage starts with the feature map output from Bottleneck and uses a series of 2×2×2 deconvolution operations to upsample the feature map, gradually restoring its spatial size to the scale of the input CT image. After each upsampling, the feature map at this scale is fused with the feature map from the encoder at the corresponding scale via a shortcut connection, and a 3×3×3 convolution operation is used to further refine the feature representation and enhance the model's ability to capture detailed information. After the decoding module completes feature processing and generates the dose prediction map, the final dose prediction head uses MAE regression loss as the optimization objective. MAE regression loss can effectively measure the average absolute error between the predicted dose distribution and the true value. It has convexity, which can ensure that there is a globally optimal solution in the optimization process, thereby continuously optimizing the model to achieve accurate prediction of the radiotherapy dose distribution. This process is largely the same as the traditional UNet 3D network decoding process, except that the input is an image feature map that incorporates text semantics, and the loss function is different. This allows text information to be integrated into the entire dose prediction process, and through reasonable loss optimization, it provides more valuable dose distribution predictions for the formulation of clinical radiotherapy plans.

[0031] When this system is in use, it innovatively introduces a multi-scale spatial attention mechanism into the end-to-end encoder-decoder framework constructed in this invention. In the encoder stage, for image features and mask features at different scales, the spatial attention module dynamically learns and strengthens the interaction information between the two in the spatial dimension, realizing the deep fusion of image and structural information at multiple levels. This design can effectively avoid the problem of structural information being lost as the network depth increases in traditional fusion methods, so that the image encoding process is always supervised and guided by structural information.

[0032] Before entering the decoder, in the Bottleneck stage, this invention further utilizes a cross-attention mechanism to perform cross-modal interaction between the encoded text features and the feature map with fused mask structure information. Through cross-attention operations, the model can adaptively learn the dependencies between different modal information, achieving efficient integration of text, image, and mask information, thereby comprehensively capturing key factors affecting radiotherapy dose distribution.

[0033] Furthermore, to enhance the efficiency of information transfer between features at different scales, this framework constructs a direct information pathway between the encoder and decoder at corresponding scales through shortcut connections. This connection method not only helps alleviate the gradient vanishing problem but also enables shallow detail information and deep semantic information to complement each other, further improving the model's understanding of complex radiotherapy scenarios and its dose prediction capabilities.

[0034] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A radiotherapy dose prediction system based on multimodal information, comprising a multi-scale cross-spatial attention encoding module, a text information representation module, a cross-modal information fusion module, and a decoding module, characterized in that: The multi-scale cross-spatial attention coding module innovatively improves the UNet 3D convolutional neural network and constructs a deep semantic fusion and hierarchical feature abstraction framework for medical image features. The multi-scale cross-spatial attention coding module adopts a dual-branch parallel coding design, which extends the traditional single-channel encoder into a dual-stream processing path for CT images and structural masks.

2. The radiotherapy dose prediction system based on multimodal information according to claim 1, characterized in that: In the feature fusion stage, the multi-scale cross-spatial attention coding module performs element-wise multiplication of the CT image features at the current scale with the spatial attention matrix generated by the mask, thereby realizing the dynamic modulation of image features by structural semantics. The multi-scale cross-spatial attention coding module dynamically adjusts the feature fusion strategy at different levels through the multi-scale cross-spatial attention mechanism.

3. The radiotherapy dose prediction system based on multimodal information according to claim 1, characterized in that: The dual-branch full-scale association design of the multi-scale cross-spatial attention coding module fully utilizes the structural information of organs at risk and target areas to guide the feature extraction of CT images. The text information representation module transforms the key text information in the radiotherapy dose prediction task into a structured feature vector that can be fused with image features.

4. The radiotherapy dose prediction system based on multimodal information according to claim 1, characterized in that: The implementation of the text information representation module includes two core steps: the first is the semantic alignment stage, which is completed using the text encoder in the CLIP model; the second is the feature dimension unification stage, which uses the embedding operation to ensure that the text features are consistent with the image feature dimensions output by the multi-scale cross-space attention encoding module.

5. The radiotherapy dose prediction system based on multimodal information according to claim 4, characterized in that: The text encoder generates a feature vector with a dimension of 512, while the image feature map after four downsamplings has a corresponding dimension of 1024.

6. The radiotherapy dose prediction system based on multimodal information according to claim 1, characterized in that: The core function of the cross-modal information fusion module is to organically fuse the structured text feature vector generated by the text information representation module with the deep image feature map output by the multi-scale cross-spatial attention coding module, so as to provide comprehensive feature support rich in multimodal semantics for dose distribution prediction in the subsequent decoding stage.

7. The radiotherapy dose prediction system based on multimodal information according to claim 1, characterized in that: The core task of the decoding module is to gradually transform the image feature map, which has been fused with textual semantics in the Bottleneck stage, into a prediction result of the radiotherapy dose distribution. After the decoding module completes feature processing to generate the dose prediction map, the final dose prediction head uses MAE regression loss as the optimization objective.

Images