Dose superposition method and device based on depth deformation registration, equipment and medium
Through deep learning model combined with Transformer and CNN, the problems of high complexity of deformation registration and local optimal solutions in the prior art are solved, and more efficient and accurate dose superposition is achieved, suitable for target areas and organs in radiation therapy. Dose evaluation.
Patent Information
- Application Number
- CN202411141063.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-22
AI Technical Summary
The existing deformation registration technology based on the Demons algorithm has high computational complexity, is prone to local optimal solutions, and is difficult to deal with complex deformation conditions, resulting in inaccurate dose superposition.
Using a deformation registration model based on deep learning, combined with Transformer encoder and CNN decoder, deformation registration is carried out through multimodal data sets, and cascading registration architecture is built, using mutual information loss function, regularization loss and segmentation loss optimization Deformation field to achieve accurate superposition of doses.
It improves the accuracy and robustness of deformation registration, reduces the computational complexity, can automatically learn and optimize the optimal registration mapping relationship between images, and significantly improves the accuracy and efficiency of dose superposition.
Smart Images

Figure CN119006424A_ABST
Abstract
Description
Technical field:
[0001] The present invention relates to the technical field of medical image registration, and in particular to a dose superposition method, device, equipment and medium based on depth deformation registration. Background technology:
[0002] Radiotherapy is one of the main treatments for malignant tumors, and more than 50% of patients with malignant tumors need to receive radiotherapy. During the treatment process, how to ensure that the tumor target area receives sufficient dose irradiation while minimizing damage to surrounding normal tissues is a key issue in radiotherapy. Dose superposition technology refers to the accumulation of dose distributions of multiple radiotherapy sessions to evaluate the total irradiation dose of the target area and organs at risk. In adaptive radiotherapy, dose superposition technology is used to evaluate the dose effects of the revised radiotherapy plan on the target area and organs at risk. The implementation of dose superposition technology depends on accurate deformation registration results. After registering images at different time points through deformation registration technology, the dose distribution of each radiotherapy can be mapped to the same coordinate system and accumulated for calculation. The accumulated dose distribution can intuitively display the total irradiation dose of the target area and organs at risk, providing an important basis for the optimization of radiotherapy plans;
[0003] The technical solution of the prior art 1 is based on the Demons deformable registration algorithm. The Demons algorithm is a non-parametric deformable registration method, which was originally derived from the optical flow algorithm. It uses multiple iterations of the optical flow algorithm and uses a Gaussian filter to smooth the displacement field to obtain the changes required for registration. The Demons algorithm uses the gradient of the reference image and the grayscale difference between the reference image and the floating image to calculate the coordinate offset of each point, thereby obtaining the coordinate offset of the entire image. Then, these offsets are used to resample and interpolate the floating image, and this process is repeated until the set number of iterations or similarity threshold is reached.
[0004] Disadvantages of prior art 1:
[0005] First, the computational complexity is high and requires a long processing time.
[0006] Second, it is easy to fall into the local optimal solution, resulting in registration failure.
[0007] 3. It relies on simple mathematical models or iterative optimization algorithms, which are difficult to handle complex deformation situations;
[0008] To this end, a dose superposition method, device, equipment and medium based on depth deformation registration are proposed. Summary of the invention:
[0009] The purpose of the present invention is to provide a dose superposition method, device, equipment and medium based on deep deformation registration. The present invention proposes a deformation registration model based on deep learning, and realizes accurate dose superposition according to this model to solve one of the problems raised in the above background technology.
[0010] First aspect: To solve the above technical problems, a technical solution adopted in this application is: a dose superposition method based on depth deformation registration, comprising the following steps:
[0011] Acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, extracting a reference dose image corresponding to the reference CT image, and extracting a floating dose image corresponding to the floating CT image;
[0012] constructing a floating rigid transformation CT image according to the preprocessed multimodal data set, importing the reference CT image and the computed tomography image sequence of the floating CT image, importing the reference dose image and the three-dimensional dose image of the floating dose image, rigidly registering the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and rigidly transforming the floating CT image to obtain a floating rigid transformation CT image;
[0013] Constructing a deep deformable registration model: the deep deformable registration model includes a Transformer encoder and a CNN decoder, inputting the reference CT image and the floating rigid transformation CT image into the deep deformable registration model for deformation registration to obtain a deformation field;
[0014] Performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image;
[0015] The floating rigid transformation dose image is deformed in response to the deformation field to obtain a floating deformation dose image, and the reference dose image and the floating deformation dose image are added pixel by pixel to obtain a dose superposition image.
[0016] Preferably, the step of constructing a deep deformable registration model specifically includes:
[0017] Combination of Transformer and CNN: The Transformer encoder is used to capture the spatial correspondence between the input floating CT image and the reference CT image, and the CNN decoder processes the information provided by the Transformer encoder into a dense displacement field;
[0018] Introducing a cascaded registration architecture to gradually correct the deformation field by connecting multiple registration models in series;
[0019] The loss of image deformation registration is calculated based on the mutual information loss function, regularization loss and segmentation loss.
[0020] Preferably, the construction of the deep deformable registration model is an unsupervised deformable registration framework, specifically comprising:
[0021] The reference CT image and the floating CT image are input into a deep neural network, the deep neural network infers a deformation field, and then the floating CT image is deformed and interpolated using a spatial transformation layer to finally obtain a deformed image;
[0022] During the training process, the loss function value is calculated by comparing the deformed image with the reference CT image, and the network parameters are updated by back propagation;
[0023] The spatial transformation layer transforms the regular spatial grid using deformation fields or transformation parameters to generate an irregular sampling grid, and uses the grid to resample the input image to achieve unsupervised learning.
[0024] Preferably, the mutual information loss function, the regularization loss and the segmentation loss are used to calculate the loss of image deformation registration, and the similarity between the deformed floating CT image and the reference CT image is calculated; the deformation field is regularized, and the organ segmentation area of the floating CT image and the reference CT image pair and the predicted deformation field are used to calculate the Dice loss to obtain the total loss function:
[0025]
[0026]
[0027] Where L sim is the image similarity loss, L reg is the regularization loss, L seg is the segmentation loss, and α, β, γ are their weight coefficients. In addition, I f ,I m , s f 、s m They represent the reference CT image, the floating CT image, the final deformation field, the organ segmentation region of the reference CT image, and the organ segmentation region of the floating CT image, respectively.
[0028] Preferably, the segmentation loss includes using the organ segmentation region s of the reference CT image and the organ segmentation region s of the floating CT image. m To introduce deconstruction constraints, directly minimize and The Dice loss between , where c represents the c-th structure, is calculated by using linear interpolation. For C channels, s mDistortion to calculate Thus, L seg The gradients are back-propagated into the network.
[0029] Preferably, the multimodal data set is preprocessed by a data warehouse tool, wherein the preprocessing includes data cleaning and data standardization, and the preprocessed data after data cleaning is aggregated according to predetermined requirements.
[0030] Preferably, the data cleaning includes missing value processing, duplicate value processing, outlier processing and data standardization;
[0031] The data collection includes extracting data, converting data and loading data:
[0032] The extracted data is to extract original data from multiple data sources; and / or;
[0033] The conversion data is used to clean, convert and standardize the data; and / or;
[0034] The loading of data is loading the processed data into a data warehouse or a database.
[0035] Second aspect: To solve the above technical problems, another technical solution adopted in this application is: a dose superposition system based on depth deformation registration, comprising:
[0036] An acquisition module, used for acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, an extracted reference dose image corresponding to the reference CT image, and an extracted floating dose image corresponding to the floating CT image;
[0037] An input module is used to construct a floating rigid transformation CT image according to the preprocessed multimodal data set, import the reference CT image and the computed tomography image sequence of the floating CT image, import the reference dose image and the three-dimensional dose image of the floating dose image, perform rigid registration on the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and perform rigid transformation on the floating CT image to obtain a floating rigid transformation CT image;
[0038] Constructing a model module, the deep deformable registration model includes a Transformer encoder and a CNN decoder, inputting the reference CT image and the floating rigid transformation CT image into the deep deformable registration model for deformation registration, and obtaining a deformation field;
[0039] A rigid transformation module, used for performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image;
[0040] The pixel-by-pixel module is used to deform the floating rigid transformation dose image in response to the deformation field to obtain a floating deformation dose image, and to perform pixel-by-pixel addition on the reference dose image and the floating deformation dose image to obtain a dose superposition image.
[0041] The third aspect: In order to solve the above-mentioned technical problems, another technical solution adopted in this application is: an electronic device, comprising a processor, a memory and a communication interface, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of any of the above-mentioned dose superposition methods based on depth deformation alignment are implemented.
[0042] Fourth aspect: In order to solve the above-mentioned technical problems, another technical solution adopted in this application is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a dose superposition method based on depth deformation alignment as described in any one of the above are implemented.
[0043] Advantages of the present invention:
[0044] 1. The deep deformation registration model has powerful feature extraction and learning capabilities, and can learn complex deformation rules from a large amount of training data, thereby achieving accurate registration of complex deformations. By building an appropriate deep learning model, key features in the image can be automatically extracted, and deformation fields can be estimated and optimized based on these features.
[0045] 2. Through end-to-end training and optimization, the optimal registration mapping relationship between images can be automatically learned. During the training process, various forms of noise and transformation can be introduced to enhance the robustness of the model. In addition, the deep learning model can also be combined with prior knowledge (such as constraints such as smoothness, consistency and topological correctness of the deformation field) for regularization processing to further improve the accuracy and robustness of the registration.
[0046] 3. The deep deformation registration model can be accelerated by parallel computing devices such as GPU, thereby significantly improving the registration speed. In addition, the deep learning model can reduce the computational complexity and improve the registration efficiency while ensuring the registration accuracy by optimizing the design of the algorithm and model architecture. This makes the deformation registration method based on deep learning more practical and feasible in practical applications.
[0047] 4. The deep deformation registration model has strong adaptive and learning capabilities, and can automatically learn the parameters and rules required for registration from the training data. This enables the deformation registration method based on deep learning to achieve a higher degree of automation and intelligence. During the training process, the model can automatically adjust the parameters to optimize the registration effect without manual intervention. In addition, the deep learning model can also be combined with other image processing technologies (such as image segmentation, feature extraction, etc.) for joint optimization to further improve the intelligence level of registration. Description of the drawings:
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 A schematic flow chart of a dose superposition method based on depth deformation registration according to the present invention;
[0050] Figure 2 It is a structural schematic diagram of an electronic device of the present invention;
[0051] Figure 3 It is a schematic diagram of the deformation registration framework based on unsupervised learning of the present invention;
[0052] Figure 4 This is a schematic diagram of the structure of the space conversion layer of the present invention;
[0053] Figure 5 It is a module schematic diagram of the dose superposition system based on depth deformation registration of the present invention. Specific implementation method:
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] This invention proposes a novel deformable registration framework based on deep learning to achieve accurate superposition of dose distribution of multiple radiotherapy treatments to evaluate the total irradiation dose of the target area and organs at risk. It uses a deep deformable registration network DeformTrans Morph and a deep deformable registration model to replace the traditional deformable registration algorithm to achieve dose superposition.
[0056] Figure 1FIG. 1 is a flow chart of a dose superposition method based on depth deformation registration according to an embodiment of the present invention. It should be noted that if there are substantially the same results, the method of the present application is not based on Figure 1 The process sequence shown is limited.
[0057] Embodiment 1
[0058] like Figure 1 A dose superposition method based on depth deformation registration is shown, comprising the following steps:
[0059] S10, acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, extracting a reference dose image corresponding to the reference CT image, and extracting a floating dose image corresponding to the floating CT image;
[0060] A CT image sequence is composed of a series of CT image planes arranged in spatial order and parallel to each other. Each image plane represents a set of sampling grids of a specific thickness, length, and width in the three-dimensional human body space. These grids are called voxels, and their values represent the density distribution sampling value of the voxel in the three-dimensional human body space, that is, the CT value.
[0061] Specifically, it includes: preprocessing of multimodal data sets, including data cleaning, data conversion (such as converting text data into numerical data), and data standardization.
[0062] S20, importing the computed tomography image sequence of the reference CT image and the floating CT image, importing the three-dimensional dose image of the reference dose image and the floating dose image according to the multimodal data set, rigidly registering the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and rigidly transforming the floating CT image to obtain a floating rigid transformation CT image;
[0063] The three-dimensional dose image is also composed of a series of dose image planes arranged in spatial order and parallel to each other. Each dose image plane is composed of a set of sampling grids of specific thickness, length and width in the three-dimensional dose space. Each grid is called a voxel, and its value represents the dose distribution sampling value at the location of the voxel in the three-dimensional dose space;
[0064] S30, constructing a deep deformation registration model: the deep deformation registration model includes a Transformer encoder and a CNN decoder, and inputting the reference CT image and the floating rigid transformation CT image into the deep deformation registration model for deformation registration to obtain a deformation field;
[0065] S40, performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image;
[0066] S50 , deforming the floating rigid transformation dose image in response to the deformation field to obtain a floating deformation dose image, and adding the reference dose image and the floating deformation dose image pixel by pixel to obtain a dose superposition image.
[0067] In this embodiment, specifically, the multimodal data set is preprocessed by a data warehouse tool, the preprocessing includes data cleaning and data standardization, and the preprocessed data after data cleaning is collected according to predetermined requirements;
[0068] The data cleaning includes missing value processing, duplicate value processing, outlier processing and data standardization;
[0069] The collection includes extracting data, transforming data, and loading data:
[0070] The extracted data is to extract original data from multiple data sources; and / or;
[0071] The conversion data is used to clean, convert and standardize the data; and / or;
[0072] The loading of data is loading the processed data into a data warehouse or a database.
[0073] 1. Missing value processing:
[0074] For fields with fewer missing values, interpolation or mean filling is used to complete them.
[0075] For fields with many missing values, consider deleting the field or using machine learning methods (such as KNN imputation) for prediction and completion.
[0076] 2. Duplicate value processing:
[0077] De-duplicate data by primary key or unique identifier to ensure data uniqueness.
[0078] 3. Outlier processing:
[0079] Use statistical methods (such as box plot method, Z-score method) to identify and handle outliers.
[0080] 4. Data Standardization:
[0081] Standardize the numerical data, such as normalization or Z-score standardization, to ensure that the data features are on the same scale.
[0082] Encode categorical data, such as One-hot encoding or Label encoding.
[0083] 5. Data collection and standardization:
[0084] ETL process:
[0085] Extraction: Extract raw data from multiple data sources to ensure the comprehensiveness and diversity of the data.
[0086] Transform: Clean, convert and standardize data to ensure consistency and comparability.
[0087] Load: Load the processed data into a data warehouse or database to facilitate subsequent analysis and modeling.
[0088] Unified data standards:
[0089] Establish unified data standards and format specifications to ensure the uniformity of data from different sources after aggregation.
[0090] Create a data dictionary to describe in detail the meaning and value range of each field.
[0091] Embodiment 2
[0092] In order to solve the above technical problems, the present application adopts another technical solution based on the first embodiment: the present invention proposes a new deformation registration model based on deep learning;
[0093] The constructing of the deep deformable registration model specifically includes:
[0094] Combination of Transformer and CNN: The Transformer encoder is used to capture the spatial correspondence between the input floating CT image and the reference CT image, and the CNN decoder processes the information provided by the Transformer encoder into a dense displacement field;
[0095] Introducing a cascaded registration architecture to gradually correct the deformation field by connecting multiple registration models in series;
[0096] The loss of image deformation registration is calculated based on the mutual information loss function, regularization loss and segmentation loss.
[0097] Specifically, the registration network model combining Transformer and CNN, registration is the process of intuitively establishing this correspondence by comparing the floating CT image to different parts of the reference CT image. The receptive field of the convolutional neural network (CNN) is very narrow. It performs convolution locally, and its receptive field grows proportionally with the depth of the CNN. Therefore, the shallow layer has a relatively small receptive field, which limits the ability of the CNN to associate distant parts between the two images. For example, if the left part of the floating CT image matches the right part of the reference CT image, if the CNN cannot see the two parts at the same time (that is, when one of the parts falls outside the field of view of the CNN), the CNN will not be able to establish an appropriate spatial correspondence between the two parts. The Transformer can better understand the spatial correspondence between the floating CT image and the reference CT image, and quickly focus on the part that needs to be deformed, because it has a large receptive field and a self-attention mechanism. Therefore, the present invention proposes a registration network model Deform Trans Morph combining Transformer and CNN;
[0098] The encoder of Deform Trans Morph first divides the input floating CT image and the reference CT image into non-overlapping 3D blocks, where the size of each block is 2×P×P×P, where P is usually set to 4.
[0099] The present invention is used represents the i-th block, where i∈{1,...,N}, is the total number of blocks. Each block is expanded into a token, and then the linear projection layer projects each token into a feature representation of arbitrary dimension, which is denoted by C:
[0100]
[0101] in Represents a linear projection, and the dimension of the output z0 is N×C.
[0102] After the linear projection layer, a Patch Merging layer and several consecutive stages of Swin-TransformerV2Block and Deformable Attention Transformer blocks are applied on the token z0.
[0103] The Swin-TransformerV2 block and the Deformable Attention Transformer block output the same number of tokens as they input, while the Patch Merging layer first merges the features of each group of 2×2×2 adjacent tokens, so they reduce the number of tokens by a factor of 2×2×2=8 (e.g., ), and then use the linear layer to project the 8C-dimensional merged features into 2C-dimensional features.
[0104] After 2 stages of Swin-TransformerV2 blocks, 2 stages of Deformable AttentionTransformer blocks, and 3 stages of Patch Merging layers between Transformer stages, the output size of the final stage of the encoder is:
[0105]
[0106] The decoder consists of consecutive upsampling and convolutional layers with a kernel size of 3×3. Each upsampled feature map in the decoding stage is merged with the corresponding feature map from the encoding path through a skip connection, followed by two consecutive convolutional layers. Since the Transformer cannot provide high-resolution feature maps and aggregate local information at lower layers;
[0107] The present invention uses a convolutional layer with a stride of 1 and a stride of 2, taking the original image pair as input to capture local information and generate high-resolution feature maps. The outputs of these layers are merged with the feature maps in the decoder to produce the deformation field. Except for the last convolutional layer, each convolutional layer is followed by a Leaky Rectified Linear Unit activation.
[0108] Swi n -TransformerV2 block:
[0109] Swin-Transformer can generate hierarchical feature maps of various resolutions by using Patch Merging layers, making it very suitable for use as a general backbone for pixel-level tasks such as image registration and segmentation. The most important component of Swin-Transformer, besides the Patch Merging layers, is the self-attention mechanism based on the shifted window.
[0110] Unlike ViT, which computes the relationship between a word unit and all other tokens at each step of the self-attention module, Swin-Transformer computes self-attention within a uniformly divided non-overlapping local window of the original feature map and the low-resolution feature map. Based on Swin-Transformer, Swin-TransformerV2 uses post-normalization and scaled cosine attention to solve the instability problem and uses the log-spaced continuous position deviation method to solve the cross-window resolution migration problem.
[0111] Different from the original Swin TransformerV2, this paper uses rectangular parallel pipeline windows to accommodate non-square images, and the shape of each window is M x ×M y ×M z At each resolution, the first Swin TransformerV2 block uses a regular window partitioning method. Starting from the voxel in the upper left corner, the feature map is evenly divided into M windows. x ×M y ×M z Non-overlapping windows of , and then local self-attention is calculated within each window. In order to introduce connections between adjacent windows, Swin TransformerV2 uses a shifted window design: in consecutive Swin TransformerV2 blocks, the window of the latter block is shifted based on the window in the previous block. With window-based attention, two consecutive Swin TransformerV2 blocks can be calculated as:
[0112]
[0113]
[0114]
[0115]
[0116] Where W_MSA and SW_MSA represent the window-based multi-head self-attention module and the moving window-based multi-head self-attention module respectively, MLP represents the multi-layer perceptron, and z l denote the output features of (S)W_MSA and MLP modules respectively. Self-attention is calculated as:
[0117]
[0118] in is the query, key, value matrix, M x M y M z is the number of words in the 3D window, B represents the relative position of the word in each window, which is a learnable parameter, and τ is a learnable scalar;
[0119] Deformable Attention Transformer block:
[0120] Since learning a deformable receptive field is very important for the image deformation registration task, and the Swin TransformerV2 block can only provide a regular receptive field like the traditional convolutional layer, the present invention introduces a Deformable Attention Transformer block in the last two stages of the encoder to handle flexible spatial positions.
[0121] The Deformable Attention Transformer block can effectively model the relationship between tokens, guided by the important regions in the feature map. These focus regions are determined by multiple sets of deformation sampling points, which are learned from the query through the offset network.
[0122] The present invention adopts trilinear interpolation to sample features from feature maps, and then feeds the sampled features to key and value projections to obtain deformable keys and values. Finally, standard multi-head attention is applied. In addition, the positions of the deformation points provide a more powerful relative position bias to promote the learning of deformable attention.
[0123] Like the Swin TransformerV2 block, the present invention extends the Deformable Attention Transformer block from 2-dimensional space to 3-dimensional space.
[0124] The Deformable Attention Transformer block replaces the Self Attention module in the Transformer with the Deformable Attention module based on the traditional Transformer.
[0125] Given an input feature map Spawn Point A uniform grid is used as a reference.
[0126] Specifically, the grid size is downsampled by a factor r from the input feature map size. The reference point values are linearly spaced three-dimensional coordinates {(0, 0, 0), ..., (D G -1, H G -1, W G -1)}, then the present invention is based on the grid shape D G ×H G ×W G Normalize it to the range [-1, 1]. To obtain the offset of each reference point, linearly project the feature map to the query word q = xW q , and then input into the lightweight sub-network θ offset(·) to generate an offset Δp = θ offset (q). In order to stabilize the training process, the present invention scales the amplitude of Δp by a predetermined factor s to prevent the offset from being too large, i.e., Δp←stanhΔp. The features are then sampled at the position of the deformation point and projected to the key word and value word using the projection matrix:
[0127]
[0128]
[0129] and Respectively represent the embedded expressions of the deformed key and value. Specifically, the present invention sets the sampling function φ(.;.) to trilinear interpolation to make it differentiable:
[0130]
[0131] where g(a, b) = max(0, 1-|ab|) and (r x , r y , r z )index Similar to Swin TransformerV2, the present invention performs multi-head attention on q, k, v and adopts the relative position R. The output formula of the mth attention head is:
[0132]
[0133] Among them, d is the dimension of each attention head, Since the deformable attention of the present invention has continuous key positions, the present invention calculates the relative displacement in the normalized range [-1, 1] and then in the parameterized deviation table Interpolation All possible offset values are covered by continuous relative displacement. Finally, the features of each attention head are merged together and passed through W o Projection obtains the final output z;
[0134] The subnetwork designed using the query feature is used to generate offsets and output the offset values of the reference points respectively. Considering that each reference point covers a local s×s×s area (s is the maximum value of the offset), the generative network should also have the perception of local features to learn reasonable offsets. Therefore, the present invention implements the subnetwork as two convolution modules with nonlinear activation, and the input features are first captured by 5×5×5 deep convolution. Then, GELU activation and 1×1×1 convolution are used to obtain the 3D offset.
[0135] Cascade Registration Architecture:
[0136] Since the patient's body position is different when the CT images are scanned at different time points, the deformation between the two images may be large, making it difficult to predict the correct deformation field at one time. Therefore, the present invention uses a cascade architecture to connect multiple registration networks in series, and the deformation field of the previous network is continuously corrected by the subsequent network, and finally a more accurate deformation field is obtained.
[0137] The present invention uses I f ,I m , Represent the reference CT image, floating CT image and deformation field respectively, and construct a prediction function F to convert I f ,I m As input to predict I m Alignment I f The present invention cascades this process by recursively performing registration on the deformed image. Deformed image I′ m It is a combination of deformation field and floating CT image, namely:
[0138]
[0139] F can be defined as:
[0140]
[0141] Here, F1 may be the same as F, but usually has a different deformation field prediction function. This recursion can be applied infinitely in theory.
[0142] Following this recursion, the floating CT image is continuously deformed so that the final prediction (which may have large deformations) can be decomposed into a cascade of progressive refinements (with small deformations). A cascade is basically a deformation prediction function (f k ), which is equivalent to the main model proposed above in the present invention, and the deformation field predicted by the k-th cascade is:
[0143]
[0144] represents the floating CT image deformed by the first k cascades, Figure 1-4 The entire cascade architecture is depicted. Assuming there are n doses in total, the final output is the combination of all predicted deformation fields:
[0145]
[0146] The final deformed image can be expressed as:
[0147]
[0148] Loss function:
[0149] The overall loss function of network training is derived from the energy function of the traditional image registration algorithm. Therefore, the present invention designs three types of loss functions: the first type is to calculate the similarity between the deformed floating CT image and the reference CT image; the second type is to regularize the deformation field to make it smooth; the third type is to calculate the Dice loss using the organ segmentation area of the CT image pair and the predicted deformation field. The present invention only needs to calculate the final composite deformation field. To find the loss, we can jointly train all cascade networks, so we can get the total loss function:
[0150]
[0151]
[0152] Where L sim is the image similarity loss, L reg is the regularization loss, L seg is the segmentation loss, and α, β, γ are their weight coefficients. In addition, I f ,I m , s f 、s m They represent the reference CT image, the floating CT image, the final deformation field, the organ segmentation region of the reference CT image, and the organ segmentation region of the floating CT image, respectively.
[0153] Image similarity loss: In order to measure the similarity between images, the present invention proposes to use mutual information as the image similarity loss function. Mutual information originates from information theory, which is a measure of shared information between two variables. In the case of image registration, these variables represent the voxel intensities of a pair of images. The more information shared between spatially corresponding voxels, the higher their mutual information. Mutual information relies on the calculation of joint entropy and marginal entropy, the latter of which relies on the calculation of probability distribution. In order to implement mutual information in a deep learning framework, the present invention cannot use histogram-based probability calculations because this is not differentiable. Therefore, the present invention relies on the Parzen window formula using a Gaussian function. By taking advantage of this, the present invention is able to implement mutual information in a deep learning framework, where the present invention utilizes the automatic differentiation capability of PyTorch. Therefore, the present invention uses the following negative normalized version:
[0154]
[0155] in It is a deformation field to I m The deformed image after deformation, isI f and ), H(I f )and is the marginal entropy.
[0156] Smoothness loss: Although optimizing similarity loss alone will encourage the deformed I m As close as possible to I f , but the resulting deformation field may not be smooth or realistic.
[0157] In order to impose smoothness in the deformation field, the present invention adds a regularization loss L to the loss function reg , to make the displacement value in one position similar to the value in its adjacent position. The present invention adopts a regularization loss for penalizing sharp bending deformation, defined as:
[0158]
[0159] Where p is the voxel position, Ω is the set of all voxel positions in the image, and u(p) is the spatial gradient of the displacement field u. The spatial gradient is approximately the forward difference, that is,
[0160] Segmentation loss: The present invention uses the organ segmentation region s of the reference CT image f and floating organ segmentation regions of CT images m To introduce deconstruction constraints, directly minimize and Dice loss between, where c represents the c-th structure:
[0161]
[0162] In order to allow the back propagation of Dice loss, the present invention converts s f and m The image volume is designed to have C channels, each of which contains a binary mask that defines the segmentation of a specific structure / organ. Then, by using linear interpolation For C channels, s m Distortion to calculate So we can use L seg The gradients are back-propagated into the network.
[0163] An unsupervised deformable registration framework is used to address the lack of a gold standard deformation field.
[0164] In this embodiment, specifically, the construction of the deep deformable registration model is an unsupervised deformable registration framework, which specifically includes:
[0165] The reference CT image and the floating CT image are input into a deep neural network (DNN), the deep neural network infers the deformation field, and then the floating CT image is deformed and interpolated using a spatial transformation layer to finally obtain a deformed image;
[0166] During the training process, the loss function value is calculated by comparing the deformed image with the reference CT image, and the network parameters are updated by back propagation;
[0167] The spatial transformation layer transforms the regular spatial grid using deformation fields or transformation parameters to generate an irregular sampling grid, and uses the grid to resample the input image to achieve unsupervised learning.
[0168] Specifically, in the prior art, in supervised learning methods, the gold standard deformation field is either generated synthetically or by traditional registration methods, which results in the generation of the gold standard deformation field being computationally expensive, and the registration accuracy of the supervised learning method depends largely on the quality of the gold standard deformation field. In view of the limitations of the supervised method, the present invention adopts an unsupervised method that does not require a gold standard deformation field.
[0169] Unsupervised methods learn a general registration representation from a training set by optimizing an energy function (loss function) on the input images, which is eventually applied to unseen images.
[0170] like Figure 3 As shown, the present invention first inputs the registration pair (reference CT image and floating CT image) into a deep neural network (DNN), the DNN infers the deformation field, and then uses the spatial transformation network (STN) to deform and interpolate the floating CT image, and finally obtains the deformed image.
[0171] During the training process, the present invention uses the deformed image and the reference CT image to calculate the loss function value, and then updates the network parameters through back propagation, thereby achieving unsupervised learning without relying on the gold standard deformation field;
[0172] like Figure 4 As shown, the spatial transformation layer transforms the regular spatial grid using the deformation field or transformation parameters (translation and rotation angle) to generate an irregular sampling grid, and uses the grid to resample the input image. Therefore, the present invention intends to use the spatial transformation layer to resample the floating CT image under the predicted deformation grid, and PyTorch currently has a corresponding implementation method.
[0173] In this embodiment, specifically, the mutual information loss function, the regularization loss and the segmentation loss are used to calculate the loss of image deformation registration, and the similarity between the deformed floating CT image and the reference CT image is calculated; the deformation field is regularized, and the organ segmentation region of the floating CT image and the reference CT image pair and the predicted deformation field are used to calculate the Dice loss, and the total loss function is obtained:
[0174]
[0175]
[0176] Where L sim is the image similarity loss, L reg is the regularization loss, L seg is the segmentation loss, and α, β, γ are their weight coefficients. In addition, I f ,I m , s f 、s m They represent the reference CT image, the floating CT image, the final deformation field, the organ segmentation region of the reference CT image, and the organ segmentation region of the floating CT image, respectively.
[0177] In this embodiment, specifically, the segmentation loss includes using the organ segmentation region s of the reference CT image. f and floating organ segmentation regions of CT images m To introduce deconstruction constraints, directly minimize and The Di ce loss between, where c represents the c-th structure, is calculated by using linear interpolation For C channels, s m Distortion to calculate Thus, L seg The gradient of is back-propagated to the network;
[0178] Embodiment 3
[0179] like Figure 5 As shown, in order to solve the above technical problems, this application adopts another technical solution based on the first embodiment:
[0180] A dose superposition system based on depth deformation registration, comprising:
[0181] An acquisition module, used for acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, an extracted reference dose image corresponding to the reference CT image, and an extracted floating dose image corresponding to the floating CT image;
[0182] An input module is used to construct a floating rigid transformation CT image according to the preprocessed multimodal data set, import the reference CT image and the computed tomography image sequence of the floating CT image, import the reference dose image and the three-dimensional dose image of the floating dose image, perform rigid registration on the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and perform rigid transformation on the floating CT image to obtain a floating rigid transformation CT image;
[0183] Constructing a model module, the deep deformable registration model includes a Transformer encoder and a CNN decoder, inputting the reference CT image and the floating rigid transformation CT image into the deep deformable registration model for deformation registration, and obtaining a deformation field;
[0184] A rigid transformation module, used for performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image;
[0185] The pixel-by-pixel module is used to deform the floating rigid transformation dose image in response to the deformation field to obtain a floating deformation dose image, and to perform pixel-by-pixel addition on the reference dose image and the floating deformation dose image to obtain a dose superposition image.
[0186] 1. Data Preparation and Preprocessing
[0187] Import CT image sequence:
[0188] Import computed tomography (CT) image sequences of reference CT images and floating CT images.
[0189] Import dose image:
[0190] Import 3D dose images of reference dose images and floating dose images.
[0191] 2. Rigid Registration
[0192] Rigid Registration:
[0193] The reference CT image and the floating CT image are rigidly registered, and a 4x4 spatial transformation matrix is calculated using an existing algorithm (such as the mutual information maximization method in ITK).
[0194] This matrix is used to perform rigid transformation on the floating CT image to obtain a floating rigid transformation CT image.
[0195] Similarly, the floating dose image is rigidly transformed using the spatial transformation matrix to obtain a floating rigid transformed dose image.
[0196] 3. Depth Deformation Registration
[0197] Deep Deformable Registration Model:
[0198] Design a deep deformable registration model (Deform Trans Morph) that adopts an unsupervised deformable registration framework.
[0199] The core of the model is the combination of Transformer and CNN: Transformer is used as an encoder to capture the spatial correspondence between the floating rigid transformation CT image and the reference CT image; CNN is used as a decoder to decode the information provided by the Transformer encoder into a dense displacement field.
[0200] A cascaded registration architecture is introduced to gradually correct the deformation field by connecting multiple registration models in series to solve the problem of excessive distortion caused by large deformation.
[0201] The loss for image deformation registration is jointly calculated using mutual information loss, regularization loss, and (optional) segmentation loss to optimize model performance.
[0202] Perform deformable registration:
[0203] The reference CT image and the floating rigid transformation CT image are input into the Deform Trans Morph model for deformation registration to obtain the deformation field.
[0204] IV. Dose deformation and superposition
[0205] Dose image deformation:
[0206] The floating rigid transformed dose image is deformed using the deformation field obtained from the deep deformable registration model to obtain a floating deformable dose image.
[0207] Dose stacking:
[0208] The reference dose image and the floating deformation dose image are added pixel by pixel to obtain the final dose superposition image. This image combines the reference dose distribution and the floating dose distribution to provide more accurate information for the formulation of radiotherapy plans.
[0209] 5. Result Evaluation and Optimization
[0210] Result evaluation:
[0211] The dose superposition images were quality assessed to check for obvious artifacts, inaccurate overlap, and other issues.
[0212] Based on the feedback from the evaluation results, the deep deformable registration model can be fine-tuned or retrained to further improve the registration accuracy and dose superposition accuracy.
[0213] For other details of the technical solutions for implementing each module in the system of the above embodiment, reference may be made to the description of a dose superposition method based on depth deformation registration in the above embodiment, which will not be repeated here.
[0214] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0215] Embodiment 4:
[0216] like Figure 2 The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 2 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0217] like Figure 2 As shown, an electronic device includes a processor, a memory and a communication interface, wherein a computer program is stored in the memory, and when the processor executes the computer program, a dose superposition method based on depth deformation registration of each embodiment of the present disclosure is implemented. The electronic device can exchange data with other devices or systems through the communication interface to realize real-time update and sharing of drug information.
[0218] The processor in the above-mentioned electronic device serves as the core of the electronic device. The processor is responsible for executing the computer program stored in the memory to realize various functions of the paperless conference terminal intelligent interaction method. The processor can use a high-performance multi-core CPU or a dedicated chip to meet the needs of complex calculations and real-time processing. The memory is used to store operating systems, applications, data, computer programs, etc. In this embodiment, the memory stores a computer program for realizing the paperless conference terminal intelligent interaction method. The memory can be RAM, ROM, Flash memory or other types of non-volatile memory. The communication interface is used to connect the electronic device with other devices or networks to realize data transmission and exchange. In this embodiment, the communication interface supports a variety of communication protocols and interface standards, such as Wi-Fi, Bluetooth, USB, Ethernet, etc., to meet the communication needs in different scenarios.
[0219] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.
[0220] Embodiment 5
[0221] According to a computer-readable storage medium of an embodiment of the present disclosure, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the various functions of the dose superposition method based on depth deformation registration of the aforementioned embodiments of the present disclosure are implemented.
[0222] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).
[0223] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.
[0224] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A dose superposition method based on depth deformation registration, It is characterized in that The following steps are involved: Acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, extracting a reference dose image corresponding to the reference CT image, and extracting a floating dose image corresponding to the floating CT image; constructing a floating rigid transformation CT image according to the preprocessed multimodal data set, importing the reference CT image and the computed tomography image sequence of the floating CT image, importing the reference dose image and the three-dimensional dose image of the floating dose image, rigidly registering the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and rigidly transforming the floating CT image to obtain a floating rigid transformation CT image; Constructing a deep deformable registration model: the deep deformable registration model includes a Transformer encoder and a CNN decoder, inputting the reference CT image and the floating rigid transformation CT image into the deep deformable registration model for deformation registration to obtain a deformation field; Performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image; The floating rigid transformation dose image is deformed in response to the deformation field to obtain a floating deformation dose image, and the reference dose image and the floating deformation dose image are added pixel by pixel to obtain a dose superposition image.
2. A dose superposition method based on depth deformation registration according to claim 1, It is characterized in that The constructing of the deep deformable registration model specifically includes: Combination of Transformer and CNN: The Transformer encoder is used to capture the spatial correspondence between the input floating CT image and the reference CT image, and the CNN decoder processes the information provided by the Transformer encoder into a dense displacement field; Introducing a cascaded registration architecture to gradually correct the deformation field by connecting multiple registration models in series; The loss of image deformation registration is calculated based on the mutual information loss function, regularization loss and segmentation loss.
3. A dose superposition method based on depth deformation registration according to claim 1, It is characterized in that The construction of the deep deformable registration model is an unsupervised deformable registration framework, which specifically includes: The reference CT image and the floating CT image are input into a deep neural network, the deep neural network infers a deformation field, and then the floating CT image is deformed and interpolated using a spatial transformation layer to finally obtain a deformed image; During the training process, the loss function value is calculated by comparing the deformed image with the reference CT image, and the network parameters are updated by back propagation; The spatial transformation layer transforms the regular spatial grid using deformation fields or transformation parameters to generate an irregular sampling grid, and uses the grid to resample the input image to achieve unsupervised learning.
4. A dose superposition method based on depth deformation registration according to claim 2, It is characterized in that The mutual information loss function, the regularization loss and the segmentation loss are used to calculate the loss of image deformation registration, and the similarity between the deformed floating CT image and the reference CT image is calculated; Regularize the deformation field, use the organ segmentation area of the floating CT image and the reference CT image pair and the predicted deformation field to calculate the Dice loss, and get the total loss function: Where L sim is the image similarity loss, L reg is the regularization loss, L seg is the segmentation loss, and α, β, γ are their weight coefficients. In addition, I f ,I m , s f 、s m They represent the reference CT image, the floating CT image, the final deformation field, the organ segmentation region of the reference CT image, and the organ segmentation region of the floating CT image, respectively.
5. A dose superposition method based on depth deformation registration according to claim 4, It is characterized in that The segmentation loss includes using the organ segmentation region s of the reference CT image f and floating organ segmentation regions of CT images m To introduce deconstruction constraints, directly minimize and The Dice loss between , where c represents the c-th structure, is calculated by using linear interpolation. For C channels, s m Distortion to calculate Thus, L seg The gradients are back-propagated into the network.
6. A dose superposition method based on depth deformation registration according to claim 1, It is characterized in that The multimodal data set is preprocessed by a data warehouse tool, wherein the preprocessing includes data cleaning and data standardization, and the preprocessed data after data cleaning is aggregated according to predetermined requirements.
7. A dose superposition method based on depth deformation registration according to claim 6, It is characterized in that The data cleaning includes missing value processing, duplicate value processing, outlier processing and data standardization; The data collection includes extracting data, converting data and loading data: The extracted data is to extract original data from multiple data sources; and / or; The conversion data is used to clean, convert and standardize the data; and / or; The loading of data is loading the processed data into a data warehouse or a database.
8. A dose superposition system based on depth deformation registration, applied to the dose superposition method based on depth deformation registration according to any one of claims 1 to 7, It is characterized in that include: An acquisition module, used for acquiring a multimodal data set and preprocessing image data, wherein the multimodal data set includes a reference CT image, a floating CT image, an extracted reference dose image corresponding to the reference CT image, and an extracted floating dose image corresponding to the floating CT image; An input module is used to construct a floating rigid transformation CT image according to the preprocessed multimodal data set, import the reference CT image and the computed tomography image sequence of the floating CT image, import the reference dose image and the three-dimensional dose image of the floating dose image, perform rigid registration on the reference CT image and the floating CT image to obtain a 4x4 spatial transformation matrix, and perform rigid transformation on the floating CT image to obtain a floating rigid transformation CT image; Constructing a model module, the deep deformable registration model includes a Transformer encoder and a CNN decoder, inputting the reference CT image and the floating rigid transformation CT image into the deep deformable registration model for deformation registration, and obtaining a deformation field; A rigid transformation module, used for performing a rigid transformation on the floating dose image according to the 4x4 spatial transformation matrix to obtain a floating rigid transformation dose image; The pixel-by-pixel module is used to deform the floating rigid transformation dose image in response to the deformation field to obtain a floating deformation dose image, and to perform pixel-by-pixel addition on the reference dose image and the floating deformation dose image to obtain a dose superposition image.
9. An electronic device, It is characterized in that The device comprises a processor, a memory and a communication interface, wherein a computer program is stored in the memory, and when the processor executes the computer program, a dose superposition method based on depth deformation registration as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the dose superposition method based on depth deformation registration according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method for generating distribution image used for observing cervix tumour radiotherapy total dose
CN103226837A
Abdomen multi-organ registration method based on adaptive multi-gating hybrid expert model
CN116993793A
Multi-task abdominal organ registration method based on mutual attention and semantic sharing
CN117036428A
Deformation registration model training and dose superposition method and device
CN117197203A
Cited By
Cervical cancer target area automatic sketching system based on deep learning
CN119724560A
