Method and device for segmenting organ at risk in radiotherapy plan, and storage medium

Abstract

The invention discloses an endangered organ segmentation method and device in a radiotherapy plan and a storage medium, and relates to the field of medical image processing, and the method comprises the steps: constructing an Adaptor module which is used for carrying out the Z-direction information modeling of a feature map extracted by a 2D convolution module; accessing the Adaptor module used for performing Z-direction information modeling on the feature map extracted by the 2D convolution module after the 2D convolution module of each resolution layer of the 2D segmentation model of the U-shaped structure, and forming an organ-at-risk segmentation model capable of extracting 2D information and 3D information; and training the endangered organ segmentation model capable of extracting the 2D information and the 3D information so as to segment the endangered organ in the CT image of the patient by using the trained endangered organ segmentation model capable of extracting the 2D information and the 3D information. The method can give consideration to the model reasoning efficiency, the model training difficulty and the segmentation precision.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, and in particular to a method, device and storage medium for segmenting organs at risk in radiotherapy planning. Background Art

[0002] For cancer patients undergoing radiotherapy, in order to more accurately target the tumor and to allow the radiation to avoid the organs around the tumor (usually referred to as organs at risk in the tumor area) as much as possible, it is necessary to identify the organs around the tumor to make a radiotherapy plan. This process is called organ delineation.

[0003] In traditional radiotherapy plans, organ delineation is performed by physicists, and manual delineation is inefficient. In a complete state, manual delineation requires manual delineation of at least four or five organs, and as many as twenty or thirty organs, for each CT slice, which takes 3-8 hours or even longer. In addition, manual delineation is easily affected by factors such as fatigue, emotional stability, and the quality of hardware equipment, resulting in unstable delineation results, and the delineation results tend to decrease with the length of each delineation task.

[0004] After the introduction of AI, the entire image is segmented using a segmentation neural network, and the organs at risk are automatically outlined through a multi-organ segmentation model or multiple single-organ segmentation models. The outline effect is stable. Compared with manual organ outlines, different image segmentation methods have different degrees of improvement in efficiency. Among them, the U-shaped structure composed of an encoder and a decoder is the mainstream medical image segmentation network, such as Figure 1 As shown in the figure, the encoder extracts feature information and reduces the dimension of the input image through a series of convolutional layers and pooling layers; the decoder restores the features extracted by the encoder for segmented organs through convolution and deconvolution operations; at each feature map resolution stage, the feature maps of the encoder and decoder are spliced ​​in the channel direction to prevent information loss during the downsampling process.

[0005] Medical imaging data in radiotherapy is usually in 3D format, including three dimensions of length, width and height. Existing automatic segmentation technologies based on neural network technology can be divided into 2D segmentation networks and 3D segmentation networks. The main difference between the two is that the convolution kernel used in the convolution operation is different. Among them, the 2D segmentation network performs segmentation of organs at risk layer by layer on CT images. For each layer, the 2D segmentation network uses a 2-dimensional convolution kernel, see Figure 2, a 3*3 convolution kernel is used to perform a 2D convolution operation on the H*W image to obtain a feature map. This method has the advantages of a small model, fast inference speed and high efficiency. However, in the process of model learning and application, since the 2D convolution operation only considers the two-dimensional spatial relationship in the H and W directions of the 3D image, the correlation between layers in the Z direction is not considered, especially the relationship between layers of the organs at risk in the Z direction. Therefore, the final segmentation accuracy is not high. The 3D segmentation network makes up for the defects of the 2D segmentation network. It uses a 3D convolution kernel, see Figure 3 , a 3*3*3 convolution kernel is used to perform a 3D convolution operation on the H*W*Z image to obtain a feature map. Since all operators focus on three-dimensional information, the model segmentation accuracy is improved. However, the model has a large number of parameters, see Figure 2 and Figure 3 In a single convolution operation with the same length and width, the number of parameters of the 3D convolution kernel is three times that of the 2D convolution kernel. Usually, medical image segmentation networks perform thousands of convolution operations. If the number of parameters increases, the difficulty of network training will increase, the inference efficiency will decrease, and higher resource requirements will be required.

[0006] In summary, how to balance model reasoning efficiency, model training difficulty, and segmentation accuracy has high research significance. Summary of the invention

[0007] The present invention provides a method, device and storage medium for segmenting organs at risk in radiotherapy planning, aiming to solve the problem that existing 2D segmentation networks and 3D segmentation networks cannot take into account model reasoning efficiency, model training difficulty and segmentation accuracy.

[0008] The present invention provides a method for segmenting organs at risk in radiotherapy planning. The method comprises: constructing an Adaptor module for modeling Z-direction information of a feature map extracted by a 2D convolution module; connecting the Adaptor module for modeling Z-direction information of a feature map extracted by the 2D convolution module after the 2D convolution module of each resolution layer of a U-shaped structure 2D segmentation model to form an organ at risk segmentation model capable of extracting 2D information and 3D information; training the organ at risk segmentation model capable of extracting 2D information and 3D information, so as to segment organs at risk in a patient's CT image by using the trained organ at risk segmentation model capable of extracting 2D information and 3D information.

[0009] Preferably, the Adaptor module includes a bottle-neck convolution submodule for preliminarily modeling Z-direction information and a self-attention submodule for performing Z-direction self-attention operation.

[0010] Preferably, the bottle-neck convolution submodule includes a 1D convolution layer for downsampling the input feature map in the Z direction, a 3D convolution layer, and a 1D convolution layer for image restoration based on the size of the input feature map, which are cascaded in sequence.

[0011] Preferably, the self-attention submodule adopts a group attention mechanism of a rolling window, and the group attention mechanism of the rolling window includes: performing a self-attention operation on the Z-direction feature data in the feature map output by the bottle-neck convolution submodule by group to obtain first interactive feature data in the Z direction; performing a rolling operation based on a rolling window on the first interactive feature data in the Z direction, and then performing a self-attention operation on the group to obtain second interactive feature data in the Z direction.

[0012] Preferably, the grouping is performed according to the following formula:

[0013]

[0014] Among them, n Z is the data dimension in the z direction of the feature map, and n is the data dimension within each group.

[0015] Preferably, during the training of the organ-at-risk segmentation model capable of extracting 2D information and 3D information, parameter updating is performed according to the following formula:

[0016]

[0017] Among them, θ is the parameter, α is the learning rate, is the loss gradient of the parameters.

[0018] Preferably, during the training of the risk organ segmentation model capable of extracting 2D information and 3D information, if the gradient disappears or is suspected of falling into a local optimal solution, the parameters are updated by increasing the learning rate.

[0019] Preferably, after increasing the learning rate, if the local optimal solution is exceeded, the gradient is calculated and the parameters are updated; if the local optimal solution cannot be exceeded, the previous gradient is used to update the parameters.

[0020] The present invention also provides a device for segmenting organs at risk in radiotherapy planning, the device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program implements the steps of the above-mentioned method for segmenting organs at risk in radiotherapy planning when executed by the processor.

[0021] The present invention also provides a storage medium, in which a program is stored. When the program is executed by a processor, the steps of the method for segmenting organs at risk in the radiotherapy plan are implemented.

[0022] The present invention builds an organ-at-risk segmentation model that can extract 2D information and 3D information. The model structure includes a 2D convolution module and an Adaptor module, and has the advantages of balancing the accuracy and efficiency of organ-at-risk segmentation. Specifically, compared with the existing 2D segmentation network, the Adaptor module focuses on the relationship between layers of organs at risk in the Z direction, thereby improving the segmentation accuracy and precision. Compared with the existing 3D segmentation network, the number of parameters is reduced, the model reasoning efficiency is improved, the difficulty of model training is reduced, and the required resource requirements are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the medical image segmentation network structure;

[0024] Figure 2 It is a schematic diagram of 2D convolution operation;

[0025] Figure 3 It is a schematic diagram of 3D convolution operation;

[0026] Figure 4 is a flow chart of the method for segmenting organs at risk in radiotherapy planning provided by the present invention;

[0027] Figure 5 It is the network structure diagram of the organ-at-risk segmentation model;

[0028] Figure 6 This is a schematic diagram of the bottle neck structure in the Adaptor module;

[0029] Figure 7 It is the operation flow chart of the attention mechanism;

[0030] Figure 8 This is a schematic diagram of the operation of the grouped attention mechanism of the rolling window;

[0031] Fig. 9 It is a schematic diagram of the gradient and learning rate adjustment process. DETAILED DESCRIPTION

[0032] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In the subsequent description, the use of suffixes such as "module", "layer" or "unit" used to represent elements is only to facilitate the description of the present invention and has no specific meaning in itself. Therefore, sometimes "module", "layer" or "unit" can be used in a mixed manner.

[0033] The present invention designs a new type of network structure of a hazard network segmentation model that integrates the structural advantages of a 2D neural network and a 3D neural network. The segmentation model has high accuracy and fast reasoning speed.

[0034] See also Figure 4 , a method for fractionating organs at risk in radiotherapy planning may include:

[0035] Step S101: construct an Adaptor module for modeling the Z-direction information of the feature map extracted by the 2D convolution module.

[0036] Step S102: the Adaptor module for modeling the Z-direction information of the feature map extracted by the 2D convolution module is connected after the 2D convolution module of each resolution layer of the U-shaped structure 2D segmentation model to form an organ-at-risk segmentation model capable of extracting 2D information and 3D information.

[0037] See also Figure 5 The endangered organ segmentation model of the present invention adds an Adaptor module after completing the convolution operation at each resolution layer of the traditional 2D segmentation model (U-shaped 2D convolutional network), and models the correlation between channels of the feature map extracted by the 2D convolutional network to solve the defect that the traditional 2D convolutional network cannot extract the correlation relationship of endangered organs in the Z direction.

[0038] The Adaptor module consists of two parts: a bottle-neck convolution submodule and a self-attention submodule.

[0039] 1. The bottle-neck convolution submodule is used to preliminarily model the Z-direction information.

[0040] The bottle-neck convolution submodule is composed of a cascade of three parts: a 1D convolution layer for downsampling the input feature map in the Z direction, a 3D convolution layer, and a 1D convolution layer for image restoration based on the size of the input feature map. Figure 6 For each resolution level, for example, each encoder level, a 1D convolution operation is first performed on the feature map of each encoder level with a size of W (width) × H (height) × Z (depth) through a 1D convolution layer, and the feature map of each encoder level with a size of W × H × Z is downsampled in the Z direction to obtain a feature map of size W × H × Z / 2. A 3D convolution operation is then performed on the downsampled feature map of size W × H × Z / 2, and finally restored to the original feature map size W × H × Z through a 1D convolution operation. Through the above operations, preliminary modeling of Z-direction information can be achieved at a small computational cost.

[0041] The number of parameters in a 3D convolution operation is P. 3D , the formula is:

[0042] P 3D =C in *V out 3*3*3=27C in V out

[0043] In the formula, C in and C out Refers to input and output channels.

[0044] The parameter quantity of the bottle-neck convolution submodule of the present invention is P bn , the formula is:

[0045]

[0046] Assumption C in and C out At this time, compared with the 3D convolution module, the bottle-neck convolution sub-model of the present invention can reduce the number of parameters by about 70%.

[0047] 2. The self-attention submodule is an improved sliding window self-attention submodule, which adopts a rolling window grouping attention mechanism to perform Z-direction self-attention operations.

[0048] The attention mechanism is similar to the human brain mechanism when humans perform image recognition or translation tasks. It only needs to focus on the key parts of the image and the core words of the sentence to complete the image recognition or translation task. The operation process of the attention mechanism is as follows Figure 7 As shown, the weight vector is calculated by Q and K, and then the weight vector is multiplied by the V vector to obtain the final output vector. The calculation method is as follows:

[0049]

[0050] Where n is the length of the Q, K, and V vectors, and the softmax function is a function commonly used for normalization in deep learning, and its form is as follows:

[0051]

[0052] The present invention adopts a self-attention mechanism, that is, the Q, K, and V vectors are the same vector.

[0053] In addition, since the data dimension of the feature map of the medical image input and the intermediate process of the network in the Z direction may be large, such as 256, 512, etc., in view of this situation, the self-attention operation of all data dimensions in the Z direction of the present invention does not conform to the actual situation (the Z direction of the endangered organ will not be distributed in too many layers), so the present invention adopts a group attention mechanism of a rolling window, specifically: the Z direction feature data in the output feature map of the bottle-neck convolution submodule is subjected to a self-attention operation by group to obtain the first interactive feature data in the Z direction, the first interactive feature data in the Z direction is subjected to a rolling operation based on a rolling window, and then the self-attention operation is performed by group to obtain the second interactive feature data in the Z direction. Among them, the data dimension divided in each group is dynamically determined by the following formula:

[0054]

[0055] Where n Z is the data dimension in the z direction of the feature map, and n is the data dimension within each group.

[0056] See also Figure 8 , the data in the Z direction are dynamically divided into multiple groups. For ease of explanation, Figure 8 Taking Group 1, Group 2, and Group 3 as examples, after the self-attention operation is performed within each group to obtain the output, the window is scrolled, and then the self-attention operation is performed again to realize information interaction between groups.

[0057] Step S103: training the OAR segmentation model capable of extracting 2D information and 3D information, so as to segment the OAR in the patient's CT image using the trained OAR segmentation model capable of extracting 2D information and 3D information.

[0058] The current neural network training method is to calculate the difference between the model output and the label, that is, the loss, calculate the gradient of the loss for each parameter, and then update the model weights through the back propagation algorithm. The specific update method is shown in the following formula:

[0059]

[0060] In the formula, θ is the parameter, α is the learning rate, that is, the step size of each parameter update. is the loss gradient of the parameters.

[0061] The present invention adopts a new model training strategy to optimize the neural network model training process. Specifically, the optimization of the model training strategy is mainly reflected in the calculation method of the gradient and the adjustment strategy of the learning rate. During the parameter update process during training, the gradient and the learning rate are dynamically adjusted according to each other's changes. Fig. 9 , adjustments include:

[0062] 1. Gradient calculation.

[0063] (1) Perform a square operation on the original calculated gradient and scale the gradient according to its size. The parameter update formula is:

[0064]

[0065] If the gradient is greater than 1, the gradient is amplified through a square operation, the parameter update is more rough, and the training process is accelerated; if the gradient is less than 1, the gradient is reduced through a square operation, and the parameters are updated cautiously to prevent the training from missing the optimal point.

[0066] (2) In order to solve the problem of falling into the local optimal solution during training, the gradient used in each parameter iteration is dynamically selected. Specifically, if the learning rate still fails to jump out of the local optimal solution after dynamic adjustment, the previous gradient is used and gradient descent is performed again. If there is no optimization, counting begins. If there is still no optimization after 100 iterations, it is considered that the optimization direction is correct and the counting continues. If there is still no optimization after counting 200, it is considered that this training has reached the theoretical optimal solution and the training is terminated early. If there is optimization, the gradient of this iteration is recorded for use next time it falls into a non-optimization state, and the count is reset to zero.

[0067] 2. Learning rate setting.

[0068] The initial learning rate is set to 1e-7, the upper limit of the learning rate is set to 1e-2, and the learning rate is dynamically adjusted during training to prevent excessive learning rate from causing large jitters in the training process.

[0069] According to the gradient change, the learning rate is adjusted to increase by 10 times. This learning rate adjustment strategy is contrary to the traditional learning rate control concept of "continuously reducing the learning rate" because the square operation is performed on the original gradient. When the original gradient is small, the gradient after the square operation has been greatly reduced, so it is necessary to increase the learning rate to prevent the parameters from being unable to be optimized. The triggering conditions for adjusting the learning rate to increase by 10 times are as follows:

[0070] (1) Gradient disappearance: the model cannot be trained because the gradient is too small;

[0071] (2) There is no optimization in the long term during training, and the system is suspected of being stuck in a local optimum.

[0072] The model training strategy of the present invention can improve the stability of the training process and the segmentation accuracy of the final model

[0073] The present invention also provides a device for segmenting organs at risk in radiotherapy planning, the device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the program implements the steps of the above-mentioned method for segmenting organs at risk in radiotherapy planning when executed by the processor.

[0074] In addition, the present invention also provides a storage medium, in which a program is stored. When the program is executed by a processor, the steps of the method for segmenting organs at risk in a radiotherapy plan as described above are implemented.

[0075] The present invention aims at the disadvantages of accuracy or efficiency of the common 2D or 3D neural network algorithms in the application scenario of organ-at-risk segmentation in radiotherapy planning. The present invention proposes a new network structure of organ-at-risk segmentation model with 2D as the main body and the Adaptor module based on the self-attention mechanism in the Z direction to capture the correlation between medical image layers. It can take into account the advantages of high efficiency and easy training of 2D network and high accuracy of 3D network. In addition, in order to further improve the segmentation accuracy of the organ-at-risk segmentation model, the present invention adopts a new model training strategy, which dynamically adjusts the learning rate and the gradient in each iteration process in a linked manner according to the real-time optimization situation during the training process.

[0076] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the present invention shall be within the scope of the present invention.

Claims

1. A method for segmenting organs at risk in radiotherapy planning, characterized in that: The method comprises: Construct an Adaptor module for modeling the Z-direction information of the feature map extracted by the 2D convolution module; The Adaptor module for modeling the Z-direction information of the feature map extracted by the 2D convolution module is connected after the 2D convolution module of each resolution layer of the U-shaped structure 2D segmentation model to form an organ-at-risk segmentation model that can extract 2D information and 3D information; The OAR segmentation model capable of extracting 2D information and 3D information is trained, so as to segment the OAR in the patient's CT image by using the trained OAR segmentation model capable of extracting 2D information and 3D information.

2. The method according to claim 1, characterized in that The Adaptor module includes a bottle-neck convolution submodule for preliminarily modeling Z-direction information and a self-attention submodule for performing Z-direction self-attention operation.

3. The method according to claim 2, characterized in that The bottle-neck convolution submodule includes a 1D convolution layer for downsampling the input feature map in the Z direction, a 3D convolution layer, and a 1D convolution layer for image restoration based on the size of the input feature map, which are cascaded in sequence.

4. The method according to claim 2, characterized in that: The self-attention submodule adopts a grouped attention mechanism of a rolling window, and the grouped attention mechanism of the rolling window includes: Performing a self-attention operation on the Z-direction feature data in the feature map output by the bottle-neck convolution submodule by grouping to obtain the first interactive feature data in the Z-direction; A scrolling operation based on a scrolling window is performed on the first interactive feature data in the Z direction, and then a self-attention operation is performed on the groups to obtain second interactive feature data in the Z direction.

5. The method according to claim 4, characterized in that Grouping is done according to the following formula: Among them, n Z is the data dimension in the z direction of the feature map, and n is the data dimension within each group.

6. The method according to any one of claims 1 to 5, characterized in that: During the training of the risk organ segmentation model capable of extracting 2D information and 3D information, the parameters are updated according to the following formula: Among them, θ is the parameter, α is the learning rate, is the loss gradient of the parameters.

7. The method according to claim 6, characterized in that During the training of the risk organ segmentation model capable of extracting 2D information and 3D information, if the gradient disappears or is suspected of falling into a local optimal solution, the parameters are updated by increasing the learning rate.

8. The method according to claim 7, characterized in that After increasing the learning rate, if the local optimal solution is exceeded, the gradient is calculated and the parameters are updated. If the local optimal solution cannot be exceeded, the previous gradient is used to update the parameters.

9. A device for segmenting organs at risk in radiotherapy planning, characterized in that: The device comprises a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the method for segmenting organs at risk in radiotherapy planning according to any one of claims 1 to 8 are implemented.

10. A storage medium, characterized in that: The storage medium stores a program, and when the program is executed by the processor, the steps of the method for segmenting organs at risk in radiotherapy planning according to any one of claims 1 to 8 are implemented.

Images