Multi-organ segmentation method and medium for intraradiation radiotherapy interventional surgery robot

Abstract

The present invention relates to a multi-organ segmentation method and medium technical solution of a radioactive particle intraradiotherapy interventional surgery robot, including: pre-segmentation and interactive optimization; wherein the pre-segmentation includes: performing full-resolution training based on image blocks on a three-dimensional organ image through a V-Net cascade neural network; wherein the interactive optimization includes: separating the organ target and the background of the three-dimensional organ image according to the pre-segmentation training result and the input bounding box. The beneficial effects of the present invention are: achieving efficient multi-organ segmentation capability, and having the ability to repair and edit the segmentation results twice.

Description

Technical Field

[0001] The present invention relates to the field of computer and medical image processing, and in particular to a multi-organ segmentation method and medium of a radioactive particle intra-radiotherapy interventional surgery robot. Background Art

[0002] Organ segmentation in medical images is a crucial task in many applications such as computer-aided diagnosis (CAD), diagnostic intervention, treatment planning and treatment execution. The delineation of the target organ contour is an essential and important part of radiotherapy planning, but the process requires a lot of tedious manual operations. In recent years, deep learning techniques for organ segmentation have received great attention. Many deep learning-based segmentation models, such as FCNs, U-Net, V-Net and their variants, have been proposed and have achieved good results in image segmentation. At present, these studies focus on the segmentation of certain specific organs or tissues, such as liver, pancreas, blood vessels, gliomas, etc. Single-category segmentation methods make it easier for the network to handle specific organs, and special strategies can be adopted according to the prior information of the target organ to further optimize the segmentation results. However, from the perspective of the requirements for organ segmentation in radiotherapy planning, it is inefficient to train multiple specific task segmentation networks. Therefore, the research focus in this field should be on multi-organ segmentation or multi-task segmentation networks. How to achieve accurate segmentation of multiple organs is still a challenging task.

[0003] Surgical robots do not have the medical background knowledge of doctors, and they cannot quickly combine medical background knowledge with patient vital signs information. Therefore, simply inputting target organ information to the robot is not complete for the entire interventional surgery. At the same time, it is necessary to inform the robot of the peripheral organ information adjacent to the target organ and the organ information of the interventional process pathway, so as to better use the robot to complete the corresponding surgical assisted operation. At the same time, although deep neural networks have greatly promoted the development of medical image processing, due to certain challenges of medical images themselves, such as poor image quality, differences in imaging protocols, and patient specificity, the fully automatic segmentation method based on deep learning has not yet been able to output sufficiently accurate and reliable results in clinical applications. Therefore, the existing segmentation algorithm cannot be directly applied to clinical diagnosis, and its segmentation needs to be further edited using other manual methods. At the same time, due to safety and ethical issues, medical imaging data sets are usually small, which also increases the difficulty of training deep neural networks for organ segmentation. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the technical problems existing in the prior art, and to provide a multi-organ segmentation method and medium for a radiation particle intra-radiotherapy interventional surgery robot. Through the organ segmentation model, efficient multi-organ segmentation capability is achieved, and the segmentation result has the ability of secondary repair and editing.

[0005] The technical solution of the present invention includes a multi-organ segmentation method for a radiation particle intraradiotherapy interventional surgery robot, characterized in that the method includes: pre-segmentation and interactive optimization; the pre-segmentation includes: performing full-resolution training based on image blocks on a three-dimensional organ image through a V-Net cascade neural network; the interactive optimization includes: separating the organ target and background of the three-dimensional organ image according to the pre-segmentation training result and the input bounding box.

[0006] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the pre-segmentation includes: preprocessing, cropping all the data of the three-dimensional organ image to the non-zero area; data enhancement, performing data enhancement on all the data of the three-dimensional organ image as training data, and the data enhancement includes random rotation, random scaling, random elastic deformation, gamma correction enhancement and mirroring; training detail optimization, using the dice loss function and the cross entropy loss function as the final loss function to train the model; post-processing, performing connectivity analysis on the segmentation labels of all annotated data on the training data.

[0007] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the preprocessing includes: obtaining the median of the collective pixel spacing of the resampled three-dimensional organ image data, using third-order spline interpolation for image data, using nearest neighbor interpolation for corresponding segmentation masks, unifying the intensity values ​​appearing in the segmentation masks in the training data set, and limiting the intensity values ​​to [0.5%, 99.5%] of the intensity of the entire data set through a clip function, and performing normalization processing according to the mean and standard deviation of all intensity values.

[0008] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the training details optimization includes: using the Adam optimizer and setting the initial learning rate to 3×10 -4 , with 250 image blocks per training round, the image block size is set to 128×128×128, the batch size is 2, and during the training process, an exponential moving average is set for the training loss and the validation loss respectively. When the exponential moving average of the training loss has not improved in the past 30 rounds, the learning rate will be reduced by 5 times; when the exponential moving average of the validation loss has not improved in the past 60 rounds and the current learning rate is less than 10 -6 , the training is terminated early; and, in the samples of the same batch of training data, more than one-third of the samples include at least one randomly selected foreground class data.

[0009] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the post-processing includes: when any class corresponding to the three-dimensional organ image data set is only located in a single connected area in all cases, the largest connected area is retained and other connected areas of the corresponding class are automatically deleted.

[0010] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the interactive optimization includes: interactive setting, respectively marking the foreground and background of the under-segmented area and the over-segmented area in the segmentation result; algorithm execution, obtaining the pre-segmentation result of the deep learning method, marking interaction according to the specified organ target, obtaining the interactive mask according to the marking information and the pre-segmentation result of the specified organ target, and extracting the original data information of the interactive layer, and using Grabcut to optimize the corresponding area of ​​the three-dimensional organ image according to the input interactive mask and the original data of the interactive layer.

[0011] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the interactive setting also includes: dividing the image pixels into a determined background BG=0, a determined foreground FG=1, a possible background PR_BG=2 and a possible foreground PR_FG=3; setting the pre-segmentation result to an initial foreground information point FG=1 and an initial background information point BG=0, and when marking the background, by setting the size of the marking brush radius, setting the points around the marked point to the possible background PR_BG=2, and the center of the marking brush to the background point BG=0; and, when marking the foreground, by setting the size of the marking brush radius, setting the points around the marked point to the possible foreground PR_FG=3, and the center of the marking brush to the foreground point FG=1.

[0012] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, wherein the optimization of the corresponding area of ​​the three-dimensional organ image by using Grabcut also includes: the optimization method is layer by layer optimization along the specified axis.

[0013] According to the multi-organ segmentation method of the radiation particle intraradiotherapy interventional surgery robot, the V-Net cascade neural network includes: using the first V-Net neural network to train the downsampled image, and the obtained segmentation result is upsampled to the original image space size again, and used as additional information and the original image as the input of the second V-Net neural network to perform full-resolution training based on image blocks.

[0014] The technical solution of the present invention also includes a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, any of the above methods is implemented.

[0015] The beneficial effects of the present invention are: achieving efficient multi-organ segmentation capability and having the ability to perform secondary repair and editing of segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is further described below in conjunction with the accompanying drawings and embodiments;

[0017] Figure 1 Shown is a flow chart according to an embodiment of the present invention;

[0018] Figure 2 The figure is a flowchart of the overall framework of multi-organ segmentation according to an embodiment of the present invention;

[0019] Figure 3 It is a schematic diagram of the cascade structure of a pre-segmentation module V-Net based on a deep learning network according to an embodiment of the present invention;

[0020] Figure 4 The figure shows a user foreground / background interaction modification interface in a Grabcut-based interaction optimization module according to an embodiment of the present invention;

[0021] Figure 5 FIG. 1 is a schematic diagram of a Grabcut interactive mask according to an embodiment of the present invention;

[0022] Figure 6 Shown is a device medium diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] This section will describe in detail the specific embodiments of the present invention. The preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the accompanying drawings is to supplement the description of the text part of the specification with graphics, so that people can intuitively and vividly understand each technical feature and the overall technical solution of the present invention, but it cannot be understood as a limitation on the scope of protection of the present invention.

[0024] In the description of the present invention, “several” means one or more, “more” means more than two, “greater than”, “less than”, “exceed”, etc. are understood as not including the number itself, and “above”, “below”, “within”, etc. are understood as including the number itself.

[0025] In the description of the present invention, the consecutive numbering of the method steps is for the convenience of review and understanding. Combined with the overall technical scheme of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical scheme of the present invention.

[0026] In the description of the present invention, unless otherwise clearly defined, words such as "setting" should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above words in the present invention in combination with the specific content of the technical solution.

[0027] Figure 1 The flowchart shown is according to an embodiment of the present invention, including: pre-segmentation and interactive optimization; wherein the pre-segmentation includes: performing full-resolution training based on image blocks on the three-dimensional organ image through the V-Net cascade neural network; wherein the interactive optimization includes: separating the organ target and the background of the three-dimensional organ image according to the pre-segmentation training result and the input bounding box.

[0028] Figure 2 The flowchart of the overall framework of multi-organ segmentation according to an embodiment of the present invention is shown. It is summarized as follows:

[0029] Deep Learning Model Reference Figure 3 , deep learning-based pre-segmentation includes:

[0030] (1) Preprocessing: First, all data will be clipped to non-zero areas. In medical images, different imaging devices or different acquisition protocols result in datasets with anisotropic voxel spacing. In order to enable the network to correctly learn spatial semantics, all patients are resampled to obtain the median of the data set voxel spacing, where third-order spline interpolation is used for image data and nearest neighbor interpolation is used for the corresponding segmentation mask. Subsequently, the intensity values ​​appearing in the segmentation mask in the training dataset are unified, and the intensity values ​​are limited to the [0.5, 99.5] percentile of the intensity of the entire dataset through the clip function, and then normalized according to the mean and standard deviation of all intensity values.

[0031] (2) Data augmentation: During the training process, the following data augmentation techniques are performed: random rotation, random scaling, random elastic deformation, gamma correction augmentation, and mirroring.

[0032] (3) Training details: The dice loss function and the cross entropy loss function are used as the final loss function to train the model. The Adam optimizer is used, and the initial learning rate is set to 3×10^(-4). It is defined that 250 image blocks are trained as one round, the image block size is set to 128×128×128, and the batch size is 2. During the training process, an exponential moving average is set for the training loss and the validation loss respectively. When the exponential moving average of the training loss has not improved in the past 30 rounds, the learning rate will be reduced by 5 times; when the exponential moving average of the validation loss has not improved in the past 60 rounds and the current learning rate is less than 10^(-6), the training will be terminated early. In addition, in order to improve the stability of network training, it is required that more than one-third of the samples in a batch contain at least one randomly selected foreground class data.

[0033] (4) Post-processing: Connectivity analysis is performed on all ground truth segmentation labels on the training data. If a class is located in a single connected region in all cases, this behavior is considered a general property of the dataset. Therefore, all connected regions of this class except the largest one are automatically deleted.

[0034] Grabcut-based interactive optimization

[0035] The Grabcut algorithm is an image segmentation algorithm based on minimum segmentation. It requires the user to input a bounding box as the segmentation target position to separate / segment the target from the background. For the Grabcut algorithm, it divides the image pixels into 4 components: the determined background BG = 0, the determined foreground FG = 1, the possible background PR_BG = 2, and the possible foreground PR_FG = 3. In this solution, the foreground and background of the specified organs are marked through an interactive interface written based on C++ / VTK. The specific execution process can be divided into the following steps:

[0036] (1) Interaction process: reference Figure 4 , the user can mark the foreground and background of the under-segmented area and over-segmented area in the current segmentation result respectively. First, set the pre-segmentation result to the initial foreground information point FG=1 and the initial background information point BG=0. When marking the background, by setting the radius of the marking brush, the points around the marked point are set to the possible background PR_BG=2, and the center of the marking brush is set to the background point BG=0. Similarly, when marking the foreground, by setting the radius of the marking brush, the points around the marked point are set to the possible foreground PR_FG=3, and the center of the marking brush is set to the foreground point FG=1.

[0037] (2) Algorithm execution: Reference Figure 5 , obtain the pre-segmentation results of the deep learning method, and then the user will mark the interaction for the specified target. The algorithm will obtain the interaction mask based on the user's marking information and the pre-segmentation results of the specified target, and use the original data information of the extracted interaction layer to use Grabcut to perform regional optimization based on the input interaction mask and the original data of the interaction layer. Since medical image data is three-dimensional data, the optimization process is optimized layer by layer along the specified axis.

[0038] Figure 3The figure shows a schematic diagram of the V-Net cascade structure of the pre-segmentation module based on the deep learning network according to an embodiment of the present invention. In order to make up for the shortcomings of the 3D image block-based training method on large-scale organs, this scheme adopts a V-Net cascade structure. One V-Net is trained on the downsampled image, and the segmentation result obtained by the first V-Net is upsampled to the original image space size again, and used as additional information and the original image as the input of the second V-Net to perform a full-resolution training process based on the image block.

[0039] Figure 6 The device medium diagram according to an embodiment of the present invention is shown. The device includes a memory 100 and a processor 200, wherein the processor 200 stores a computer program, and the computer program is used to perform: pre-segmentation and interactive optimization; wherein the pre-segmentation includes: performing full resolution training based on image blocks on the three-dimensional organ image through the V-Net cascade neural network; wherein the interactive optimization includes: separating the organ target and the background of the three-dimensional organ image according to the pre-segmentation training result and the input bounding box. Among them, the memory 100 is used to store data.

[0040] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0041] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0042] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0043] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0044] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present invention.

Claims

1. A multi-organ segmentation method for a radiation particle intra-radiotherapy interventional surgery robot, characterized in that: The method includes: Pre-segmentation and interaction optimization; The pre-segmentation includes: performing full-resolution training based on image blocks on the three-dimensional organ image through a V-Net cascade neural network; The pre-segmentation further includes: pre-processing, cutting all data of the three-dimensional organ image to a non-zero area; Data enhancement, performing data enhancement on all data of the three-dimensional organ image as training data, wherein the data enhancement includes random rotation, random scaling, random elastic deformation, gamma correction enhancement and mirroring; Optimize training details, using dice loss function and cross entropy loss function as the final loss function to train the model; Post-processing: Connectivity analysis of segmentation labels of all annotated data on the training data; After the pre-segmentation, a pre-segmentation result is obtained; The interactive optimization includes: separating the organ target and the background of the three-dimensional organ image according to the pre-segmentation training result and the input bounding box; The interactive optimization also includes: Interactively set to mark the foreground and background of the under-segmented area and the over-segmented area in the pre-segmentation result respectively; The algorithm is executed to obtain the pre-segmentation results of the deep learning method, and the interaction is marked according to the specified organ target. The interaction mask is obtained according to the marking information and the pre-segmentation results of the specified organ target. The original data information of the extracted interaction layer is used to optimize the corresponding area of ​​the three-dimensional organ image according to the input interaction mask and the original data of the interaction layer using Grabcut; The interactive setting also includes: The image pixels are divided into determined background BG=0, determined foreground FG=1, possible background PR_BG=2 and possible foreground PR_FG=3; The pre-segmentation result is set as the initial foreground information point FG=1 and the initial background information point BG=0. When marking the background, the points around the marked point are set as the possible background PR_BG=2 by setting the radius of the marking brush, and the center of the marking brush is set as the background point BG=0; and when marking the foreground, the points around the marked point are set as the possible foreground PR_FG=3 by setting the radius of the marking brush, and the center of the marking brush is set as the foreground point FG=1; The optimization of the corresponding region of the three-dimensional organ image by using Grabcut also includes: The optimization method is to optimize layer by layer along the specified axis.

2. The method for segmenting multiple organs by a radioactive particle intraoperative radiotherapy interventional surgery robot according to claim 1, characterized in that: The pre-processing comprises: The median of the collective pixel spacing of the resampled three-dimensional organ image data is obtained, third-order spline interpolation is used for the image data, nearest neighbor interpolation is used for the corresponding segmentation mask, the intensity values ​​appearing in the segmentation mask in the training data set are unified, and the intensity values ​​are limited to [0.5%, 99.5%] of the intensity of the entire data set through the clip function, and normalized according to the mean and standard deviation of all intensity values.

3. The method for segmenting multiple organs by a radioactive particle intraoperative radiotherapy interventional surgery robot according to claim 1, characterized in that: The training detail optimization includes: Through the Adam optimizer, and set the initial learning rate to 3×10 -4 , with 250 image blocks per training round, the image block size is set to 128×128×128, the batch size is 2, and during the training process, an exponential moving average is set for the training loss and the validation loss respectively. When the exponential moving average of the training loss has not improved in the past 30 rounds, the learning rate will be reduced by 5 times; when the exponential moving average of the validation loss has not improved in the past 60 rounds and the current learning rate is less than 10 -6 , the training is terminated early; and, in the samples of the same batch of training data, more than one-third of the samples include at least one randomly selected foreground class data.

4. The method for segmenting multiple organs by a radioactive particle intraoperative radiotherapy interventional surgery robot according to claim 1, characterized in that: The post-processing includes: When any class corresponding to the three-dimensional organ image data set is located in only a single connected area in all cases, the largest connected area is retained, and other connected areas of the corresponding class are automatically deleted.

5. The method for segmenting multiple organs by a radioactive particle intraoperative radiotherapy interventional surgery robot according to claim 1, characterized in that: The V-Net cascade neural network includes: The downsampled image is trained using the first V-Net neural network, and the obtained pre-segmentation result is upsampled again to the original image space size and used as additional information and the original image as the input of the second V-Net neural network for full-resolution training based on image blocks.

6. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 5 when executed by a processor.

Images