Multi-scale attention and accumulation learning-based multi-task registration and segmentation method

Abstract

The invention discloses a multi-scale attention and accumulated learning-based multi-task registration and segmentation method, which realizes mutual promotion and enhancement between registration and segmentation tasks. According to the method, efficient feature coding between tasks is realized through a double-flow sharing encoder, a multi-scale attention module guides a model to pay attention to a challenging area, the accuracy of registration and segmentation is further improved through an accumulated learning strategy, multi-task registration and segmentation of medical images are efficiently realized, and the accuracy of registration and segmentation is improved. And the registration and segmentation precision superior to that of the current mainstream method is obtained. The method has important clinical application significance and reference value for medical image-assisted disease diagnosis, focus monitoring and surgical navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multi-task registration and segmentation method based on multi-scale attention and accumulation learning. Background Art

[0002] Lung cancer is one of the most serious diseases that threaten human life. The early clinical manifestations of lung cancer are isolated spherical nodules, namely lung nodules. In assisting doctors in the efficient diagnosis of lung nodules, accurate registration and segmentation methods play a vital role.

[0003] Medical image registration and segmentation are two of the most critical technologies in the field of medical image processing, and they are widely used in clinical practice. Among them, medical image registration technology is a key technology used in disease diagnosis, radiotherapy and surgical navigation by modeling a deformation field to distort the floating image and align it with the reference image. Medical image segmentation technology is used to segment specific organs or lesions and is widely used in intraoperative auxiliary positioning and lesion growth monitoring.

[0004] Although medical image registration and segmentation methods have made remarkable progress, they still face some challenges. For example, registration is usually modeled in an unsupervised form, which is essentially an ill-posed problem; and in the absence of prior information, the feasibility of registration will be greatly reduced. The segmentation of specific organs or lesions requires the supervision of segmentation labels, and the acquisition of segmentation labels requires a certain amount of time, effort and expertise; this is a costly process, especially for 3D medical images.

[0005] Registration and segmentation are also highly correlated and complementary: the deformation field generated by registration can provide deformation augmentation for the segmentation task, and the anatomical prior information obtained by segmentation can provide auxiliary supervision for registration. In addition, the two tasks are usually modeled as the same structure, both containing an encoder and a decoder. Therefore, applying multi-task learning to registration and segmentation can achieve mutual promotion and enhancement between the two tasks. However, registration usually requires splicing input images for feature encoding, while segmentation does not. The different encoding forms of registration and segmentation make it impossible to establish an efficient multi-task feature sharing mechanism between the two tasks. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and proposes a multi-task registration and segmentation method based on multi-scale attention and cumulative learning, which realizes the mutual promotion and enhancement between the registration and segmentation tasks, and obtains performance that is superior to the most advanced multi-task registration and segmentation methods. It has important clinical application significance and reference value for medical image-assisted disease diagnosis, lesion monitoring and surgical navigation.

[0007] To achieve the above object, the technical solution provided by the present invention is: a multi-task registration and segmentation method based on multi-scale attention and cumulative learning, comprising the following steps:

[0008] 1) Inputting the reference image and the floating image into a dual-stream shared encoder for feature extraction, and obtaining encoding feature maps of the reference image and the floating image respectively;

[0009] 2) Input the encoded feature maps of the reference image and the floating image into the multi-scale attention module to extract multi-scale anatomical attention;

[0010] 3) Share the multi-scale anatomical attention to the registration decoder and the segmentation decoder. The registration decoder outputs the deformation field from the floating image to the reference image, and the segmentation decoder outputs the segmentation results of the reference image and the floating image.

[0011] 4) The registration process consists of the similarity loss between the reference image and the deformed image Regularization loss of deformation field The segmentation process is jointly supervised by the segmentation results of the reference image and the floating image and the segmentation loss of the corresponding segmentation labels. Supervision is performed; among them, consistency loss is adopted Ensure the consistency of registration and segmentation in anatomical structure.

[0012] Further, in step 1), the dual-stream shared encoder is constructed with Swin Transformer as the backbone network; the dual-stream shared encoder is defined as DSSE (,), and the encoded feature maps of the reference image and the floating image extracted by it are expressed as:

[0013] F1,F2,…,F 10 =DSSE(f,m)

[0014] Where F1, F2, ..., F5 represent the encoded feature maps of different resolution levels extracted from the reference image, F6, F7, ..., F 10 Represents the encoded feature maps at different resolution levels extracted from the floating image; f is the reference image and m is the floating image.

[0015] Further, in step 2), the multi-scale attention module is defined as MSAM(,), and the multi-scale anatomical attention extracted by it is expressed as:

[0016]

[0017] In the formula, is a multi-scale anatomical attention map for segmentation decoding, A multi-scale anatomical attention map for registration decoding; represents the segmentation decoding feature map from the lower resolution level, F i Represents the encoded feature map from the current resolution level.

[0018] Further, in step 3), the registration decoder and the segmentation decoder both apply the accumulation learning strategy, and are constructed with convolution as the backbone network; the decoding process of the registration decoder applying the accumulation learning strategy is expressed as follows:

[0019]

[0020] Where CONV(·) represents the convolution operation, RH(·) is the registration head, and UPS(·) represents the upsampling operation; i Decode the feature map for registration at the current resolution level, is the registered decoded feature map from the next lower resolution level; φ i and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level; Multi-scale anatomical attention map for floating images, Represents a deformation operation;

[0021] The decoding process of the segmentation decoder using the accumulation learning strategy is as follows:

[0022]

[0023] In the formula, SH(·) is the segmentation head, s i Decode the feature map for the segmentation at the current resolution level, Decode feature maps for segmentation from one lower resolution level; and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level.

[0024] Further, in step 4), the similarity loss between the reference image and the deformed image is Specifically expressed as:

[0025]

[0026] In the formula, MSE(,) is the mean square error operator; is the deformed image, which is generated by deforming the floating image m through the deformation field φ; Ω is the three-dimensional space where the reference image and the floating image are located, p is the pixel in Ω, p∈Ω;

[0027] The regularization loss of the deformation field Specifically expressed as:

[0028]

[0029] In the formula, is the gradient operator, ||·|| 2 is an L2 norm operator, and the deformation field φ is defined as the sum of the identity transformation I and the displacement field u, that is, φ = I + u; therefore, by calculating the pixel displacement gradient in the displacement field u The L2 norm of is used to regularize the deformation field φ;

[0030] The segmentation loss Specifically expressed as:

[0031]

[0032] In the formula, f l and m l are the segmentation labels of the reference image and the floating image, respectively, and f s and m s The segmentation results of the reference image and the floating image respectively; represents the Dice similarity loss function, represents the binary cross entropy loss function;

[0033] The consistency loss Specifically expressed as:

[0034]

[0035] In the formula, is the floating image segmentation result after deformation, which is the floating image segmentation result m s Generated after deformation by deformation field φ.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] 1. The present invention adopts a multi-task form to realize the joint learning of registration and segmentation, making full use of the correlation and complementarity between the registration and segmentation tasks to achieve mutual promotion and enhancement between the two tasks.

[0038] 2. The present invention realizes efficient feature sharing between registration and segmentation tasks. Through the proposed dual-stream shared encoder, it breaks the barriers of different coding forms of registration and segmentation, and adopts a unified feature coding form to realize efficient feature sharing between the two tasks.

[0039] 3. The present invention performs well in challenging registration and segmentation areas. The proposed multi-scale attention module extracts multi-scale anatomical attention and shares it with the registration decoder and segmentation decoder, guiding the model to focus on challenging registration and segmentation areas and achieving higher registration and segmentation accuracy.

[0040] 4. The present invention applies the accumulation learning strategy to the registration and segmentation decoders, optimizes the registration process by realizing coarse-to-fine registration, and optimizes the segmentation process by fusing multi-scale segmentation maps, further improving the accuracy of registration and segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the present invention.

[0042] Figure 2 The figure shows the visualized 3D registration results of the method of the present invention and the comparative method.

[0043] Figure 3 This is a visualized 3D segmentation result diagram of the method of the present invention and the comparative method. DETAILED DESCRIPTION

[0044] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0045] like Figure 1 As shown, this embodiment discloses a multi-task registration and segmentation method based on multi-scale attention and accumulation learning, which realizes mutual promotion and enhancement between tasks through multi-task learning. The specific implementation of this method includes the following steps:

[0046] 1) Inputting the reference image and the floating image into a dual-stream shared encoder for feature extraction, and obtaining encoding feature maps of the reference image and the floating image respectively;

[0047] The dual-stream shared encoder is constructed with Swin Transformer as the backbone network; the dual-stream shared encoder is defined as DSSE (,), and the encoded feature maps of the reference image and the floating image extracted by it are expressed as:

[0048] F1,F2,…,F 10 =DSSE(f,m)

[0049] Where F1, F2, ..., F5 represent the encoded feature maps of different resolution levels extracted from the reference image, F6, F7, ..., F 10 Represents the encoded feature maps at different resolution levels extracted from the floating image; f is the reference image and m is the floating image.

[0050] 2) Input the encoded feature maps of the reference image and the floating image into the multi-scale attention module to extract multi-scale anatomical attention;

[0051] The multi-scale attention module is defined as MSAM(,), and the multi-scale anatomical attention extracted by it is expressed as:

[0052]

[0053] In the formula, is a multi-scale anatomical attention map for segmentation decoding, A multi-scale anatomical attention map for registration decoding; represents the segmentation decoding feature map from the lower resolution level, F i Represents the encoded feature map from the current resolution level.

[0054] 3) Share the multi-scale anatomical attention to the registration decoder and the segmentation decoder. The registration decoder outputs the deformation field from the floating image to the reference image, and the segmentation decoder outputs the segmentation results of the reference image and the floating image.

[0055] The registration decoder and the segmentation decoder both apply the accumulation learning strategy and are constructed with convolution as the backbone network; the decoding process of the registration decoder applying the accumulation learning strategy is expressed as follows:

[0056]

[0057] Where CONV(·) represents the convolution operation, RH(·) is the registration head, and UPS(·) represents the upsampling operation; i Decode the feature map for registration at the current resolution level, is the registered decoded feature map from the next lower resolution level; φ i and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level; Multi-scale anatomical attention map for floating images, Represents a deformation operation;

[0058] The decoding process of the segmentation decoder using the accumulation learning strategy is as follows:

[0059]

[0060] In the formula, SH(·) is the segmentation head, s i Decode the feature map for the segmentation at the current resolution level, Decode feature maps for segmentation from one lower resolution level; and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level.

[0061] 4) The registration process consists of the similarity loss between the reference image and the deformed image Regularization loss of deformation field The segmentation process is jointly supervised by the segmentation results of the reference image and the floating image and the segmentation loss of the corresponding segmentation labels. Supervision is performed; among them, consistency loss is adopted Ensure the consistency of registration and segmentation in anatomical structure;

[0062] The similarity loss between the reference image and the deformed image Specifically expressed as:

[0063]

[0064] In the formula, MSE(,) is the mean square error operator; is the deformed image, which is generated by deforming the floating image m through the deformation field φ; Ω is the three-dimensional space where the reference image and the floating image are located, p is the pixel in Ω, p∈Ω;

[0065] The regularization loss of the deformation field Specifically expressed as:

[0066]

[0067] In the formula, is the gradient operator, ||·|| 2 is an L2 norm operator, and the deformation field φ is defined as the sum of the identity transformation I and the displacement field u, that is, φ = I + u; therefore, by calculating the pixel displacement gradient in the displacement field u The L2 norm of is used to regularize the deformation field φ;

[0068] The segmentation loss Specifically expressed as:

[0069]

[0070] In the formula, f l and m l are the segmentation labels of the reference image and the floating image, respectively, and f s and m s The segmentation results of the reference image and the floating image respectively; represents the Dice similarity loss function, represents the binary cross entropy loss function;

[0071] The consistency loss Specifically expressed as:

[0072]

[0073] In the formula, is the floating image segmentation result after deformation, which is the floating image segmentation result m s Generated after deformation by deformation field φ.

[0074] The effect of the multi-task registration and segmentation method based on multi-scale attention and accumulation learning in this embodiment will be further illustrated by the following experiments:

[0075] 1. Experimental conditions:

[0076] Experimental environment: The computing platform of this experiment is NVIDIA Tesla V100 (32GB) GPU and Intel Xeon Processor (Skylake, IBRS) CPU. PyTorch is used to build a deep learning framework and train neural networks.

[0077] Experimental data: This example uses 420 chest follow-up CT scans from 210 high-risk lung cancer patients as experimental data. The dataset includes not only CT images, but also corresponding lung parenchyma segmentation labels and registration key point data.

[0078] Evaluation indicators: The evaluation indicators of the registration performance of the method are the target registration error TRE (mm) and the lung parenchyma Dyess similarity coefficient Lung DSC, and the evaluation indicators of the segmentation performance are the intersection over union ratio IoU and the average surface distance ASD (mm). Among them, the target registration error TRE (mm) calculates the average Euclidean distance between key points and evaluates the accuracy of global image registration. The smaller the value, the higher the registration accuracy; the lung parenchyma Dyess similarity coefficient Lung DSC measures the overlap of the lung parenchyma. The larger the value, the higher the overlap of the lung parenchyma; the intersection over union ratio IoU calculates the ratio of the intersection to the union between the segmentation result and the segmentation label. The larger the value, the higher the overlap between the two; the average surface distance ASD (mm) quantifies the distance between the segmentation result and the segmentation label boundary. The smaller the value, the higher the overlap between the boundaries of the two.

[0079] Comparison method: In order to verify the effectiveness of the registration and segmentation of this method, the experiment selected several of the most advanced methods for comparative experiments, including three registration methods (VoxelMorph-diff, TransMorph-diff, LapIRN), three segmentation methods (3D U-net, UNet++, nnU-Net), and two multi-task registration and segmentation methods (Deep Atlas, Cross-stitch).

[0080] 2. Experimental results:

[0081] The quantitative results of the registration and segmentation indicators of the method of the present invention and the comparative method are shown in Table 1 below:

[0082] Table 1: Quantitative results of registration and segmentation indicators of different methods (average value)

[0083]

[0084]

[0085] It can be seen from Table 1 that the TRE (mm) value obtained by the method of the present invention is the smallest, and the Lung DSC value is the largest, which proves that the method of the present invention has the highest registration accuracy compared with the comparison method; moreover, the IoU value obtained by the present invention is the largest, and the ASD (mm) value is only larger than that of nnU-Net, but smaller than that of other methods, which proves that the method of the present invention has excellent segmentation accuracy.

[0086] The visual 3D registration results of the method of the present invention and the comparative method are shown in Figure 2 As shown in the figure, the 3D superposition of the deformable image and the reference image lung parenchyma label generated by the registration of each method is specifically shown. The deformable image lung parenchyma label in each sub-image comes from: (a) floating image segmentation label, (b) VoxelMorph-diff, (c) TransMorph-diff, (d) LapIRN, (e) Deep Atlas, (f) Cross-stitch, and (g) the method of the present invention. Figure 2 It can be found that before registration, there is a large difference between the lung parenchyma labels of the floating image and the reference image, especially the area near the diaphragm in the lower right corner has a large deformation. Compared with the comparison method, the method of the present invention achieves the highest registration accuracy, especially for areas with large deformation.

[0087] The visualized 3D segmentation results of the method of the present invention and the comparative method are shown in FIG. Figure 3 As shown in the figure, the 3D superposition of the segmentation results and segmentation labels of each method is specifically shown. The segmentation results in each sub-figure come from: (a) 3D U-net, (b) UNet++, (c) nnU-Ne, (d) Deep Atlas, (e) Cross-stitch, and (f) the method of the present invention. Figure 3 It can be found that, compared with the comparison method, the method of the present invention obtains the segmentation result with the highest accuracy, especially in the upper left area of ​​the edge of the lung parenchyma.

[0088] The above experimental results show that the present invention can effectively realize multi-task registration and segmentation of medical images, and achieve the highest registration and segmentation accuracy. The present invention realizes efficient feature encoding between tasks through a dual-stream shared encoder, guides the model to focus on challenging areas through a multi-scale attention module, and further improves the accuracy of registration and segmentation through an accumulation learning strategy, realizing mutual promotion and enhancement between registration and segmentation tasks, which has important reference value for practical clinical applications and is worthy of promotion.

[0089] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task registration and segmentation method based on multi-scale attention and accumulation learning, characterized by: The following steps are involved: 1) Inputting the reference image and the floating image into a dual-stream shared encoder for feature extraction, and obtaining encoding feature maps of the reference image and the floating image respectively; 2) Input the encoded feature maps of the reference image and the floating image into the multi-scale attention module to extract multi-scale anatomical attention; 3) Share the multi-scale anatomical attention to the registration decoder and the segmentation decoder. The registration decoder outputs the deformation field from the floating image to the reference image, and the segmentation decoder outputs the segmentation results of the reference image and the floating image. 4) The registration process consists of the similarity loss between the reference image and the deformed image Regularization loss of deformation field The segmentation process is jointly supervised by the segmentation results of the reference image and the floating image and the segmentation loss of the corresponding segmentation labels. Supervision is performed; among them, consistency loss is adopted Ensure the consistency of registration and segmentation in anatomical structure.

2. The multi-task registration and segmentation method based on multi-scale attention and accumulation learning according to claim 1, characterized in that: In step 1), the dual-stream shared encoder is constructed with Swin Transformer as the backbone network; the dual-stream shared encoder is defined as DSSE (,), and the encoded feature maps of the reference image and the floating image extracted by it are expressed as: F1,F2,…,F 10 =DSSE(f,m) Where F1, F2, ..., F5 represent the encoded feature maps of different resolution levels extracted from the reference image, F6, F7, ..., F 10 Represents the encoded feature maps at different resolution levels extracted from the floating image; f is the reference image and m is the floating image.

3. The multi-task registration and segmentation method based on multi-scale attention and accumulation learning according to claim 2, characterized in that: In step 2), the multi-scale attention module is defined as MSAM(,), and the multi-scale anatomical attention extracted by it is expressed as: In the formula, is a multi-scale anatomical attention map for segmentation decoding, A multi-scale anatomical attention map for registration decoding; represents the segmentation decoding feature map from the lower resolution level, F i Represents the encoded feature map from the current resolution level.

4. The multi-task registration and segmentation method based on multi-scale attention and accumulation learning according to claim 3, characterized in that: In step 3), the registration decoder and the segmentation decoder both apply the accumulation learning strategy, and are constructed with convolution as the backbone network; the decoding process of the registration decoder applying the accumulation learning strategy is expressed as follows: Where CONV(·) represents the convolution operation, RH(·) is the registration head, and UPS(·) represents the upsampling operation; i Decode the feature map for registration at the current resolution level, is the registered decoded feature map from the next lower resolution level; φ i and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level; is the multi-scale anatomical attention map of the floating image, ο represents the deformation operation; The decoding process of the segmentation decoder using the accumulation learning strategy is as follows: In the formula, SH(·) is the segmentation head, s i Decode the feature map for the segmentation at the current resolution level, Decode feature maps for segmentation from one lower resolution level; and are the deformation fields of the current resolution level before and after upsampling, is the deformation field from a lower resolution level.

5. The multi-task registration and segmentation method based on multi-scale attention and accumulation learning according to claim 4, characterized in that: In step 4), the similarity loss between the reference image and the deformed image is Specifically expressed as: In the formula, MSE(,) is the mean square error operator; is the deformed image, which is generated by deforming the floating image m through the deformation field φ; Ω is the three-dimensional space where the reference image and the floating image are located, p is the pixel in Ω, p∈Ω; The regularization loss of the deformation field Specifically expressed as: In the formula, is the gradient operator, ||·|| 2 is an L2 norm operator, and the deformation field φ is defined as the sum of the identity transformation I and the displacement field u, that is, φ = I + u; therefore, by calculating the pixel displacement gradient in the displacement field u The L2 norm of is used to regularize the deformation field φ; The segmentation loss Specifically expressed as: In the formula, f l and m l are the segmentation labels of the reference image and the floating image, respectively, and f s and m s They are the segmentation results of the reference image and the floating image respectively; represents the Dice similarity loss function, represents the binary cross entropy loss function; The consistency loss Specifically expressed as: In the formula, is the floating image segmentation result after deformation, which is the floating image segmentation result m s Generated after deformation by deformation field φ.

Images