Multi-phase fusion organ segmentation method and device based on non-local attention mechanism

Abstract

The present application discloses a method and device for organ segmentation based on a multi-phase fusion non-local attention mechanism. The method performs image segmentation on a global-scale multi-phase data pair and a local-scale multi-phase data pair respectively by presetting a multi-phase fusion organ segmentation network model to obtain a global-scale segmentation result and a local-scale segmentation result, and fuses the global-scale segmentation result and the local-scale segmentation result to obtain a segmentation result of the organ to be segmented. In the present application, a cross-phase non-local attention fusion mechanism module in a preset multi-phase fusion organ segmentation network model is used to complement the features of the two phases according to the positional relationship and depth relationship between the features of the two phases, and the complementarity of the two phases is fully utilized so that the features of the two phases can be effectively fused. A multi-scale segmentation framework is used to fuse the global-scale segmentation result and the local-scale segmentation result, effectively utilizing global context information to improve the accuracy of organ segmentation.

Description

Technical Field

[0001] The present application relates to the technical field of multi-phase organ segmentation, and in particular to a multi-phase fusion organ segmentation method and device based on a non-local attention mechanism. Background Art

[0002] In the field of medical image segmentation, due to the limited capabilities of existing CT imaging technology, single-phase CT scans often have difficulty accurately locating the contours of organs. Different phases can emphasize different details of organ boundaries. Therefore, referring to image data from different phases is an effective strategy to identify organ boundaries as completely as possible. For example, many guidelines explicitly recommend the use of dual-phase CT contrast-enhanced imaging for the pancreas, including arterial and venous phases. Image data from both phases are important in clinical diagnosis, especially for cancer. Image data from the arterial phase helps to detect tumors, and image data from the venous phase helps to show invasion of surrounding structures. Dual-phase image data provides a reliable imaging basis for clinical treatment. Therefore, multi-phase fusion segmentation methods have been used at home and abroad for automatic organ segmentation.

[0003] The existing multi-phase fusion segmentation method has the following two disadvantages, which leads to low accuracy of organ segmentation:

[0004] (1) The input of existing multi-phase fusion segmentation methods is either a 3D patch of a medical image or three adjacent slices. However, neither 3D patches nor slices can capture the powerful three-dimensional characteristics of medical images, nor can they effectively utilize global context information.

[0005] (2) Due to the inconsistency between multiple periods of image data, the multiple periods of image data cannot be effectively fused. Summary of the invention

[0006] The present application provides a multi-phase fusion organ segmentation method and device based on a non-local attention mechanism to improve the accuracy of organ segmentation.

[0007] In a first aspect, the present application provides a multi-phase fusion organ segmentation method based on a non-local attention mechanism, comprising:

[0008] Extract the centroid of each multi-phase image data to be segmented respectively to obtain the centroid of the organ to be segmented;

[0009] Cutting each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented;

[0010] Performing deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs;

[0011] The registered multi-phase data pairs are downsampled to obtain global-scale multi-phase data pairs, and the registered multi-phase data pairs are sliced ​​and segmented to obtain local-scale multi-phase data pairs. The global-scale multi-phase data pairs are respectively feature extracted through two training paths of the preset multi-phase fusion organ segmentation network model to obtain global features of the two phases. The global features of the two phases are complemented according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result. The local-scale multi-phase data pairs are respectively feature extracted through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases. The local features of the two phases are complemented according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result. The global-scale segmentation result and the local-scale segmentation result are weightedly fused to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network.

[0012] Optionally, the step of extracting the centroid of each of the multi-phase image data to be segmented to obtain the centroid of the organ to be segmented includes:

[0013] According to the preset organ segmentation model, each of the multi-phase image data to be segmented is subjected to organ segmentation to obtain an initial segmentation result;

[0014] The centroid of the organ to be segmented is determined according to the initial segmentation result.

[0015] Optionally, the step of cutting each multi-phase image data to be segmented according to the centroid to obtain multi-phase data pairs includes:

[0016] According to the centroid, upper and lower N slices of the organ to be segmented are extracted from each multi-phase image to be segmented, and two phase data containing all slices of the organ to be segmented are obtained as a multi-phase data pair, wherein N is a positive integer.

[0017] Optionally, the step of complementing the global features of the two phases according to the position relationship and the depth relationship between the global features of the two phases to obtain a global scale segmentation result includes:

[0018] According to each position feature of one phase in the global features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the deep features in the global features of the two phases are complemented to obtain a global-scale segmentation result.

[0019] Optionally, the step of complementing the local features of the two phases according to the position relationship and the depth relationship between the local features of the two phases to obtain a local scale segmentation result includes:

[0020] According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the depth features in the local features of the two phases are complemented to obtain a local scale segmentation result.

[0021] Optionally, the backbone network of the preset multi-phase fusion organ segmentation network model consists of a 3D encoder and a 2D decoder.

[0022] Optionally, the multi-phase image data to be segmented are multi-phase image data of CT or multi-modal data of MRI.

[0023] In a second aspect, the present application provides a multi-phase fusion organ segmentation device based on a non-local attention mechanism, comprising:

[0024] A centroid extraction module is used to extract the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented;

[0025] A cutting module, used for cutting each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented;

[0026] A registration module, used for performing deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs;

[0027] A fusion module is used to downsample the registered multi-phase data pairs to obtain global-scale multi-phase data pairs, slice and segment the registered multi-phase data pairs to obtain local-scale multi-phase data pairs, extract features from the global-scale multi-phase data pairs through two training paths of a preset multi-phase fusion organ segmentation network model to obtain global features of the two phases, complement the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result, extract features from the local-scale multi-phase data pairs through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases, complement the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result, and perform weighted fusion of the global-scale segmentation result and the local-scale segmentation result to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network.

[0028] Optionally, the centroid extraction module includes:

[0029] The segmentation submodule is used to perform organ segmentation on each of the multi-phase image data to be segmented according to a preset organ segmentation model to obtain an initial segmentation result;

[0030] A determination submodule is used to determine the centroid of the organ to be segmented according to the initial segmentation result.

[0031] Optionally, the cutting module is specifically used for:

[0032] According to the centroid, upper and lower N slices of the organ to be segmented are extracted from each multi-phase image to be segmented, and two phase data containing all slices of the organ to be segmented are obtained as a multi-phase data pair, wherein N is a positive integer.

[0033] Optionally, the fusion module is specifically used to:

[0034] According to each position feature of one phase in the global features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the deep features in the global features of the two phases are complemented to obtain a global-scale segmentation result.

[0035] Optionally, the fusion module is specifically used to:

[0036] According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the depth features in the local features of the two phases are complemented to obtain a local scale segmentation result.

[0037] Optionally, the backbone network of the preset multi-phase fusion organ segmentation network model consists of a 3D encoder and a 2D decoder.

[0038] Optionally, the multi-phase image data to be segmented are multi-phase image data of CT or multi-modal data of MRI.

[0039] In a third aspect, the present application provides a readable medium comprising execution instructions. When a processor of an electronic device executes the execution instructions, the electronic device executes any method described in the first aspect.

[0040] In a fourth aspect, the present application provides an electronic device, comprising a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor executes any method described in the first aspect.

[0041] It can be seen from the above technical scheme that the present application provides a multi-phase fusion organ segmentation method based on a non-local attention mechanism. In the present method, the centroid of each multi-phase image data to be segmented is extracted to obtain the centroid of the organ to be segmented, and each multi-phase image data to be segmented is cut according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all the slices of the organ to be segmented, and the multi-phase data pairs are deformably aligned to obtain the aligned multi-phase data pairs, and the aligned multi-phase data pairs are downsampled to obtain the global-scale multi-phase data pairs, and the aligned multi-phase data pairs are sliced ​​to obtain the local-scale multi-phase data pairs, and two training paths of the preset multi-phase fusion organ segmentation network model are used. The paths perform feature extraction on the global-scale multi-phase data pairs respectively to obtain the global features of the two phases, and complement the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain the global-scale segmentation result; the two training paths of the preset multi-phase fusion organ segmentation network model perform feature extraction on the local-scale multi-phase data pairs respectively to obtain the local features of the two phases, and complement the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain the local-scale segmentation result; the global-scale segmentation result and the local-scale segmentation result are weightedly fused to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network. In the technical solution of the present application, the cross-phase non-local attention fusion mechanism module in the preset multi-phase fusion organ segmentation network model complements the features of the two phases according to the positional relationship and depth relationship between the features of the two phases, suppresses the influence of the two phases not strictly aligned with the features, ensures the consistency between the features of the two phases, and makes full use of the complementarity of the two phases so that the features of the two phases can be effectively fused. In addition, a multi-scale segmentation framework is used to fuse the global scale segmentation results and the local scale segmentation results, which can effectively utilize the global context information and improve the accuracy of organ segmentation.

[0042] The further effects of the above-mentioned non-conventional preferred manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1It is a flowchart of a multi-phase fusion organ segmentation method based on a non-local attention mechanism in an embodiment of the present application;

[0045] Figure 2 This is a flowchart of the multi-phase fusion organ segmentation method based on the non-local attention mechanism proposed in this application;

[0046] Figure 3 This is a schematic diagram of the structure of a multi-phase fusion organ segmentation device based on a non-local attention mechanism in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0049] In order to solve the problem in the prior art that global context information cannot be effectively utilized and that multi-phase image data cannot be effectively fused due to inconsistencies between the multi-phase image data, resulting in low accuracy of organ segmentation.

[0050] The present application provides a multi-phase fusion organ segmentation method based on a non-local attention mechanism. In the method, the centroid of each multi-phase image data to be segmented is extracted to obtain the centroid of the organ to be segmented, and each multi-phase image data to be segmented is cut according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented, and the multi-phase data pairs are deformably registered to obtain the registered multi-phase data pairs, and the registered multi-phase data pairs are downsampled to obtain global-scale multi-phase data pairs, and the registered multi-phase data pairs are sliced ​​to obtain local-scale multi-phase data pairs, and the global scale is respectively registered by two training paths of the preset multi-phase fusion organ segmentation network model. The local-scale segmentation result is obtained by performing feature extraction on the local-scale multi-phase data pairs, and the global features of the two phases are obtained. The global features of the two phases are complemented according to the positional relationship and depth relationship between the global features of the two phases to obtain the global-scale segmentation result. The local-scale segmentation result and the local-scale segmentation result are weightedly fused, and the segmentation result of the organ to be segmented is obtained. The preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network. In the technical solution of the present application, the cross-phase non-local attention fusion mechanism module in the preset multi-phase fusion organ segmentation network model complements the features of the two phases according to the positional relationship and depth relationship between the features of the two phases, suppresses the influence of the two phases not strictly aligned with the features, ensures the consistency between the features of the two phases, and makes full use of the complementarity of the two phases so that the features of the two phases can be effectively fused. In addition, a multi-scale segmentation framework is used to fuse the global scale segmentation results and the local scale segmentation results, which can effectively utilize the global context information and improve the accuracy of organ segmentation.

[0051] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0052] See also Figure 1 , shows a multi-phase fusion organ segmentation method based on a non-local attention mechanism in an embodiment of the present application. In this embodiment, the method is applied to an electronic device, and the method may include the following steps:

[0053] S101: Extracting the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented.

[0054] In order to address the two shortcomings of the prior art in the background technology, this application uses the power of deep learning to develop a scientific, accurate, and clinically meaningful multi-phase fusion organ segmentation model. The multi-phase fusion organ segmentation model is an end-to-end multi-phase fusion organ segmentation model based on a non-local attention mechanism, which can effectively combine the network characteristics of 2D and 3D and can effectively deal with the inconsistency between multi-phase image data.

[0055] In view of the shortcoming of the existing technology that global context information cannot be effectively utilized, this application adopts a multi-scale segmentation framework, designs a global scale model and a local scale model to perform targeted learning of global information and local information, and finally fuses the results of the two networks as the final segmentation result, so as to achieve the purpose of effectively utilizing global context information.

[0056] In view of the shortcoming in the prior art that there are inconsistencies between multi-period image data, which leads to the inability to effectively fuse multi-period image data, this application designs a cross-period non-local attention fusion mechanism module to complement the features of multi-period images, which can suppress non-calibrated areas and improve the robustness of the multi-period fusion organ segmentation model. It is introduced in detail below.

[0057] Since the spatial resolution of the original multi-phase image data to be segmented is 512×512, the number of slices is different. For example, in abdominal scanning, there is a difference in the number of scanning layers between the arterial phase image data and the venous phase image data. The scanning range of the arterial phase image data is mostly the abdomen, and the scanning range of the venous phase image data is mostly the abdomen and pelvis. This causes scanning differences between image data of different phases.

[0058] Due to the difference in image data scanning at different phases, it is extremely difficult to directly align the multi-phase image data. Therefore, in order to reduce the difficulty of alignment caused by the difference in scanning, before alignment, this application proposes a centroid extraction strategy, that is, to extract the centroid of each multi-phase image data to be segmented, and obtain the centroid of the organ to be segmented, and then align based on the centroid, which can reduce the difficulty of alignment. Figure 2 This is a flowchart of the multi-phase fusion organ segmentation method based on the non-local attention mechanism proposed in this application, see Figure 2 , Figure 2 The multi-phase images in are the multi-phase image data to be segmented.

[0059] Wherein, step S101 may include:

[0060] According to the preset organ segmentation model, each of the multi-phase image data to be segmented is subjected to organ segmentation to obtain an initial segmentation result;

[0061] Determine the centroid of the organ to be segmented based on the initial segmentation results.

[0062] Exemplarily, the multi-phase image data to be segmented may be multi-phase image data of CT (computed tomography) or multi-modal data of MRI (Magnetic Resonance Imaging).

[0063] Continue to see Figure 2 , use the pre-trained preset organ segmentation model to perform organ segmentation on each multi-phase image data to be segmented, and obtain an initial segmentation result, that is, segment the organ to be segmented from each multi-phase image data to be segmented, and then determine the centroid of the organ to be segmented according to the initial segmentation result, wherein the preset organ segmentation model can be any organ segmentation model in the prior art, and the embodiment of the present invention does not make any limitation to this, and the preset organ segmentation model is Figure 2 Organ pre-segmentation network in .

[0064] S102: Cutting each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented.

[0065] After obtaining the centroid, each multi-phase image data to be segmented is cut according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all the slices of the organ to be segmented, thereby ensuring that all the cut data contain all the slices of the organ to be segmented.

[0066] The above step S102 may be: extracting upper and lower N slices of the organ to be segmented from each multi-phase image to be segmented according to the centroid, and obtaining two phase data containing all slices of the organ to be segmented as a multi-phase data pair, wherein N is a positive integer.

[0067] Take N slices from top to bottom according to the centroid. The purpose of this is to ensure the consistency of data from different phases at the same level. Thus, two phases of data containing all slices of the organ to be segmented are obtained as multi-phase data pairs, where N is a positive integer. Figure 2 , the two phase data can be Figure 2 Phase one and phase two in the period.

[0068] S103: Perform deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs.

[0069] After obtaining the multi-phase data pairs, the multi-phase data pairs can be deformably registered to obtain the registered multi-phase data pairs. Figure 2 The deformable registration method may be any deformable registration method in the prior art, and the embodiment of the present invention does not impose any limitation on this.

[0070] S104: down-sampling the registered multi-phase data pairs to obtain global-scale multi-phase data pairs, slicing and segmenting the registered multi-phase data pairs to obtain local-scale multi-phase data pairs, extracting features from the global-scale multi-phase data pairs through two training paths of a preset multi-phase fusion organ segmentation network model to obtain global features of the two phases, complementing the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result, extracting features from the local-scale multi-phase data pairs through two training paths of a preset multi-phase fusion organ segmentation network model to obtain local features of the two phases, complementing the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result, performing weighted fusion on the global-scale segmentation result and the local-scale segmentation result to obtain a segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network.

[0071] In an embodiment of the present invention, a multi-phase fusion organ segmentation network model is proposed, that is, a preset multi-phase fusion organ segmentation network model obtained through pre-training.

[0072] Continue to see Figure 2 , the preset multi-phase fusion organ segmentation network model is a dual-path full convolutional network, that is, Figure 2 The multi-phase fusion organ segmentation network in the preset multi-phase fusion organ segmentation network model contains two paths. Figure 2 The upper layer in the multi-phase fusion organ segmentation network is one path, and the lower layer is another path. Each path inputs the data of one phase in the multi-phase data pair. That is to say, the data input by the preset multi-phase fusion organ segmentation network model is paired, that is, the multi-phase data comes from the same patient.

[0073] The network in each path is modeled based on the encoder-decoder architecture, and central pruning is performed in the skip-layer connection between the encoder and decoder to remove redundant features of each block of the encoder. Skip-layer connections between features of two paths with the same resolution are also added to facilitate information exchange between data of different phases. Figure 2 In the figure, the dotted path and the solid path represent the training paths of different phases, and the arrows represent the information transfer between the data of different phases. The backbone network of the preset multi-phase fusion organ segmentation network model consists of a 3D encoder and a 2D decoder. This not only effectively utilizes the powerful 3D characteristics of medical images but also reduces the parameters of the network.

[0074] During the training process of the preset multi-phase fusion organ segmentation network model, multi-scale supervision will be performed to strengthen the training of the encoder. Among them, the multi-scale supervision adopts a multi-scale segmentation framework, which includes a global scale model that obtains global context features, that is, global information, and a local scale model that obtains local details, that is, local information.

[0075] Since strict alignment between multi-phase image data, that is, consistency between multi-phase image data, is crucial for accurate organ segmentation, the prior art methods cannot effectively integrate multi-phase data due to the loose alignment between image data of different phases. To solve this problem, the embodiment of the present invention develops a cross-phase non-local attention mechanism module to capture the dependencies between the features of multiple phases and further calibrate the features of multiple phases.

[0076] Continue to see Figure 2 The preset multi-period fusion organ segmentation network model includes an inter-period non-local attention mechanism module, and the inter-period non-local attention mechanism module consists of three parts: a position attention module, a depth attention module and a feature fusion module.

[0077] Among them, the position attention module: For most multi-phase segmentation tasks, a pair of non-strictly aligned images, i.e., the registered multi-phase data pair, is obtained through the existing registration algorithm. This pair of images is highly correlated in the corresponding neighborhood. Therefore, the relevant neighborhood features of one phase are used as a guide to complement the feature representation of each position of another phase. The purpose is to explore the position correlation of different phases and improve the robustness to non-aligned features.

[0078] Deep Attention Module: In the feature extraction stage of different paths, it is difficult to maintain semantic consistency in the features of multi-phase image data. Therefore, features with richer channels are used to guide the consistency of semantic information on different paths.

[0079] That is, the cross-phase non-local attention mechanism module can use the position correlation and depth correlation between the data of two phases to enhance the semantic feature representation. The position correlation of the cross-phase non-local attention mechanism module is to explore the position dependency information between different phases by estimating the correlation between each position of one phase and the corresponding neighboring position of another phase. The depth correlation is achieved by calculating the cross-phase attention value between channels of different phases.

[0080] Feature fusion module: The fusion module is designed to effectively integrate and refine the features of different phases.

[0081] After obtaining the registered multi-phase data pairs, the registered multi-phase data pairs are downsampled to obtain global-scale multi-phase data pairs, and the registered multi-phase data pairs are sliced ​​and segmented to obtain local-scale multi-phase data pairs. The global-scale multi-phase data pairs are respectively extracted with two training paths of the preset multi-phase fusion organ segmentation network model to obtain global features of the two phases. The global features of the two phases are complemented according to the positional relationship and depth relationship between the global features of the two phases to obtain the global-scale segmentation result. The local-scale multi-phase data pairs are respectively extracted with two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases. The local features of the two phases are complemented according to the positional relationship and depth relationship between the local features of the two phases to obtain the local-scale segmentation result. The global-scale segmentation result and the local-scale segmentation result are weightedly fused to obtain the segmentation result of the organ to be segmented.

[0082] The global scale segmentation result obtained by complementing the global features of the two phases according to the position relationship and depth relationship between the global features of the two phases may include:

[0083] According to each position feature of one phase in the global features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, and the attention value of the cross-phase between the same channels of the two phases is calculated. According to the attention value, the deep features in the global features of the two phases are complemented to obtain the global scale segmentation result.

[0084] The above-mentioned method of obtaining the local scale segmentation result by complementing the local features of the two phases according to the position relationship and depth relationship between the local features of the two phases may include:

[0085] According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, and the attention value of the cross-phase between the same channels of the two phases is calculated. According to the attention value, the deep features in the local features of the two phases are complemented to obtain the local scale segmentation result.

[0086] Among them, the above-mentioned method of calculating the attention value of the cross phase between the same channels of two phases can be any attention value calculation method, and the present invention does not impose any limitation on this.

[0087] That is to say, the position feature of one phase is used to complement the position feature of another phase, and the complementary method can be any method in the prior art, so that the features of the two phases can be strictly aligned after the position complementation, that is, they have consistency, and the depth features of the two phases are complemented by the attention value, so that the features of the two phases can be strictly aligned after the depth complementation, that is, they have consistency. As a result, the features of the two phases are corrected by the cross-phase non-local attention mechanism model, which improves the robustness to non-aligned features. Among them, the method of calculating the attention value of the cross-phase between the same channels of the two phases can be any attention value calculation method in the prior art, and the embodiment of the present invention does not impose any limitation on this.

[0088] In summary, the present application provides a multi-phase fusion organ segmentation method based on a non-local attention mechanism, which can extract the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented, and cut each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented, and deformably align the multi-phase data pairs to obtain the aligned multi-phase data pairs, downsample the aligned multi-phase data pairs to obtain global-scale multi-phase data pairs, and slice and segment the aligned multi-phase data pairs to obtain local-scale multi-phase data pairs, and respectively align the whole multi-phase fusion organ segmentation network model by two training paths preset in the pre-set multi-phase fusion organ segmentation network model. Feature extraction is performed on local-scale multi-phase data pairs to obtain global features of the two phases. The global features of the two phases are complemented according to the positional relationship and depth relationship between the global features of the two phases to obtain the global-scale segmentation result. Feature extraction is performed on local-scale multi-phase data pairs respectively through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases. The local features of the two phases are complemented according to the positional relationship and depth relationship between the local features of the two phases to obtain the local-scale segmentation result. The global-scale segmentation result and the local-scale segmentation result are weightedly fused to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network. In the technical solution of the present application, the cross-phase non-local attention fusion mechanism module in the preset multi-phase fusion organ segmentation network model complements the features of the two phases according to the positional relationship and depth relationship between the features of the two phases, suppresses the influence of the two phases not strictly aligned with the features, ensures the consistency between the features of the two phases, and makes full use of the complementarity of the two phases so that the features of the two phases can be effectively fused. In addition, a multi-scale segmentation framework is used to fuse the global scale segmentation results and the local scale segmentation results, which can effectively utilize the global context information and improve the accuracy of organ segmentation.

[0089] At the same time, the centroid extraction strategy proposed in this application can extract the centroid of each multi-phase image data to be segmented respectively, obtain the centroid of the organ to be segmented, and then perform alignment based on the centroid, which effectively improves the accuracy of alignment between image data of different phases.

[0090] In addition, the end-to-end dual-path fully convolutional network proposed in this application, that is, the preset multi-phase fusion organ segmentation network model, can effectively interact with the features of the two phases, effectively combine local information and global information, give full play to the 3D performance of medical images and reduce parameters. The proposed cross-phase non-local attention mechanism module can further correct the features of the two phases and improve the robustness to non-aligned features.

[0091] Based on the above multi-phase fusion organ segmentation method based on non-local attention mechanism, see Figure 3 The present application also provides a multi-phase fusion organ segmentation device based on a non-local attention mechanism, the device comprising:

[0092] A centroid extraction module 301 is used to extract the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented;

[0093] A cutting module 302 is used to cut each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented;

[0094] A registration module 303, configured to perform deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs;

[0095] The fusion module 304 is used to downsample the registered multi-phase data pairs to obtain global-scale multi-phase data pairs, slice and segment the registered multi-phase data pairs to obtain local-scale multi-phase data pairs, extract features from the global-scale multi-phase data pairs through two training paths of the preset multi-phase fusion organ segmentation network model to obtain global features of the two phases, complement the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result, extract features from the local-scale multi-phase data pairs through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases, complement the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result, and perform weighted fusion of the global-scale segmentation result and the local-scale segmentation result to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network.

[0096] It can be seen that the device can extract the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented, cut each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all the slices of the organ to be segmented, perform deformable registration on the multi-phase data pair to obtain the registered multi-phase data pair, downsample the registered multi-phase data pair to obtain a global-scale multi-phase data pair, perform slice segmentation on the registered multi-phase data pair to obtain a local-scale multi-phase data pair, and perform feature extraction on the global-scale multi-phase data pair through two training paths of the preset multi-phase fusion organ segmentation network model. , obtain the global features of the two phases, complement the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain the global-scale segmentation result, extract features from the local-scale multi-phase data pairs respectively through the two training paths of the preset multi-phase fusion organ segmentation network model, and obtain the local features of the two phases, complement the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain the local-scale segmentation result, perform weighted fusion on the global-scale segmentation result and the local-scale segmentation result to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path fully convolutional network. In the technical solution of the present application, the cross-phase non-local attention fusion mechanism module in the preset multi-phase fusion organ segmentation network model complements the features of the two phases according to the positional relationship and depth relationship between the features of the two phases, suppresses the influence of the two phases not strictly aligned with the features, ensures the consistency between the features of the two phases, and makes full use of the complementarity of the two phases so that the features of the two phases can be effectively fused. In addition, a multi-scale segmentation framework is used to fuse the global scale segmentation results and the local scale segmentation results, which can effectively utilize the global context information and improve the accuracy of organ segmentation.

[0097] In one implementation, the centroid extraction module 301 may include:

[0098] The segmentation submodule is used to perform organ segmentation on each of the multi-phase image data to be segmented according to a preset organ segmentation model to obtain an initial segmentation result;

[0099] A determination submodule is used to determine the centroid of the organ to be segmented according to the initial segmentation result.

[0100] In one implementation, the cutting module 302 may be specifically used for:

[0101] According to the centroid, upper and lower N slices of the organ to be segmented are extracted from each multi-phase image to be segmented, and two phase data containing all slices of the organ to be segmented are obtained as a multi-phase data pair, wherein N is a positive integer.

[0102] In one implementation, the fusion module 304 is specifically configured to:

[0103] According to each position feature of one phase in the global features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the deep features in the global features of the two phases are complemented to obtain a global-scale segmentation result.

[0104] In one implementation, the fusion module 304 is specifically configured to:

[0105] According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the depth features in the local features of the two phases are complemented to obtain a local scale segmentation result.

[0106] In one implementation, the backbone network of the preset multi-phase fusion organ segmentation network model consists of a 3D encoder and a 2D decoder.

[0107] In one implementation, the multi-phase image data to be segmented are multi-phase image data of CT or multi-modal data of MRI.

[0108] Figure 4 It is a structural diagram of an electronic device provided in an embodiment of the present application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0109] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0110] The memory is used to store execution instructions. Specifically, the execution instructions are computer programs that can be executed. The memory may include internal memory and non-volatile memory, and provides execution instructions and data to the processor.

[0111] In one possible implementation, the processor reads the corresponding execution instructions from the non-volatile memory into the memory and then runs them, and can also obtain the corresponding execution instructions from other devices to form a multi-phase fusion organ segmentation device based on the non-local attention mechanism at the logical level. The processor executes the execution instructions stored in the memory to implement the multi-phase fusion organ segmentation method based on the non-local attention mechanism provided in any embodiment of the present application through the executed execution instructions.

[0112] The above application Figure 1The multi-phase fusion organ segmentation method based on the non-local attention mechanism provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0113] The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0114] An embodiment of the present application also proposes a readable medium, which stores execution instructions. When the stored execution instructions are executed by a processor of an electronic device, the electronic device can execute the multi-phase fusion organ segmentation method based on the non-local attention mechanism provided in any embodiment of the present application.

[0115] The electronic device described in the above embodiments may be a computer.

[0116] Those skilled in the art should understand that the embodiments of the present application can be provided as methods or computer program products. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware.

[0117] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0118] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0119] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-phase fusion organ segmentation method based on non-local attention mechanism, characterized in that: include: Extract the centroid of each multi-phase image data to be segmented respectively to obtain the centroid of the organ to be segmented; Cutting each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented; Performing deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs; Down-sampling the registered multi-phase data pairs to obtain global-scale multi-phase data pairs, slicing and segmenting the registered multi-phase data pairs to obtain local-scale multi-phase data pairs, extracting features from the global-scale multi-phase data pairs through two training paths of a preset multi-phase fusion organ segmentation network model to obtain global features of the two phases, complementing the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result, extracting features from the local-scale multi-phase data pairs through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases, complementing the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result, weighted fusion of the global-scale segmentation result and the local-scale segmentation result to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path full convolution network; The step of cutting each multi-phase image data to be segmented according to the centroid to obtain multi-phase data pairs comprises: Extracting upper and lower N slices of the organ to be segmented from each multi-phase image to be segmented according to the centroid, and obtaining two phase data containing all slices of the organ to be segmented as a multi-phase data pair, wherein N is a positive integer; The step of complementing the global features of the two phases according to the position relationship and the depth relationship between the global features of the two phases to obtain a global scale segmentation result comprises: Complementing the position features of the corresponding neighborhood of the other phase according to each position feature of one phase in the global features of the two phases, calculating the attention value of the cross-phase between the same channels of the two phases, and complementing the deep features in the global features of the two phases according to the attention value to obtain a global scale segmentation result; The step of complementing the local features of the two phases according to the position relationship and the depth relationship between the local features of the two phases to obtain a local scale segmentation result comprises: According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the depth features in the local features of the two phases are complemented to obtain a local scale segmentation result.

2. The method according to claim 1, characterized in that The step of extracting the centroid of each of the multi-phase image data to be segmented to obtain the centroid of the organ to be segmented comprises: According to the preset organ segmentation model, each of the multi-phase image data to be segmented is subjected to organ segmentation to obtain an initial segmentation result; The centroid of the organ to be segmented is determined according to the initial segmentation result.

3. The method according to claim 1, characterized in that The backbone network of the preset multi-phase fusion organ segmentation network model consists of a 3D encoder and a 2D decoder.

4. The method according to claim 1, characterized in that The multi-phase image data to be segmented are multi-phase image data of CT or multi-modal data of MRI.

5. A multi-phase fusion organ segmentation device based on a non-local attention mechanism, comprising: A centroid extraction module is used to extract the centroid of each multi-phase image data to be segmented to obtain the centroid of the organ to be segmented; A cutting module, used for cutting each multi-phase image data to be segmented according to the centroid to obtain a multi-phase data pair, wherein each phase data in the multi-phase data pair contains all slices of the organ to be segmented; A registration module, used for performing deformable registration on the multi-phase data pairs to obtain registered multi-phase data pairs; A fusion module, used for downsampling the registered multi-phase data pairs to obtain global-scale multi-phase data pairs, slicing and segmenting the registered multi-phase data pairs to obtain local-scale multi-phase data pairs, extracting features from the global-scale multi-phase data pairs through two training paths of a preset multi-phase fusion organ segmentation network model to obtain global features of the two phases, complementing the global features of the two phases according to the positional relationship and depth relationship between the global features of the two phases to obtain a global-scale segmentation result, extracting features from the local-scale multi-phase data pairs through two training paths of the preset multi-phase fusion organ segmentation network model to obtain local features of the two phases, complementing the local features of the two phases according to the positional relationship and depth relationship between the local features of the two phases to obtain a local-scale segmentation result, and weighted fusion of the global-scale segmentation result and the local-scale segmentation result to obtain the segmentation result of the organ to be segmented, wherein the preset multi-phase fusion organ segmentation network model is a dual-path full convolution network; The cutting module is specifically used for: Extracting upper and lower N slices of the organ to be segmented from each multi-phase image to be segmented according to the centroid, and obtaining two phase data containing all slices of the organ to be segmented as a multi-phase data pair, wherein N is a positive integer; The fusion module is specifically used for: Complementing the position features of the corresponding neighborhood of the other phase according to each position feature of one phase in the global features of the two phases, calculating the attention value of the cross-phase between the same channels of the two phases, and complementing the deep features in the global features of the two phases according to the attention value to obtain a global scale segmentation result; The fusion module is specifically used for: According to each position feature of one of the local features of the two phases, the position feature of the corresponding neighborhood of the other phase is complemented, the attention value of the cross-phase between the same channels of the two phases is calculated, and according to the attention value, the depth features in the local features of the two phases are complemented to obtain a local scale segmentation result.

6. The device according to claim 5, characterized in that The centroid extraction module comprises: The segmentation submodule is used to perform organ segmentation on each of the multi-phase image data to be segmented according to a preset organ segmentation model to obtain an initial segmentation result; A determination submodule is used to determine the centroid of the organ to be segmented according to the initial segmentation result.

7. An electronic device, characterized in that: include: A processor and a memory storing execution instructions, when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 4.

Images