Construction method and application of multi-modal medical image registration segmentation model

Abstract

The invention discloses a multi-modal medical image registration segmentation model construction method and application thereof, and the construction method comprises the steps: obtaining intraoperative ultrasonic images and preoperative CT images of a plurality of cervical cancer patients, and forming an image data set; preprocessing the image data set, and dividing the preprocessed image data set to obtain a training set, a verification set and a test set; training the registration decoding branch and the segmentation decoding branch based on a loss function by adopting the training set to obtain an initial registration segmentation model; verifying the initial registration segmentation model by adopting the verification set to obtain a middle registration segmentation model; and evaluating the intermediate registration segmentation model by using the test set to obtain a target registration segmentation model. According to the method, the preoperative high-resolution CT image and the intraoperative ultrasonic navigation image are introduced to be fused, so that the precision of automatic segmentation of the tumor target region and the endangered organ of the cervical cancer is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method for constructing a multimodal medical image registration and segmentation model and its application. Background Art

[0002] Cervical cancer is a common gynecological malignancy, and its morbidity and mortality are high among malignant tumors of the female reproductive system. At present, painless afterloading therapy based on CT or ultrasound image guidance and represented by three-dimensional implantation technology has become a hot technology for precision radiotherapy of cervical cancer.

[0003] During the post-treatment of cervical cancer, doctors need to develop personalized treatment plans based on the specific conditions of the patients. Medical image registration technology can help doctors understand the location, size, shape and other information of the tumor more accurately, so as to develop a more reasonable and effective treatment plan. In recent years, the research on deep learning in medical image registration can be roughly divided into two technical routes, namely supervised learning and unsupervised learning. Compared with supervised learning that requires the use of real deformation fields as training data, unsupervised learning can use a large amount of unlabeled raw data for autonomous learning to explore the intrinsic relationship between images, and has a wider range of applications. However, the current research on image registration based on unsupervised learning is mainly focused on single-modality medical images, and the exploration of multimodal image registration needs to be strengthened. Due to the differences in the sources, acquisition parameters and imaging principles of different modal images, multimodal image registration based on unsupervised learning faces greater challenges in image feature extraction, similarity measurement and network structure optimization. Therefore, how to make the deep learning network better adapt to the anatomical characteristics and imaging characteristics of different modal images has become an urgent problem to be solved in the research of multimodal image registration.

[0004] Target and OAR contouring is a key step in the design of cervical cancer afterloading treatment plan. Traditional manual contouring methods take a lot of time. At present, many studies have tried to use different deep learning models to automatically segment the contours of cervical cancer targets and OARs. Although deep learning models have achieved remarkable results in the automatic segmentation of cervical cancer target and OAR contours, the above studies are all based on high-resolution CT and MRI images, and there is little research on relatively low-resolution ultrasound images. Therefore, how to achieve rapid segmentation of cervical cancer targets and OARs based on ultrasound navigation images has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for constructing a multimodal medical image registration and segmentation model and its application.

[0006] In a first aspect, an embodiment of the present invention provides a method for constructing a multimodal medical image registration and segmentation model, comprising:

[0007] Acquire intraoperative ultrasound images and preoperative CT images of multiple cervical cancer patients to form an image data set;

[0008] Preprocessing the image data set, and dividing the preprocessed image data set to obtain a training set, a validation set, and a test set;

[0009] Using the training set, training the registration decoding branch and the segmentation decoding branch based on the loss function to obtain an initial registration and segmentation model;

[0010] Among them, the initial registration and segmentation model includes an ultrasound image encoding module, a CT image encoding module, a feature fusion module, a registration decoding branch, a feature alignment module and a segmentation decoding branch; the loss function is a weighted sum of the registration loss and the segmentation loss, the registration loss includes mutual information loss and smoothing loss, and the segmentation loss includes cross entropy loss and Dice loss; the ultrasound image encoding module is used to perform feature learning on the intraoperative ultrasound image to obtain a first feature; the CT image encoding module is used to perform feature learning on the preoperative CT image to obtain a second feature; the feature fusion module is used to perform feature fusion on the first feature and the second feature to obtain a third feature; the registration decoding branch is used to perform registration based on the first feature, the second feature and the third feature to obtain a predicted deformation field; the feature alignment module is used to perform feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature; the segmentation decoding branch is used to segment the fourth feature to obtain a predicted probability map;

[0011] Using the verification set to verify the initial registration and segmentation model to obtain an intermediate registration and segmentation model;

[0012] The test set is used to evaluate the intermediate registration and segmentation model to obtain a target registration and segmentation model.

[0013] Preferably, the network architecture of the ultrasound image encoding module and the CT image encoding module is the same, both of which are multi-scale convolutional networks.

[0014] Preferably, the feature fusion module comprises: a first fusion branch, a second fusion branch and an SK convolution layer;

[0015] The first fusion branch includes a mathematical operation layer and a first convolution layer, which is used to fuse the position information of the first feature and the second feature to obtain a first fused feature;

[0016] The second fusion branch includes a cross-modal attention mechanism layer and a second convolution layer, which is used to fuse the semantic relevance information of the first feature and the second feature to obtain a second fused feature; wherein the convolution kernel sizes of the first convolution layer and the second convolution layer are the same;

[0017] The SK convolution layer is used to fuse the first fusion feature and the second fusion feature to obtain a third feature.

[0018] Preferably, fusing the position information of the first feature and the second feature to obtain a first fused feature includes:

[0019] Calculating the first feature and the second feature according to the mathematical operation layer to obtain a plurality of position information features; wherein the calculation at least includes summing, subtracting, product and taking the maximum value;

[0020] The plurality of position information codes are concatenated, and a convolution operation is performed on the concatenated position information features according to the first convolution layer to obtain a first fusion feature.

[0021] Preferably, fusing the semantic relevance information of the first feature and the second feature to obtain a second fused feature includes:

[0022] Performing attention calculation on the first feature and the second feature according to the cross-modal attention mechanism layer to obtain a first cross-modal attention feature of the first feature to the second feature and a second cross-modal attention feature of the second feature to the first feature;

[0023] The first cross-modal attention feature and the second cross-modal attention feature are concatenated, and a convolution operation is performed on the concatenated cross-modal attention feature according to the second convolution layer to obtain a second fusion feature.

[0024] Preferably, the registration decoding branch and the segmentation decoding branch have the same network architecture, both comprising a plurality of densely connected convolution blocks and an output convolution block.

[0025] Preferably, the feature alignment module includes a third convolutional layer and a fourth convolutional layer;

[0026] The performing feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature includes:

[0027] Extracting data from the predicted deformation field according to the third convolutional layer to obtain spatial matching information;

[0028] Splicing the spatial matching information and the first feature, and aligning the spliced ​​features according to the fourth convolutional layer to obtain an aligned feature;

[0029] The alignment feature and the second feature are fused to obtain a fourth feature.

[0030] Preferably, the fourth convolutional layer includes at least deformable convolution, batch normalization and Leaky ReLU.

[0031] In a second aspect, an embodiment of the present invention provides a multimodal medical image registration and segmentation method, wherein the target registration and segmentation model is constructed by the construction method described above, and comprises:

[0032] Acquire intraoperative ultrasound images and preoperative CT images of the target object;

[0033] Using the ultrasound image encoding module in the target registration and segmentation model to perform feature learning on the intraoperative ultrasound image to obtain a first feature;

[0034] Using the CT image encoding module in the target registration and segmentation model to perform feature learning on the preoperative CT image to obtain a second feature;

[0035] Using a feature fusion module in the target registration and segmentation model to fuse the first feature and the second feature to obtain a third feature;

[0036] Using the registration decoding branch in the target registration segmentation model to perform registration based on the first feature, the second feature and the third feature to obtain a predicted deformation field;

[0037] Using a feature alignment module in the target registration and segmentation model to perform feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature;

[0038] The fourth feature is segmented using the segmentation decoding branch in the target registration segmentation model to obtain a prediction probability map.

[0039] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute a method for constructing a multimodal medical image registration and segmentation model as described above and / or a multimodal medical image registration and segmentation method as described above.

[0040] Compared with the prior art, the method for constructing a multimodal medical image registration and segmentation model and its application in an embodiment of the present invention have the following beneficial effects: independent encoding modules are constructed for CT images and ultrasound images, which can retain the unique information of images of different modalities as much as possible; in the design of the feature fusion module, the semantic relevance information fusion is strengthened by constructing a cross-modal attention mechanism, and the SK convolution is introduced to selectively fuse features based on position information with features based on semantic relevance, so that the model can better utilize the complementary information of multimodal data; for the registration and segmentation tasks, independent decoding branches are constructed, and the feature alignment module is used to provide the spatial matching information in the registration decoding branch to the segmentation decoding branch, effectively improving the segmentation accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a method for constructing a multimodal medical image registration and segmentation model according to an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the structure of the initial registration and segmentation model of an embodiment of the present invention;

[0043] Figure 3 is a schematic diagram of the internal structure of each convolution block in an ultrasound image encoding module or a CT image encoding module according to an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the structure of a feature fusion module according to an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of the structure of the cross-modal attention mechanism layer of an embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of a processing mechanism of a context block according to an embodiment of the present invention;

[0047] Figure 7 It is a structural schematic diagram of a feature alignment module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0049] In the description of the present invention, it should be understood that the terms "first" and "second" etc. are used in the present invention to distinguish different objects rather than to describe a specific order.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a method for constructing a multimodal medical image registration and segmentation model, comprising the steps of:

[0051] S1, obtaining intraoperative ultrasound images and preoperative CT images of multiple cervical cancer patients to form an image data set;

[0052] Specifically, this embodiment obtains intraoperative ultrasound images and preoperative CT images of 200 cervical cancer patients in stage IIB-IVA (FIGO 2018 staging) who were initially diagnosed without distant metastasis and received radical radiotherapy from the hospital database. The tumor target area and the outline of the organ at risk were manually outlined on the intraoperative ultrasound images and preoperative CT images, respectively, to form an image data set.

[0053] S2, preprocessing the image data set, and dividing the preprocessed image data set into a training set, a validation set, and a test set;

[0054] Specifically, in the preprocessing, the intraoperative ultrasound images and preoperative CT images were first resampled to unify the pixel spacing to 1 mm, 1 mm, and 1 mm. Then, all intraoperative ultrasound images and preoperative CT images were normalized. Then, the intraoperative ultrasound images and preoperative CT images of the same patient were used as a group of image pairs for rigid registration. The CT sequence was registered to the ultrasound image space. Finally, the image pairs were center cropped to ensure that the size of the image pairs in all directions could be divided by the overall downsampling factor of 32.

[0055] Furthermore, in this embodiment, the preprocessed image data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0056] S3, using the training set, training the registration decoding branch and the segmentation decoding branch based on the loss function to obtain an initial registration and segmentation model;

[0057] It should be noted that the present invention constructs a registration and segmentation model based on the idea of ​​hard feature sharing. Multi-scale features are extracted from the input ultrasound image and CT image through an independent encoding module. While providing overall anatomical structure information, the endangered organ structure and position information required for registration and segmentation are emphasized, thereby achieving mutual promotion of registration and segmentation tasks.

[0058] Specifically, Figure 2 As shown, the initial registration and segmentation model includes an ultrasound image encoding module, a CT image encoding module, a feature fusion module, a registration decoding branch, a feature alignment module and a segmentation decoding branch.

[0059] In order to retain the unique information of different modal images as much as possible, the initial registration and segmentation model designs independent coding modules, namely, ultrasound image coding module and CT image coding module, for intraoperative ultrasound images and preoperative CT images, respectively, to extract the feature information of each input modality separately. Specifically, the ultrasound image coding module is used to perform feature learning on intraoperative ultrasound images to obtain the first feature; the CT image coding module is used to perform feature learning on preoperative CT images to obtain the second feature. Among them, the ultrasound image coding module performs feature learning on the anatomical structure and pathological tissue of the intraoperative ultrasound image, and the CT image coding module performs feature learning on the anatomical structure and pathological tissue of the preoperative CT image. It can be understood that the first feature and the second feature are corresponding multi-scale features.

[0060] Furthermore, the network architecture of the ultrasound image encoding module and the CT image encoding module is the same, both of which are multi-scale convolutional networks. Figure 2 The network architecture of the two encoding modules is introduced in detail: Both encoding modules include an input convolution block and five convolution blocks, where I m Indicates the intraoperative ultrasound image, i.e., the floating image. f Represents the preoperative CT image, i.e., a fixed image. Each convolutional block is interspersed with a maximum pooling layer to achieve feature downsampling and mine sequence information of different scales. Figure 3 As shown in the figure, the internal structure of each convolution block includes two convolution layers (Conv(k=3)) with a convolution kernel size of 3, followed by a batch normalization layer (Batch Norm) and a leaky ReLU. The number of convolution layers in the input convolution block is 16, and the number of convolution layers in the five convolution blocks is 32, 64, 128, 256, and 512 respectively.

[0061] The feature fusion module is one of the characteristic designs of the present invention, which aims to explore the correlation between preoperative CT images and intraoperative ultrasound images, as well as complementary information at different scales at the tissue and organ levels. The feature fusion module is used to fuse the first feature and the second feature to obtain the third feature. It adopts two feature fusion methods: fusion based on position information and fusion based on semantic relevance. In addition, the SK convolution is used to selectively fuse the features based on position information with the features based on semantic relevance, so as to adaptively screen the features.

[0062] Specifically, Figure 4 As shown, the feature fusion module includes: a first fusion branch, a second fusion branch and an SK convolution layer (SK Conv). f represents the second feature, x mRepresents the first feature, (C, D, H, W) is used to describe the four dimensions of the feature, C represents the number of channels, D represents the dimension, H represents the height, and W represents the width. Represents a concatenation operation.

[0063] The first fusion branch includes a mathematical operation layer and a first convolution layer, which is used to perform position information fusion on the first feature and the second feature to obtain a first fused feature. Specifically, the first feature and the second feature are fused with position information to obtain a first fused feature, including: calculating the first feature and the second feature according to the mathematical operation layer to obtain multiple position information features. Among them, the calculation includes at least summing, subtracting, product and taking the maximum value. This embodiment uses four predefined functional relationships to represent the interaction relationship between images of different modalities, including summing (+), subtracting (-), product (×) and taking the maximum value (Max). Multiple position information codes are spliced, and the spliced ​​position information features are convolved according to the first convolution layer to obtain the first fused feature.

[0064] The second fusion branch includes a cross-modal attention mechanism layer and a second convolution layer, which are used to perform semantic relevance information fusion on the first feature and the second feature to obtain a second fused feature. Among them, the convolution kernel size of the first convolution layer and the second convolution layer is the same, and the convolution kernel size of the first convolution layer and the second convolution layer in this embodiment is 3. Specifically, the first feature and the second feature are semantically fused to obtain a second fused feature, including: performing attention calculation on the first feature and the second feature according to the cross-modal attention mechanism layer to obtain a first cross-modal attention feature of the first feature to the second feature and a second cross-modal attention feature of the second feature to the first feature; splicing the first cross-modal attention feature and the second cross-modal attention feature, and performing a convolution operation on the spliced ​​cross-modal attention feature according to the second convolution layer to obtain a second fused feature.

[0065] In order to strengthen the learning of long-distance semantics by convolutional neural networks, the present invention designs a cross-modal attention mechanism layer to respectively calculate the first cross-modal attention feature of the first feature to the second feature and the second cross-modal attention feature of the second feature to the first feature.

[0066] Specifically, Figure 5 As shown in Figure 2, the cross-modal attention mechanism layer includes a context modeling module and an extraction module. a and x b represents the first and second characteristics. That is, if x a represents the first feature, then x b represents the second feature; if x a represents the second characteristic, then x bIndicates the first feature.

[0067] Furthermore, the context modeling module includes multiple context blocks with different downsampling multiples (Context (s = 4), Context (s = 2) and Context (s = 1)), where s represents the downsampling multiple. Global context attention features of different scales are obtained through context blocks, and the global context attention features of different scales are spliced ​​together and input into the extraction module for processing, where Indicates the splicing operation. Figure 6 The processing mechanism of the context block is introduced in detail: In the context block, the input feature x is pooled using the average pooling layer (AP(s×s×s)) a and x b Downsample to the original The convolution layer (Conv(1×1×1)) with a convolution kernel size of 1 is used to linearly map and compress the features to obtain a tensor with a shape of 1×D×H×W in the embedding space, and then reshape it into a shape of DHW×1×1×1. The Softmax layer is then used to obtain the weight tensor of each spatial position, and the weight tensor is combined with the input feature x reshaped into a shape of DHW×1×1×1. a Perform matrix multiplication to get x a x b The global contextual attention features of . Among them, Represents a matrix multiplication operation.

[0068] Furthermore, for the extraction module, a multi-layer perceptron structure similar to SE Block is used to increase nonlinearity while retaining the original number of input channels, and aggregate the global context attention features into the features of each spatial position, so that the model can extract richer and more expressive feature representations and improve its performance in cross-modal tasks. Specifically, Figure 5 As shown in Figure 1, the internal structure of the extraction module includes two convolutional layers (Conv(1×1×1)) with a convolution kernel size of 1, and layer normalization calculation (Layer Norm) and Leaky ReLU between the two convolutional layers. Finally, the convolution operation is used to extract the input feature x a And the input feature x b The mutual cross-modal attention features are fused and the size of the feature map is downsampled to The convolution kernel size of the convolution operation in this embodiment is 3. Represents a fusion operation.

[0069] The SK convolution layer is used to fuse the first fusion feature and the second fusion feature to obtain the third feature. The first fusion feature based on location information and the second fusion feature based on semantic relevance are selectively fused through the three operators Split, Fuse and Select, so as to adaptively filter features and enable the model to better utilize the complementary information of multimodal data.

[0070] The registration decoding branch is used to perform registration based on the first feature, the second feature, and the third feature to obtain a predicted deformation field. Figure 2 As shown in the figure, the registration decoding branch includes five densely connected convolution blocks and an output convolution block, which concatenates the first feature, the second feature, and the third feature, and organically combines the features under the guidance of the registration task to obtain the input of each densely connected convolution block. The number of convolution layers in the five densely connected convolution blocks is 256, 128, 64, 32, and 16, respectively, and the number of convolution layers in the output convolution block is 16. represents the predicted deformation field.

[0071] The feature alignment module is another characteristic design of the present invention, which aims to make full use of the rich spatial matching information in the output features of the registration decoding branch to promote the segmentation work. Specifically, the feature alignment module is used to align the features based on the first feature, the second feature and the predicted deformation field to obtain the fourth feature. Further, if Figure 7 As shown in the figure, the feature alignment module includes the third convolutional layer and the fourth convolutional layer, and its three inputs are the predicted deformation field First feature And the second feature Performing feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature, including: performing data extraction on the predicted deformation field according to the third convolutional layer to obtain spatial matching information, wherein the third convolutional layer in this embodiment is a convolutional layer with a convolution kernel size of 1 (Conv(1×1×1)); splicing the spatial matching information and the first feature, and performing feature alignment on the spliced ​​features according to the fourth convolutional layer to obtain an aligned feature, wherein the fourth convolutional layer includes at least deformable convolution, batch normalization and Leaky ReLU, and the fourth convolutional layer (DConv) in this embodiment includes deformable convolution, batch normalization and Leaky ReLU; fusing the aligned feature and the second feature to obtain the fourth feature.

[0072] The segmentation decoding branch is used to segment the fourth feature to obtain a prediction probability map. The network architecture of the registration decoding branch and the segmentation decoding branch is the same, both of which include multiple densely connected convolution blocks and an output convolution block. Specifically, Figure 2As shown in the figure, the segmentation decoding branch includes five densely connected convolution blocks and one output convolution block. The fourth feature is used as the input of each densely connected convolution block, and finally the predicted probability map of the background class and the foreground class is calculated as the output using the convolution layer with a convolution kernel size of 1 and the Softmax function. Among them, the number of convolution layers in the five densely connected convolution blocks is 256, 128, 64, 32 and 16 respectively, the number of convolution layers in the output convolution block is 16, and p pred It should be noted that in the above process, the features of the corresponding level of the registration decoding branch will be transferred to the segmentation decoding branch to achieve the fusion of CT features and ultrasound features under the guidance of registration information, thereby reducing the influence of the position deviation of the two images on the segmentation decision.

[0073] The following is a detailed introduction to the construction process of the initial registration and segmentation model loss function:

[0074] The predicted deformation field output by the registration decoding branch is used to transform the intraoperative ultrasound image, and the registration loss is calculated. The registration loss includes mutual information loss and smoothing loss. The predicted probability map output by the segmentation decoding branch is compared with the manually annotated organ region, and the segmentation loss is calculated. The segmentation loss includes cross entropy loss and Dice loss. The registration loss and segmentation loss are combined together through the weight coefficient to form the loss function of the initial registration segmentation model, that is, the loss function is the weighted sum of the registration loss and the segmentation loss. Specifically, this embodiment uses the following formula to represent the loss function:

[0075]

[0076] Among them, L total represents the loss function, represents the registration loss, L seg (s gt ,p pred ) represents the segmentation loss, α represents the weight coefficient, I f ,I m , s gt 、p pred They represent preoperative CT image, intraoperative ultrasound image, predicted deformation field, manually annotated organ region and predicted probability map respectively.

[0077] S4, using the validation set to validate the initial registration and segmentation model to obtain an intermediate registration and segmentation model;

[0078] Specifically, the validation set is used to verify the initial registration and segmentation model based on the loss function to obtain an intermediate registration and segmentation model.

[0079] S5. Use the test set to evaluate the intermediate registration and segmentation model to obtain the target registration and segmentation model.

[0080] Specifically, the test set is used to evaluate the intermediate registration and segmentation model based on the evaluation index to obtain the target registration and segmentation model. The evaluation index includes the registration result evaluation index and the segmentation result evaluation index.

[0081] Furthermore, the evaluation indicators of the registration results include the Dice similarity coefficient and the percentage of non-positive values ​​in the Jacobian matrix determinant. The Dice similarity coefficient is a set similarity metric used to calculate the similarity of two samples. Its value range is (0,1) and is positively correlated with the registration result. That is, when its value is closer to 1, the similarity of the two samples is higher, indicating that the registration effect is better.

[0082] In image registration, the Jacobian matrix determinant is an important evaluation indicator used to measure the topological characteristics of the deformation field. The value of the Jacobian matrix determinant at each point reflects the change in the image volume at that point:

[0083] 1) When the value of the Jacobian matrix determinant is equal to 1, it means that the volume of the image at that point has not changed;

[0084] 2) When the value is greater than 1, it means that the image is dilated at that point;

[0085] 3) When the value is between 0 and 1, it means that the image is reduced at that point;

[0086] 4) When the value is less than or equal to 0, it means that the image is folded at that point, which is an undesirable situation because it may mean that the image is severely distorted during the registration process.

[0087] Therefore, the percentage of non-positive values ​​in the Jacobian matrix determinant can be used to measure the degree of folding in the entire deformation field. The higher the percentage of non-positive values, the more serious the image distortion during the registration process and the worse the registration effect.

[0088] Furthermore, the evaluation indicators of the segmentation results include the Dice similarity coefficient and the 95% Hausdorff distance. The Dice similarity coefficient is a set similarity measurement indicator used to calculate the similarity of two samples. Its value range is (0,1) and is positively correlated with the segmentation result. That is, when its value is closer to 1, the similarity of the two samples is higher, indicating that the segmentation effect is better. For the 95% Hausdorff distance, the smaller its value is, the more similar the two sets are, and the better the segmentation effect is.

[0089] An embodiment of the present invention provides a method for constructing a multimodal medical image registration and segmentation model, which constructs independent encoding modules for CT images and ultrasound images, and can retain the unique information of different modal images as much as possible; in the design of the feature fusion module, the semantic relevance information fusion is strengthened by constructing a cross-modal attention mechanism, and the SK convolution is introduced to selectively fuse features based on position information with features based on semantic relevance, so that the model can better utilize the complementary information of multimodal data; for the registration and segmentation tasks, independent decoding branches are constructed, and the feature alignment module is used to provide the spatial matching information in the registration decoding branch to the segmentation decoding branch, effectively improving the segmentation accuracy of the model.

[0090] The embodiment of the present invention provides a multimodal medical image registration and segmentation method, which uses the above-mentioned construction method to construct a target registration and segmentation model, including the steps of:

[0091] 1) Obtaining intraoperative ultrasound images and preoperative CT images of the target object;

[0092] 2) using the ultrasound image encoding module in the target registration and segmentation model to perform feature learning on the intraoperative ultrasound image to obtain the first feature;

[0093] 3) Using the CT image encoding module in the target registration and segmentation model to perform feature learning on the preoperative CT image to obtain the second feature;

[0094] 4) Using the feature fusion module in the target registration and segmentation model to fuse the first feature and the second feature to obtain a third feature;

[0095] 5) using the registration decoding branch in the target registration segmentation model to perform registration based on the first feature, the second feature and the third feature to obtain a predicted deformation field;

[0096] 6) using a feature alignment module in the target registration and segmentation model to perform feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature;

[0097] 7) Use the segmentation decoding branch in the target registration and segmentation model to segment the fourth feature and obtain a predicted probability map.

[0098] It should be noted that, for the specific limitations of a multimodal medical image registration and segmentation method, please refer to the limitations of a method for constructing a multimodal medical image registration and segmentation model mentioned above. The two have the same functions and effects and will not be repeated here.

[0099] An embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method for constructing a multimodal medical image registration and segmentation model as described above and / or the method for constructing a multimodal medical image registration and segmentation as described above. For related technical solutions, please refer to the method for constructing a multimodal medical image registration and segmentation model as described above, which will not be repeated here.

[0100] In summary, an embodiment of the present invention provides a method for constructing a multimodal medical image registration and segmentation model and its application, and constructs independent encoding modules for CT images and ultrasound images, which can retain the unique information of different modal images as much as possible; in the design of the feature fusion module, the semantic relevance information fusion is strengthened by constructing a cross-modal attention mechanism, and the SK convolution is introduced to selectively fuse features based on position information with features based on semantic relevance, so that the model can better utilize the complementary information of multimodal data; for the registration and segmentation tasks, independent decoding branches are constructed, and the feature alignment module is used to provide the spatial matching information in the registration decoding branch to the segmentation decoding branch, effectively improving the segmentation accuracy of the model.

[0101] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referenced to each other, and each embodiment focuses on the differences from other embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A method for constructing a multimodal medical image registration and segmentation model, characterized in that: include: Acquire intraoperative ultrasound images and preoperative CT images of multiple cervical cancer patients to form an image data set; Preprocessing the image data set, and dividing the preprocessed image data set to obtain a training set, a validation set, and a test set; Using the training set, training the registration decoding branch and the segmentation decoding branch based on the loss function to obtain an initial registration and segmentation model; Among them, the initial registration and segmentation model includes an ultrasound image encoding module, a CT image encoding module, a feature fusion module, a registration decoding branch, a feature alignment module and a segmentation decoding branch; the loss function is a weighted sum of the registration loss and the segmentation loss, the registration loss includes mutual information loss and smoothing loss, and the segmentation loss includes cross entropy loss and Dice loss; the ultrasound image encoding module is used to perform feature learning on the intraoperative ultrasound image to obtain a first feature; the CT image encoding module is used to perform feature learning on the preoperative CT image to obtain a second feature; the feature fusion module is used to perform feature fusion on the first feature and the second feature to obtain a third feature; the registration decoding branch is used to perform registration based on the first feature, the second feature and the third feature to obtain a predicted deformation field; the feature alignment module is used to perform feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature; the segmentation decoding branch is used to segment the fourth feature to obtain a predicted probability map; Using the verification set to verify the initial registration and segmentation model to obtain an intermediate registration and segmentation model; The test set is used to evaluate the intermediate registration and segmentation model to obtain a target registration and segmentation model.

2. The construction method according to claim 1, characterized in that: The network architecture of the ultrasound image encoding module and the CT image encoding module is the same, both of which are multi-scale convolutional networks.

3. The construction method according to claim 1, characterized in that: The feature fusion module includes: a first fusion branch, a second fusion branch and an SK convolution layer; The first fusion branch includes a mathematical operation layer and a first convolution layer, which is used to fuse the position information of the first feature and the second feature to obtain a first fused feature; The second fusion branch includes a cross-modal attention mechanism layer and a second convolution layer, which is used to fuse the semantic relevance information of the first feature and the second feature to obtain a second fused feature; wherein the convolution kernel sizes of the first convolution layer and the second convolution layer are the same; The SK convolution layer is used to fuse the first fusion feature and the second fusion feature to obtain a third feature.

4. The construction method according to claim 3, characterized in that: The fusing the position information of the first feature and the second feature to obtain a first fused feature includes: Calculating the first feature and the second feature according to the mathematical operation layer to obtain a plurality of position information features; wherein the calculation at least includes summing, subtracting, product and taking the maximum value; The plurality of position information codes are concatenated, and a convolution operation is performed on the concatenated position information features according to the first convolution layer to obtain a first fusion feature.

5. The construction method according to claim 3, characterized in that: The fusing the semantic relevance information of the first feature and the second feature to obtain a second fused feature includes: Performing attention calculation on the first feature and the second feature according to the cross-modal attention mechanism layer to obtain a first cross-modal attention feature of the first feature to the second feature and a second cross-modal attention feature of the second feature to the first feature; The first cross-modal attention feature and the second cross-modal attention feature are concatenated, and a convolution operation is performed on the concatenated cross-modal attention feature according to the second convolution layer to obtain a second fusion feature.

6. The construction method according to claim 1, characterized in that: The registration decoding branch and the segmentation decoding branch have the same network architecture, both including a plurality of densely connected convolution blocks and an output convolution block.

7. The construction method according to claim 1, characterized in that: The feature alignment module includes a third convolution layer and a fourth convolution layer; The step of performing feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature includes: Extracting data from the predicted deformation field according to the third convolutional layer to obtain spatial matching information; Splicing the spatial matching information and the first feature, and aligning the spliced ​​features according to the fourth convolutional layer to obtain an aligned feature; The alignment feature and the second feature are fused to obtain a fourth feature.

8. The construction method according to claim 7, characterized in that: The fourth convolutional layer includes at least deformable convolution, batch normalization and Leaky ReLU.

9. A multimodal medical image registration and segmentation method, characterized in that: The target registration and segmentation model constructed by the construction method according to any one of claims 1 to 8 comprises: Acquire intraoperative ultrasound images and preoperative CT images of the target object; Using the ultrasound image encoding module in the target registration and segmentation model to perform feature learning on the intraoperative ultrasound image to obtain a first feature; Using the CT image encoding module in the target registration and segmentation model to perform feature learning on the preoperative CT image to obtain a second feature; Using a feature fusion module in the target registration and segmentation model to fuse the first feature and the second feature to obtain a third feature; Using the registration decoding branch in the target registration segmentation model to perform registration based on the first feature, the second feature and the third feature to obtain a predicted deformation field; Using a feature alignment module in the target registration and segmentation model to perform feature alignment based on the first feature, the second feature and the predicted deformation field to obtain a fourth feature; The fourth feature is segmented using the segmentation decoding branch in the target registration segmentation model to obtain a prediction probability map.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed by the processor, the device where the storage medium is located is controlled to execute a method for constructing a multimodal medical image registration and segmentation model as described in any one of claims 1 to 8 and / or a multimodal medical image registration and segmentation method as described in claim 9.

Images