Image registration and segmentation joint optimization method, system, device and medium

Abstract

The invention discloses an image registration and segmentation joint optimization method, system, device and medium, and relates to the technical field of image registration and segmentation, and the method comprises the steps: obtaining an original medical image, and employing a dynamic pairing strategy to obtain a floating image and a fixed image; the method comprises the following steps: constructing an image registration and segmentation joint optimization model, wherein the model comprises an encoder sharing weight, a segmentation decoder, a registration decoder and an uncertainty estimator; and inputting the floating image and the fixed image into an image registration and segmentation joint optimization model to obtain a registered image and a tag thereof, inputting the fixed image and the registered image into an uncertainty estimator to predict the uncertainty of registration, and minimizing a loss function to obtain an updated image registration and segmentation joint optimization model. According to the method, the problem of class imbalance of different anatomical structure labels can be relieved, and the forward action of the layered segmentation feature map in the deformation subfield generation process can be promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration and segmentation, and particularly relates to a method, system, device and medium for jointly optimizing image registration and segmentation. Background Art

[0002] Medical image registration aims to find an optimal transformation (or optimal deformation field) to achieve precise alignment of two or more images acquired by different imaging devices, at different times, of different patients or from different viewpoints in terms of spatial position and anatomical position. As the basis for research such as medical image fusion, medical image reconstruction, and standard atlas matching, medical image registration plays a crucial role in clinical applications such as disease diagnosis, atlas analysis, surgical guidance, and radiotherapy. However, due to the influence of factors such as lesions, tumor growth, breathing, and heartbeat on the structure or region of tissues and organs, significant non-rigid deformations (such as stretching, twisting, and local deformation) are likely to occur in tissues and organs, resulting in complex deformations between the acquired medical images and thus affecting the registration accuracy.

[0003] Currently, based on fully supervised medical image registration algorithms, the generation of a gold standard deformation field by using traditional registration methods is used to guide the training of deep learning networks. Although the registration efficiency is improved, it is still difficult to exceed the performance of traditional algorithms in terms of registration accuracy. Unsupervised medical image registration methods mainly include single-stage registration methods, cascaded network registration methods, and pyramid registration methods. Different from fully supervised methods, these unsupervised methods remove the guidance of the gold standard deformation field and use a spatial transformation network (STN) to perform grid sampling on the deformation field generated by the registration network to achieve image registration.

[0004] The single-stage method performs poorly in processing medical images with complex deformations. The cascaded network method alleviates the registration problem of complex deformations to a certain extent by cascading multiple registration networks, but at the same time increases the video memory overhead. The pyramid registration method generates multi-scale features through a two-stream encoder, generates deformation sub-fields at different levels in the decoding stage, and finally fuses these deformation sub-fields, effectively alleviating the registration challenges under complex deformations. However, the errors of different deformation sub-fields may gradually propagate with the progress of hierarchical fusion, forming cumulative errors.

[0005] The registration method based on weak supervision guides the training of the registration network through the segmentation labels of anatomical structures, making full use of the corresponding relationships of anatomical structures, thereby effectively improving the registration performance. Some existing algorithms attempt to jointly optimize the training of the segmentation network and the pyramid registration network, which can not only complete the medical image registration task but also perform the medical image segmentation task. However, these algorithms usually fail to distinguish the primary and secondary tasks, resulting in suboptimal registration accuracy. Currently, there are also some algorithms that have achieved the quantification of the uncertainty of the registration results and used this to guide the training of the registration network. However, these methods usually only optimize the registration parameters by weighting the mean square error loss, which may affect the final registration accuracy. Summary of the Invention

[0006] An object of the present invention is to provide an image registration and segmentation joint optimization method, system, device and medium for the deficiencies of the above-mentioned existing technologies to solve the problems in the existing technologies.

[0007] The present invention specifically provides the following technical solutions: Obtain the floating image and its label and the fixed image and its label of the original medical image; Construct an image registration and segmentation joint optimization model, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator; Process the floating image and the fixed image through the encoder to obtain a hierarchical image feature map. Input the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and the final segmentation result. Input the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation subfield, and perform a deformation fusion operation to finally generate a deformation field. Based on this deformation field, use the spatial transformation network to perform a spatial transformation on the floating image and its label to obtain the registered image and its label; Input the fixed image and the registered image into the uncertainty estimator to predict the uncertainty of the registration. And calculate the registration network loss function of the image registration and segmentation joint optimization model and the loss function of the uncertainty estimator through the uncertainty of the registration, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label; According to the registration network loss function and the loss function of the uncertainty estimator, use the Adam optimizer to update the model parameters to obtain the updated image registration and segmentation joint optimization model.

[0008] Preferably, the obtaining of the floating image and its label and the fixed image and its label of the original medical image is specifically: Obtain the original medical image; Adopt a dynamic pairing strategy. For the single-modal data in the original medical images, randomly select two images for pairing. For the multi-temporal data in the original medical images, select images from different time frames for pairing to obtain the floating images and their labels, and the fixed images and their labels.

[0009] Preferably, the construction of the joint optimization model for image registration and segmentation is specifically as follows: Construct a shared-weight encoder, where each stage of the shared-weight encoder consists of a convolutional module and a downsampling module; among them, the convolutional module consists of a convolutional layer with a stride of 1, an instance normalization layer, and an activation function layer; the downsampling module consists of a convolutional layer with a stride of 2, an instance normalization layer, and an activation function. Construct a segmentation decoder, where the segmentation decoder includes a deformable convolutional module and a normal convolutional layer. The deformable convolutional module dynamically adjusts the positions of the sampling points by introducing learnable offsets to learn the features of different anatomical structures. In the nth layer, a segmentation label estimation module is added to predict the segmentation label, and the segmentation label estimation module consists of a convolutional layer and a Softmax activation function layer. Construct a registration decoder, where each stage of the registration decoder includes a fusion module, a deformation field estimation module, and an upsampling layer. The fusion module consists of a convolutional module and a deformable convolutional layer; the deformation field estimation module is a convolutional layer; the convolutional module consists of a convolutional layer, an instance normalization layer, and an activation function layer. Construct an uncertainty estimator, where the uncertainty estimator consists of a feedforward neural network module and a convolutional layer, and the feedforward neural network module consists of two fully connected layers and an activation function layer.

[0010] Preferably, the calculation of the registration network loss function of the joint optimization model for image registration and segmentation is specifically as follows: Obtain the fixed image and the uncertainty-weighted similarity loss with the registered image ; obtain the smooth regularization constraint loss of the deformation field ; obtain the predicted floating image segmentation label map and the smooth regularization constraint loss of the deformation field ; obtain the predicted floating image segmentation label map , the predicted fixed image segmentation label map and the floating image ground truth label , the fixed image ground truth label of the segmentation loss ; obtain the fusion loss among all losses ; Calculate the registration network loss function of the joint optimization model for image registration and segmentation through the sum of the uncertainty-weighted similarity loss, the smooth regularization constraint loss, the segmentation loss, and the fusion loss ; wherein is the registration uncertainty, indicating to stop the gradient calculation for the registration uncertainty and represents a composite operation implemented using a spatial transformation network.

[0011] Preferably, the specific expression of the segmentation loss is: ; wherein L FocalDice the loss consists of a focal loss and a Soft-Dice loss and the specific expression is: .

[0012] Preferably, the specific expression of the fusion loss is: ; wherein is the label after the registration of the floating image.

[0013] Preferably, the loss function of the uncertainty estimator is the negative log-likelihood loss of the exponential .

[0014] The present invention provides a system for jointly optimizing image registration and segmentation, comprising: an image acquisition module for acquiring a floating image and its label and a fixed image and its label of an original medical image; a model construction module for constructing a joint optimization model for image registration and segmentation, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator; an image registration module for processing the floating image and the fixed image through the encoder to obtain a hierarchical image feature map, inputting the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and a final segmentation result, inputting the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation sub-field, and performing a deformation fusion operation to finally generate a deformation field, and based on this deformation field, using a spatial transformation network to perform a spatial transformation on the floating image and its label to obtain a registered image and its label; the registered image and its label A loss function calculation module, which is used to input a fixed image and a registered image into an uncertainty estimator to predict the uncertainty of registration, and calculate the registration network loss function of the image registration and segmentation joint optimization model and the loss function of the uncertainty estimator through the registration uncertainty, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label. A model optimization module, which updates the model parameters by using an Adam optimizer according to the registration network loss function and the loss function of the uncertainty estimator, and obtains an updated image registration and segmentation joint optimization model.

[0015] The present invention provides a computer device, including a memory and a processor. A program is stored in the memory. When the program is executed by the processor, the processor executes the steps of the above-mentioned image registration and segmentation joint optimization method.

[0016] The present invention provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image registration and segmentation joint optimization method are implemented.

[0017] Compared with the prior art, the present invention has the following remarkable advantages: By constructing an image registration and segmentation joint optimization model, in the process of predicting the deformation field, the floating image and the fixed image of the original image are processed by the model to extract hierarchical image feature maps and perform segmentation. The deformation sub-field is generated by combining the hierarchical image feature maps and the segmentation feature maps, thereby reducing the deformation sub-field error to obtain the registered image and its label. At the same time, the uncertainty of the generated segmentation label result is estimated, the corresponding loss function is calculated, and the model parameters are optimized, so that the finally generated registration result is more reliable, closer to the fixed image and its label, and has better effects. Therefore, the model proposed by the present invention can improve the accuracy and reliability of medical image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of an image registration and segmentation joint optimization method provided by the present invention; Figure 2 It is a network framework diagram of an image registration and segmentation joint optimization model guided by uncertainty provided by the present invention; Figure 3 It is a framework diagram of a shared-weight encoder provided by the present invention; Figure 4 It is a framework diagram of a segmentation decoder provided by the present invention; Figure 5 It is a framework diagram of a registration decoder provided by the present invention; Figure 6The framework diagram of the uncertainty estimator provided by the present invention; Figure 7 The diagram of the registration qualitative analysis result provided by the present invention; wherein Figure 7 (a1) to Figure 7 (a5) of is the diagram of the registration qualitative analysis result without the uncertainty estimator, Figure 7 (b1) to Figure 7 (b5) of is the diagram of the registration qualitative analysis result with the uncertainty estimator, wherein Figure 7 (a1) of and Figure 7 (b1) of are floating images, Figure 7 (a2) of and Figure 7 (b2) of are registered images, Figure 7 (a3) of and Figure 7 (b3) of are fixed images, Figure 7 (a4) of and Figure 7 (b4) of are deformation fields, Figure 7 (a5) of and Figure 7 (b5) of are uncertainty maps, and Figure 7 (a5) of has no image because it is generated without the uncertainty estimator. Detailed implementation manners

[0019] Next, in combination with the accompanying drawings in the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0020] As Figure 1 shown, a method for jointly optimizing image registration and segmentation in this embodiment includes the following steps: Step S1: Obtain the original medical images, and adopt a dynamic pairing strategy: for single-modal data, randomly select two images for pairing, and for multi-temporal data, select images of different time frames for pairing, and finally obtain the floating image and its label and the fixed image and its label.

[0021] Obtaining the original medical images further includes resampling, cropping, and normalizing the original medical images and labels to obtain normalized images of a specified size and their corresponding segmentation labels, and dividing the preprocessed images into a training set, a validation set, and a test set according to a ratio.

[0022] In this embodiment, 4D cardiac cine MRI images of 150 patients from the ACDC dataset are selected as the original medical images. According to the dynamic pairing strategy, the ED (end-diastolic) frame image and the ES (end-systolic) frame image are selected as the floating image and the fixed image respectively. The paired images and corresponding labels are resampled with a voxel spacing of mm, cropped to the size of centered on the heart. Finally, max-min normalization is used to adjust the range of image voxel values to . The preprocessed image data is divided into 9:1:5, with 90 cases as the training set, 10 cases as the validation set, and 50 cases as the test set.

[0023] Step S2: As shown in Figure 2 , construct a joint optimization model for image registration and segmentation, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator.

[0024] Construct the encoder with shared weights. As shown in Figure 3 , each stage of the encoder with shared weights consists of a convolutional module and a downsampling module; among them, the convolutional module consists of a convolutional layer with a stride of 1, an instance normalization layer, and an activation function layer; the downsampling module consists of a convolutional layer with a stride of 2, an instance normalization layer, and an activation function.

[0025] Construct the segmentation decoder. As shown in Figure 4 , the segmentation decoder contains a deformable convolutional module and ordinary convolutional layers. The deformable convolutional module dynamically adjusts the positions of sampling points by introducing learnable offsets to learn the features of different anatomical structures. The ordinary convolutional layers have the same structure as the convolutional modules in the encoder with shared weights, and a segmentation label estimation module is added to the nth layer to predict the segmentation label. The segmentation label estimation module consists of a convolutional layer and a Softmax activation function layer.

[0026] Construct the registration decoder. As shown in Figure 5 , each stage of the registration decoder contains a fusion module, a deformation field estimation module, and an upsampling layer. The fusion module consists of a convolutional module and a deformable convolutional layer; the deformation field estimation module is a convolutional layer; the convolutional module consists of a convolutional layer, an instance normalization layer, and an activation function layer.

[0027] Construct the uncertainty estimator. As shown in Figure 6 , the uncertainty estimator consists of a feedforward neural network module and a convolutional layer. The feedforward neural network module consists of two fully connected layers and an activation function layer, and a Dropout layer is used for the output of the feedforward neural network module to control overfitting of the model.

[0028] In this implementation, as shown in Figure 3As shown, the number of stages n is set to 4, and the convolutional kernel size of all convolutional layers of the shared-weight encoder is , the stride of the convolutional layer of the convolutional module is set to 1, while the stride of the convolutional layer of the downsampling module is set to 2, and the parameter of the activation function is set to 0.1. The floating image and the fixed image are input into the first stage, and the output of the first stage is , the output of the second stage is , the output of the third stage is , the output of the fourth stage is , the output of each stage is used as the input of the next stage, and the output of the fourth stage is used as the input of the segmentation decoder.

[0029] The convolutional kernel of the convolutional layer of the deformable convolutional module and the ordinary convolutional module in the segmentation decoder is , the stride is 1, and the parameters of all activation functions are set to 0.1.

[0030] All convolutional module parameters and deformable convolutional layers in the fusion module of the registration decoder are the same as those of the segmentation decoder. The upsampling layer is implemented using trilinear interpolation, and the convolutional kernel of the convolutional layer of the deformation field estimation module is , the stride is 1.

[0031] The implementation of the two fully connected layers of the feedforward neural network in the uncertainty estimator is a convolutional layer with a convolutional kernel of and a stride of 1. The last convolutional layer is the same as the convolutional layer of the segmentation decoder.

[0032] Step S3: Process the floating image and the fixed image through the encoder to obtain a hierarchical image feature map. Input the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and the final segmentation result. Input the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation sub-field, and perform a deformation fusion operation to finally generate a deformation field. Based on this deformation field, use the spatial transformation network to perform a spatial transformation on the floating image and its label to obtain the registered image and its label.

[0033] As Figure 2 shown, the specific steps of processing the floating image and the fixed image are as follows: Step 3.1: Input the floating image and the fixed image into the encoder with shared weights to obtain the hierarchical features of the floating image and the hierarchical features of the fixed image.

[0034] Step 3.2: Input and obtained in Step 3.1 into the segmentation decoder respectively to generate a hierarchical segmentation feature map and The segmentation feature map of the last layer and are input into the segmentation label estimation module to obtain the fixed image and the floating image of the segmentation label result and . As Figure 4 shown, in one embodiment is 4, that is, it includes 4 deformable convolution modules and ordinary convolution layers, and each deformable convolution module and ordinary convolution layer obtains a hierarchical segmentation feature map through upsampling.

[0035] Step 3.3: The image hierarchical features obtained in Step 3.1 and , the hierarchical segmentation feature maps obtained in Step 3.2 and and the initial deformation field are input into the registration decoder to obtain the registration deformation sub-field , which is the final deformation field .

[0036] Specifically, as Figure 5 shown, in this implementation, the number of stages n is set to 4. Therefore the initial value of is 4, and then it decreases by 1 stage by stage. In each stage, through the fusion module, , , and are fused, input into the fusion module of the next stage through upsampling, and the registration deformation sub-field is generated through the deformation field estimator module. When the fusion module fuses, for and , they are concatenated in the channel dimension and then input into a convolution module. For and , they are concatenated in the channel dimension and then input into a convolution module. And the input features and the outputs of the two convolution modules are concatenated in the channel dimension, and the concatenated result is processed successively through a convolution module and a deformable convolution to finally output the features. Since the initial deformation field has no deformation sub-field of the previous stage, therefore is a all-zero matrix and has the same dimension as . The deformation field of the subsequent stages is obtained by the deformation fusion operation and upsampling Up of the generated in the previous stage. The specific expression is as follows: ; where, Represents a composite operation implemented using the Spatial Transformer Network (STN).

[0037] Step 3.4: Use the Spatial Transformer Network to perform spatial transformation on the floating image and the corresponding label to obtain the registered image and its label .

[0038] Step S4: Input the fixed image and the registered image into the uncertainty estimator to predict the registration uncertainty, and calculate the registration network loss function of the image registration and segmentation joint optimization model and the loss function of the uncertainty estimator based on the registration uncertainty, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label.

[0039] As Figure 2 shown, the registration network loss function of the image registration and segmentation joint optimization model is specifically , and the calculation process is as follows: Obtain the uncertainty-weighted similarity loss between the fixed image and the registered image ; Obtain the smooth regularization constraint loss of the deformation field ; Obtain the predicted floating image segmentation label map , the predicted fixed image segmentation label map and the floating image ground truth label , the fixed image ground truth label segmentation loss ; Obtain the fusion loss between all losses; Calculate the registration network loss function of the image registration and segmentation joint optimization model through the sum of the uncertainty-weighted similarity loss, the smooth regularization constraint loss, the segmentation loss, and the fusion loss ; where is the registration uncertainty, represents stopping the gradient calculation for the registration uncertainty , and represents a composite operation implemented using the Spatial Transformer Network.

[0040] The segmentation loss is calculated based on the following formula: ; The FocalDice loss is composed of the focal loss and the Soft-Dice loss : ; The fusion loss Calculated based on the following formula: ; Total registration network loss Calculated based on the following formula: ; is the weight of the smooth regularization constraint loss, is the weight of the segmentation loss, is the weight of the fusion loss.

[0041] Loss function of the uncertainty estimator is Negative log-likelihood loss of the exponent .

[0042] In this implementation, during the training phase, the paired images and labels are input into the registration network, and the total registration network loss and the loss of the uncertainty estimator are used for optimization training for 150 epochs. The hyperparameters of the loss function , , , are set to 0.01, 0.5, 1, and 0.5. The initial learning rates of the registration network and the uncertainty estimator are both set to 0.0001, and are adaptively updated according to the following formula: ; By applying the FocalDice loss to the generated segmentation label results, not only can the class imbalance problem of different anatomical structure labels be alleviated, but also the positive effect of the hierarchical segmentation feature map in the deformation subfield generation process can be promoted.

[0043] Step S5: According to the registration network loss function and the uncertainty estimator loss function, use the Adam optimizer to update the model parameters to obtain an updated joint optimization model of image registration and segmentation.

[0044] Use the Adam optimizer to update the network parameters, and use the validation set to evaluate the Dice metric to select and save the optimal model parameters. In the test phase, the paired images , and the corresponding labels , in the test set are input into the registration network and the uncertainty estimator trained in step S4 to obtain the registered image and the label . Subsequently, calculate and The Dice Similarity Coefficient (DSC), Average Symmetric Surface Distance (ASSD), 95% Hausdorff Distance (HD95), and the percentage of folded voxels in the deformation field ) were quantitatively analyzed, and finally, the obtained registration results and uncertainty quantification images were used for qualitative analysis.

[0045] Table 1 Quantitative analysis results obtained after registering the test set based on the trained registration network

[0046] From Table 1 and Figure 7 it can be seen that the present invention uses an uncertainty estimator to generate uncertainty quantification images, and by weighting the similarity loss of registration, improves the accuracy of the registered images, making them closer to the fixed image and its label, with better effects. Specifically, the registration results show a higher DSC, a higher overlap degree of the labeled area, and smaller HD95 and ASSD values, indicating that the structural contours of the labeled area are more consistent with the fixed image. In addition, the deformation field has fewer folded voxels, achieving smoother deformation, and the visualized heat map of uncertainty is evenly distributed, further verifying the high reliability of the registration results.

[0047] This method takes the registration task as the main line and the segmentation task as the auxiliary, and uses a pyramid structure for registration. The registration decoder effectively reduces the error of the deformation sub-field at each level by fusing the segmentation features extracted by the segmentation decoder, thus alleviating the complex deformation problem between the input images and ensuring that the registration deformation field conforms to the structural correspondence relationship between the images. In addition, the method introduces an uncertainty estimator to predict the uncertainty during the registration process, and uses this uncertainty to weight the registration similarity loss, further guiding the optimization of the registration decoder, thereby improving the reliability of the registration results.

[0048] The present invention proposes a joint optimization system for image registration and segmentation, including: an image acquisition module, a model construction module, an image registration module, a loss function calculation module, and a model optimization module.

[0049] Among them, the image acquisition module is used to acquire the floating image of the original medical image and its label, and the fixed image and its label; the model construction module is used to construct a joint optimization model for image registration and segmentation, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator; the image registration module is used to process the floating image and the fixed image through the encoder to obtain a hierarchical image feature map, input the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and the final segmentation result, input the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation sub-field, and perform a deformation fusion operation to finally generate a deformation field. Based on this deformation field, use a spatial transformation network to perform a spatial transformation on the floating image and its label to obtain the registered image and its label; the loss function calculation module is used to input the fixed image and the registered image into the uncertainty estimator to predict the uncertainty of the registration, and calculate the registration network loss function of the joint optimization model for image registration and segmentation and the loss function of the uncertainty estimator through the uncertainty of the registration, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label; according to the registration network loss function and the loss function of the uncertainty estimator, use the Adam optimizer to update the model parameters to obtain an updated joint optimization model for image registration and segmentation.

[0050] The present invention also provides a computer device, including a memory and a processor. When a program stored in the memory is executed by the processor, the processor executes the steps of an image registration and segmentation joint optimization method.

[0051] According to the disclosed embodiments, the computer device can communicate with one or more external devices (such as a keyboard, a pointing device, Bluetooth communication, etc.), or communicate with any device that enables the computing device to communicate with one or more other computing devices (such as a router, a demodulator, etc.).

[0052] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, which should all be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for jointly optimizing image registration and segmentation, characterized in that, Including: Obtaining a floating image of the original medical image and its label, and a fixed image and its label; Constructing a joint optimization model for image registration and segmentation, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator; Processing the floating image and the fixed image through the encoder to obtain a hierarchical image feature map, inputting the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and a final segmentation result, inputting the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation sub-field, and performing a deformation fusion operation to finally generate a deformation field. Based on this deformation field, using a spatial transformation network to perform a spatial transformation on the floating image and its label to obtain a registered image and its label; Inputting the fixed image and the registered image into the uncertainty estimator to predict the uncertainty of registration, and calculating the registration network loss function of the joint optimization model for image registration and segmentation and the loss function of the uncertainty estimator through the uncertainty of registration, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label; Updating the model parameters by using an Adam optimizer according to the registration network loss function and the loss function of the uncertainty estimator to obtain an updated joint optimization model for image registration and segmentation.

2. The joint optimization method for image registration and segmentation according to claim 1, characterized in that The obtaining of the floating image of the original medical image and its label, and the fixed image and its label is specifically: Obtaining the original medical image; Adopting a dynamic pairing strategy, randomly selecting two images for pairing for the single-modal data in the original medical image, and selecting images of different time frames for pairing for the multi-temporal data in the original medical image to obtain the floating image and its label and the fixed image and its label.

3. The joint optimization method for image registration and segmentation according to claim 1, characterized in that The constructing of the joint optimization model for image registration and segmentation is specifically: Constructing an encoder with shared weights, each stage of the encoder with shared weights consisting of a convolutional module and a downsampling module; wherein, the convolutional module consists of a convolutional layer with a stride of 1, an instance normalization layer, and an activation function layer; the downsampling module consists of a convolutional layer with a stride of 2, an instance normalization layer, and an activation function; Constructing a segmentation decoder, the segmentation decoder including a deformable convolutional module and a common convolutional layer, the deformable convolutional module dynamically adjusting the positions of sampling points by introducing learnable offsets to learn the features of different anatomical structures, and adding a segmentation label estimation module to the nth layer to predict the segmentation label, the segmentation label estimation module consisting of a convolutional layer and a Softmax activation function layer; Constructing a registration decoder, each stage of the registration decoder including a fusion module, a deformation field estimation module, and an upsampling layer, the fusion module consisting of a convolutional module and a deformable convolutional layer; the deformation field estimation module being a convolutional layer; the convolutional module consisting of a convolutional layer, an instance normalization layer, and an activation function layer; Constructing an uncertainty estimator, the uncertainty estimator consisting of a feedforward neural network module and a convolutional layer, the feedforward neural network module consisting of two fully connected layers and an activation function layer.

4. The joint optimization method for image registration and segmentation according to claim 1, wherein The calculating of the registration network loss function of the joint optimization model for image registration and segmentation is specifically: Obtain a fixed image and the registered image with uncertainty weighted similarity loss ; Obtain a deformation field with smooth regularization constraint loss ; Obtain the predicted floating image segmentation label map , the predicted fixed image segmentation label map and the floating image ground truth , the fixed image ground truth with segmentation loss ; Obtain the fusion loss between all losses ; Calculate the registration network loss function of the image registration and segmentation joint optimization model by the sum of the uncertainty weighted similarity loss, the smooth regularization constraint loss, the segmentation loss, and the fusion loss ; wherein is the registration uncertainty, indicating to stop gradient calculation for the registration uncertainty and indicating a composite operation implemented using a spatial transformation network.

5. The combined optimization method for image registration and segmentation according to claim 4, wherein The segmentation loss has the following specific expression: ； Among them, L FocalDice the loss is composed of focal loss and Soft-Dice loss and the specific expression is: 。 6. The joint optimization method for image registration and segmentation according to claim 5, wherein The fusion loss has the following specific expression: ； Among them, is the label after floating image registration.

7. The combined optimization method for image registration and segmentation according to claim 1, wherein The loss function of the uncertainty estimator is the negative log-likelihood loss of the exponent .

8. An image registration and segmentation joint optimization system, characterized in that, Including: An image acquisition module, configured to acquire a floating image and its label of an original medical image, and a fixed image and its label; A model construction module, configured to construct a joint optimization model for image registration and segmentation, including an encoder with shared weights, a segmentation decoder, a registration decoder, and an uncertainty estimator; An image registration module, configured to process the floating image and the fixed image through the encoder to obtain a hierarchical image feature map, input the hierarchical image feature map into the segmentation decoder to obtain a segmentation feature map and a final segmentation result, input the hierarchical image feature map and the segmentation feature map into the registration decoder to obtain a registration deformation sub-field, and perform a deformation fusion operation to finally generate a deformation field. Based on this deformation field, use a spatial transformation network to perform a spatial transformation on the floating image and its label to obtain a registered image and its label; A loss function calculation module, configured to input the fixed image and the registered image into the uncertainty estimator to predict the uncertainty of registration, and calculate a registration network loss function of the joint optimization model for image registration and segmentation and a loss function of the uncertainty estimator through the uncertainty of registration, the floating image and its label, the fixed image and its label, the final segmentation result, and the registered image and its label; A model optimization module, configured to update model parameters by using an Adam optimizer according to the registration network loss function and the loss function of the uncertainty estimator to obtain an updated joint optimization model for image registration and segmentation; 9. A computer device, characterized in that, It includes a memory and a processor. A program is stored in the memory. When the program is executed by the processor, the processor is caused to execute the steps of an image registration and segmentation joint optimization method according to any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of an image registration and segmentation joint optimization method according to any one of claims 1 to 7.

Images