Image registration segmentation method and system

Abstract

The application discloses an image registration segmentation method and system, comprising the following steps: S1, providing a segmentation registration joint network model; S2, pre-training a registration network and a segmentation network; S3, parameter optimization is performed on the registration network; S4, parameter optimization is performed on the segmentation network; S5, steps S2 and S3 are repeated alternately until a trained segmentation registration joint network model is finally obtained, and the model is used for image segmentation registration. The image registration segmentation method and system provided by the application start from two points of global anatomical structure contour and local structure information, on one hand, the constraint of organ labels is used to guide image registration, and the registration and the segmentation network are combined, and the label guidance of all images can be completed only by a small amount of manual annotation; on the other hand, msMIND is proposed to constrain the local structure of images to guide image registration, and high-precision multi-modal medical image registration can be realized.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and in particular to an image registration and segmentation method and system. Background Technology

[0002] Multimodal medical images have significant differences in grayscale distribution, which leads to low registration accuracy and slow computation speed in traditional multimodal medical image registration.

[0003] Deep learning registration methods automatically learn features between multimodal images from large-sample medical images, showing great potential to improve the accuracy of multimodal medical image registration. However, existing deep learning registration methods for multimodal images suffer from the following problems:

[0004] 1. Large-scale pixel-by-pixel annotation results are required, but high-quality annotation data is difficult to obtain, which restricts the improvement of registration accuracy.

[0005] 2. Existing deep learning registration methods for multimodal medical images are difficult to perceive large deformation regions between multimodal medical images, which limits the registration accuracy of large deformation regions.

[0006] Therefore, it is now necessary to improve existing technologies to provide more reliable solutions. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an image registration and segmentation method and system that addresses the shortcomings of the prior art.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an image registration and segmentation method, comprising the following steps:

[0009] S1. Provide a joint segmentation and registration network model, which includes a registration network and a segmentation network;

[0010] S2. The registration network is trained in an unsupervised manner using all M pairs of CT-MR image pairs as pre-training, and the CT and MR segmentation networks are trained in a recurrent self-training manner using N pairs of labeled CT-MR image pairs as pre-training; wherein, the CT-MR image pairs include CT images and MR images.

[0011] S3, move all I mr and I ct The deformation field is sequentially input into the registration network. Through deformation field to I mr Deformation generation predicts images use with I ct , With L ctThe structural similarity is used to guide the parameter optimization of the registration network;

[0012] Among them, I mr and I CT These represent MR and CT images, respectively; L represents the image segmentation label, where... It's a manual label from the CT scan. The labels are calculated using the CT algorithm;

[0013] in, It is a manually labeled tag, serving as the gold standard; It is the segmentation result of the segmentation network on unlabeled data; It is an artificial label for MR. These are the CT labels obtained by the algorithm; L seg mr These are the MR tags obtained by the algorithm;

[0014] in, Artificial labels representing MR images, Artificial labels representing CT images, This indicates the segmentation result of the MR image; This indicates the segmentation results of a CT image;

[0015] Among them, L mr L represents mr express and The set of φ, where φ represents the image deformation field. L represents mr φ represents L mr A collection of image distortions, L ct Lct represents L gt ct and L seg ct A collection;

[0016] S4, move all I mr and I ct The data is sequentially input into a segmentation network to generate segmentation results. and When manual labeling exists, utilize With L gt ct / mr The structural similarity is used to optimize the segmentation network parameters; when no manual labels are available, the segmentation results are obtained by applying a registration deformation field to the floating image. and The same characteristics constrain the parameter optimization of the segmentation network;

[0017] S5. Repeat steps S2 and S3 alternately until a trained segmentation and registration joint network model is obtained. Use this model for image segmentation and registration.

[0018] Preferably, the segmentation network is a Seg-Net segmentation network, and the Seg-Net segmentation network uses a floating image I. mr Fixed Image I ct and corresponding manual labels and As input, it is used to optimize the segmentation and registration joint network model.

[0019] Preferably, the registration network is a Reg-Net registration network, and the Reg-Net registration network uses a floating image I. mr Fixed Image I ct As input, the corresponding deformation field is output. The optimal fit is obtained through spatial transformation of the deformation field. Output the segmented label image or

[0020] Preferably, the segmentation network Seg-Net receives the input image I mr Or I ct It predicts the contours of the organs to be segmented, and uses the characteristic that the predicted contours should be the same as the true contours to adjust the segmentation network parameters θ. seg Optimize, θ seg The calculation formula is as follows:

[0021] θ seg =arg min(E seg (L seg ,L gt (2)

[0022] Among them, E seg L represents the loss function for the segmentation. seg L represents the segmentation result of the image. gt This indicates the true label of the image.

[0023] Preferably, in the registration network Reg-Net, the msMIND descriptor is used to constrain the local structure of the image to guide image registration.

[0024] Preferably, during training, when artificial labels exist, the following CT-MR labeled loss function set is used for parameter optimization; when no artificial labels exist, the following CT-MR unlabeled loss function set is used for parameter optimization; the specific formula is as follows:

[0025]

[0026]

[0027] Among them, E smoothIt is the regularization term that maintains the smoothness of the deformation field, E msMIND E represents a local measure describing the similarity between CT and MR images. dice This represents a global measure describing the similarity of organizational labels in an image.

[0028] Preferably, E dice Replace with E softdice E softdice For E dice Improvements, E softdice The calculation formula is as follows:

[0029]

[0030] Where k represents a segmentation label, x is a voxel position, and S k and These are the two segmentation results to be compared.

[0031] Preferably, in the registration network Reg-Net, any local patch centered at y and of size m*m*m is selected from the multi-scale feature map S(I,x), all voxels in the patch are transformed into three-dimensional polar coordinates, and the local patch is divided into 32 bins according to 8 angles, 2 radial intervals, and 2 heights, with the number of bins being n.

[0032] The feature responses are ultimately aggregated to the maximum extent within the bins surrounding each local patch. Average pooling is then performed on the feature values ​​represented by all voxels within each bin to obtain 32 values ​​associated with the center y, representing the feature descriptor msMIND at that point y. By sequentially sliding the local patch window across the multi-scale feature map S(I,x), the local feature response msMIND(I,x,y) at any point on the feature map can be obtained. The calculation formula is as follows:

[0033]

[0034] The formula for the multi-scale autocorrelation feature map S(I,x) is as follows:

[0035]

[0036] Among them, D p (I,x i ,x j ) represents two patches centered at point x and spaced apart by a distance p: I p(xi) and I p(xj) The sum of the root mean square deviations, where α and β are both decimals between 0 and 1, and α + β = 1:

[0037]

[0038] V(I,x) is an estimate of the noise, defined as the average distance across all patches:

[0039]

[0040] The present invention also provides an image registration and segmentation system, which performs image registration and segmentation using the method described above.

[0041] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0042] The beneficial effects of this invention are:

[0043] The image registration and segmentation method and system provided by this invention start from two aspects: global anatomical structure contours and local structural information. On the one hand, it uses organ labels to constrain the registration to guide image registration and combines the registration with the segmentation network, so that the label guidance of all images can be completed with only a small amount of manual annotation. On the other hand, it proposes msMIND (an improved modality-independent image structure descriptor) to constrain the local structure of the image to guide image registration, which can achieve high-precision multimodal medical image registration. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the structure of the segmentation and registration joint network model of the present invention;

[0045] Figure 2 This is a schematic diagram of the Seg-Net network structure of the present invention;

[0046] Figure 3 This is a detailed diagram of the additional scSE-block structure in this invention;

[0047] Figure 4 This is a schematic diagram of the Reg-Nett network structure of the present invention;

[0048] Figure 5 Here is a detailed structural diagram of the spatial attention module Sa-block in this invention;

[0049] Figure 6 This is a schematic diagram illustrating the principle of the MIND descriptor.

[0050] Figure 7 This is a schematic diagram of multi-scale autocorrelation feature map acquisition in this invention;

[0051] Figure 8 This refers to the process in this invention of extracting local support windows from a multi-scale feature map and converting them to two-dimensional polar coordinates. Detailed Implementation

[0052] The present invention will be further described in detail below with reference to embodiments, so that those skilled in the art can implement it based on the description.

[0053] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0054] Example 1

[0055] An image registration and segmentation method includes the following steps:

[0056] S1. Provide a joint segmentation and registration network model, which includes a registration network and a segmentation network;

[0057] S2. The registration network is trained in an unsupervised manner using all M pairs of CT-MR image pairs as pre-training, and the CT and MR segmentation networks are trained in a recurrent self-training manner using N pairs of labeled CT-MR image pairs as pre-training; wherein, the CT-MR image pairs include CT images and MR images.

[0058] S3, move all I mr and I ct The deformation field is sequentially input into the registration network. Through deformation field to I mr Deformation generation predicts images use with I ct , With L ct The structural similarity is used to guide the parameter optimization of the registration network;

[0059] Among them, I mr and I CT These represent MR and CT images, respectively; L represents the image segmentation label, where... It's a manual label from the CT scan. The labels are calculated using the CT algorithm;

[0060] in, It is a manually labeled tag, serving as the gold standard; It is the segmentation result of the segmentation network on unlabeled data; It is an artificial label for MR. These are the CT labels obtained by the algorithm; L seg mr These are the MR tags obtained by the algorithm;

[0061] in, Artificial labels representing MR images, Artificial labels representing CT images, This indicates the segmentation result of the MR image; This indicates the segmentation results of a CT image;

[0062] Among them, L mr L represents mr express and The set of φ, where φ represents the image deformation field. L represents mr φ represents L mr A collection of image distortions, L ct Lct represents L gt ct and L seg ct A collection;

[0063] S4, move all I mr and I ct The data is sequentially input into a segmentation network to generate segmentation results. and When manual labeling exists, utilize With L gt ct / mr The structural similarity is used to optimize the segmentation network parameters; when no manual labels are available, the segmentation results are obtained by applying a registration deformation field to the floating image. and The same characteristics constrain the parameter optimization of the segmentation network;

[0064] S5. Repeat steps S2 and S3 alternately until a trained segmentation and registration joint network model is obtained. Use this model for image segmentation and registration.

[0065] The joint segmentation and registration network framework, along with its segmentation and registration subnets, can each accomplish their respective tasks but still suffer from several problems. The registration network requires a large number of image labels as global constraints to improve registration accuracy, but obtaining a large number of accurate labels is difficult. Similarly, the segmentation network needs a large number of labels as supervisory information to constrain segmentation, but also faces the challenge of obtaining labels. Therefore, this invention concludes that labels are a crucial factor affecting the results of these two specific tasks. To address these issues, this invention combines segmentation and registration within the joint segmentation and registration framework, allowing each to provide the necessary label information, thereby improving registration accuracy and reducing the label dependence of segmentation.

[0066] The framework of this invention includes two steps: segmentation and registration. A segmentation step follows a registration step, forming a complete training process. That is, while training one network, the other network remains fixed. Through iterative training, the segmentation and registration networks mutually promote and optimize each other. It is worth noting that in this invention, only the networks used in the first two steps (Step 2 and Step 3) are pre-trained results; in subsequent alternating training processes, the networks used are the optimized results from the previous step. The manually labeled input networks in this invention... The number N is much smaller than the total number of images (I mr &I ct The quantity M. When segmenting networks process data without human labels, this invention can consider the deformation field... It represents and The true deformation relationship between the two is known. Therefore, when the CT and MR segmentation networks are continuously optimized to make the results best match the deformation relationship of the deformation field, this invention can be considered to have achieved better results in the segmentation of both CT and MR images.

[0067] Since segmentation and registration tasks have different task characteristics and model complexities, directly optimizing the entire framework at the beginning of training may not be very effective. Therefore, this invention does not choose to train jointly from scratch, but first pre-trains the segmentation and registration subnetworks until both subnetworks reach stable performance. This invention assumes that N out of M images have artificial labels (N << M). For segmentation, this invention first trains the segmentation network on N labeled data using a recurrent self-training method, outputting a pre-trained segmentation model; for registration, this invention uses formula (1) to perform unsupervised training on all data, outputting a pre-trained registration model. After this, the joint segmentation and registration framework of this invention begins training by minimizing the loss function. The purpose of this is to ensure that the data entering the joint segmentation and registration network can be optimized in the correct direction at the beginning.

[0068] When N=M, the framework of this invention contains all the manually labeled data, representing the upper limit of the model's performance. Note that the pre-trained network is only used in the first iteration; subsequent iterations use the optimized network from the previous step.

[0069] 1. Multimodal registration

[0070] Multimodal registration is a registration problem used to deform a floating image so that the deformed result matches the structure of a fixed image. Its input is the floating image I. mr and fixed image I ct This network can predict the transformation relationship between two modal images and use deformation fields. This indicates that the deformation field Used for I mr Deformation generation predicts images Ideally with I ct They should have the same shape and structure; this invention optimizes the registration network based on this. By maximizing the similarity measure, the optimal network parameters θ are learned. reg To train the network, see formula (1).

[0071]

[0072] Among them, E reg It is a measure of image similarity, E smooth It is a regularization term that keeps the deformation field smooth.

[0073] Multimodal registration requires spatial alignment of images with different modalities. However, multimodal images exhibit significant differences in intensity and texture, making it difficult to obtain modality-independent feature representations. Therefore, designing modality-independent feature representations to eliminate the influence of modality differences on registration is of great importance. To address this issue, this invention attempts to represent modality-independent features from both image structure and image label perspectives. MIND, defined as a structure of local self-similarity representation, has already been applied in deep learning-based multimodal registration. This invention further enriches the local structural information and improves the robustness of the structural representation by multi-scale and polar coordinate transformation. In addition, label-driven methods are often used to bypass traditional intensity-based image similarity measures. However, label acquisition is difficult and often requires time-consuming and laborious annotation by professionals. Therefore, this invention uses a segmentation-registration joint method to generate pseudo-labels to achieve label-driven multimodal registration with fewer labels.

[0074] Unlike most current multimodal registration methods, this invention employs two modality-independent features simultaneously, constraining the registration from two aspects. This invention uses segmentation labels as a global constraint, learning voxel correspondences from higher-level label correspondences; and uses neighborhood structure as a local constraint, learning voxel correspondences from the surrounding structure correspondences of voxels. This ensures both the alignment of the region of interest boundary and the alignment of the organizational structures inside and outside the boundary. Therefore, segmentation labels are considered as a global constraint and added to the measure function, working together with local structure to constrain the registration. The specific loss function can be rewritten as follows:

[0075]

[0076] Among them, I mr Table MR image segmentation labels, I ct Segmentation labels for CT images Represents the deformation field;

[0077] Among them, E smooth It is the regularization term that maintains the smoothness of the deformation field, E msMIND E represents a local measure describing the similarity between CT and MR images. dice This represents a global measure describing the similarity of organizational labels in an image.

[0078] In this invention, the segmentation network is Seg-Net, and the Seg-Net uses a floating image I. mr Fixed Image I ct and corresponding manual labels and As input, it is used to optimize the joint segmentation and registration network model. The registration network is Reg-Net, which uses a floating image I... mr Fixed Image I ct As input, the corresponding deformation field is output. The optimal fit is obtained through spatial transformation of the deformation field. Output the segmented label image or

[0079] 2. Segmentation and Registration Joint Network Model Framework

[0080] Reference Figure 1 The joint network model structure for segmentation and registration is as follows: Reg-Net and Seg-Net are the registration and segmentation sub-networks used in this paper, respectively. The registration sub-network adopts MultiSsa_RNet, which takes floating and fixed images as input and outputs the corresponding deformation fields. This allows for deformation of floating images and their labels. The segmentation subnetwork uses a recurrently self-trained nnUNet, taking images of two modalities as input and outputting segmented labels for both modalities. The red dashed box illustrates the interdependence between segmentation and registration: the pseudo-labels generated by the segmentation network constrain the global contour of the registration; the deformation relationship generated by the registration network constrains the segmentation of the two modalities. Furthermore, the red dashed box also includes the msMIND cross-modal descriptor designed for registration in this invention, used to improve registration accuracy by leveraging local structural information. It is important to note that only the segmentation and registration network used in the first iteration is a pre-trained network; subsequent iterations use the optimized results from the previous step. Additionally, the manually generated labels input to the network in this invention... The number is much smaller than the total number of images (I mr &I ct )quantity.

[0081] Multimodal image registration requires establishing spatial voxel correspondences between a pair of moving and stationary images of different modalities. From a machine learning perspective, ground truth transformation relationships available for learning are unavailable; therefore, this invention selects labels as weakly supervised information to supervise registration. However, label information still requires annotation by professionals using anatomical knowledge, such as identifying identical organs, lesions, or other structures. Extensive manual annotation is time-consuming, labor-intensive, and often difficult to obtain. Therefore, this invention utilizes a combination of segmentation and registration to generate pseudo-labels, thereby improving the accuracy of both segmentation and registration, and ultimately achieving simultaneous completion of both tasks.

[0082] This invention proposes a multimodal joint segmentation and registration framework for mapping preoperative MR images to intraoperative CT images. The method generates multimodal image labels within the framework to guide image registration. The framework comprises two independent networks: a weakly supervised registration network and a semi-supervised segmentation network, which are simultaneously improved through alternating learning.

[0083] Given a pair of 3D images, let I mr Represents a floating image, I ct This represents a fixed image. Furthermore, this invention also provides a small number of N artificially labeled pairs within all M pairs of CT-MR image pairs. and (N << M), used to perform accurate supervision of segmentation and registration. This invention assumes that all image pairs are affine aligned, leaving only the nonlinear parts that need registration.

[0084] The overall network framework of this invention is as follows: Figure 1 As shown. The framework of this invention includes two sub-networks: segmentation and registration, namely the registration network Reg-Net and the segmentation network Seg-Net. From an overall perspective, the entire framework uses a floating image I... mr Fixed Image I ct And a small number of corresponding manual labels and As input, the optimized segmentation and registration model is used as output. For image registration, this invention employs the 3D registration network Reg-Net, taking both floating and fixed 3D images as input and outputting the corresponding deformation field. The optimal fit is obtained through spatial transformation of the deformation field. That is, for x represents any voxel in the image, R 3 This indicates that x is defined in a 3-dimensional space. with I ct (x) corresponds to similar anatomical structures in the same location, i.e. Deformation field between floating and fixed images It can be represented as a three-dimensional image with three channels, each representing a corresponding displacement component. For image segmentation, the 3D segmentation network Seg-Net uses image I... mr Or I ct Input is the segmented label image, output is the segmented label image. or

[0085] Given the characteristics of these two tasks, both focusing on common organ labels, their high correlation can enhance each other's performance. Segmentation can provide auxiliary anatomical information for registration, helping to find the correspondence between regions of interest; simultaneously, segmentation can utilize pseudo-labels provided by registration to increase the number of labels and thus improve segmentation accuracy. Therefore, this invention connects these two tasks in its architecture by designing a loss function that interacts with segmentation and registration, such as... Figure 1 The red dashed box indicates the training process for segmentation and registration. Detailed training steps and the loss function will be described later.

[0086] 3. Segment registration baseline network

[0087] 3.1 Segmentation Network

[0088] The purpose of a segmentation network is to isolate organ portions from medical images. Its input is a 3D image, and its output is the segmentation result of its region of interest. In the task of this invention, two segmentation networks with different parameters need to be trained for different modalities of images. The segmentation network receives input image I. mr Or I ct It predicts the contours of the organs to be segmented, and uses the characteristic that the predicted contours should be the same as the true contours to adjust the segmentation network parameters θ. seg Optimization can be performed, see formula (3).

[0089] θ seg =arg min(E seg (L seg ,L gt (3)

[0090] Among them, E seg This represents the loss function for the segmentation.

[0091] Where L seg L represents the segmentation result of the image. gt This represents the true label of the image. According to formula (3), the most important thing in training is the existence of accurate labels, which is an important indicator for measuring the quality of segmentation. Most previous segmentation networks do indeed need segmentation labels as supervision information to effectively train the network. However, accurate segmentation labels require a large number of professionals to perform time-consuming and labor-intensive annotation, so labeled data is often difficult to obtain.

[0092] This invention uses pseudo-labels to reduce the number of manually generated labels. Specifically, the segmentation result of the floating image is transformed to obtain the label of the fixed image, resulting in a pseudo-label for the fixed image. The manually generated labels and pseudo-labels are combined to form the labels of all training data, which are then used as supervisory information for training.

[0093] This invention designs a segmentation network, Seg-Net, with an architecture similar to nnUNet, such as... Figure 2 As shown, Seg-Net takes a 3D image as input and outputs the corresponding segmentation of the region of interest. To further improve the model's performance, this invention uses a spatial and channel attention parallel module, scSE-block, as shown... Figure 3 As shown, information in unimportant spatial locations is suppressed by multiplying it by a smaller value; the same method is used to suppress information in unimportant channels. The original feature maps are processed through cSE-block and sSE-block respectively to generate recalibrated feature maps. The two are then added together and passed through an activation function to output the final feature map, which improves the effectiveness of feature information and benefits the model's performance.

[0094] 3.2 Registration Network

[0095] In the experiments of this invention, the registration network adopted a structure similar to 3D-Unet. However, unlike 3D-Unet, this invention utilizes multi-resolution theory to transform it into a coarse-to-fine optimization problem, while incorporating a spatial attention mechanism to capture structural differences. This resulted in MultiSsa_Rnet being implemented as the registration network Reg-Net in this paper. Figure 4 As shown, Reg-Net takes a stitched pair of 3D images as input and outputs the deformation field between the two images. Besides the main network structure, multi-resolution theory is applied to the registration process to solve the large deformation problem through coarse-to-fine optimization. Spatial attention captures the most relevant contextual semantic information by preserving relevant responses and removing irrelevant responses, aggregating contextual information at different levels to highlight large deformation regions and refine the deformation field. The specific structure is as follows... Figure 5 As shown. Please note that this invention only provides a brief description of the various modules in the network, but the specific design is not the focus of this article.

[0096] 4 Loss Function

[0097] Based on the joint segmentation registration scheme proposed above, this invention proposes a loss function applied to this joint segmentation registration framework, which consists of two parts. E reg Let E represent the registration loss function. According to formula (1), E msMIND and E dice The predicted image is compared with the original fixed image I in terms of both local structure and global contour.ct As similar as possible, E smooth This makes the deformation field smoother. E seg This represents the loss function for the segmentation. According to formula (3), E dice When labeled, the segmentation results are made similar to those with manual labels; when unlabeled, the segmentation results of the two modalities are made to converge through a deformation field. The specific formula is:

[0098]

[0099]

[0100] Among them, E smooth It is the regularization term that maintains the smoothness of the deformation field, E msMIND E represents a local measure describing the similarity between CT and MR images. dice This represents a global measure describing the similarity of organizational labels in an image.

[0101] 4.1 Multi-scale Modal Independent Neighborhood Descriptor

[0102] 4.1.1 Background and Overview

[0103] Unlike traditional descriptors that rely on visual attributes such as grayscale or gradients, the MIND descriptor represents local structure by observing the similarity between corresponding patches based on the geometric layout that retains local self-similarity across modal images. For example... Figure 6 As shown in (a), given an image I and a voxel x in it, a local support window is formed by defining a 6-neighborhood of x at a fixed distance r. The MIND descriptor can be represented within the local support window as follows:

[0104]

[0105] D p (I,x,x+r) is the self-similarity distance between the central patch and the patch at a fixed distance from the voxel point, and V(I,x) is an estimate of the local variance. To ensure the descriptor receives a high response for the same structure and a low response for different structures, MIND is set to D. p The Gaussian function.

[0106] Figure 6 In the images above, (a) shows the two-dimensional 4-neighborhood of the MIND descriptor; (b) shows the two-dimensional 4-neighborhood of the MINDSSC descriptor; and (c) shows the two-dimensional 8-neighborhood of the msMIND descriptor. In these three images, the red square represents the central patch, and the red line connects the two points used to calculate self-similarity. For ease of illustration, only the two-dimensional case is shown here.

[0107] To address the absolute dependency of the central voxel in MIND, a design was implemented that allows self-similarity calculations among the surrounding 6 neighborhoods to generate MINDSSC. For example... Figure 6 As shown in (b), this invention selects 12 pairs of points in a 6-neighborhood. The selection of points is based on choosing each point and any of its surrounding adjacent points as a pair. The self-similar surface is determined by calculating the Euclidean distance between the point pairs, and then the MINDSSC descriptor is obtained by processing with a Gaussian function.

[0108] Inspired by MIND and MINDSSC, the msMIND descriptor of this invention also utilizes 6-neighborhood self-similarity to determine the representation of local structure, such as... Figure 6 As shown in (c). The difference is that this invention simultaneously aggregates the autocorrelation responses of 6-neighborhoods at multiple scales to construct a pyramid-shaped self-similar surface. Subsequently, this invention further transforms the feature map generated from the multi-scale self-similar surface using polar coordinate pooling within a local support window. Finally, by clustering each point on the feature map, a new feature description, namely the msMIND proposed in this invention, is generated.

[0109] 4.1.2 Multiscale Autocorrelation

[0110] Existing algorithms for calculating self-similar surfaces only use a single scale, resulting in a fixed receptive field and difficulty in handling structural changes at different scales in images. To address this issue, this invention constructs autocorrelation surfaces at different scales. Specifically, this invention... Figure 7 In the local support window shown in figure 'a', 12 patches are selected around the central voxel x. Green represents the first layer of 6-neighborhood points, and blue represents the second layer of 6-neighborhood points. By calculating the Euclidean distance between each patch and the center point, these 12 points form the self-similar surface of the local window. More commonly, the self-similar surface is calculated for every point within the window, resulting in a self-similar surface with the same size as the original window. By calculating the Gaussian function of this self-similar surface, the msMIND feature response at the center point is obtained.

[0111] Figure 7 In the diagram: (a) Expanding the 4-neighborhood local support window. (b) The process of the local support window sliding sequentially across image I, where H*W is the size of image I. (c) The multi-scale feature map S(I,x) of image I generated after the local support window slides across the image, with a size of H*M, the same as the size of image I, and 12 channels. Note that only the two-dimensional case is shown here for ease of illustration.

[0112] To use msMIND in image registration, this invention implements it in a vectorized manner. Specifically, this invention arbitrarily selects voxel x in image I, and selects two patches x in a 6-neighborhood of x.i and x j Let image I be shifted a distance p in the directions of these two patches. Subtract the two resulting images, take the square of each voxel, and perform a convolution operation using a kernel of size p*p*p. This will yield a new image where the value at position x is the mean squared error D between the images shifted in the two selected directions. p (I,x i ,x j The invention selects 12 pairs of directions and calculates their Gaussian functions to obtain the features in the two selected directions. Based on the description, this invention selects 12 pairs of directions; calculating the features in these 12 pairs of directions forms a 12-dimensional channel to obtain the first layer of 6-neighborhood features. To calculate the multi-scale autocorrelation features, this invention expands p to 2p to obtain the second layer of 6-neighborhood features; a weighted sum of the two yields the multi-scale autocorrelation features. This invention sequentially scans the entire image using the local window, as shown... Figure 7 As shown in b, the multi-scale autocorrelation feature map S(I,x) of the entire image can be obtained. Figure 7 As shown in c, the formula is as follows:

[0113]

[0114] Where D p (I,x i ,x j ) represents two patches centered at point x and spaced apart by a distance p. p (x i ) and I p (x j The sum of the root mean squares between α and β, where α and β are both decimals between 0 and 1, and α + β = 1:

[0115] D p (I,x i ,x j )=∑(I p (x i )-I p (x j )) 2 (10)

[0116] V(I,x) is an estimate of the noise, defined as the average distance across all patches:

[0117]

[0118] 4.1.3 Log-polar coordinate feature clustering

[0119] To encode the feature responses on autocorrelation surfaces, this invention clusters voxels in the same region within polar coordinate space. Since the voxel density is higher near the central voxel, this results in greater stability; it also eliminates the need to directly compute the central voxel, reducing its dependence. This method of representing features in polar coordinates has already been applied to some descriptors in the field of computer vision and has proven effective.

[0120] Specifically, such as Figure 8 As shown, this invention selects an arbitrary local patch centered at y with a size of m*m*m from the multi-scale feature map S(I,x), transforms all voxels in the patch to three-dimensional polar coordinates (cylindrical coordinates), and divides the local patch into 32 bins according to 8 angles, 2 radial intervals, and 2 heights, with the number of bins being n. The feature responses are ultimately clustered to the maximum extent within the bins surrounding each local patch. Average pooling is performed on the feature values ​​represented by all voxel points within each bin to obtain 32 values ​​associated with the center y, representing the feature descriptor msMIND at point y. This invention obtains the local feature response msMIND(I,x,y) of any point on the feature map by sliding the local patch window sequentially in the multi-scale feature map S(I,x), as follows:

[0121]

[0122] Figure 8 The process of extracting local support windows from a multi-scale feature map S(I,x) and transforming it to two-dimensional polar coordinates. This invention divides the map into 8 angles and 2 radial intervals, using these 16 bins to aggregate local features at the center point. Note that for ease of illustration, only a two-dimensional example is used here.

[0123] 4.1.4 Pooling Selection and Nonlinear Mapping

[0124] A single max-pooling strategy can lose matching details, resulting in poor discriminative ability. In contrast, the descriptors of this invention employ average pooling within each bin to provide a more representative feature representation, mitigating the impact of potential outliers to some extent and thus preserving matching details.

[0125] This invention generates a multi-scale self-similar surface and then uses a Gaussian function to perform a nonlinear mapping on the self-similar surface to obtain a feature map. This results in a lower response when the patches are different and a higher response when the patches are the same. Compared to the original self-similar surface, this reduces the influence of outliers and better represents local features. This also explains why polar coordinate aggregation using the feature map is more accurate than polar coordinate aggregation using the self-similar surface.

[0126] 4.2 Cross-entropy loss (dice)

[0127] To overcome the class imbalance problem and mitigate propagation errors caused by false labels, this invention E dice Replace with multi-category dieE softdice E softdice For E dice The improvement uses multi-class Dice loss to address inherent imbalances and propagation errors.

[0128]

[0129] Where k represents a segmentation label, x is a voxel position, and S k and These are the two segmentation results to be compared.

[0130] Example 2

[0131] This embodiment provides an image registration and segmentation system, which uses the method of Embodiment 1 to perform image registration and segmentation.

[0132] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiment 1.

[0133] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details.

Claims

1. An image registration and segmentation method, characterized in that, Includes the following steps: S1. Provide a joint segmentation and registration network model, which includes a registration network and a segmentation network; S2. The registration network is trained in an unsupervised manner using all M pairs of CT-MR image pairs as pre-training, and the CT and MR segmentation networks are trained in a recurrent self-training manner using N pairs of labeled CT-MR image pairs as pre-training; wherein, the CT-MR image pairs include CT images and MR images. S3, move all I mr and I ct The deformation field is sequentially input into the registration network. Through deformation field to I mr Deformation generation predicts images use with I ct , With L ct The structural similarity is used to guide the parameter optimization of the registration network; Among them, I mr and I CT These represent MR and CT images, respectively; L represents the image segmentation label, where... It's a manual label from the CT scan. The labels are calculated using the CT algorithm; in, It is a manually labeled tag, serving as the gold standard; It is the segmentation result of the segmentation network on unlabeled data; It is an artificial label for MR. These are CT labels obtained by the algorithm; These are the MR tags obtained by the algorithm; in, Artificial labels representing MR images, Artificial labels representing CT images, This indicates the segmentation result of the MR image; This indicates the segmentation results of a CT image; Among them, L mr L represents mr express and The set of φ, where φ represents the image deformation field. L represents mr φ represents L mr A collection of image distortions, L ct Lct represents and A collection; S4, move all I mr and I ct The data is sequentially input into a segmentation network to generate segmentation results. and When manual labeling exists, utilize and The structural similarity is used to optimize the segmentation network parameters; when no manual labels are available, the segmentation results are obtained by applying a registration deformation field to the floating image. and The same characteristics constrain the parameter optimization of the segmentation network; S5. Repeat steps S2 and S3 alternately until a well-trained segmentation and registration joint network model is obtained. Use this model for image segmentation and registration. The segmentation network Seg-Net receives the input image I mr or I ct It predicts the contours of the organs to be segmented, and uses the characteristic that the predicted contours should be the same as the true contours to adjust the segmentation network parameters θ. seg Optimize, θ seg The calculation formula is as follows: θ seg =argmin(E seg (L seg ,L gt )) (2) Among them, E seg L represents the loss function for the segmentation. seg L represents the segmentation result of the image. gt Indicates the true label of the image; During training, when manually labeled data exists, the following CT-MR labeled loss function set is used for parameter optimization; when manually labeled data does not exist, the following CT-MRulabeled loss function set is used for parameter optimization; the specific formulas are as follows: Among them, E smooth It is the regularization term that maintains the smoothness of the deformation field, E msMIND E represents a local measure describing the similarity between CT and MR images. dice E represents a global measure describing the similarity of organizational labels in an image. reg It is a measure of image similarity.

2. The image registration and segmentation method according to claim 1, characterized in that, The segmentation network is Seg-Net, which uses a floating image I. mr Fixed Image I ct and corresponding manual labels and As input, it is used to optimize the segmentation and registration joint network model.

3. The image registration and segmentation method according to claim 2, characterized in that, The registration network is Reg-Net, and Reg-Net uses a floating image I. mr Fixed Image I ct As input, the corresponding deformation field is output. The optimal fit is obtained through spatial transformation of the deformation field. Output the segmented label image or 4. The image registration and segmentation method according to claim 3, characterized in that, In the registration network Reg-Net, the msMIND descriptor is used to constrain the local structure of the image to guide image registration.

5. The image registration and segmentation method according to claim 1, characterized in that, in, E dice Replace with E softdice E softdice For E dice Improvements, E softdice The calculation formula is as follows: Where k represents a segmentation label, x is a voxel position, and S k and These are the two segmentation results to be compared.

6. The image registration and segmentation method according to claim 5, characterized in that, In the registration network Reg-Net, an arbitrary local patch centered at y and of size m*m*m is selected from the multi-scale feature map S(I,x). All voxels in the patch are transformed into three-dimensional polar coordinates, and the local patch is divided into 32 bins according to 8 angles, 2 radial intervals, and 2 heights, with the number of bins being n. The feature responses are ultimately aggregated to the maximum extent within the bins surrounding each local patch. Average pooling is then performed on the feature values ​​represented by all voxels within each bin to obtain 32 values ​​associated with the center y, representing the feature descriptor msMIND at that point y. By sequentially sliding the local patch window across the multi-scale feature map S(I,x), the local feature response msMIND(I,x,y) at any point on the feature map can be obtained. The calculation formula is as follows: The formula for the multi-scale autocorrelation feature map S(I,x) is as follows: Among them, D p (I,x i ,x j ) represents two patches centered at point x and spaced apart by a distance p: I p(xi) and I p(xj) The sum of the root mean squares, where α and β are both decimals between 0 and 1, and α + β = 1; D p (I,x i ,x j )=Σ(I p (x i )-I p (x j )) 2 (10) V(I,x) is an estimate of the noise, defined as the average distance across all patches:

7. An image registration and segmentation system, characterized in that, It uses the method described in any one of claims 1-6 to perform image registration and segmentation.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-6.

Images