A segmentation and co-registration method, device, equipment and storage medium

Abstract

The application discloses a segmentation and registration joint method, device and equipment and a storage medium. A segmentation and registration joint learning network framework with potential feature information exchange and sharing between tasks is adopted, and an intra-image segmentation result and an inter-image motion deformation field are simultaneously predicted in single-step optimization. A multi-task linking module exchanges and transfers feature information of different tasks, and by introducing a loss function item between tasks, the topology structure constraint of the registration process is enhanced, and the supervision constraint of the segmentation process is enhanced with the aid of deformation information. In the feature extraction stage, spatial attention fusion modules and multi-scale spatial attention fusion modules jointly model the topology information of key regions of an image, and the topology information is transmitted to a registration branch from a segmentation branch. In the feature recovery stage, a velocity field fusion module integrates feature information from the segmentation and registration tasks, and an inter-image velocity field is predicted at each resolution level, so that the differential homeomorphic elastic registration is realized.

Description

Technical Field

[0001] This invention relates to the field of image segmentation and registration technology, and in particular to a segmentation joint registration method, apparatus, device, and storage medium based on spatial and deformation bidirectional multi-scale fusion and multi-task learning. Background Technology

[0002] Segmentation and registration are two fundamental tasks in medical image processing, widely applied in various clinical scenarios such as medical image analysis, disease screening and diagnosis, and robot-assisted surgery. Medical image segmentation clusters image pixels based on the similarity of features such as grayscale, color, and texture, dividing the image plane into several non-overlapping regions. Pixels within a region have consistent features, while pixel features differ between different regions. Most segmentation algorithms are sensitive to image intensity, and their performance is limited on images with low contrast or poor image quality. Furthermore, learning-based segmentation algorithms heavily rely on large-scale segmentation labels.

[0003] Medical image registration aims to find a spatial transformation between pairs of images to be registered, such that the deformed image obtained after this transformation is applied to the floating image is spatially as similar to the target image to the greatest extent possible. Essentially, it seeks the optimal solution for this spatial transformation. Many conventional registration algorithms fail to capture high-level semantic and anatomical correspondences and lack topological constraints on key regions during the deformation process.

[0004] Both of these approaches are essentially single-task methods, which use one task to provide prior information or additional auxiliary constraints for the other task, without taking into account the complementary correlation and mutual promotion potential between tasks.

[0005] Correspondingly, other methods go further by establishing a joint segmentation and registration framework, combining parameter optimization processes from two different tasks using iterative optimization-based algorithms such as Bayesian statistical models and discrete Markov random field models. However, the computational cost of parameter tuning for traditional iterative optimization-based algorithms is high, which greatly limits their application in clinical scenarios.

[0006] To circumvent this problem, deep learning-based joint segmentation and registration methods have been extensively studied in recent years. These methods leverage deep learning networks to build a joint learning framework, jointly optimizing two tasks while ensuring high computational efficiency and short inference time. They model organ topology and capture the motion deformation fields between the images to be registered. Deep learning-based joint segmentation and registration methods mainly include two categories: alternating optimization and single-step optimization. In alternating optimization methods, each task also uses an independent network for alternating optimization, using inter-task loss functions to satisfy implicit constraints between different tasks in an unsupervised manner.

[0007] Such works aim to optimize two sub-networks sequentially using an alternating training strategy. While this ensures lower computational costs, its effectiveness is affected by the optimization order and the criteria used for switching tasks. Furthermore, the independent sub-networks do not consider feature correlations between tasks, limiting the joint framework's use of feature space information from different tasks.

[0008] In contrast, single-step optimization methods aim to simultaneously estimate the segmentation result and the registration deformation field in a single optimization. Predicting both tasks simultaneously in a single step avoids the impact of different optimization orders on the joint learning performance. Although the different task networks remain independent, this approach allows for further exploration of the conditional correlations between tasks compared to iteratively optimizing individual tasks. Furthermore, some recent works have further integrated different task sub-networks into a unified framework based on single-step optimization. Compared to works that rely solely on single-step optimization to achieve task joint operation with independent task networks, these works focus on task information exchange at the network architecture level. However, while focusing on deeper task correlations, they neglect the negative impact of inherent differences between tasks on the performance of the unified joint network. Segmentation aims to extract structural topological information of key regions within an image, while registration aims to extract motion deformation information of the same regions between images. These two tasks have different focuses and optimization directions in feature extraction. Directly using the same feature extraction results as input for different tasks may cause performance differences or even degradation, i.e., a seesaw effect.

[0009] During the training phase, a large number of training samples are used to improve the network's ability to capture and model motion deformation fields. During the testing phase, the pre-trained network is used to predict the motion deformation between input image pairs. This greatly reduces the time cost of registration while ensuring registration accuracy, reducing the registration time to the order of seconds.

[0010] It can be seen that most current learning-based joint learning algorithms combine tasks by introducing inter-task loss functions, employing alternating training strategies to sequentially optimize segmentation and registration tasks, or directly optimizing in a single step to simultaneously predict the results of both tasks. However, existing methods that fuse segmentation and registration into a unified joint network have room for improvement in considering the correlation and inherent differences between different tasks. Existing deep learning-based segmentation joint registration methods still have the following shortcomings:

[0011] 1. Although alternating optimization methods can guarantee lower computational costs, their effectiveness is affected by the optimization order and the criteria for switching tasks. Furthermore, the relationship between tasks is only constrained by the corresponding loss function terms, which is insufficient to fully describe the characteristics of different tasks.

[0012] 2. Although single-step optimization methods can predict two tasks simultaneously in a single step and avoid the impact of different optimization orders on the joint learning effect, the networks for different tasks are independent of each other and do not take into account the feature correlation between tasks, which limits the use of feature space information of different tasks by the joint framework.

[0013] 3. Some works further integrate different task sub-networks into a unified framework based on single-step optimization. Although they focus on task relevance at a deeper level, simply mixing and integrating different tasks ignores the negative impact of the inherent differences between different tasks on the performance of the unified joint network. Summary of the Invention

[0014] In view of the above problems, the present invention provides a segmentation joint registration method, apparatus, device, and storage medium for overcoming or at least partially solving the above problems. It is applicable to the field of organ topology modeling and deformation analysis based on medical image guidance.

[0015] This invention provides the following solution:

[0016] A segmentation joint registration method, comprising:

[0017] Obtain the image pair to be registered, which includes a floating image and a fixed image;

[0018] The floating image and the fixed image are input into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result.

[0019] The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module.

[0020] The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information.

[0021] The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network.

[0022] The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information.

[0023] The velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field; the target deformation field is used to apply to the intra-image segmentation result to obtain the inter-image registration result.

[0024] Preferably, the velocity field fusion module is used to fuse the segmentation-guided velocity field and the registration-guided velocity field of each resolution layer in a resolution-by-resolution hierarchical prediction manner to obtain the fused velocity field.

[0025] Preferably, the next resolution level performs a new round of deformation field prediction based on the deformation results obtained from the previous resolution level.

[0026] Preferably, the attention fusion module includes a spatial attention fusion module and a multi-scale spatial attention fusion module.

[0027] Preferably, the registration sub-network calculates the target deformation field based on the fused velocity field, including:

[0028] The fused velocity field is scaled and squared before being integrated to obtain the displacement field, and the target deformation field is calculated based on the displacement field.

[0029] A segmentation joint registration device, comprising:

[0030] A registration image pair acquisition unit is used to acquire a pair of images to be registered, wherein the pair of images to be registered includes a floating image and a fixed image;

[0031] The result prediction unit is used to input the floating image and the fixed image into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result.

[0032] The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module.

[0033] The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information.

[0034] The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network.

[0035] The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information.

[0036] The velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field; the target deformation field is used to apply to the intra-image segmentation result to obtain the inter-image registration result.

[0037] A segmentation joint registration device, the device comprising a processor and a memory:

[0038] The memory is used to store program code and transmit the program code to the processor;

[0039] The processor is used to execute the above-described segmentation joint registration method according to the instructions in the program code.

[0040] A computer-readable storage medium for storing program code for performing the segmentation joint registration method described above.

[0041] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0042] This application provides a segmentation and registration joint learning network framework with inter-task potential feature information exchange and sharing. In single-step optimization, it simultaneously predicts the intra-image segmentation result and the inter-image motion deformation field. A multi-task linking module facilitates the exchange and transmission of feature information from different tasks. By introducing inter-task loss function terms, it enhances the topological constraints of the registration process while simultaneously strengthening the supervision constraints of the segmentation process using deformation information. The multi-task linking module comprises a spatial attention fusion module, a multi-scale spatial attention fusion module, and a velocity field fusion module. In the feature extraction stage, the spatial attention fusion module and the multi-scale spatial attention fusion module jointly model the topological information of key image regions, which is then passed from the segmentation branch to the registration branch. In the feature recovery stage, the velocity field fusion module integrates feature information from the segmentation and registration tasks, predicting the inter-image velocity field at each resolution level to achieve differential homeomorphic elastic registration.

[0043] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0045] Figure 1 This is a flowchart of a segmentation joint registration method provided in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of the algorithm workflow provided in an embodiment of the present invention;

[0047] Figure 3 This is a diagram of the target segmentation and registration joint learning network framework provided in an embodiment of the present invention;

[0048] Figure 4 This is a comparison chart of the segmentation performance of various methods on the OASIS-1 dataset provided in the embodiments of the present invention;

[0049] Figure 5 This is a comparison chart of the registration performance of various methods on the OASIS-1 dataset provided in this embodiment of the invention;

[0050] Figure 6 This is an evaluation chart of the performance of various comparative methods on the OASIS-1 dataset provided in this embodiment of the invention;

[0051] Figure 7 This is a schematic diagram of a segmentation joint registration device provided in an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram of a segmentation and joint registration device provided in an embodiment of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0054] Image segmentation refers to the process of segmenting an image into an n-dimensional space. The purpose of the deep learning-based segmentation algorithm for image I is to automatically infer the probability p of each voxel x∈I belonging to one of the different anatomical structures Y. i ,i∈[1,2,…,Y], and then the image is further subdivided into multiple image sub-regions according to the type of target structure, that is, the segmentation mask S, as shown in Formula 1.

[0055]

[0056] in, Let θ represent the objective function modeled by operations such as convolution and nonlinear transformations in the segmentation network, and let θ represent the trainable segmentation network weights. The segmentation network uses N sets of segmentation gold standards. Image I is used as training data, and the network prediction result S is compared with the segmentation gold standard. The difference is used to constrain optimization to obtain the optimal network weights.

[0057]

[0058] Image registration refers to the process of matching a pair of images that belong to the same n-dimensional space. Images f and m to be registered have the same anatomical structure, where f represents the fixed image and m represents the floating image. The spatial mapping from the floating image m to the fixed image f is defined as φ: Space mapping It consists of the identity transformation Id and the displacement field u. In the deep learning-based registration algorithm, the displacement field u is predicted by a convolutional neural network:

[0059]

[0060] in, The objective function is represented by operations such as convolution and nonlinear transformations in the registration network, and ψ represents the trainable registration network weights. Due to the gold standard deformation field... In real-world scenarios, obtaining the image f is difficult. Most existing deep learning-based registration algorithms employ unsupervised training methods to obtain the image f with the deformed image. The similarity differences between them and the deformation field regularization term are used as constraints to optimize and obtain the optimal network weights.

[0061]

[0062] Both of these approaches are essentially single-task methods, which use one task to provide prior information or additional auxiliary constraints for the other task, without taking into account the complementary correlation and mutual promotion potential between tasks.

[0063] The method provided in this application embodiment simultaneously realizes the joint operation and information sharing between different tasks in terms of both network structure and loss function; it makes full use of the potential feature representations of segmentation and registration tasks to achieve information complementarity and effect promotion between different tasks, while taking into account the imbalance of optimization difficulty between different tasks.

[0064] See Figure 1 This invention provides a segmentation joint registration method, such as... Figure 1 As shown, the method may include:

[0065] S101: Obtain the image pair to be registered, the image pair to be registered includes a floating image and a fixed image;

[0066] S102: Input the floating image and the fixed image into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result;

[0067] The target segmentation and registration joint learning network includes a segmentation sub-network, a registration sub-network, and a multi-task linking module located between the segmentation sub-network and the registration sub-network; the multi-task linking module includes an attention fusion module and a velocity field fusion module; further, the attention fusion module includes a spatial attention fusion module and a multi-scale spatial attention fusion module.

[0068] The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information.

[0069] The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network.

[0070] The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information.

[0071] The velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field. The target deformation field is applied to the intra-image segmentation result to obtain the inter-image registration result. Further, the velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field of each resolution layer in a resolution-level prediction manner to obtain a fused velocity field. The next resolution layer performs a new round of deformation field prediction based on the deformation result obtained from the previous resolution layer.

[0072] The registration sub-network calculates the target deformation field based on the fused velocity field, including:

[0073] The fused velocity field is scaled and squared before being integrated to obtain the displacement field, and the target deformation field is calculated based on the displacement field.

[0074] The segmentation and joint registration method provided in this application differs from existing mainstream segmentation and joint registration frameworks. This system not only bridges the outputs of the two sub-tasks at the loss function level, but also strengthens the information exchange between the latent feature representations of the segmentation and registration tasks in the feature space at the network structure level by designing a multi-task connection module. For the end-to-end segmentation network, the network's encoder can model the latent representation of the image to be segmented in the feature space.

[0075] Under the constraints of the segmentation gold standard, as training progresses, this latent representation can focus on the foreground region, i.e., the region of interest. This discriminative feature information is of great significance for the registration task, as it can provide spatial location information and target structure topology information for global registration between images, enhance the constraint on the alignment of the region of interest, improve registration accuracy, and strengthen the topological preservation of the target structure.

[0076] The following is combined with Figure 2 as well as Figure 3 The segmentation and joint registration method provided in the embodiments of this application will be described in detail.

[0077] This segmentation joint registration multi-task learning method based on bidirectional multi-scale fusion of spatial and deformation data includes the following steps:

[0078] Step 1: Construct a 3D medical image dataset, perform data preprocessing operations such as histogram equalization and intensity normalization, and divide the dataset into training and testing sets. To ensure sufficient validation of the method's generalization ability across various modalities of medical imaging, experiments were conducted on publicly available datasets for three commonly used modalities: MR, CT, and ultrasound. Each original dataset underwent preprocessing operations such as size cropping, intensity normalization, and histogram equalization. The preprocessed datasets were then divided into training and testing sets according to a specific ratio.

[0079] Step 2: Construct a segmentation and joint registration framework that integrates spatial and deformation information in both directions. A schematic diagram is attached. Figure 2 The entire framework consists of a registration subnetwork, a segmentation subnetwork, and a multi-task linking module. The multi-task linking module comprises a spatial attention fusion module, a multi-scale spatial attention fusion module, and a velocity field fusion module, fusing feature information from different tasks during the feature encoding and feature recovery stages. The two images of the image pair to be registered are separately fed into the segmentation subnetwork, then stitched together and fed into the registration subnetwork.

[0080] Step 3: In the feature encoding stage, the multi-scale feature information obtained by the segmentation sub-network based on the input floating image and fixed image is sequentially sent to the spatial attention fusion module and the multi-scale spatial attention fusion module in the multi-task linking module. After fusion based on the attention mechanism, feature information with segmentation task specific information is obtained and sent to the registration sub-network feature encoder.

[0081] Step 4: In the feature recovery stage, the feature results obtained from the floating image branch and the fixed image branch of the fusion segmentation sub-network (contour) are used for segmentation-guided velocity field prediction. This prediction, along with the registration-guided velocity field predicted by the registration sub-network (more detailed full-image), is fed into the velocity field fusion module to obtain the deformation field corresponding to the fused velocity field. This fused velocity field is then applied to the floating image branch of the segmentation sub-network to obtain the deformed feature image. At the next resolution level, a new round of deformation field prediction is performed based on the deformation results obtained at the previous resolution level.

[0082] Specifically, the entire framework receives a floating image m and a fixed image f as input, and outputs the segmentation results S of the two images. m and S f And the deformation field φ between the two. Specifically, the floating image m and the fixed image f are input separately into the segmentation sub-network for segmentation result prediction, and the two are concatenated in the channel dimension and input into the registration sub-network to calculate their corresponding velocity field νf. After scaling and squaring operations, the displacement field u is obtained by integrating the velocity field νf, and the final deformation field φ is calculated based on the displacement field u.

[0083] Step 5: The final network output yields the segmentation results of the floating and fixed images, as well as the deformation field between them. Various loss functions are calculated, and the joint learning network is iteratively optimized. The network is trained using the training set data until convergence, and the network model parameters are saved. Different training strategies and hyperparameter combinations are explored, and an ablation experiment is designed to determine the optimal training strategy and hyperparameter combination.

[0084] Step 6: In the testing phase, import the trained network model parameters and, based on the input image pair to be registered, achieve rapid simultaneous prediction of segmentation results and registration deformation fields. Simultaneously, predict the segmentation results of both images and the deformation field between them.

[0085] In summary, the segmentation and registration method provided in this application employs a joint learning network framework for segmentation and registration that facilitates the exchange and sharing of potential feature information between tasks. In single-step optimization, it simultaneously predicts the intra-image segmentation result and the inter-image motion deformation field. The multi-task linking module facilitates the exchange and transfer of feature information from different tasks, and by introducing inter-task loss function terms, it enhances the topological constraints of the registration process while simultaneously strengthening the supervision constraints of the segmentation process through deformation information. The multi-task linking module comprises a spatial attention fusion module, a multi-scale spatial attention fusion module, and a velocity field fusion module. In the feature extraction stage, the spatial attention fusion module and the multi-scale spatial attention fusion module jointly model the topological information of key image regions, which is then passed from the segmentation branch to the registration branch. In the feature recovery stage, the velocity field fusion module integrates feature information from the segmentation and registration tasks, predicting the inter-image velocity field at each resolution level to achieve differential homeomorphic elastic registration.

[0086] To verify the effectiveness of the method provided in this application, extensive experimental verification was conducted on the publicly available 3D brain MR dataset OASIS-1. Furthermore, to quantitatively compare the advantages and disadvantages of the method provided in this application with existing methods, four different representative algorithms—DeepAtlas, U-ResNet, RSegNet, and SegisNet—were selected for comparative experiments.

[0087] Compared to single-task optimization methods for segmentation / registration, the BFM-Net method provided in this application can achieve more refined segmentation while improving the deformation accuracy of the registration task, and promoting better alignment of key structural regions in floating images to fixed images.

[0088] With the help of the multi-task linking module, the segmentation and registration tasks achieved better information exchange and sharing. The network fully captured the potential feature associations between tasks and explored the potential of each task through joint optimization, thereby improving the segmentation / registration accuracy.

[0089] BFM-Net achieved significant improvements over single-task methods across three different labels: CSF, GM, and WM.

[0090] The Dice index for segmentation tasks improved by an average of 3.2% and a maximum of 5.4% (CSF), while the ASSD decreased by an average of 0.118 mm and a maximum of 0.181 mm (CSF).

[0091] The registration task improved the Dice index by an average of 6.2% and a maximum of 9.0% (GM), and reduced the ASSD by an average of 0.126mm and a maximum of 0.246mm (WM).

[0092] Compared to existing joint learning methods, the BFM-Net provided in this application embodiment can achieve varying degrees of improvement in both segmentation and registration tasks.

[0093] Figure 4 The figure shows a comparison of the segmentation performance of various methods on the OASIS-1 dataset. From left to right, the columns represent the segmentation performance of each comparison method and our proposed method, BFM-Net. SegNet represents a segmentation network optimized for a single task. The rightmost column shows the corresponding gold standard for segmentation.

[0094] Figure 5 The image shows a comparison of the registration performance of various methods on the OASIS-1 dataset. Moving Image and Fixed Image represent the floating and fixed images, respectively. RegNet represents the registration network optimized for a single task.

[0095] Figure 6 The image shows the performance evaluation of various methods on the OASIS-1 dataset. (a1 / 2) and (d1 / 2), (b1 / 2) and (e1 / 2), (c1 / 2) and (f1 / 2) represent segmentation and registration tasks on CSF, GM, and WM labels, respectively. The radar chart reflects the performance of each method on different tasks and labels in terms of Dice, Jaccard, Precision, Recall, and Accuracy, while the box plot reflects the performance of each method on the ASSD metric.

[0096] The figure reflects the performance of each method on different tasks and labels in terms of Dice, Jaccard, Precision, Recall, and Accuracy metrics.

[0097] Figure 6Different lines in the graph correspond to different methods: DeepAtlas (D), U-ResNet (U), SegisNet (S), RSegNet (R), and BFM-Net (B). The numerical ranges of each coordinate axis within each graph are the same, and different colors are used to label the specific performance of the method provided in this application (B) and the comparative methods on various metrics. When labeling the comparative methods, only the results of the best-performing method are compared with the method provided in the embodiments of this application.

[0098] As can be seen, the box plot reflects the performance of each method on the ASSD metric. Whether in segmentation or registration tasks, BFM-Net achieves the lowest ASSD while maintaining the highest Dice score. In segmentation, BFM-Net minimizes missegmentation and reduces discontinuities in the segmentation results. In registration, BFM-Net aligns key structures as closely as possible while minimizing the probability of abnormal distortions.

[0099] See Figure 5 This application embodiment can also provide a segmentation joint registration device, such as... Figure 5 As shown, the device may include:

[0100] The image pair registration unit 701 is used to acquire an image pair to be registered, the image pair to be registered including a floating image and a fixed image;

[0101] The result prediction unit 702 is used to input the floating image and the fixed image into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result.

[0102] The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module.

[0103] The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information.

[0104] The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network.

[0105] The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information.

[0106] The velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field; the target deformation field is used to apply to the intra-image segmentation result to obtain the inter-image registration result.

[0107] This application embodiment can also provide a segmentation joint registration device, the device including a processor and a memory:

[0108] The memory is used to store program code and transmit the program code to the processor;

[0109] The processor is used to execute the steps of the segmentation joint registration method described above according to the instructions in the program code.

[0110] like Figure 8 As shown in the embodiment of this application, a segmentation joint registration device may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0111] In this embodiment, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0112] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the segmentation joint registration method.

[0113] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions:

[0114] Obtain the image pair to be registered, which includes a floating image and a fixed image;

[0115] The floating image and the fixed image are input into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result.

[0116] The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module.

[0117] The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information.

[0118] The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network.

[0119] The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information.

[0120] The velocity field fusion module is used to fuse the segmentation-oriented velocity field and the registration-oriented velocity field to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field; the target deformation field is used to apply to the intra-image segmentation result to obtain the inter-image registration result.

[0121] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0122] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0123] Communication interface 12 can be an interface for the communication module, used to connect with other devices or systems.

[0124] Of course, it should be noted that, Figure 8 The structure shown does not constitute a limitation on the segmentation joint registration device in the embodiments of this application. In practical applications, the segmentation joint registration device may include more than Figure 8 More or fewer components as shown, or combinations of certain components.

[0125] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the segmentation joint registration method described above.

[0126] It should be noted that, in the embodiments provided in this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0127] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0128] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A segmentation joint registration method, characterized in that, include: Obtain the image pair to be registered, which includes a floating image and a fixed image; The floating image and the fixed image are input into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result. The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module. The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information. The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network. The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information. The velocity field fusion module is used to fuse the velocity field with segmentation guidance and the velocity field with registration guidance to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field. The target deformation field is applied to the intra-image segmentation results to obtain the inter-image registration results.

2. The segmentation and joint registration method according to claim 1, characterized in that, The velocity field fusion module is used to fuse the segmentation-guided velocity field and the registration-guided velocity field of each resolution layer in a resolution-by-resolution hierarchical prediction manner to obtain the fused velocity field.

3. The segmentation and joint registration method according to claim 2, characterized in that, The next resolution level performs a new round of deformation field prediction based on the deformation results obtained from the previous resolution level.

4. The segmentation and joint registration method according to claim 1, characterized in that, The attention fusion module includes a spatial attention fusion module and a multi-scale spatial attention fusion module.

5. The segmentation and joint registration method according to claim 1, characterized in that, The registration sub-network calculates the target deformation field based on the fused velocity field, including: The fused velocity field is scaled and squared before being integrated to obtain the displacement field, and the target deformation field is calculated based on the displacement field.

6. A segmentation and joint registration device, characterized in that, include: A registration image pair acquisition unit is used to acquire a pair of images to be registered, wherein the pair of images to be registered includes a floating image and a fixed image; The result prediction unit is used to input the floating image and the fixed image into the target segmentation and registration joint learning network for image segmentation and image registration processing, so as to simultaneously predict the intra-image segmentation result and the inter-image registration result. The target segmentation and registration joint learning network includes a segmentation subnetwork, a registration subnetwork, and a multi-task linking module located between the segmentation subnetwork and the registration subnetwork; the multi-task linking module includes an attention fusion module and a velocity field fusion module. The segmentation sub-network is used to calculate multi-scale feature information and a segmentation-oriented velocity field based on the floating image and the fixed image, so as to calculate the intra-image segmentation result based on the multi-scale feature information. The attention fusion module is used to fuse the received multi-scale feature information using an attention mechanism to obtain feature information with segmentation task specific information, and to send the feature information with segmentation task specific information into the registration sub-network. The registration sub-network is used to calculate a velocity field with registration guidance based on the stitched floating image, the fixed image, and the feature information with segmentation task-specific information. The velocity field fusion module is used to fuse the velocity field with segmentation guidance and the velocity field with registration guidance to obtain a fused velocity field, so that the registration sub-network can calculate the target deformation field based on the fused velocity field. The target deformation field is applied to the intra-image segmentation results to obtain the inter-image registration results.

7. A segmentation and joint registration device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the segmentation joint registration method according to any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the segmentation joint registration method according to any one of claims 1-5.

Images