Image Registration Method, Apparatus, Device, and Storage Medium

Abstract

The present application discloses an image registration method, apparatus, device, and storage medium, which relate to the field of artificial intelligence. The method includes: obtaining a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image; inputting the pair of sample images into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image; inputting the pair of sample images into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image, where the first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times; and training the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and particularly to an image registration method, apparatus, device, and storage medium. Background Art

[0002] Image registration refers to the process of aligning one or more images in space through spatial transformation so that they are spatially aligned with a reference image.

[0003] The image registration process is a process of finding the optimal spatial transformation, that is, the optimal deformation field. For each pair of images to be registered, the solution space of the deformation field is not unique. Therefore, in the training stage of an image registration network based on deep learning, to constrain the solution space of the deformation field, a spatial regularization term is added to constrain the deformation field.

[0004] However, in the process of training an image registration network, there are also significant differences in the deformation field at different training times. In the related art, only the deformation field in space is constrained, resulting in poor training effects of the image registration network, and further affecting the accuracy of image registration based on the image registration network. Summary of the Invention

[0005] The embodiments of the present application provide an image registration method, apparatus, device, and storage medium, which can improve the performance of the deformation field predicted by the image registration network, and further improve the accuracy of image registration. The technical solutions are as follows:

[0006] On the one hand, the embodiments of the present application provide an image registration method, which includes:

[0007] Obtain a sample image pair, where the sample image pair includes a sample image to be registered and a sample target image;

[0008] Input the sample image pair into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image;

[0009] Input the sample image pair into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image, where the first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times;

[0010] Train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

[0011] On the other hand, an embodiment of the present application provides an image registration device, which includes:

[0012] A sample image acquisition module, configured to acquire a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image;

[0013] A first registration module, configured to input the pair of sample images into a first registration network, obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image;

[0014] A second registration module, configured to input the pair of sample images into a second registration network, obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image, where the first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times;

[0015] A training module, configured to train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

[0016] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory. At least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image registration method as described in the above aspect.

[0017] On the other hand, a computer-readable storage medium is provided. At least one instruction, at least one program, a code set, or an instruction set is stored in the readable storage medium, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image registration method as described in the above aspect.

[0018] On the other hand, an embodiment of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image registration method provided in the above aspect.

[0019] The beneficial effects brought by the technical solutions provided by the embodiments of the present application at least include:

[0020] The to-be-registered image and the target image are respectively subjected to image registration through the first registration network and the second registration network to obtain deformation fields corresponding to network parameters at different training times. Finally, the to-be-registered image is deformed based on different deformation fields to obtain different registered images. Furthermore, the first registration network can be trained through different registered images and the target image, so that the network parameters obtained by the first registration network at different training times tend to be stable, that is, the deformation fields predicted by the first registration network at different training times tend to be consistent. Adding time constraints during the training process helps to improve the training effect of the first registration network, and further improves the accuracy of the deformation field predicted by the first registration network. Description of the Drawings

[0021] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 Shows a schematic diagram of the principle of the image registration method provided by the embodiments of the present application;

[0023] Figure 2 Shows a schematic diagram of the implementation environment provided by an exemplary embodiment of the present application;

[0024] Figure 3 Shows a flowchart of the image registration method provided by an exemplary embodiment of the present application;

[0025] Figure 4 Shows a flowchart of the image registration method provided by another exemplary embodiment of the present application;

[0026] Figure 5 Shows a flowchart of the image registration method provided by another exemplary embodiment of the present application;

[0027] Figure 6 Is an implementation schematic diagram of the training process of the first registration network shown in an exemplary embodiment;

[0028] Figure 7 Shows a flowchart of the image registration method provided by another exemplary embodiment of the present application;

[0029] Figure 8 Is an implementation schematic diagram of the image registration process using the first registration network shown in an exemplary embodiment;

[0030] Figure 9 Is a structural block diagram of the image registration device provided by an exemplary embodiment of the present application;

[0031] Figure 10 The structural schematic diagram of a computer device provided by an exemplary embodiment of the present application is shown. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0033] Artificial Intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, a theory, method, technology, and application system that can perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0034] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0035] Computer Vision (CV) technology is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to perform machine vision such as target recognition, tracking, and measurement on targets, and further perform graphic processing to make the images processed by the computer more suitable for human eyes to observe or be transmitted to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes technologies such as image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition. The image registration method involved in the embodiments of the present application, that is, the application of computer vision technology in the field of image processing, can increase the training time constraint during the process of training a registration network based on a sample image, improve the training effect of the registration network, and further improve the prediction deformation field performance of the registration network.

[0036] In the related art, during the training phase of an image registration network based on unsupervised learning, the image registration network is trained using the following loss function:

[0037]

[0038] where I m is the moving image, is the deformed moving image, and I f is the fixed image, quantifies the spatial difference between the fixed image and the deformed moving image. However, since there are multiple solutions for the deformation field during the image registration process, that is, the solution space of the deformation field is not unique, a spatial regularization loss is imposed to constrain the solution space, and λ is the regularization strength.

[0039] It can be seen that in the related art, only spatial constraints are imposed on the deformation field. However, during the training process, there are also significant differences in the deformation fields predicted by the image registration network at different training times (i.e., during the iterative process). The related art does not consider the differences in the deformation fields over time. Therefore, in the embodiments of the present application, a second registration network is introduced to constrain the solution space of the deformation field in terms of time and optimize the training effect of the image registration network, that is, improve the performance of the image registration network.

[0040] As Figure 1 shown, a first registration network 102 and a second registration network 103 are provided. The sample image pair 101 is input into the first registration network 102 and the second registration network 103 respectively to obtain a first deformation field 104 and a second deformation field 105. Then, based on the first deformation field 104 and the second deformation field 105, the image to be registered 106 is deformed to obtain a first registered image 107 and a second registered image 108. Since the network parameters of the first registration network 102 and the second registration network 103 are the network parameters at different training times, training the first registration network 102 based on the first registered image 107 and the second registered image 108 can constrain the solution space of the deformation field predicted by the first registration network 102 in terms of time, thereby improving the performance of the first registration network 102. When using the trained first registration network 102 for image registration, the accuracy of image registration is improved.

[0041] Moreover, the image registration method provided by the embodiments of the present application can be used in the training process of the image registration network, and the trained image registration network can be used in any image registration scenario.

[0042] When applied to the medical scenario, for the same organ of a patient, images containing accurate anatomical information can be collected using different devices, such as Computed Tomography (CT), Positron Emission Computed Tomography (PET), etc.; or medical images at different time periods can be collected, and the collected images are subjected to image registration to fuse the information in each image, so that medical staff can more accurately observe the changes of the lesion and the organ from various angles, which helps in medical diagnosis, formulating surgical plans, and the process of radiotherapy planning.

[0043] When applied to the target tracking scenario, image registration can be performed on target images of the same target collected at different angles and different times, and then the information in each image is fused to obtain the target movement trajectory, which helps in target tracking.

[0044] Of course, in addition to the above application scenarios, the image registration method provided in the embodiments of the present application can also be applied to other image registration scenarios, and the embodiments of the present application do not limit the specific application scenarios.

[0045] Figure 2 FIG. shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment includes a computer device 210 and a server 220. Among them, the computer device 210 and the server 220 perform data communication through a communication network. Optionally, the communication network can be a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0046] The computer device 210 is an electronic device with an image registration requirement. This electronic device can be a smart phone, a tablet computer, a personal computer, etc., and the present embodiment does not limit this. Figure 2 In this, the computer device 210 is taken as an example of a computer used by medical staff for illustration.

[0047] In some embodiments, an application program with an image registration function is installed in the computer device 210. When it is necessary to perform image registration on a pair of target images (for example, medical images scanned at different times, medical images scanned by different devices), the user inputs the pair of target images into the application program, thereby uploading the pair of target images to the server 220, and the server 220 performs image registration and feeds back the registration result.

[0048] The server 220 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.

[0049] In some embodiments, the server 220 is used to provide image registration services for the application programs installed in the computer device 210. Optionally, an image registration network 221 is set in the server 220. In a possible implementation manner, after receiving the target image pair 211 sent by the computer device 210, the server 220 uses the image registration network 221 to perform image registration to obtain a deformation field 222, and uses the deformation field 222 to perform deformation processing on the image to be registered to obtain a registered image 223, and returns the registered image 223 to the computer device 210 so that the computer device 210 can display the image registration result.

[0050] Of course, in other possible implementation manners, the image registration network can also be deployed on the side of the computer device 210, and the computer device 210 can implement image registration locally without relying on the server 220. This embodiment does not limit this. And the image registration network can be trained on the server side or on the computer device side for the deployment of the image registration network. For the sake of convenience of description, the following embodiments are described by taking the image registration method as being executed by the computer device as an example.

[0051] Please refer to Figure 3 , which shows a flowchart of an image registration method provided by an exemplary embodiment of the present application. This embodiment is described by taking the method as being used for the computer device as an example. The method includes the following steps.

[0052] Step 301, obtain a sample image pair, where the sample image pair includes a sample image to be registered and a sample target image.

[0053] The sample image pair refers to an image pair of the same object collected under different conditions. Among them, the different conditions may include at least one of different times, different angles, or different acquisition devices. For example, the sample image pair includes medical images of the lungs of the same patient collected during the exhalation and inhalation phases.

[0054] Optionally, any one of the sample image pairs can be used as the sample target image. For example, if the sample image pair contains Image A and Image B, when Image A is used as the sample target image, Image B can be spatially transformed to align it with Image A in space; when Image B is used as the sample target image, Image A can be spatially transformed to align it with Image A in space.

[0055] Step 302: Input the sample image pair into the first registration network to obtain the first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain the first registered image.

[0056] Optionally, the first registration network can adopt a U-Net network, which includes an encoder for extracting high-level semantic features and an encoder for restoring the original size and generating prediction results. In this embodiment, a three-dimensional registration prediction network is constructed using the U-Net network to register three-dimensional images. In addition, the first registration network can also be other structural networks, which are not limited in this embodiment.

[0057] After inputting the sample image pair into the first registration network, the first deformation field can be obtained. Among them, based on the first deformation field, the sample image to be registered is spatially transformed so that the sample image to be registered is aligned with the sample target image in space, thereby achieving the purpose of fusing the information in the two images.

[0058] Optionally, during the process of deforming the sample image to be registered, a Spatial Transformer Networks (STN) can be used to deform the sample image to be registered. That is, the first deformation field and the sample image to be registered are input into the STN, and the STN uses the first deformation field to deform the sample image to be registered to obtain the first registered image.

[0059] Step 303: Input the sample image pair into the second registration network to obtain the second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain the second registered image. The first network parameter of the first registration network and the second network parameter of the second registration network are network parameters at different training times.

[0060] During the training process of the first registration network, there are significant differences in the deformation fields predicted at different training times. Therefore, in order to constrain the solution space of the deformation field in time, in the embodiment of the present application, a second registration network is set to predict the deformation field between the sample image pairs, that is, the second deformation field is predicted. Since the network parameters of the first registration network and the second registration network are network parameters at different training times, deformation fields at different training times can be predicted.

[0061] After predicting the second deformation field, the STN network can also be used to perform deformation processing on the sample image to be registered to obtain the second registered image. Furthermore, the first registered image and the second registered image at different training times can be obtained. Therefore, when training the first registration network based on the first registered image and the second registered image, the deformation field can be constrained in terms of time.

[0062] Optionally, the network structure of the first registration network is the same as that of the second registration network, and only the network parameters used have differences in time.

[0063] Step 304: Train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

[0064] After obtaining the first deformation field, the first registered image, and the second registered image, the first registration network can be trained based on each feature, thereby realizing the addition of constraints on the deformation fields obtained at different training times during the training process.

[0065] Optionally, during the training process, the total loss can be determined based on the features of the first deformation field, the first registered image, the second registered image, and the sample target image, and the first registration network can be trained using the gradient descent or backpropagation algorithm, that is, the network parameters of the first registration network are adjusted until the training conditions are met.

[0066] In summary, in the embodiments of the present application, the first registration network and the second registration network are used to perform image registration on the image to be registered and the target image respectively, to obtain deformation fields corresponding to network parameters at different training times. Finally, after performing deformation processing on the image to be registered based on different deformation fields, different registered images are obtained. Furthermore, the first registration network can be trained using different registered images and the target image, so that the network parameters obtained by the first registration network at different training times tend to be stable, that is, the deformation fields predicted by the first registration network at different training times tend to be consistent. Adding time constraints during the training process helps to improve the training effect of the first registration network, and further improves the accuracy of the deformation field predicted by the first registration network.

[0067] In the embodiments of the present application, the temporal information during the training process is used to optimize the first registration network. To constrain the differences in deformation fields at different training times, a time regularization loss is introduced on the basis of the similarity loss and the spatial regularization loss. Therefore, the first registration network is trained based on the three together, so that during the training process, constraints are imposed on the first registration network in both the spatial and temporal dimensions. The following will illustrate this process with exemplary embodiments.

[0068] Please refer to Figure 4, which shows a flowchart of an image registration method provided by an exemplary embodiment of the present application. In this embodiment, the method is described by taking a computer device as an example. The method includes the following steps.

[0069] Step 401: Obtain a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image.

[0070] Step 402: Input the pair of sample images into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image.

[0071] Step 403: Input the pair of sample images into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image. The first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times.

[0072] The implementation manners of steps 401 to 403 can refer to the above steps 301 to 303, and will not be elaborated in this embodiment.

[0073] Step 404: Determine a similarity loss based on the first registered image and the sample target image.

[0074] Since image registration is to register the sample image to be registered and the sample target image, to improve the accuracy of the first registration network in predicting the deformation field, the similarity loss can be determined according to the spatial difference between the first registered image and the sample target image, and then the first registration network can be trained using this similarity loss.

[0075] Image registration can be divided into single-modal image registration and multi-modal image registration. Among them, single-modal image registration refers to registering a pair of sample images from the same imaging device. For example, both images in the pair of sample images are from CT, while multi-modal image registration refers to registering sample images from different imaging devices. For example, one image in the pair of sample images is from CT and the other is from PET. In a possible implementation manner, to be applicable to both single-modal image registration and multi-modal image registration simultaneously, the dissimilarity of the Modality Independent Neighbourhood Descriptor (MIND) is used to quantify the similarity loss.

[0076] Among them, MIND is:

[0077]

[0078] I is an image, x is the position of a voxel in the image, r is a distance vector, V(I,x) is an estimate of the local variance, and D p (I,x,x+r) represents the L2 distance between the image patch centered at x and the image patch centered at x+r.

[0079] During the training process of the first registration network, the aim is to minimize the difference between the MIND features of the first registered image and the MIND features of the sample target image. Therefore, in one possible implementation, the loss function of the similarity loss is:

[0080]

[0081] where I ws is the first registered image, I f is the sample target image, and N represents the number of voxels in the image.

[0082] Step 405: Determine a spatial regularization loss based on the first deformation field. The spatial regularization loss is used to impose a spatial constraint on the deformation field.

[0083] During the process of registering the sample image to be registered with the sample target image, when deforming the sample image to be registered, i.e., performing a spatial transformation, the spatial transformation can be carried out in various ways, such as affine transformation, projective transformation, and bending transformation, etc. Therefore, the solution space of the deformation field of the spatial transformation of the sample image to be registered is not unique. That is, during the process of the first registration network registering the sample image to be registered with the sample target image, there are many differences in the predicted deformation field in space. Therefore, it is necessary to constrain the deformation field in space.

[0084] In one possible implementation, the deformation field is constrained by setting the spatial regularization loss, and the first registration network is trained using the spatial regularization loss to ensure the smoothness of the deformation field output by the first registration network. Optionally, the spatial regularization loss can be measured according to the gradient of the first deformation field, and its loss function is as follows:

[0085]

[0086] where represents the gradient of the first deformation field.

[0087] Step 406: Determine a temporal regularization loss based on the first registered image and the second registered image. The temporal regularization loss is used to impose a training time constraint on the deformation field.

[0088] Since the network parameters of the first registration network and the second registration network are the network parameters at different training times, there is a time difference in the predicted deformation fields of the first registration network and the second registration network. The difference in the predicted deformation fields of the first registration network and the second registration network can be measured by the spatial similarity between the first registered image and the second registered image. That is, the temporal regularization loss can be determined by the spatial similarity between the first registered image and the second registered image, so as to impose a temporal training time constraint on the deformation field of the first registration network.

[0089] In a possible implementation manner, since the temporal regularization loss is measured according to the spatial similarity between the first registered image and the second registered image, the temporal regularization loss can also be determined according to the difference degree between the MIND features of the first registered image and the second registered image. The loss function is as follows:

[0090]

[0091] where I ws is the first registered image, I wt is the second registered image, and N is the number of voxels in the image.

[0092] Step 407: Train the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.

[0093] After determining the similarity loss, the spatial regularization loss, and the temporal regularization loss, the first registration network is trained based on the three of them. The accuracy of the predicted deformation field of the first registration network is improved by the similarity loss, while the spatial regularization loss and the temporal regularization loss can respectively impose spatial constraints and temporal constraints on the deformation field, optimizing the training effect of the first registration network.

[0094] In the related art, a fixed weight is set for the spatial regularization loss to control the regularization intensity. During the training process, for each training sample, that is, each pair of sample images, their training difficulties are not the same. That is, there are differences in the number of uncertain deformation fields generated when predicting the deformation field between the pair of sample images. When there are more solutions to the predicted deformation field, it is necessary to reduce the unnecessary deformation complexity, that is, a stronger regularization is needed to constrain the solution space of the deformation field. When there are fewer solutions to the predicted deformation field, the intensity of the regularization constraint can be reduced. It can be seen that if a fixed weight is used, that is, a fixed regularization intensity is set, it cannot adapt to each training sample. Correspondingly, the performance of the first registration network obtained by training is poor.

[0095] Therefore, in the embodiments of the present application, the loss weights of the spatial regularization loss and the temporal regularization loss will be adjusted according to the uncertainty of the predicted deformation field for each sample image, so that they are applicable to different training samples. In a possible implementation manner, the first loss weight of the spatial regularization loss and the second loss weight of the temporal regularization loss are determined by the second registration network. That is, the second registration network can not only be used to determine the temporal regularization loss, but also be used to estimate the uncertainty of the predicted deformation field of the sample image pair, and then be used to adaptively adjust the loss weight. Determining each loss weight by the second registration network may include the following steps:

[0096] Step 1: Perform n forward predictions on the sample image pair through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, where n is an integer greater than or equal to 2.

[0097] Since it is necessary to determine the uncertainty of the predicted deformation field, that is, to judge the trend of the number of predicted deformation fields, therefore, it is necessary to perform n forward predictions on the sample image through the second registration network, and then obtain n deformation fields between the sample image pairs, so that the uncertainty of the deformation field can be determined according to the dispersion degree of the n deformation fields, and finally the loss weight of the spatial regularization loss, that is, the intensity of the spatial regularization, can be determined based on the uncertainty of the deformation field.

[0098] Step 2: Perform deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registered images.

[0099] Since the temporal regularization loss is determined according to the spatial similarity between the first registered image and the second registered image, therefore, the intensity of the temporal regularization can be determined by the uncertainty of the registered image obtained after performing deformation processing on the sample image to be registered through the predicted deformation field of the second registration network. That is, perform deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registered images, and determine the loss weight of the temporal regularization loss according to the uncertainty of the n third registered images.

[0100] Step 3: Determine the first loss weight based on the n third deformation fields.

[0101] Optionally, the first loss weight is the loss weight of the spatial regularization loss. The process of determining the first loss weight according to the n third deformation fields is as follows:

[0102] Step a: Determine the average deformation field and the deformation field standard deviation of the n third deformation fields.

[0103] Optionally, the uncertainty of the n third deformation fields is determined according to the dispersion degree of the n third deformation fields. Therefore, first determine the average deformation field of the n third deformation fields and the standard deviation of the deformation field.

[0104] Among them, the average deformation field of the n third deformation fields is:

[0105]

[0106] Among them, c represents the c-th channel of the deformation field (i.e., the displacements in the x, y, and z directions), and i represents the i-th forward prediction. represents the third deformation field obtained by the i-th forward prediction.

[0107] The standard deviation of the deformation fields of the n third deformation fields is:

[0108]

[0109] Step b: Determine the deformation field uncertainty based on the standard deviation of the deformation field and the average deformation field.

[0110] In a possible implementation manner, the deformation field uncertainty is determined according to the absolute value of the ratio of the standard deviation of the deformation field to the average deformation field. The method is as follows:

[0111]

[0112] Step c: Determine the first loss weight based on the deformation field uncertainty, and the first loss weight has a positive correlation with the deformation field uncertainty.

[0113] Determine the first loss weight according to the deformation field uncertainty. When the deformation field uncertainty is larger, it indicates that the dispersion degree of the predicted deformation field is larger, that is, the deformation fields between the predicted sample images tend to produce more uncertain predictions. Therefore, a larger first loss weight needs to be set to constrain the solution space of the deformation field, that is, the first loss weight has a positive correlation with the deformation field uncertainty.

[0114] In a possible implementation manner, the method for determining the first loss weight based on the deformation field uncertainty is as follows:

[0115]

[0116] Among them, λ φ is the first loss weight, Ⅱ(·) is the indicator function, that is, when it satisfies , it is set to "1", and when , it is set to "0", and v represents the v-th voxel; represents the volume size, that is, the sum of the deformation field uncertainties corresponding to each voxel , k 1 is the scaling scalar value of the first loss weight, which is used to give the weight an empirically reasonable upper limit; τ 1 is the threshold for selecting the uncertain target.

[0117] That is, by setting a threshold for the deformation field uncertainty, the first loss weight is determined according to the proportion of the deformation field uncertainty greater than the threshold in the deformation field uncertainties corresponding to all voxels. When the proportion is larger, that is, the uncertainty is greater, the corresponding first loss weight is larger, that is, the spatial regularization strength is stronger.

[0118] Step 4: Determine the second loss weight based on n third registered images.

[0119] Optionally, the second loss weight is the loss weight of the temporal regularization loss. The steps for determining the second loss weight based on n third registered images are as follows:

[0120] Step a: Determine the average registered image and the standard deviation of the registered images of the n third registered images.

[0121] Optionally, in the same way as determining the deformation field uncertainty above, the uncertainty of the n third registered images is determined according to the degree of dispersion of the n third registered images. Therefore, first, the average registered image and the standard deviation of the registered images of the n third registered images are obtained.

[0122] Among them, the average registered image of the n third registered images is:

[0123]

[0124] Among them, is the third registered image obtained by deforming the sample to-be-registered image through the third deformation field obtained by the i-th forward prediction.

[0125] The standard deviation of the registered images of the n third registered images is:

[0126]

[0127] Step b: Determine the registered image uncertainty based on the standard deviation of the registered images and the average registered image.

[0128] Optionally, the registered image uncertainty is also determined according to the absolute value of the ratio of the standard deviation of the registered images to the average registered image. The method is as follows:

[0129]

[0130] Step c: Determine the second loss weight based on the registered image uncertainty. The second loss weight has a positive correlation with the registered image uncertainty.

[0131] Determine the first loss weight according to the registered image uncertainty. When the registered image uncertainty When it is larger, it indicates that the discreteness of the registered image obtained based on the predicted deformation field is greater. Correspondingly, the discreteness of the predicted deformation field is also greater, that is, the deformation fields between the predicted sample image pairs tend to produce more uncertain predictions. Therefore, a larger second loss weight needs to be set to constrain the solution space of the deformation field, that is, the second loss weight is positively correlated with the uncertainty of the deformation field.

[0132] In a possible implementation manner, the method for determining the second loss weight based on the uncertainty of the registered image is as follows:

[0133]

[0134] Among them, λ c is the second loss weight, Ⅱ(·) is an indicator function, that is, when is satisfied, it is set to "1", and when is satisfied, it is set to "0", v represents the v-th voxel; represents the volume size, that is, the sum of the uncertainties of the registered images corresponding to each voxel , k 2 is the scaling scalar value of the second loss weight, which is used to give the weight an empirically reasonable upper limit; τ 2 is the threshold for selecting the uncertain target.

[0135] That is, by setting a threshold for the uncertainty of the registered image, the second loss weight is determined according to the proportion of the uncertainty of the registered image greater than the threshold in the uncertainties of the registered images corresponding to all voxels. When the proportion is larger, that is, the uncertainty is greater, the corresponding second loss weight is larger, that is, the time regularization strength is stronger, so as to constrain the difference in the deformation field in time during the training of the first registration network.

[0136] After determining the first loss weight and the second loss weight, training the first registration network based on the similarity loss, spatial regularization loss, and time regularization loss includes the following steps:

[0137] Step 1: Based on the similarity loss, spatial regularization loss, first loss weight, time regularization loss, and second loss weight, calculate the total loss by weighting.

[0138] The first loss weight is the loss weight of the spatial regularization loss, and the second loss weight is the loss weight of the time regularization loss. Then, the total loss is obtained by weighting according to the similarity loss, spatial regularization loss, time regularization loss, and the corresponding weights. The loss function is as follows:

[0139]

[0140] Among them, is the similarity loss, is the spatial regularization loss, and λ φ is the first loss weight, is the temporal regularization loss, and λ c is the second loss weight.

[0141] Step 2: Train the first registration network based on the total loss.

[0142] After determining the total loss, train the first registration network based on the total loss. When the loss function reaches the convergence condition, the training of the first registration network can be completed.

[0143] In this embodiment, the temporal regularization loss is introduced, and finally the first registration network is trained based on the similarity loss, the temporal regularization loss, and the spatial regularization loss, so as to constrain the solution space of the deformation field in both the temporal and spatial dimensions, optimize the training effect of the first registration network, and further improve the accuracy of the first registration network in predicting the deformation field.

[0144] Moreover, in this embodiment, a first loss weight is set for the spatial regularization loss, and a second loss weight is set for the temporal regularization loss. The uncertainty of the deformation field and the uncertainty of the registered image are determined based on the n - th forward prediction results of the second registration network. Then, according to the uncertainty of the deformation field and the uncertainty of the registered image, the training difficulty of the training samples is judged, so as to adjust the first loss weight and the second loss weight to make the weights adapt to the current training samples, and further optimize the training effect of the first registration network, which helps to improve the accuracy of the first registration network in predicting the deformation field.

[0145] Please refer to Figure 5 , which shows the flowchart of an image registration method provided by another exemplary embodiment of the present application. This embodiment is described by taking the method being used in a computer device as an example. The method includes the following steps.

[0146] Step 501: Obtain a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image.

[0147] Step 502: Input the pair of sample images into the first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image.

[0148] The implementation manners of Step 501 and Step 502 can refer to the above - mentioned Step 301 and Step 302, and will not be elaborated in this embodiment.

[0149] Step 503: Add random perturbations to the pair of sample images.

[0150] The inputs of the first registration network and the second registration network are the same training samples, that is, the same pair of sample images. During the training process, if the deformation fields predicted by the first registration network and the second registration network are consistent under different perturbations, the prediction performance of the first registration network and the second registration network is better, that is, the prediction accuracy and robustness of the network are stronger. Therefore, to further improve the prediction performance of the first registration network, when inputting the pair of sample images into the second registration network, random perturbations are added to the pair of sample images to enable the first registration network and the second registration network to predict the deformation field between the pair of sample images under different perturbations.

[0151] Optionally, the added random perturbation can be random Gaussian noise, that is, random Gaussian noise is added to both the sample image to be registered and the sample target image input to the second registration network.

[0152] Step 504: Input the pair of sample images with added random perturbations into the second registration network to obtain the second deformation field between the sample image to be registered and the sample target image.

[0153] After inputting the pair of sample images with added random Gaussian noise into the second registration network, the second deformation field is obtained. The second deformation field is the deformation field predicted after being perturbed, while the first deformation field is the deformation field predicted without being perturbed.

[0154] Step 505: Perform deformation processing on the sample image to be registered without added random perturbations through the second deformation field to obtain the second registered image.

[0155] In a possible implementation manner, after obtaining the second deformation field, since the STN network is used to perform deformation processing on the sample image to be registered and interpolation operations are required during the deformation processing, to avoid the influence of random perturbations on this interpolation operation, when inputting the second deformation field and the sample image to be registered into the STN network, the sample image to be registered without added random perturbations is input, that is, the sample image to be registered without added random perturbations is deformed to obtain the second registered image.

[0156] Step 506: Train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

[0157] The process of training the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image can refer to Steps 404 to 407 in the above embodiment, and will not be elaborated in this embodiment.

[0158] It should be noted that when determining the similarity loss based on the first registered image and the sample target image, the sample target image used is the sample target image without added random perturbations.

[0159] Step 507: Update the second network parameters of the second registration network based on the first network parameters of the first registration network.

[0160] After each training, the first network parameters of the first registration network will be updated accordingly. The second network parameters of the second registration network are network parameters at different training times from the first network parameters, and the second network parameters of the second registration network are updated following the first network parameters of the first registration network.

[0161] Optionally, perform EMA update on the current second network parameters of the second registration network based on the first network parameters to obtain the updated second network parameters.

[0162] Among them, when updating the second network parameters based on the first network parameters, the second network parameters are updated using Exponential Moving Average (EMA) as follows:

[0163] θ′ k = αθ′ k-1 + (1 - α)θ k

[0164] Among them, θ′ k is the second network parameter at the k-th time, θ k is the first network parameter at the k-th time, and α is the EMA decay parameter, which can take a value of 0.99.

[0165] Optionally, the initial parameters of the first network parameters and the second network parameters are the same. After the first network parameters are updated, the second network parameters are updated accordingly.

[0166] In this embodiment, when inputting the sample image pair into the second registration network, random perturbations are added to train the first registration network, so that when the first registration network predicts the deformation field between the same image pair under different perturbations, the prediction results can tend to be consistent, thereby improving the accuracy and robustness of the first registration network in predicting the deformation field.

[0167] In the above embodiment, when inputting the sample image pair into the second registration network, random perturbations are input to improve the robustness of the first registration network. The second registration network is also used for uncertainty estimation. During this process, n forward predictions need to be performed. To further improve the accuracy of the network in predicting the deformation field, during the n forward predictions, random perturbations are also added to the sample image pair, and then the second registration network performs n forward predictions on the sample image pair with added random perturbations to obtain n third deformation fields between the sample image to be registered and the sample target image. To avoid the influence of noise on the difference operation, during the deformation process, the sample image to be registered without added random perturbations is deformed through the n third deformation fields to obtain n third registered images.

[0168] In a possible implementation, Figure 6 As shown, the training process of the first registration network is as follows: the input sample image to be registered I m With sample target image I f To the first registration network 601 and the second registration network 602, wherein the second network parameters of the second registration network 602 are EMA updated based on the first network parameters of the first registration network 601, and noise ξ is added to the sample image pair input to the second registration network 602.

[0169] The deformation field φ is predicted by the first registration network 601 s , according to the deformation field φ s Determining the spatial regularization loss And the deformation field φ s And the sample image to be registered without adding noise I m Input to STN network 603 to obtain the first registered image I ws , so that according to the first registration image I ws With sample target image I f Determining Similarity Loss The deformation field φ is predicted by the second registration network 602 t , the deformation field φ t And the sample image to be registered without adding noise I m Input to STN network 603 to obtain the second registered image I wt , so that according to the first registered image I ws With the second registration image I wt Determining the Temporal Regularization Loss

[0170] Determining Similarity Loss Spatial Regularization Loss And the temporal regularization loss After that, the loss weight can also be determined by the second registration network 602. The second registration network 602 is used to perform n forward predictions to obtain n third deformation fields, and the deformation field uncertainty is estimated based on the n third deformation fields to obtain the deformation field uncertainty According to the uncertainty of the deformation field Determine the first loss weight λ φ After obtaining n third deformation fields, the n third deformation fields are compared with the sample image to be registered without adding noise I m Input into STN network 603, obtain n third registered images, and estimate the uncertainty of the registered image based on the n third registered images to obtain the uncertainty of the registered image Therefore, according to the uncertainty of the registered image Determine the second loss weight λ c , and finally based on the similarity loss spatial regularization loss the first loss weight λ φ , temporal regularization loss and the second loss weight λ c train the first registration network 601.

[0171] The training process of the first registration network is described in the above embodiments. After the first registration network is trained, the first registration network can be used for image registration, which will be described below with an exemplary embodiment.

[0172] Please refer to Figure 7 , which shows a flowchart of an image registration method provided by another exemplary embodiment of the present application. This embodiment is described by taking the method being used in a computer device as an example. The method includes the following steps.

[0173] Step 701, obtain a target image pair, where the target image pair includes an image to be registered and a reference image.

[0174] After the first registration network is trained, the first registration network can be used for image registration to align the image to be registered with the reference image in space, and then fuse the image information in the image to be registered and the reference image.

[0175] Optionally, when using the first registration network for image registration, two images can be registered, and any one of them can be used as the reference image; or multiple images can be registered, that is, multiple images are subjected to a spatial transformation to be aligned with a specified reference image in space.

[0176] Schematically, as Figure 8 shown, the target image pair includes the image to be registered I 1 and the reference image I 2 , which are respectively medical images of the lungs of the same patient collected at different times.

[0177] Step 702, input the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image.

[0178] Use the trained first registration network to perform image registration on the image to be registered and the reference image to obtain a target deformation field, and then perform a spatial transformation on the image to be registered based on the target deformation field.

[0179] Schematically, as Figure 8 shown, use the first registration network 801 to perform image registration to obtain a target deformation field φ 1 .

[0180] Step 703: Perform deformation processing on the image to be registered through the target deformation field to obtain the target registered image.

[0181] Optionally, when performing deformation processing on the image to be registered through the target deformation field, the STN network can also be used for spatial transformation. The target deformation field and the image to be registered are input into the STN network to obtain the target registered image, and the target registered image is spatially aligned with the reference image.

[0182] Schematically, as Figure 8 shown, after obtaining the target deformation field φ 1 it and the image to be registered I 1 are jointly input into the STN802 to obtain the target registered image I 3 .

[0183] When performing image registration using the first registration network trained by the solution provided in the above embodiment, the accuracy of the registration result can be improved. Taking the target image pairs for registration as the chest single-modal CT images of the same patient's lungs in the exhalation and inhalation phases as an example, the quantitative analysis of the registration effects of the iterative optimization method SyN and the deep learning-based benchmark method VM in the related technology and the registration effect of the first registration network after training in the embodiment of the present application is shown in Table 1:

[0184] Table 1

[0185] Solution Dice(%) ASD(mm) Percentage of folded voxels (%) Initial moving image 86.37 2.51 - SyN 86.67 2.27 0.09％ VM(λ=1) 90.77 1.90 0.06％ VM(λ=3) 90.45 1.94 <0.001％ VM(λ=5) 89.84 2.02 <0.0005％ The solution of this application 91.40 1.67 <0.0005％

[0186] As shown in Table 1, the deformation fields predicted by SyN, VM (weights of spatial regularization loss λ = 1, 3, 5), and the solution of the present application are respectively applied to the initially moving segmentation mask, and the Dice score and the average surface distance (ASD) are used to quantify the accuracy of registration. The Dice score is positively correlated with the registration accuracy, and the ASD is negatively correlated with the registration accuracy.

[0187] Moreover, the registration effect is also quantified by the percentage of folded voxels. The percentage of folded voxels refers to the proportion of the voxels lost when performing deformation processing on the image based on the predicted deformation field. The higher the proportion, the more image information is lost, that is, the worse the registration effect.

[0188] It can be seen that the solution of the present application can significantly improve the accuracy of image registration compared with the related technology solutions.

[0189] Figure 9 is the structural block diagram of the image registration device provided by an exemplary embodiment of the present application. As Figure 9 shown, the device includes:

[0190] The sample image acquisition module 901 is used to acquire a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image;

[0191] The first registration module 902 is used to input the pair of sample images into a first registration network, obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image;

[0192] The second registration module 903 is used to input the pair of sample images into a second registration network, obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image. The first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times;

[0193] The training module 904 is used to train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

[0194] Optionally, the training module 904 includes:

[0195] The first loss determination unit is used to determine a similarity loss based on the first registered image and the sample target image;

[0196] The second loss determination unit is used to determine a spatial regularization loss based on the first deformation field. The spatial regularization loss is used to impose spatial constraints on the deformation field;

[0197] The third loss determination unit is used to determine a temporal regularization loss based on the first registered image and the second registered image. The temporal regularization loss is used to impose training time constraints on the deformation field;

[0198] The training unit is used to train the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.

[0199] Optionally, the device further includes:

[0200] The weight determination module is used to determine a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss through the second registration network.

[0201] Optionally, the training unit is further used to:

[0202] Based on the similarity loss, the spatial regularization loss, the first loss weight, the temporal regularization loss, and the second loss weight, calculate the total loss by weighting.

[0203] Train the first registration network based on the total loss.

[0204] Optionally, the weight determination module includes:

[0205] A prediction unit for performing n forward predictions on the sample image pair through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, where n is an integer greater than or equal to 2;

[0206] A first processing unit for deforming the sample image to be registered through the n third deformation fields to obtain n third registered images;

[0207] A first weight determination unit for determining the first loss weight based on the n third deformation fields;

[0208] A second weight determination unit for determining the second loss weight based on the n third registered images.

[0209] Optionally, the first weight determination unit is further configured to:

[0210] Determine the average deformation field and the deformation field standard deviation of the n third deformation fields;

[0211] Based on the deformation field standard deviation and the average deformation field, determine the deformation field uncertainty;

[0212] Determine the first loss weight based on the deformation field uncertainty, and the first loss weight has a positive correlation with the deformation field uncertainty.

[0213] Optionally, the second weight determination unit is further configured to:

[0214] Determine the average registered image and the registered image standard deviation of the n third registered images;

[0215] Based on the registered image standard deviation and the average registered image, determine the registered image uncertainty;

[0216] Determine the second loss weight based on the registered image uncertainty, and the second loss weight has a positive correlation with the registered image uncertainty.

[0217] Optionally, the prediction unit is further configured to:

[0218] Add random perturbations to the sample image pair;

[0219] Performing n forward predictions on the sample image pair with added random perturbations through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image;

[0220] The first processing unit is further configured to:

[0221] Performing deformation processing on the sample image to be registered without added random perturbations through the n third deformation fields to obtain the n third registered images.

[0222] Optionally, the second registration module 902 includes:

[0223] A perturbation adding unit for adding random perturbations to the sample image pair;

[0224] A registration unit for inputting the sample image pair with added random perturbations into the second registration network to obtain the second deformation field between the sample image to be registered and the sample target image;

[0225] A second processing unit for performing deformation processing on the sample image to be registered without added random perturbations through the second deformation field to obtain the second registered image.

[0226] Optionally, the device further includes:

[0227] A parameter updating module for updating the second network parameters of the second registration network based on the first network parameters of the first registration network.

[0228] Optionally, the parameter updating module is further configured to:

[0229] Performing EMA update on the current second network parameters of the second registration network based on the first network parameters to obtain the updated second network parameters.

[0230] Optionally, the device further includes:

[0231] A target image acquisition module for acquiring a target image pair, where the target image pair includes an image to be registered and a reference image;

[0232] A third registration module for inputting the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image;

[0233] A deformation processing module for performing deformation processing on the image to be registered through the target deformation field to obtain a target registered image.

[0234] In summary, in the embodiments of the present application, the first registration network and the second registration network are used to perform image registration on the image to be registered and the target image respectively, so as to obtain the deformation fields corresponding to the network parameters at different training times. Finally, based on different deformation fields, the image to be registered is deformed to obtain different registered images. Furthermore, the first registration network can be trained by using different registered images and the target image, so that the network parameters obtained by the first registration network at different training times tend to be stable, that is, the deformation fields predicted by the first registration network at different training times tend to be consistent. Adding time constraints during the training process helps to improve the training effect of the first registration network, and further improves the accuracy of the deformation field predicted by the first registration network.

[0235] It should be noted that: for the device provided in the above embodiment, only the above division of each functional module is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device provided in the above embodiment and the method embodiment belong to the same concept, and the implementation process is detailed in the method embodiment, which will not be repeated here.

[0236] Please refer to Figure 10 , which shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application. Specifically: the computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory 1002 and a read only memory 1003, and a system bus 1005 connecting the system memory 1004 and the central processing unit 1001. The computer device 1000 further includes a basic input / output system (Input / Output, I / O system) 1006 for facilitating the transfer of information between various components within the computer, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0237] The basic input / output system 1006 includes a display 1008 for displaying information and input devices 1009 such as a mouse and a keyboard for user input. Among them, both the display 1008 and the input devices 1009 are connected to the central processing unit 1001 through an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may further include an input / output controller 1010 for receiving and processing inputs from multiple other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, a printer, or other types of output devices.

[0238] The large-capacity storage device 1007 is connected to the central processing unit 1001 through a large-capacity storage controller (not shown) connected to the system bus 1005. The large-capacity storage device 1007 and its associated computer-readable medium provide non-volatile storage for the computer device 1000. That is to say, the large-capacity storage device 1007 may include a computer-readable medium (not shown) such as a hard disk or a drive.

[0239] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic tape cartridges, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage media is not limited to the above several types. The above system memory 1004 and large-capacity storage device 1007 may be collectively referred to as memory.

[0240] The memory stores one or more programs, the one or more programs are configured to be executed by one or more central processing units 1001, the one or more programs contain instructions for implementing the above method, and the central processing unit 1001 executes the one or more programs to implement the methods provided by the above various method embodiments.

[0241] According to various embodiments of the present application, the computer device 1000 may also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1000 may be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or in other words, the network interface unit 1011 may also be used to connect to other types of networks or remote computer systems (not shown).

[0242] The memory further includes one or more programs, the one or more programs are stored in the memory, and the one or more programs contain steps for performing the methods executed by the computer device in the embodiments of the present application.

[0243] An embodiment of the present application further provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image registration method described in any of the above embodiments.

[0244] An embodiment of the present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image registration method provided in the above aspect.

[0245] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the computer-readable storage medium can be the computer-readable storage medium included in the memory in the above embodiments; it can also exist separately and be a computer-readable storage medium not assembled into the terminal. At least one instruction, at least one program, a code set or an instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image registration method described in any of the above method embodiments.

[0246] Optionally, the computer-readable storage medium may include: ROM, RAM, solid state drive (SSD, Solid State Drives) or optical disc, etc. Among them, RAM may include resistive random access memory (ReRAM, Resistance Random Access Memory) and dynamic random access memory (DRAM, Dynamic Random Access Memory). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0247] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disc, etc.

[0248] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image registration method, characterized in that, the method includes: Obtain a pair of sample images, where the pair of sample images includes a sample image to be registered and a sample target image; Input the pair of sample images into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image; Input the pair of sample images into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image, where the first network parameters of the first registration network and the second network parameters of the second registration network are network parameters at different training times; Train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

2. The method according to claim 1, characterized in that, the training of the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image includes: Determine a similarity loss based on the first registered image and the sample target image; Determine a spatial regularization loss based on the first deformation field, where the spatial regularization loss is used to impose a spatial constraint on the deformation field; Determine a temporal regularization loss based on the first registered image and the second registered image, where the temporal regularization loss is used to impose a training time constraint on the deformation field; Train the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.

3. The method according to claim 2, characterized in that, the method further includes: Determine a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss through the second registration network; the training of the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss includes: Based on the similarity loss, the spatial regularization loss, the first loss weight, the temporal regularization loss, and the second loss weight, calculate a total loss by weighted calculation; Train the first registration network based on the total loss.

4. The method according to claim 3, characterized in that, the determination of the first loss weight of the spatial regularization loss and the second loss weight of the temporal regularization loss through the second registration network includes: Perform n forward predictions on the pair of sample images through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, where n is an integer greater than or equal to 2; Perform deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registered images; Determine the first loss weight based on the n third deformation fields; Determine the second loss weight based on the n third registered images.

5. The method according to claim 4, characterized in that, Determining the first loss weight based on the n third deformation fields includes: Determining the average deformation field and the deformation field standard deviation of the n third deformation fields; Based on the deformation field standard deviation and the average deformation field, determining the deformation field uncertainty; Determining the first loss weight based on the deformation field uncertainty, and the first loss weight has a positive correlation with the deformation field uncertainty.

6. The method according to claim 4, wherein, Determining the second loss weight based on the n third registered images includes: Determining the average registered image and the registered image standard deviation of the n third registered images; Based on the registered image standard deviation and the average registered image, determining the registered image uncertainty; Determining the second loss weight based on the registered image uncertainty, and the second loss weight has a positive correlation with the registered image uncertainty.

7. The method according to claim 4, wherein, The step of obtaining the n third deformation fields between the sample to-be-registered image and the sample target image by performing n forward predictions on the sample image pair through the second registration network includes: Adding random perturbations to the sample image pair; Performing n forward predictions on the sample image pair with added random perturbations through the second registration network to obtain the n third deformation fields between the sample to-be-registered image and the sample target image; The step of deforming the sample to-be-registered image through the n third deformation fields to obtain n third registered images includes: Deforming the sample to-be-registered image without added random perturbations through the n third deformation fields to obtain the n third registered images.

8. The method according to any one of claims 1 to 7, wherein, The step of inputting the sample image pair into the second registration network to obtain the second deformation field between the sample to-be-registered image and the sample target image, and deforming the sample to-be-registered image through the second deformation field to obtain the second registered image includes: Adding random perturbations to the sample image pair; Inputting the sample image pair with added random perturbations into the second registration network to obtain the second deformation field between the sample to-be-registered image and the sample target image; Deforming the sample to-be-registered image without added random perturbations through the second deformation field to obtain the second registered image.

9. The method according to any one of claims 1 to 7, wherein, After training the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image, the method further includes: Updating the second network parameters of the second registration network based on the first network parameters of the first registration network.

10. The method according to claim 9, wherein, Updating the second network parameters of the second registration network based on the first network parameters of the first registration network includes: Based on the first network parameter, perform EMA update on the current second network parameter of the second registration network to obtain the updated second network parameter.

11. The method according to any one of claims 1 to 7, wherein, the method further includes: obtain a target image pair, where the target image pair includes an image to be registered and a reference image; input the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image; perform deformation processing on the image to be registered through the target deformation field to obtain a target registered image.

12. An image registration device, wherein, the device includes: a sample image acquisition module, configured to acquire a sample image pair, where the sample image pair includes a sample image to be registered and a sample target image; a first registration module, configured to input the sample image pair into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field to obtain a first registered image; a second registration module, configured to input the sample image pair into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registered image, where the first network parameter of the first registration network and the second network parameter of the second registration network are network parameters at different training times; a training module, configured to train the first registration network based on the first deformation field, the first registered image, the second registered image, and the sample target image.

13. A computer device, wherein, the computer device includes a processor and a memory, and at least one instruction, at least one program, a code set, or an instruction set is stored in the memory, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image registration method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein, at least one instruction, at least one program, a code set, or an instruction set is stored in the readable storage medium, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the image registration method according to any one of claims 1 to 11.

Images