Image registration method and device and computer equipment

Abstract

The invention relates to an image registration method and device and computer equipment. The method comprises the following steps: acquiring a multi-modal medical image group; the multi-modal medical image group comprises a reference medical image and at least one mobile medical image; inputting the multi-modal medical image group into a deformation field prediction model, and predicting a prediction deformation field between each mobile medical image and the reference medical image through the deformation field prediction model; the deformation field prediction model uses a label of a sample reference medical image as weak supervision information, and performs constraint training based on an evaluation result of a prediction sample deformation field output in a training process; and completing registration of the corresponding mobile medical image and the reference medical image based on each prediction deformation field. By adopting the method, the accuracy of the registration result can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method, device and computer equipment. Background Art

[0002] Image registration plays a vital role in radiotherapy planning, surgical planning, and pathology analysis.

[0003] Image registration can be performed on image pairs or on image groups. Taking a multimodal medical image group as an example, in the related art, when registering a multimodal medical image group, the medical image registration is usually performed based on a similarity measure between the medical images in the multimodal medical image group.

[0004] However, the related technology has the problem of inaccurate registration results when registering a multimodal medical image group. Summary of the invention

[0005] Based on this, it is necessary to provide an image registration method, device and computer equipment to address the above technical problems, which can improve the accuracy of the registration results.

[0006] In a first aspect, the present application provides an image registration method, comprising:

[0007] Acquire a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one moving medical image;

[0008] The multimodal medical image group is input into the deformation field prediction model, and the deformation field prediction model is used to predict the predicted deformation field between each mobile medical image and the reference medical image; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process;

[0009] Based on each predicted deformation field, the corresponding mobile medical image is registered with the reference medical image.

[0010] In one embodiment, the process of constructing the deformation field prediction model includes:

[0011] Acquire a multimodal sample medical image group and a corresponding real anatomical structure label map; the multimodal sample medical image group includes at least one sample moving medical image and a sample reference medical image;

[0012] Inputting a multimodal sample medical image group into an initial deformation field prediction model to obtain a plurality of predicted sample deformation fields;

[0013] The deformation fields of multiple predicted samples are evaluated by using an initial deformation field evaluation model to obtain evaluation results;

[0014] Based on the evaluation results, the parameters of the initial deformation field prediction model and the initial deformation field evaluation model are adjusted and iteratively executed to obtain the deformation field prediction model.

[0015] In one embodiment, multiple predicted sample deformation fields are evaluated by using an initial deformation field evaluation model to obtain evaluation results, including:

[0016] Based on the multiple predicted sample deformation fields, the corresponding sample moving medical image is registered with the sample reference medical image to obtain the registered sample medical image;

[0017] Extracting the sample intensity vector of each coordinate point in the registered sample medical image;

[0018] The sample intensity vector is input into the initial deformation field evaluation model to obtain the predicted anatomical structure probability value of each coordinate point, the structure corresponding to the maximum probability value among the predicted anatomical structure probability values ​​is used as the predicted anatomical structure label, and the predicted anatomical structure label is used as the evaluation result.

[0019] In one embodiment, adjusting the parameters of the initial deformation field prediction model based on the evaluation results includes:

[0020] Determine the loss function of the initial deformation field prediction model based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point;

[0021] Based on the loss function of the initial deformation field prediction model, the parameters of the initial deformation field prediction model are adjusted.

[0022] In one embodiment, the loss function includes a similarity measurement loss function, a label smoothness loss function, a deformation field smoothness loss function, and a discreteness loss function;

[0023] Based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point, the loss function of the initial deformation field prediction model is determined, including:

[0024] Based on multiple predicted sample deformation fields, a deformation field smoothness loss function is obtained; and, based on the true anatomical structure label map and the sample intensity vector of each coordinate point, a discreteness loss function is obtained; and, based on the true anatomical structure label map and the predicted anatomical structure probability value of each coordinate point, a similarity measurement loss function is obtained; and, based on the predicted anatomical structure label of each coordinate point, a label smoothness loss function is obtained.

[0025] In one embodiment, the process of constructing the initial deformation field assessment model includes:

[0026] Extracting a set of intensity vectors of a multimodal sample medical image group based on a true anatomical structure label map of a reference image, and determining the mean and covariance of the Gaussian components of each anatomical structure;

[0027] Based on the mean and covariance of the Gaussian components of each anatomical structure, the Gaussian mixture model components corresponding to each anatomical structure are constructed;

[0028] Based on the intensity vector set of the multimodal sample medical image group, the Gaussian mixture model component weight corresponding to each anatomical structure is constructed;

[0029] According to the Gaussian mixture model components corresponding to each anatomical structure and the corresponding Gaussian mixture model component weights, an initial deformation field evaluation model is constructed.

[0030] In one embodiment, the method further comprises:

[0031] Segment the anatomical structure of the registered medical image group to obtain a segmentation result;

[0032] Based on the segmentation results, the image group registration process is evaluated.

[0033] In one embodiment, the image group registration process is evaluated based on the segmentation result, including:

[0034] Calculating the similarity coefficient between the segmentation result and the anatomical structure of the reference medical image;

[0035] Based on the similarity coefficient, the image registration process is evaluated.

[0036] In a second aspect, the present application also provides an image registration device, comprising:

[0037] An acquisition module, configured to acquire a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one moving medical image;

[0038] A prediction module is used to input the multimodal medical image group into the deformation field prediction model, and predict the predicted deformation field between each mobile medical image and the reference medical image through the deformation field prediction model; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process;

[0039] The registration module is used to register the corresponding mobile medical image to the reference medical image based on each predicted deformation field to obtain a registered medical image group.

[0040] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the content of any one embodiment of the image registration method in the first aspect is implemented.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content of any one embodiment of the image registration method in the first aspect above.

[0042] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the content of any one embodiment of the image registration method in the first aspect above.

[0043] The above-mentioned image registration method, device and computer equipment obtain a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one mobile medical image; the multimodal medical image group is input into a deformation field prediction model, and the deformation field prediction model is used to predict the predicted deformation field between each mobile medical image and the reference medical image; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information, and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process; and the corresponding mobile medical image and the reference medical image are registered based on each predicted deformation field. In the process of registering the medical image group, the deformation field prediction model can be used to simultaneously analyze multiple medical images in a medical image group, and multiple predicted deformation fields can be quickly obtained, so that the registration process of the image group can be quickly completed based on multiple predicted deformation fields. In theory, as the number of medical images in the medical image group increases, the number of modalities in the image group will also increase accordingly, and the improvement of complementary information is more conducive to the registration process of the image group. In this way, the time for image registration can be greatly reduced, the efficiency of image group registration can be improved, and the complexity of the image group registration process can be reduced. At the same time, since the deformation field prediction model uses the labels of the sample reference medical images as weak supervision information during the training process, multiple medical images in the multimodal medical image group can be quickly and accurately aligned to the common anatomical space. In addition, the deformation field prediction model is constrained training based on the evaluation results of the output predicted sample deformation field. Then, the predicted deformation field output by the deformation field prediction model after constraint training has a higher accuracy, and the accuracy of image registration can be improved based on the predicted deformation field. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] FIG1 is a diagram of an application environment of an image registration method according to an embodiment;

[0046] FIG2 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0047] FIG3 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0048] FIG4 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0049] FIG5 is a schematic diagram of a process of constructing a Gaussian mixture model corresponding to an anatomical structure in one embodiment;

[0050] FIG6 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0051] FIG7 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0052] FIG8 is a schematic diagram of a training process of a deformation field prediction model in one embodiment;

[0053] FIG9 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0054] FIG10 is a schematic diagram of a flow chart of an image registration method in one embodiment;

[0055] FIG11 is a schematic diagram of a medical image group obtained by different registration methods in one embodiment;

[0056] FIG. 12 is a structural block diagram of an image registration device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Before introducing the technical solution of the present application in detail, a brief introduction to the background technology of the present application is first given.

[0059] In clinical practice, doctors usually need to combine medical images of patients in different modalities to obtain more comprehensive patient information. Due to differences in acquisition methods, resolution, and contrast of different modal images, as well as the movement of patients during acquisition, the anatomical structure positions and shapes of different modal images will be distorted and inconsistent. Therefore, it is necessary to group medical images of different modalities to align multiple medical images in space. In this way, it is convenient for doctors to accurately compare and analyze multiple subsequent medical images, so as to have a more comprehensive understanding of the patient's pathological condition.

[0060] It can be seen that multimodal medical image group registration plays a vital role in radiotherapy planning, surgical planning, and pathological analysis. Multimodal medical image group registration involves spatial transformation and morphological matching between images to ensure that they are aligned in the same anatomical coordinate system. How to align a set of multimodal medical images to a reference image at the same time is a challenging problem.

[0061] In the related art, when registering a multimodal medical image group, the similarity measure between the two medical images is usually determined based on the statistical dependence between the two medical images in the multimodal medical image group, so as to perform medical image registration. However, when the number of images included in the multimodal medical image group is large, the pairwise registration needs to register the two medical images in sequence, and its registration efficiency is low. At the same time, there are many challenges in directly extending the pairwise registration method to the group registration of multiple images. Existing group registration methods usually estimate the required spatial transformation through an intensity-based similarity measure. For example, assuming that every two images are conditionally independent given a third image, the similarity measure can be obtained by calculating the sum of all pairwise mutual information in the deformed image as a similarity measure. However, this method has a large computational complexity as the number of images in each group increases. In addition, since the connection between each moving image is not taken into account during the pairwise registration process, the smoothness between the registered image groups is poor, and there is a problem of inaccurate registration results.

[0062] In another implementation, a multivariate Gaussian mixture model (GMM) is used to construct a GMM for each group of images to achieve group registration, but an appropriate number of Gaussian components K needs to be pre-selected. Since the number of anatomical structures in each group of images may be inconsistent, an inappropriate parameter K will lead to registration failure, limiting its applicability in batch group registration. In addition, when the initial misalignment of small anatomical structure regions in the image group is large, the GMM constructed based on a single image group may not accurately model the anatomical structure corresponding to each Gaussian component and de-disperse, resulting in group registration failure.

[0063] In view of the above problems, the present application provides an image registration method, device and computer equipment, which can improve the accuracy of the registration results. Of course, the technical solution provided in the embodiments of the present application is not limited to solving the above problems, but also has other technical effects, which can be specifically described in the following embodiments. The technical solution of the present application is described in detail below.

[0064] The image registration method provided in the embodiment of the present application can be applied in the application environment as shown in FIG. 1 .

[0065] The computer device in the application environment can be a server, a personal computer, a laptop computer, a smart phone, a tablet computer, a smart mobile phone, etc. The computer device may include a processor, a memory and a network interface connected via a system bus or wirelessly. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the image registration process. The network interface of the computer device is used to communicate with an external terminal through a network connection, and the computer program is executed by the processor to implement an image registration method.

[0066] In an exemplary embodiment, as shown in FIG2 , an image registration method is provided, which is described by taking the method applied to the computer device in FIG1 as an example, and includes the following steps 201 to 203. Among them:

[0067] S201, acquiring a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one mobile medical image.

[0068] Among them, the multimodal medical image group can be an image generated by different medical imaging devices or different acquisition parameters. For example, different medical imaging devices can be magnetic resonance imaging devices (MRI), computed tomography (CT) and positron emission tomography (PET). Medical images acquired by different acquisition parameters of the same MRI device can be T1-weighted images and T2-weighted images. It should be noted that the multimodal medical image group refers to medical images of the same patient or the same scanning part of the same patient. The medical images in the medical image group can be two-dimensional medical images or three-dimensional medical images. For example, the multimodal medical image group can be a brain image under multiple modalities, or a multimodal carotid artery image with inconsistent intensity mapping.

[0069] In an embodiment of the present application, a multimodal medical image group is stored in a Picture Archiving and Communication Systems (PACS), and a computer device can obtain multiple medical images from the PACS and use the multiple medical images as a multimodal medical image group. Alternatively, the computer device can also send image acquisition instructions to different medical imaging devices, and different medical imaging devices can perform image acquisition for the same patient or the same scanning part of the same patient according to the image acquisition instructions, and send the acquired multiple medical images to the computer device. The present application does not limit the method for obtaining a multimodal medical image group.

[0070] It is understandable that the reference medical image can be any medical image in the multimodal medical image group, and other medical images except the reference medical image can be used as mobile medical images. For example, the multimodal medical image group includes 5 medical images of different modalities. If the first medical image is the reference medical image, then the 2nd to 5th medical images are all mobile medical images.

[0071] S202, input the multimodal medical image group into the deformation field prediction model, and predict the predicted deformation field between each mobile medical image and the reference medical image through the deformation field prediction model; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information, and performs constrained training based on the evaluation results of the predicted sample deformation field output by the training process.

[0072] The deformation field refers to the information used to describe the local shape change of the image during the registration process, and can represent the displacement vector of each pixel or voxel during the registration process.

[0073] In an embodiment of the present application, after acquiring a multimodal medical image group, the computer device can use the multimodal medical image group as an input signal and input it into a deformation field prediction model. The deformation field prediction model can extract the motion feature information of each mobile medical image in the multimodal medical image group and the reference feature information of the reference medical image, and determine the predicted deformation field corresponding to each mobile medical image and the reference medical image based on the difference between each motion feature information and the reference feature information.

[0074] Among them, the deformation field prediction model can be any machine learning model, such as a deep learning model, a random forest model, a decision tree model, etc. Among them, the network in the machine learning model can be a separate neural network, or a cascade network composed of at least two neural networks. The embodiments of the present application are not limited to this. Furthermore, the deformation field prediction model is obtained by pre-training using sample medical images, and the training process can be supervised learning, unsupervised learning, or semi-supervised learning.

[0075] It should be noted that during the training process, the deformation field prediction model uses the labels of the sample reference medical images as weak supervision information, and the predicted sample deformation field output by each training process needs to be evaluated and retrained based on the evaluation results. Through this iterative process, the trained deformation field prediction model can be made more accurate.

[0076] S203: completing registration of the corresponding mobile medical image with the reference medical image based on each predicted deformation field.

[0077] In the embodiment of the present application, for the registration process between any mobile medical image and the reference medical image, the computer device can move the pixel position of the mobile medical image according to the corresponding predicted deformation field to register the mobile medical image to the reference medical image. When the registration of all mobile medical images with the reference medical image is completed, a registered medical image group is obtained.

[0078] In the above-mentioned image registration method, a multimodal medical image group is obtained; the multimodal medical image group includes a reference medical image and at least one mobile medical image; the multimodal medical image group is input into a deformation field prediction model, and the deformation field prediction model is used to predict the predicted deformation field between each mobile medical image and the reference medical image; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information, and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process; and the corresponding mobile medical image and the reference medical image are registered based on each predicted deformation field. In the process of registering the medical image group, the deformation field prediction model can be used to simultaneously analyze multiple medical images in a medical image group, and multiple predicted deformation fields can be quickly obtained, so that the registration process of the image group can be quickly completed based on multiple predicted deformation fields. In theory, as the number of medical images in the medical image group increases, the number of modalities in the image group will also increase, and the improvement of complementary information is more conducive to the registration process of the image group. In this way, the time of image registration can be greatly reduced, the efficiency of image group registration is improved, and the complexity of the image group registration process is reduced. At the same time, since the deformation field prediction model uses the labels of sample reference medical images as weak supervision information during the training process, multiple medical images in the multimodal medical image group can be quickly and accurately aligned to the common anatomical space. In addition, the deformation field prediction model is constrained training based on the evaluation results of the output predicted sample deformation field. Therefore, the predicted deformation field output by the deformation field prediction model after constraint training is more accurate, and the accuracy of image registration can be improved based on the predicted deformation field.

[0079] The above embodiment is an introduction to the actual application process of the deformation field prediction model. Next, the training process of the deformation field prediction model is described in detail through an embodiment. As shown in FIG3 , the construction process of the deformation field prediction model includes the following steps:

[0080] S301, obtaining a multimodal sample medical image group and a corresponding real anatomical structure label map; the multimodal sample medical image group includes at least one sample moving medical image and a sample reference medical image.

[0081] In an embodiment of the present application, a multimodal sample medical image group is stored in a Picture Archiving and Communication Systems (PACS), and a computer device can obtain multiple medical images from the PACS and use the multiple medical images as a multimodal sample medical image group. Alternatively, the computer device can also send image acquisition instructions to different medical imaging devices, and different medical imaging devices can perform image acquisition for the same patient or the same scanning part of the same patient according to the image acquisition instructions, and send the acquired multiple medical images to the computer device to obtain a multimodal sample medical image group. The present application does not limit the method for obtaining a multimodal sample medical image group.

[0082] It should be noted that during the training process of the deformation field prediction model, the multimodal sample medical image group includes a real anatomical structure label map, which is obtained by annotating the anatomical structure to which each pixel point in the sample reference medical image belongs.

[0083] S302: Input the multimodal sample medical image group into an initial deformation field prediction model to obtain a plurality of predicted sample deformation fields.

[0084] In an embodiment of the present application, a computer device can use a multimodal sample medical image group as an input signal and input it into an initial deformation field prediction model. The initial deformation field prediction model can extract sample movement feature information of each sample movement medical image in the multimodal sample medical image group, and extract sample reference feature information of the sample reference medical image in the multimodal sample medical image group. The displacement difference between each sample movement feature information and the sample reference feature information is calculated, and the displacement difference is used as the predicted sample deformation field between each sample movement medical image and the sample reference medical image.

[0085] S303, evaluating the deformation fields of multiple predicted samples by using the initial deformation field evaluation model to obtain evaluation results.

[0086] In an embodiment of the present application, after obtaining multiple predicted sample deformation fields, the computer device can input the multiple predicted sample deformation fields into the initial deformation field evaluation model. The multiple predicted sample deformation fields are compared with the real sample deformation field through the initial deformation field evaluation model, and based on the comparison results, the evaluation results of the multiple predicted sample deformation fields are determined. Alternatively, the anatomical structure of the registered medical image group can be determined based on the multiple predicted sample deformation fields through the initial deformation field evaluation model. And the anatomical structure of the registered medical image group is compared with the real anatomical structure label map, and based on the comparison results, the evaluation results of the multiple predicted sample deformation fields are determined. The embodiment of the present application does not limit the way in which the multiple predicted sample deformation fields are evaluated through the initial deformation field evaluation model.

[0087] S304: adjusting parameters of the initial deformation field prediction model and the initial deformation field evaluation model based on the evaluation result, and performing the adjustment iteratively to obtain the deformation field prediction model.

[0088] In an embodiment of the present application, after obtaining the evaluation result, the computer device can adjust the parameters of the initial deformation field prediction model multiple times until the output result of the initial deformation field prediction model meets the current conditions. The output result of the initial deformation field prediction model is then input into the initial deformation field evaluation model, and the parameters of the initial deformation field evaluation model are adjusted once using the output result of the initial deformation field evaluation model. Based on the output result of the adjusted initial deformation field evaluation model, the parameters of the initial deformation field prediction model are adjusted multiple times, and the execution is iterative until the output result of the initial deformation field prediction model meets the preset conditions, thereby obtaining the deformation field prediction model.

[0089] It should be noted that during the training process of the deformation field prediction model and the deformation field evaluation model, the parameters of the initial deformation field prediction model are adjusted multiple times, and then the parameters of the initial deformation field evaluation model are adjusted once on this basis, thus completing an iterative process. Multiple iterative processes are completed in the above manner until the output result of the initial deformation field prediction model meets the preset conditions, and the deformation field prediction model is obtained.

[0090] Alternatively, the training stop condition of the deformation field prediction model may also be that the number of iterations reaches a preset threshold. For example, when the number of iterations reaches 300, the prediction model output by the deformation field prediction model becomes more accurate. At this time, the training process of the deformation field prediction model is completed and the deformation field prediction model is obtained.

[0091] In the above-mentioned image registration method, a multimodal sample medical image group and a corresponding real anatomical structure label map are obtained; the multimodal sample medical image group includes at least one sample moving medical image and a sample reference medical image; the multimodal sample medical image group is input into the initial deformation field prediction model to obtain multiple predicted sample deformation fields; the multiple predicted sample deformation fields are evaluated by the initial deformation field evaluation model to obtain evaluation results; the parameters of the initial deformation field prediction model and the initial deformation field evaluation model are adjusted based on the evaluation results, and iterative execution is performed to obtain the deformation field prediction model. In the process of training the initial deformation field prediction model using the multimodal sample medical image group, the method can evaluate the training results through the initial deformation field evaluation model, so that the parameters of the initial deformation field prediction model can be adjusted in a targeted manner according to the evaluation results, and the parameters of the initial deformation field evaluation model are also adjusted during the iteration process. In other words, the accuracy of the deformation field prediction and deformation field evaluation increases continuously during the training process, and after the iterative execution is completed, the obtained deformation field prediction model will also be more accurate.

[0092] In one embodiment, the specific contents of the construction process of the above deformation field assessment model are introduced, as shown in FIG4 , and the specific contents include:

[0093] S401, extracting a set of intensity vectors of a multimodal sample medical image group based on a real anatomical structure label map of a reference image, and determining a mean value and covariance of Gaussian components of each anatomical structure.

[0094] For example, assume that M multimodal sample medical image groups are needed during training. Represents each multimodal sample medical image group, defined in spatial coordinates The purpose of image group registration is to align the medical image group I into a common anatomical space In the image group registration process, N-1 mobile medical images need to be registered to the reference medical image, and the space where the reference medical image is located is the common anatomical space. Each reference medical image in the multimodal sample medical image has an anatomical label map S ref , each coordinate point Each corresponds to an anatomical label value k, , K represents the number of anatomical label types.

[0095] The multimodal sample medical image group includes N sample medical images of different modalities. Each coordinate point x corresponds to N different pixel values, and an N-dimensional intensity vector is obtained. Represents a set of pixel values. Multimodal sample medical image group at coordinate point Through the deformation field The resampled intensity vector can be expressed as ,in, Indicates image group I i At coordinate point x, through the deformation field The pixel value after resampling.

[0096] FIG5 is a schematic diagram showing the process of constructing a Gaussian mixture model corresponding to anatomy. For each multimodal medical image group in the figure, a Gaussian mixture model with an anatomical label map S is used. ref The set of intensity vectors for each anatomical structure can be calculated, represents the coordinate space corresponding to the anatomical structure k. The set of intensity vectors corresponding to the anatomical structure k can be expressed as , Since not every image group has K kinds of anatomical structures, let Indicates the number of specific anatomical structures in a medical image group. Each medical image group can be To avoid the prior bias of the intensity vector set of a single image group, all objects in the training set can be treated as a whole to calculate the multivariate Gaussian distribution as prior information. Specifically, the intensity vectors obtained from the M training set image groups can be combined according to the anatomical label category to obtain K overall intensity vector sets. Each intensity vector set can be expressed as: , that is, the size is An array of . represents the number of coordinate points corresponding to the anatomical structure k in the M training set image groups. Therefore, the average value of the Gaussian component of each anatomical structure can be expressed as:

[0097]

[0098] The Gaussian component covariance of each anatomical structure can be expressed as:

[0099]

[0100] S402: constructing Gaussian mixture model components corresponding to each anatomical structure based on the Gaussian component mean and covariance of each anatomical structure.

[0101] Continuing to refer to FIG5 , each circle in the figure represents an anatomical structure type, and there are 5 anatomical structure types in the figure, and the number of corresponding Gaussian components is also 5. Since the intensity vector contains complementary information from all modality images, the intensity vector set corresponding to different anatomical structures should have specific properties.

[0102] Intensity Vector Contains N variables and satisfies multivariate Gaussian distribution, and its probability density function can be expressed as:

[0103]

[0104]

[0105] Intensity Vector Conditional probability distribution belonging to Gaussian component k It can be expressed as:

[0106]

[0107] In order to construct the deformation field assessment model, it is necessary to calculate the parameters corresponding to each Gaussian component. The parameters corresponding to each Gaussian component can be estimated by the expectation maximization (EM) algorithm, which optimizes the parameters of each component through unsupervised iteration until convergence, but it is inevitable that certain errors will occur. Therefore, in order to reduce the error in the parameter calculation process. The intensity vector can be classified into K corresponding feature spaces, and the mean, covariance and prior probability corresponding to each Gaussian component can be calculated.

[0108] S403, constructing a Gaussian mixture model component weight corresponding to each anatomical structure based on the intensity vector set of the multimodal sample medical image group.

[0109] In the embodiment of the present application, the weight corresponding to the Gaussian component of each anatomical structure can be expressed as:

[0110]

[0111] S404: Obtain a deformation field evaluation model according to the Gaussian component of each anatomical structure and the corresponding weight.

[0112] Before group registration, the deformation field =0, therefore, the mean, covariance and weight of the multivariate Gaussian distribution corresponding to each Gaussian component can be directly calculated from the set of intensity vectors corresponding to the undeformed training set image group. As the training process progresses, the deformation field The optimization can be iterated gradually, and each group of sample moving medical images is gradually aligned with the corresponding sample reference medical images. In this case, the mean value and covariance of the initial deformation field evaluation model need to be optimized. For example, after the initial deformation field prediction model is trained 10 times, the initial deformation field evaluation model can be optimized once.

[0113] The above-mentioned real anatomical structure label map based on the reference image extracts the intensity vector set of the multimodal sample medical image group, determines the Gaussian component mean and covariance of each anatomical structure; constructs the Gaussian mixture model component corresponding to each anatomical structure based on the Gaussian component mean and covariance of each anatomical structure; constructs the Gaussian mixture model component weight corresponding to each anatomical structure based on the intensity vector set of the multimodal sample medical image group; constructs the initial deformation field evaluation model according to the Gaussian mixture model component corresponding to each anatomical structure and the corresponding Gaussian mixture model component weight. The method extracts the intensity vector set in the multimodal sample medical image group, and then accurately obtains the Gaussian component mean and covariance of each anatomical structure based on the extracted intensity vector set, so that the Gaussian mixture model component corresponding to each anatomical structure can be accurately constructed, and then based on the Gaussian mixture model component weight, the initial deformation field evaluation model suitable for deformation field evaluation can be accurately determined. In addition, by constructing a model composed of multiple Gaussian mixture model components as the initial deformation field evaluation model, the anatomical structure distribution deviation caused by a single image can be avoided.

[0114] During the training process, the accuracy of the deformation field prediction model is increased due to the participation of the deformation field evaluation model. Then, in one embodiment, as shown in FIG6 , the specific contents of the evaluation results obtained by evaluating the deformation fields of multiple prediction samples by the initial deformation field evaluation model include:

[0115] S501 , based on a plurality of predicted sample deformation fields, registering a corresponding sample mobile medical image to a sample reference medical image to obtain a registered sample medical image.

[0116] In the embodiment of the present application, for the registration process between any sample moving medical image and the sample reference medical image, the computer device can move the pixel position of the sample moving medical image according to the corresponding predicted sample deformation field to register the sample moving medical image to the sample reference medical image. When the registration of all sample moving medical images and the sample reference medical image is completed, the registered sample medical image is obtained.

[0117] S502, extracting a sample intensity vector of each coordinate point in the registered sample medical image.

[0118] The sample intensity vector usually refers to a specific coordinate point, which contains the intensity value of the point in different modes or channels.

[0119] In an embodiment of the present application, after the sample medical image group is registered, the computer device can obtain the pixel value of each pixel on the registered sample medical image, and use the pixel value of each pixel in different images as the sample intensity vector of each coordinate point.

[0120] S503, input the sample intensity vector into the initial deformation field evaluation model to obtain the predicted anatomical structure probability value for each coordinate point, use the structure corresponding to the maximum probability value among the predicted anatomical structure probability values ​​as the predicted anatomical structure label, and use the predicted anatomical structure label as the evaluation result.

[0121] In the embodiment of the present application, after obtaining the sample intensity vector, the computer device can use the sample intensity vector as an input signal and input it into the initial deformation field evaluation model. The sample intensity vector is analyzed by the initial deformation field evaluation model to determine the predicted anatomical structure probability value of each coordinate point. Afterwards, the structure corresponding to the maximum probability value is determined from the predicted anatomical structure probability value, and the structure is used as the anatomical structure label.

[0122] In the above-mentioned image registration method, at least one sample mobile medical image is registered to a sample reference medical image based on multiple predicted sample deformation fields to obtain a registered sample medical image; a sample intensity vector of each coordinate point in the registered sample medical image is extracted; the sample intensity vector is input into a deformation field evaluation model to obtain a predicted anatomical structure probability value of each coordinate point, and the structure corresponding to the maximum probability value among the predicted anatomical structure probability values ​​is used as a predicted anatomical structure label, and the predicted anatomical structure label is used as an evaluation result. The method can achieve spatial consistency between different images by registering the sample mobile medical image to the sample reference medical image based on multiple predicted sample deformation fields. This enables medical images from different sources, different times or different devices to be compared and analyzed in the same anatomical coordinate system, so that the sample intensity vector of each coordinate point can be accurately obtained, and an evaluation is performed based on the initial deformation field evaluation model, and the predicted anatomical structure label of each coordinate point is used as an evaluation result.

[0123] In one embodiment, as shown in FIG. 7 , the specific contents of adjusting the parameters of the initial deformation field prediction model based on the evaluation results include:

[0124] S601, determining the loss function of the initial deformation field prediction model based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields and the sample intensity vector of each coordinate point.

[0125] In an embodiment of the present application, the computer device can calculate the overall loss function based on the real anatomical structure label map of each coordinate point, the predicted anatomical structure label of each coordinate point, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point. Alternatively, the computer device can also calculate multiple different types of loss functions based on the real anatomical structure label map of each coordinate point, the predicted anatomical structure label of each coordinate point, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point. Then, based on each loss function type and the corresponding weight, the overall loss function is calculated.

[0126] S602: Adjust parameters of the initial deformation field prediction model based on the loss function of the initial deformation field prediction model.

[0127] In an embodiment of the present application, after obtaining the loss function of the initial deformation field prediction model, the computer device can calculate the gradient of the loss function to the initial deformation field prediction model, and update the parameters of the initial deformation field prediction model according to the optimization algorithm. Repeat the above process, in each iteration, substitute the new parameters into the initial deformation field prediction model for prediction, calculate the loss function, and then update the parameters again until the loss function converges or reaches a predetermined number of iterations.

[0128] In the above-mentioned image registration method, the loss function of the initial deformation field prediction model is determined based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point; based on the loss function of the initial deformation field prediction model, the parameters of the initial deformation field prediction model are adjusted. This method can obtain a more comprehensive loss function by comprehensively considering the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point, so as to accurately adjust the parameters of the initial deformation field prediction model based on the loss function, so that the accuracy and speed of the model training process are improved.

[0129] The loss function includes a similarity measurement loss function, a label smoothness loss function, a deformation field smoothness loss function and a discreteness loss function. In one embodiment, the loss function of the initial deformation field prediction model is determined based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, multiple predicted sample deformation fields and the sample intensity vector of each coordinate point. The specific content includes:

[0130] Based on multiple predicted sample deformation fields, a deformation field smoothness loss function is obtained; and, based on the true anatomical structure label map and the sample intensity vector of each coordinate point, a discreteness loss function is obtained; and, based on the true anatomical structure label map and the predicted anatomical structure probability value of each coordinate point, a similarity measurement loss function is obtained; and, based on the predicted anatomical structure label of each coordinate point, a label smoothness loss function is obtained.

[0131] In the embodiment of the present application, the overall loss function can be defined as:

[0132] L total = L sim + L dispersion + L label +

[0133] Among them, L sim represents the similarity measurement loss function; L dispersion represents the discreteness loss function; L label represents the label smoothness loss function; represents the deformation field smoothing loss function; The parameters of the initial deformation field prediction model are The predicted sample deformation field of , , and They represent the coefficients of the similarity metric loss function, the discreteness loss function, the label smoothness loss function, and the deformation field smoothness loss function, respectively. Their main purpose is to reduce the differences between the above four loss functions and balance the proportions between the four loss functions so that each loss function can make a certain contribution to the overall loss function. The four loss functions constrain each other, and each loss function has a certain contribution to the overall loss function, which makes the overall loss function more accurate and can improve the accuracy of model training.

[0134] It should be emphasized that , , and It can be set manually, or the optimal value of each coefficient can be automatically calculated by dynamic grid search or random search. Regardless of how the coefficients of the loss function are defined, the four loss functions need to be converted to similar orders of magnitude, or even to the same order of magnitude.

[0135] Among them, different training samples correspond to different coefficients of the loss function. For example, for the first training sample, , , and It can be 1, 2, 5 and 10. For the second training sample, , , and It can be 3, 7, 10 and 15.

[0136] After completing the registration of multiple sample medical image groups based on multiple predicted sample deformation fields, the label S of each image group reference image is used. ref As weak supervision information, the sample intensity vector of each coordinate point of the registered medical image group can be expressed as n k,j , , and The anatomical structure label ratio of each image group can be calculated based on the specific anatomical labels, that is, .

[0137] make represents the intensity vector corresponding to image group j. Based on the Bayesian formula, the posterior probability distribution can be expressed as:

[0138]

[0139] The similarity measurement loss function based on the initial deformation field evaluation model can be obtained by S ref The cross entropy calculation of the unique hot encoding and the negative log-likelihood of the posterior probability distribution is:

[0140]

[0141] Discreteness loss function L dispersion The dispersion of the feature space can be measured by calculating the variance of the set of intensity vectors corresponding to each anatomical structure:

[0142]

[0143] The class with the maximum probability can be used to predict the label value of the coordinate point, and then the predicted label map S can be obtained in this way pred Assuming that the neighborhood pixels of an image group should be as similar as possible, the neighborhood label values ​​of the predicted labels should be as similar as possible, and the label smoothness loss function L label It can be expressed as:

[0144]

[0145] in, Represents the predicted label map S pred The predicted label value at coordinate point x, x i represents the neighborhood coordinates of the coordinate point x. If is equal to 0, otherwise it is equal to 1. For a three-dimensional image, n represents a six-neighborhood; for a two-dimensional image, n represents a four-neighborhood.

[0146] FIG8 is a schematic diagram of the training process of the deformation field prediction model. The multimodal sample medical image group is input into the initial deformation field prediction model, and the initial deformation field prediction model outputs the predicted sample deformation field. Then, based on the predicted sample deformation field, the loss function is determined. The loss function is used to adjust the parameters of the initial deformation field prediction model until the output result of the initial deformation field prediction model meets the conditions, the deformation field prediction model training process is completed, and the deformation field prediction model is obtained.

[0147] In the above image registration method, based on multiple predicted sample deformation fields, a deformation field smoothing loss function is obtained; and based on the real anatomical structure label map and the sample intensity vector of each coordinate point, a discreteness loss function is obtained; and based on the real anatomical structure label map and the predicted anatomical structure probability value of each coordinate point, a similarity measurement loss function is obtained; and based on the predicted anatomical structure label of each coordinate point, a label smoothness loss function is obtained. In the deformation field prediction model training process, the deformation field smoothing loss function can make the smoothness between multiple mobile medical images better, avoiding the situation where the pixel points between two adjacent mobile images change too much; and the discreteness loss function can make the aggregation of the same anatomical structure better; the label smoothness loss function can optimize the neighborhood similarity between multiple image groups; the similarity measurement loss function can guide the multimodal image group to align to the same anatomical space coordinate system, and the four loss functions are combined to train the deformation field prediction model, which can comprehensively improve the prediction accuracy of the deformation field prediction model.

[0148] After the medical image group registration is completed, the image group registration process can also be evaluated. In one embodiment, as shown in FIG9 , the evaluation process may include the following steps:

[0149] S701, performing segmentation processing on the anatomical structure of the registered medical image group to obtain a segmentation result.

[0150] In the embodiment of the present application, after obtaining the registered medical image group, the computer device can use the image segmentation algorithm to perform segmentation processing according to the anatomical structure of the registered medical image group, and divide the image regions belonging to the same anatomical structure into one category to obtain the segmentation result. For example, the image segmentation algorithm can be threshold segmentation, clustering algorithm, deep learning-based segmentation, TransBTS algorithm, etc.

[0151] S702: Evaluate the image group registration process according to the segmentation result.

[0152] In an embodiment of the present application, after obtaining the segmentation result, the computer device can compare the segmentation result with the anatomical structure label of the reference medical image in the multimodal medical image to determine whether the anatomical structure of each pixel in the registered medical image group is the same as the anatomical structure label. And obtain different numbers of pixels, compare the different numbers of pixels with a preset threshold, if it is greater than or equal to the preset threshold, it means that the accuracy of the image group registration is high; if it is less than the preset threshold, it means that the accuracy of the image group registration is poor. Alternatively, the computer device can also calculate the similarity between the segmentation result and the anatomical structure of the reference medical image, and evaluate the image group registration process based on the similarity.

[0153] In the above-mentioned image registration method, the anatomical structure of the registered medical image group is segmented to obtain a segmentation result; and the image group registration process is evaluated based on the segmentation result. The method is based on the segmentation of the anatomical structure of the registered medical image group, and based on the segmentation result, the image group registration process can be accurately evaluated, so that the quality of the image group registration result can be determined based on the evaluation result.

[0154] Next, the specific content of evaluating the image group registration process according to the segmentation result is introduced through an embodiment, as shown in FIG10 , and the specific content includes:

[0155] S801, calculating the similarity coefficient between the segmentation result and the anatomical structure of the reference medical image.

[0156] Among them, the similarity coefficient is usually used to quantitatively evaluate the matching degree between the segmentation result and the anatomical structure in the reference medical image. Common similarity coefficients can be Dice coefficient (Dice Similarity Coefficient) and Jaccard coefficient (Jaccard Index).

[0157] In an embodiment of the present application, since the segmentation result is to divide the registered medical image into regions of different anatomical types, the computer device can compare the pixel positions corresponding to different anatomical types in the registered medical image group with the corresponding pixel positions in the reference medical image to obtain the similarity coefficient between different anatomical structures.

[0158] S802: Evaluate the registration process of the evaluation image group based on the similarity coefficient.

[0159] In an embodiment of the present application, if the similarity coefficient is high, it means that the segmentation result is closer to the anatomical structure of the reference medical image, and the image group registration effect is better; conversely, if the similarity coefficient is low, it means that the segmentation result is greatly different from the anatomical structure of the reference medical image, and the image group registration effect is not good. Therefore, the computer device can evaluate the image group registration process according to the similarity coefficient. Among them, the evaluation type can be directly the similarity coefficient, or it can be a level divided based on the similarity coefficient. For example, the similarity coefficient is 90%, and the corresponding level is the first level, indicating that the registration effect of the image group registration process is good; the similarity coefficient is 70%, and the corresponding level is the second level, indicating that the registration effect of the image group registration process is average; the similarity coefficient is 40%, and the corresponding level is the third level, indicating that the registration effect of the image group registration process is poor.

[0160] In the above-mentioned image registration method, the similarity coefficient between the segmentation result and the anatomical structure of the reference medical image is calculated; based on the similarity coefficient, the image group registration process is evaluated. This method uses the anatomical structure of the reference medical image as a reference, and by calculating the similarity coefficient between the segmentation result and the anatomical structure of the reference medical image, the performance and accuracy of the image group registration process can be objectively evaluated.

[0161] Next, an example is used to illustrate the image registration process.

[0162] For example, the multimodal medical image set is image data of the carotid artery, including initially misaligned 3D-T1, T1c, T2, and time of flight (TOF) carotid artery magnetic resonance images. Each magnetic resonance image has a 1×1×1 cubic millimeter (mm 3 ), cut into 128×128×32 blocks, and finally normalized to [0,1]. The image size is not strictly limited and can be adjusted according to actual needs. Each image group uses T1 as the reference medical image. The anatomical structure labels of the reference medical image include lumen, wall, calcification, lipid and hemorrhage. Different multimodal medical image groups do not necessarily include these five anatomical structures, and may be equal to or less than these five anatomical structures.

[0163] The overall registration process includes: the training process, the actual application process and the testing process of the deformation field prediction model. Then, multiple medical image groups can be obtained and divided into training sets, validation sets and test sets. For example, the number of medical image groups can be 230, and 160 / 25 / 45 image groups can correspond to the training set, validation set and test set respectively.

[0164] For the training process, the algorithm framework corresponding to the training process can be written in a programming language. For example, it can be written in Pytorch and deployed on a graphics processing unit (GPU) of model A800.

[0165] During the training process, the optimizer uses a learning rate of 1×e -4 The Adam optimizer was used and trained for 500 rounds with a batch size of 1. The hyperparameters of the overall loss function and the weights of each loss function were set to 2, 1, and 100, respectively, and the number of anatomical structures was set to 5. The above hardware conditions and specific parameters are only examples and are not strictly limited to fixed selections. Size adjustment and hardware replacement can be performed according to actual data. The deformation field prediction model is trained through the above parameters.

[0166] In actual application, 45 medical image groups are input into the trained deformation field prediction model in turn. The deformation field prediction model is used to predict the deformation field in each medical image group, and the registration of the medical image groups is completed based on the prediction results.

[0167] During the test, the registered medical image group may be segmented, and the image group registration process may be evaluated based on the similarity coefficient between the segmentation result and the anatomical structure of the reference medical image.

[0168] After completing the image group registration by different methods, the image group registration results of different methods are compared. Table 1 is an evaluation comparison of the image group registration results of different methods. The different group registration methods include: no registration, Ants-SyN registration, APE registration, VoxelMorph registration, and registration based on the deformation field prediction model. It can be seen from Table 1 that the registration based on the deformation field prediction model has the highest similarity coefficient and the best evaluation result. In other words, the registration based on the deformation field prediction model can achieve better alignment of large-scale anatomical structures and local areas in most anatomical structures.

[0169] Table 1

[0170]

[0171] Figure 11 is a schematic diagram of a medical image group obtained by different registration methods. The registration methods here correspond to the five methods in Table 1. Each column in the figure represents the registration result of a registration method, and each row represents the registration result of an anatomical structure under different registration methods. It can be seen from the figure that the registration result based on the deformation field prediction model is more accurate.

[0172] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides an image registration device for implementing the above-mentioned image registration method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image registration device embodiments provided below can refer to the limitations of the image registration method above, and will not be repeated here.

[0174] In an exemplary embodiment, as shown in FIG12 , an image registration device is provided, comprising: an acquisition module 11, a prediction module 12 and a registration module 13, wherein:

[0175] An acquisition module 11 is used to acquire a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one moving medical image;

[0176] A prediction module 12 is used to input the multimodal medical image group into the deformation field prediction model, and predict the predicted deformation field between each mobile medical image and the reference medical image through the deformation field prediction model; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process;

[0177] The registration module 13 is used to complete the registration of the corresponding mobile medical image and the reference medical image based on each predicted deformation field.

[0178] In one embodiment, the above-mentioned image registration device further includes: a sample acquisition module, a sample prediction module, an evaluation module and an iterative execution module, wherein:

[0179] A sample acquisition module, used to acquire a multimodal sample medical image group and a corresponding real anatomical structure label map; the multimodal sample medical image group includes at least one sample moving medical image and a sample reference medical image;

[0180] A sample prediction module, used for inputting a multimodal sample medical image group into an initial deformation field prediction model to obtain a plurality of predicted sample deformation fields;

[0181] An evaluation module is used to evaluate the deformation fields of multiple predicted samples through an initial deformation field evaluation model to obtain an evaluation result;

[0182] The iterative execution module is used to adjust the parameters of the initial deformation field prediction model and the initial deformation field evaluation model based on the evaluation results, and iteratively execute to obtain the deformation field prediction model.

[0183] In one embodiment, the evaluation module comprises: a registration unit, an extraction unit and an evaluation unit, wherein:

[0184] A registration unit, used for registering the corresponding mobile medical image to the sample reference medical image based on the multiple predicted sample deformation fields to obtain a registered sample medical image;

[0185] An extraction unit, used for extracting a sample intensity vector of each coordinate point in the registered sample medical image;

[0186] The evaluation unit is used to input the sample strength vector into the initial deformation field evaluation model to obtain the predicted anatomical structure probability value of each coordinate point, take the structure corresponding to the maximum probability value among the predicted anatomical structure probability values ​​as the predicted anatomical structure label, and take the predicted anatomical structure label as the evaluation result.

[0187] In one embodiment, the above-mentioned iterative execution module includes: a determination unit and an adjustment unit, wherein:

[0188] A determination unit, for determining a loss function of an initial deformation field prediction model based on a predicted anatomical structure label of each coordinate point, a true anatomical structure label map, a predicted anatomical structure probability value of each coordinate point, a plurality of predicted sample deformation fields, and a sample intensity vector of each coordinate point;

[0189] The adjustment unit is used to adjust the parameters of the initial deformation field prediction model based on the loss function of the initial deformation field prediction model.

[0190] In one embodiment, the above-mentioned determination unit is also used to obtain a deformation field smoothness loss function based on multiple predicted sample deformation fields; and, based on the real anatomical structure label map and the sample intensity vector of each coordinate point, obtain a discreteness loss function; and, based on the real anatomical structure label map and the predicted anatomical structure probability value of each coordinate point, obtain a similarity measurement loss function; and, based on the predicted anatomical structure label of each coordinate point, obtain a label smoothness loss function.

[0191] In one embodiment, the above-mentioned image registration device further includes: a determination module, a component construction module, a weight construction module and a model acquisition module, wherein:

[0192] A determination module, for extracting a set of intensity vectors of a multimodal sample medical image group based on a true anatomical structure label map of a reference image, and determining a mean value and covariance of a Gaussian component of each anatomical structure;

[0193] A component construction module, for constructing a Gaussian mixture model component corresponding to each anatomical structure based on the Gaussian component mean and covariance of each anatomical structure;

[0194] A weight construction module, for constructing a Gaussian mixture model component weight corresponding to each anatomical structure based on a set of intensity vectors of a multimodal sample medical image group;

[0195] The model acquisition module is used to construct an initial deformation field evaluation model according to the Gaussian mixture model components corresponding to each anatomical structure and the corresponding Gaussian mixture model component weights.

[0196] In one embodiment, the image registration device further includes: a segmentation module and an evaluation module, wherein:

[0197] A segmentation module is used to segment the anatomical structure of the registered medical image group to obtain a segmentation result;

[0198] The evaluation module is used to evaluate the image group registration process according to the segmentation results.

[0199] In one embodiment, the evaluation module is further used to calculate a similarity coefficient between the segmentation result and the anatomical structure of the reference medical image; and based on the similarity coefficient, the evaluation image registration process is evaluated.

[0200] Each module in the above-mentioned image registration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0201] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the content of any one embodiment of the above-mentioned image registration method is implemented.

[0202] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the content of any one embodiment of the above-mentioned image registration method is implemented.

[0203] In one embodiment, a computer program product is provided, including a computer program, which implements the content of any one of the embodiments of the above-mentioned image registration method when executed by a processor.

[0204] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0205] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0206] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0207] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image registration method, characterized in that: The method comprises: Acquire a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one moving medical image; The multimodal medical image group is input into a deformation field prediction model, and the deformation field prediction model is used to predict the predicted deformation field between each of the mobile medical images and the reference medical image; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process; Based on each of the predicted deformation fields, the corresponding mobile medical image is registered with the reference medical image.

2. The method according to claim 1, characterized in that The construction process of the deformation field prediction model includes: Acquire a multimodal sample medical image group and a corresponding real anatomical structure label map; the multimodal sample medical image group includes at least one sample moving medical image and a sample reference medical image; Inputting the multimodal sample medical image group into an initial deformation field prediction model to obtain a plurality of predicted sample deformation fields; Evaluating the deformation fields of the plurality of predicted samples by using an initial deformation field evaluation model to obtain an evaluation result; The parameters of the initial deformation field prediction model and the initial deformation field evaluation model are adjusted based on the evaluation result, and the adjustment is performed iteratively to obtain the deformation field prediction model.

3. The method according to claim 2, characterized in that The multiple predicted sample deformation fields are evaluated by the initial deformation field evaluation model to obtain evaluation results, including: Based on the multiple predicted sample deformation fields, registering the corresponding sample mobile medical image to the sample reference medical image to obtain a registered sample medical image; Extracting a sample intensity vector of each coordinate point in the registered sample medical image; The sample intensity vector is input into the initial deformation field evaluation model to obtain the predicted anatomical structure probability value for each coordinate point, the structure corresponding to the maximum probability value among the predicted anatomical structure probability values ​​is used as the predicted anatomical structure label, and the predicted anatomical structure label is used as the evaluation result.

4. The method according to claim 3, characterized in that The adjusting the parameters of the initial deformation field prediction model based on the evaluation result includes: Determine the loss function of the initial deformation field prediction model based on the predicted anatomical structure label of each coordinate point, the true anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, the multiple predicted sample deformation fields and the sample intensity vector of each coordinate point; Based on the loss function of the initial deformation field prediction model, the parameters of the initial deformation field prediction model are adjusted.

5. The method according to claim 4, characterized in that The loss function includes a similarity measurement loss function, a label smoothness loss function, a deformation field smoothness loss function and a discreteness loss function; The step of determining the loss function of the initial deformation field prediction model based on the predicted anatomical structure label of each coordinate point, the real anatomical structure label map, the predicted anatomical structure probability value of each coordinate point, the multiple predicted sample deformation fields, and the sample intensity vector of each coordinate point includes: Based on the multiple predicted sample deformation fields, the deformation field smoothness loss function is obtained; and, based on the real anatomical structure label map and the sample intensity vector of each coordinate point, the discreteness loss function is obtained; and, based on the real anatomical structure label map and the predicted anatomical structure probability value of each coordinate point, the similarity measurement loss function is obtained; and, based on the predicted anatomical structure label of each coordinate point, the label smoothness loss function is obtained.

6. The method according to any one of claims 2 to 5, characterized in that: The construction process of the initial deformation field evaluation model includes: Extracting a set of intensity vectors of a multimodal sample medical image group based on a true anatomical structure label map of a reference image, and determining the mean and covariance of the Gaussian components of each anatomical structure; Constructing Gaussian mixture model components corresponding to each anatomical structure based on the Gaussian component mean and covariance of each anatomical structure; Based on the set of intensity vectors of the multimodal sample medical image group, constructing a Gaussian mixture model component weight corresponding to each anatomical structure; The initial deformation field evaluation model is constructed according to the Gaussian mixture model components corresponding to each anatomical structure and the corresponding Gaussian mixture model component weights.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Segmenting the anatomical structure of the registered medical image group to obtain a segmentation result; Based on the segmentation results, the image group registration process is evaluated.

8. The method according to claim 7, characterized in that The step of evaluating the image group registration process according to the segmentation result includes: Calculating a similarity coefficient between the segmentation result and the anatomical structure of the reference medical image; Based on the similarity coefficient, the image registration process is evaluated.

9. An image registration device, characterized in that: The device comprises: An acquisition module, configured to acquire a multimodal medical image group; the multimodal medical image group includes a reference medical image and at least one moving medical image; A prediction module, used for inputting the multimodal medical image group into a deformation field prediction model, and predicting the predicted deformation field between each of the mobile medical images and the reference medical image through the deformation field prediction model; the deformation field prediction model uses the label of the sample reference medical image as weak supervision information and performs constraint training based on the evaluation result of the predicted sample deformation field output by the training process; A registration module is used to complete the registration of the corresponding mobile medical image with the reference medical image based on each of the predicted deformation fields.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images