Image registration model method and device, equipment and storage medium

Abstract

The invention discloses an image registration model method and device, equipment and a storage medium, and relates to the field of medical imaging. The image registration model method comprises the steps that a sample data set is acquired, the sample data set comprises a plurality of sample pairs, each sample pair comprises a first image and a second image which are located in different spaces, iterative training is carried out on a to-be-trained image registration model based on the plurality of sample pairs until a training ending condition is met, and a trained image registration model is obtained. Through combination of multi-scale registration and symmetric registration, the registration network is more robust and efficient.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a method, device, equipment and storage medium for an image registration model. Background Art

[0002] Image registration is of great significance in the field of medical imaging. Through image registration of different imaging modalities, multiple imaging information can be integrated to provide a more comprehensive diagnostic basis.

[0003] Registering CT images of the same patient collected at different time points helps monitor disease progression or treatment effectiveness. By comparing the registered images, changes in the lesion area can be accurately identified, supporting doctors to make timely interventions.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art: in the current training process of the registration model, generally only the registration of the image in the target space is considered, resulting in poor robustness of the registration model and low efficiency of the model stock process. Summary of the invention

[0005] The present invention provides a method, apparatus, device and storage medium for an image registration model to improve the robustness of the registration model and the efficiency of the training process.

[0006] According to one aspect of the present invention, a method for training an image registration model is provided, comprising:

[0007] Acquire a sample data set, the sample data set including a plurality of sample pairs, the sample pairs including a first image and a second image located in different spaces;

[0008] The image registration model to be trained is iteratively trained based on multiple samples until the training end condition is met to obtain a trained image registration model, wherein each iterative training process includes:

[0009] For any set of sample pairs, the first image and the second image in the sample pair are spliced ​​to obtain spliced ​​data; the spliced ​​data is input into the image registration model in the training process to obtain a first registration parameter; based on the first registration parameter, a second registration parameter is determined, and the second registration parameter is an inverse registration parameter of the first registration parameter;

[0010] Performing registration processing on the second image based on the first registration parameter to obtain a first registered image; performing registration processing on the first image based on the second registration parameter to obtain a second registered image; generating a first loss term based on the first registered image and the first image, and generating a second loss term based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function;

[0011] The model parameters of the image registration model in the training process are adjusted based on the target loss function.

[0012] According to another aspect of the present invention, there is provided an image registration method, comprising:

[0013] Acquire the image to be processed and the target space image; perform data splicing on the image to be processed and the target space image to obtain spliced ​​data;

[0014] Input the spliced ​​data into a pre-trained image registration model to obtain registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model provided in any embodiment of the present invention;

[0015] The image to be processed is registered based on the registration parameters to obtain a registered image corresponding to the image to be processed, and the registered image and the target space image are located in the same space.

[0016] According to another aspect of the present invention, there is provided a training device for an image registration model, comprising:

[0017] A sample acquisition module, used to acquire a sample data set, wherein the sample data set includes a plurality of sample pairs, wherein the sample pairs include a first image and a second image located in different spaces;

[0018] A model training module is used to iteratively train an image registration model to be trained based on multiple sample pairs until a training end condition is met to obtain a trained image registration model, wherein each iterative training process includes: for any group of sample pairs, splicing the first image and the second image in the sample pair to obtain spliced ​​data; inputting the spliced ​​data into the image registration model in the training process to obtain a first registration parameter; determining a second registration parameter based on the first registration parameter, the second registration parameter being an inverse registration parameter of the first registration parameter; performing registration processing on the second image based on the first registration parameter to obtain a first registered image; performing registration processing on the first image based on the second registration parameter to obtain a second registered image; generating a first loss term based on the first registered image and the first image, and generating a second loss term based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function; the target loss function adjusts the model parameters of the image registration model in the training process.

[0019] According to another aspect of the present invention, there is provided an image registration device, comprising:

[0020] An image acquisition module is used to acquire the image to be processed and the target space image; and perform data splicing on the image to be processed and the target space image to obtain spliced ​​data;

[0021] A registration parameter determination module, used for inputting the spliced ​​data into a pre-trained image registration model to obtain the registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model provided in any embodiment of the present invention;

[0022] The image registration module is used to perform registration processing on the image to be processed based on the registration parameters to obtain a registered image corresponding to the image to be processed, and the registered image and the target space image are located in the same space.

[0023] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0024] at least one processor; and

[0025] a memory communicatively connected to at least one processor; wherein,

[0026] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the image registration model training method or image registration method provided in any embodiment of the present invention.

[0027] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the image registration model training method or image registration method provided by any embodiment of the present invention when executed.

[0028] The technical solution of the embodiment of the present invention is to obtain a sample data set, wherein the sample data set includes multiple sample pairs, wherein the sample pairs include a first image and a second image located in different spaces, and obtain a target image and an image to be registered with the same size; iteratively train an image registration model to be trained based on the multiple sample pairs until a training end condition is met, and obtain a trained image registration model, so that the learned parameters are more robust and efficient, thereby increasing the robustness and efficiency of the registration model; in each iterative training process, for any group of sample pairs, a first registration parameter is obtained by processing the sample pairs, and an inverse registration parameter of the first registration parameter, i.e., a second registration parameter, is determined based on the first registration parameter; forward registration processing is performed on the second image through the first registration parameter, and inverse registration processing is performed on the first image through the second registration parameter; and model parameters of the registration model are adjusted based on loss functions respectively determined by the forward registration processing and the inverse registration processing, thereby improving the accuracy and comprehensiveness of the loss function, further accelerating the convergence efficiency of the registration model training process, and increasing the robustness and efficiency of the image registration model.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a flow chart of a training method for an image registration model provided in Embodiment 1 of the present invention;

[0032] Figure 2 is a structural schematic diagram of an image registration model provided by an embodiment of the present invention;

[0033] Figure 3 is a flow chart of another method for training an image registration model provided in Embodiment 2 of the present invention;

[0034] Figure 4 is a structural schematic diagram of another image registration model provided by an embodiment of the present invention;

[0035] Figure 5 is a flow chart of an image registration method provided by Embodiment 3 of the present invention;

[0036] Figure 6 It is a structural schematic diagram of a training device for an image registration model provided by a fourth embodiment of the present invention;

[0037] Figure 7 is a structural schematic diagram of an image registration device provided by Embodiment 5 of the present invention;

[0038] Figure 8 It is a structural schematic diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] Embodiment 1

[0042] Figure 1 This is a flow chart of a training method for an image registration model provided in the first embodiment of the present invention. This embodiment is applicable to the case where spatially inconsistent images are aligned in space. The method can be executed by a training device for an image registration model. The training device for the image registration model can be implemented in the form of hardware and / or software. The training device for the image registration model can be configured on a variety of terminal smart devices, such as computers, etc. Figure 1 As shown, the method includes:

[0043] S110 , obtaining a sample data set, where the sample data set includes a plurality of sample pairs, and the sample pairs include a first image and a second image located in different spaces.

[0044] In this embodiment, the first image and the second image may be medical images located in different spaces, wherein the first image and the second image may be medical images of the same region of interest of the same object, for example, the first image and the second image may be a computed tomography (CT) image and a magnetic resonance imaging (MRI) image, etc. The above-mentioned object may be a human body or an animal body, which is not limited here, and the region of interest of the object may be determined according to the acquisition requirements of the medical image, for example, it may include but is not limited to the brain, trunk, etc.

[0045] In some embodiments, the first image and the second image in the sample pair may be acquired by a medical imaging acquisition device, for example, they may be medical images acquired from the same region of interest of the same object at different time points, or they may be medical images acquired from the same region of interest of the same object by different acquisition devices. For example, the first image is a CT image of the left front leg of a rabbit on the first day after surgery, and the second image may be a CT image of the left front leg of the rabbit acquired by the same device on the third day after surgery.

[0046] In some embodiments, the first image in the sample pair may be acquired by a medical image acquisition device or read from an image library, wherein the image library is used to store historically acquired medical images. The second image in the sample pair may be obtained by spatial transformation based on the first image. The second image may be generated by one or more of rotational transformation, translational transformation, and elastic transformation.

[0047] Based on the above embodiment, the method further includes performing image enhancement processing on the second image in each sample pair.

[0048] Specifically, the first image is subjected to a spatial transformation, such as a rotation transformation, a translation transformation, and an elastic transformation, to obtain a second image, and the second image is subjected to image enhancement processing, and the image enhancement processing may include one or more of noise addition and smoothing. The second image subjected to image enhancement processing and the first image form an image pair. Optionally, the second image subjected to different image enhancement processing and the first image form different image pairs, and on the basis of enhancing the generalization performance of the image pair and the robustness of the image registration model through image enhancement processing, the number of sample pairs is increased and the method of obtaining the sample pairs is simplified.

[0049] Exemplarily, the sample data set includes N sample pairs, each of which is represented by a pair of three-dimensional CT images (i.e., the first image and the second image) with a size of h×w×d, which can be obtained by, for example, resizing the first image and the second image in each sample pair, respectively, wherein the resizing method can be image resizing, or image cropping, etc. All data are divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The training set is used to train the registration model, the validation set is used to select the best performing model from the trained registration models, and the test set is used to evaluate the performance of the registration model.

[0050] S120. Iteratively train the image registration model to be trained based on multiple sample pairs until the training end condition is met to obtain a trained image registration model; wherein each iterative training process includes: for any group of sample pairs, splicing the first image and the second image in the sample pair to obtain spliced ​​data; inputting the spliced ​​data into the image registration model in the training process to obtain a first registration parameter; determining the second registration parameter based on the first registration parameter, the second registration parameter being the inverse registration parameter of the first registration parameter; performing registration processing on the second image based on the first registration parameter to obtain a first registered image; performing registration processing on the first image based on the second registration parameter to obtain a second registered image; generating a first loss term based on the first registered image and the first image, and generating a second loss term based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function; adjusting model parameters of the image registration model in the training process based on the target loss function.

[0051] In this embodiment, an image registration model to be trained is created in advance, model parameters of the image registration model to be trained are initialized, and the initialized image registration model is trained based on a sample data set to obtain a trained image registration model.

[0052] The image registration model may be, for example, a U-net model or a transformer model. The model structure of the image registration model is not limited here and may be created according to user needs. Optionally, the image registration model may include an encoder and a decoder. The encoder and the decoder may each include a plurality of network blocks. The network blocks in the encoder and the decoder are jump-connected. Optionally, the network block may be a residual network block. For example, see Figure 2 , Figure 2 It is a structural schematic diagram of an image registration model provided by an embodiment of the present invention.

[0053] In any iterative training process, the spliced ​​data of the first image and the second image in the sample pair is used as the input data of the image registration model in the training process, and the spliced ​​data is processed by the image registration model in the training process to obtain the first registration parameter. Wherein, the splicing of the first image and the second image in the sample pair can be achieved by splicing the data in the channel dimension. Wherein, the first image and the second image can be RGB images, and accordingly, the first image and the second image have data of three color channel dimensions respectively; the first image and the second image can be grayscale images, and accordingly, the first image and the second image have data of one channel dimension respectively. Exemplarily, the first image and the second image are 1×h×w×d data respectively, and the spliced ​​data is 2×h×w×d data. Optionally, the first image and the second image have a certain stitching order, and the stitching order may be the stitching order of the image corresponding to the target space (which may be marked as a fixed image) and the image to be spatially registered (which may be marked as a moving image). For example, the order of the fixed image may be in front and the order of the moving image may be in the back. This may be set according to processing requirements. Taking the first image as a fixed image and the second image as a moving image as an example, in the stitching data, the data of the first image is in front and the data of the second image is in the back to obtain the stitching data.

[0054] The first registration parameter is used to characterize the spatial conversion relationship between the second image and the first image, that is, the registration parameter for spatial conversion of the second image to the space where the first image is located. The second registration parameter is obtained by inverse processing of the first registration parameter, and characterizes the spatial conversion relationship between the first image and the second image, that is, the registration parameter for spatial conversion of the first image to the space where the second image is located.

[0055] Optionally, the first registration parameter is inversely transformed to obtain the second registration parameter to simulate the inverse process of registering the first image to the space where the second image is located, thereby simplifying the process of obtaining the second registration parameter. Here, the first registration parameter and the second registration parameter can be in matrix form, respectively, wherein the first registration parameter and the second registration parameter can be h×w×d respectively.

[0056] Specifically, the first registration parameter extracted by the image registration model during the training process can be represented by G. Optionally, the inverse transformation of the first registration parameter can be to assign the negative value of the first registration parameter to the second registration parameter, so that the second registration parameter is -G; Optionally, the inverse transformation of the first registration parameter can be to obtain G by performing an inverse operation on the first registration parameter. -1 , and the second registration parameter is assigned to G -1 By performing inverse transformation processing on the first registration parameter, the image inverse transformation process is omitted, thereby improving the efficiency and accuracy of image registration model training.

[0057] In this embodiment, the first registered image is an image obtained after the second image is processed for image registration using the first registration parameters. The second registered image is an image obtained after the first image is processed for image registration using the second registration parameters, for example, the first registered image can be obtained by performing matrix multiplication based on the first registration parameters and the matrix form data corresponding to the second image, and the second registered image can be obtained by performing matrix multiplication based on the second registration parameters and the matrix form data corresponding to the first image. The first image is used as the label corresponding to the first registered image, and the second image is used as the label corresponding to the second registered image, and a loss function is generated to adjust the model parameters of the image registration model during the training process.

[0058] The loss function includes multiple loss terms, wherein the first loss term is the loss term formed between the first registered image and the first image, and the second loss term is the loss term formed between the second registered image and the second image. The first loss term and the second loss term can be, for example, mean square loss, which is not limited here. The target loss function is a standard used to measure the training effect of the image registration model. The smaller the target loss function, the better the training effect of the image registration model; and vice versa. The first loss term and the second loss term are calculated by one or more of accumulation, cumulative averaging and weighting to obtain the target loss function. The corresponding images are registered by forward registration parameters and reverse registration parameters to obtain two different loss terms. The two different loss terms are accumulated, summed and averaged to obtain the target loss function reflecting the training effect of the model.

[0059] Specifically, the parameters of the image registration model are adjusted by the target loss function based on the minimum principle of the target loss function. The above training process is iterated so that the target loss function reaches the minimum or converges, and the training end condition is met to obtain a trained image registration model.

[0060] In the above embodiment, the above training process is performed multiple times to obtain multiple trained image registration models, and the performance of the multiple trained image registration models is evaluated based on the validation set, and the image registration model with the best performance is determined as the final image registration model. The performance parameters to be evaluated may include image registration model precision, accuracy, recall rate, F1 parameter, etc.

[0061] The technical solution of this embodiment is to obtain a sample data set, which includes multiple sample pairs, and the sample pairs include a first image and a second image located in different spaces, so as to obtain a target image and an image to be registered with the same size; iteratively train the image registration model to be trained based on the multiple sample pairs until the training end condition is met, so as to obtain a trained image registration model, so that the learned parameters are more robust and efficient, thereby increasing the robustness and efficiency of the registration model; in each iterative training process, for any group of sample pairs, a first registration parameter is obtained by processing the sample pairs, and an inverse registration parameter of the first registration parameter, i.e., a second registration parameter, is determined based on the first registration parameter, and a forward registration process is performed on the second image through the first registration parameter, and an inverse registration process is performed on the first image through the second registration parameter, and the model parameters of the registration model are adjusted based on the loss functions respectively determined by the forward registration process and the inverse registration process, thereby improving the accuracy and comprehensiveness of the loss function and increasing the robustness and efficiency of the image registration model.

[0062] Embodiment 2

[0063] Figure 3 is a flow chart of another method for training an image registration model provided by Embodiment 2 of the present invention. This embodiment is an optimization of the above embodiment. Figure 3 As shown, the method includes:

[0064] S210 , obtaining a sample data set, where the sample data set includes a plurality of sample pairs, and the sample pairs include a first image and a second image located in different spaces.

[0065] S220 , down-sampling the first image and the second image in any sample pair respectively to obtain a plurality of first images of different scales and a plurality of second images of different scales.

[0066] In this embodiment, the first image and the second image are downsampled at least twice respectively. The downsampling times corresponding to the first image and the second image are the same. If the first image is downsampled three times, the second image is also downsampled three times. The downsampling scales corresponding to the first image and the second image are the same, that is, the first image and the second image are downsampled by the same multiple. If the first image is downsampled by 2 times and 4 times respectively, the second image is also downsampled by 2 times and 4 times respectively. For example, the size of the first image is h×w×d, and the size of the second image is h×w×d. The first image is downsampled by multiple multiples, such as 2 times and 4 times, to obtain first images with image sizes of h / 2×w / 2×d / 2 and h / 4×w / 4×d / 4 respectively; the second image is downsampled by multiple times with the same number of times and the same scale, such as 2 times and 4 times, to obtain second images with image sizes of h / 2×w / 2×d / 2 and h / 4×w / 4×d / 4 respectively. By downsampling the first image and the second image at least twice to obtain multiple first images and second images of different scales, data features of different scales in the image can be better extracted, so that the image registration model can extract the global features and detail features of the image, and provide more comprehensive and accurate feature data for the processing process of the image registration model.

[0067] S230, iteratively training the image registration model to be trained based on multiple sample pairs until the training end condition is met to obtain a trained image registration model; wherein each iterative training process includes: for any group of sample pairs, performing data splicing on the first image and the second image of the same scale in the sample pair to obtain spliced ​​data; inputting the spliced ​​data of different scales into the image registration model in the training process to obtain first registration parameters of different scales; determining second registration parameters of different scales based on the first registration parameters of different scales, the second registration parameters being the inverse registration parameters of the first registration parameters; based on the first registration parameters of different scales The method comprises the following steps: performing registration processing on the second images of corresponding scales based on the second registration parameters of different scales to obtain a plurality of first registered images of different scales; performing registration processing on the first images of corresponding scales based on the second registration parameters of different scales to obtain a plurality of second registered images of different scales; generating a plurality of first loss terms based on the plurality of first registered images of different scales and the plurality of first images of different scales, and generating a plurality of second loss terms based on the plurality of second registered images of different scales and the plurality of second images of different scales, wherein the plurality of first loss terms and the plurality of second loss terms form a target loss function; and adjusting the model parameters of the image registration model in the training process based on the target loss function.

[0068] Optionally, data of the first image and the second image of the same scale are spliced ​​to obtain spliced ​​data of multiple scales, and the spliced ​​data of multiple scales are input into an image registration model in a training process to obtain a first registration parameter.

[0069] Specifically, the spliced ​​data may splice at least two first images and second images of the same scale in the channel dimension. For example, the first image and the second image of size h×w×d are spliced ​​in the channel dimension, the first image and the second image of size h / 2×w / 2×d / 2 are spliced ​​in the channel dimension, and the first image and the second image of size h / 4×w / 4×d / 4 are spliced ​​in the channel dimension, and then respectively input into the residual convolution network to obtain the features of their respective outputs. The features corresponding to the first image and the second image of size h×w×d are input into the next layer of residual convolution network, and the output is spliced ​​with the features corresponding to the first image and the second image of size h / 2×w / 2×d / 2 in the channel dimension, and the spliced ​​result is input into the next layer of residual convolution network, and the output result is spliced ​​with the features corresponding to the first image and the second image of size h / 4×w / 4×d / 4 in the channel dimension, and the spliced ​​result is input into the subsequent residual convolution network until the output G of the last layer is obtained, that is, the first registration parameter G is obtained.

[0070] Optionally, down-sampling processing is performed on the first registration parameter and the second registration parameter to obtain a plurality of first registration parameters of different scales and a plurality of second registration parameters of different scales.

[0071] Specifically, the first registration parameter and the second registration parameter are downsampled at least twice, and the number of downsampling times is the same. For example, if the first registration parameter is downsampled three times, then the second registration parameter is also downsampled three times; the downsampling scales of the first registration parameter and the second registration parameter are the same, that is, the first registration parameter and the second registration parameter are downsampled by the same multiple, for example, if the first registration parameter is downsampled by 2 times and 4 times respectively, then the second registration parameter is also downsampled by 2 times and 4 times respectively. By downsampling the first registration parameter and the second registration parameter at least twice, multiple first registration parameters and second registration parameters of different scales are obtained, thereby accelerating the convergence efficiency of the registration model training process.

[0072] Exemplarily, the first registration parameter G is downsampled by 2 times and 4 times respectively to obtain the first registration parameters G of different scales. 2 and G 4 When the second registration parameter is assigned the negative value of the first registration parameter G, the second registration parameter is -G, and the second registration parameter -G is downsampled by 2 times and 4 times respectively to obtain the second registration parameters -G of different scales. 2 and -G 4When the second registration parameter is given the inverse transformation of the first registration parameter G, the second registration parameter is G -1 , for the second registration parameter G -1 Downsampling by 2 and 4 times is performed respectively to obtain the second registration parameters G of different scales. 2 -1 and G 4 -1 Optionally, the second image is registered based on a plurality of first registration parameters of different scales to obtain first registered images of multiple scales; and the first image is registered based on a plurality of second registration parameters of different scales to obtain second registered images of multiple scales.

[0073] Specifically, the first registration parameter and the second image are registered at least twice to obtain at least two first registered images, the first registration parameter and the second image have the same scale each time the registration is performed, and the first registration parameter and the second image have different scales between at least two registration processes; the second registration parameter and the first image are registered at least twice to obtain at least two second registered images, the second registration parameter and the first image have the same scale each time the registration is performed, and the second registration parameter and the first image have different scales between at least two registration processes. For example, when the scale of the first registration parameter is G, the second image with a size of h×w×d is registered to obtain a first registered image with a size of h×w×d; when the scale of the first registration parameter is G 2 When the scale of the first registration parameter is G, the second image with a size of h / 2×w / 2×d / 2 is registered to obtain a first registered image with a size of h / 2×w / 2×d / 2. 4 When the scale of the second registration parameter is -G, the second image with a size of h / 4×w / 4×d / 4 is registered to obtain a first registered image with a size of h / 4×w / 4×d / 4. For example, when the scale of the second registration parameter is -G, the first image with a size of h×w×d is registered to obtain a second registered image with a size of h×w×d; when the scale of the second registration parameter is -G 2 When the first image with a size of h / 2×w / 2×d / 2 is registered, a second registered image with a size of h / 2×w / 2×d / 2 is obtained; when the scale of the second registration parameter is -G 4 When the first image with a size of h / 4×w / 4×d / 4 is registered, a second registered image with a size of h / 4×w / 4×d / 4 is obtained. Figure 4 It is a structural schematic diagram of another image registration model provided by an embodiment of the present invention.

[0074] Optionally, a plurality of first loss terms are generated based on first registered images and first images corresponding to a plurality of scales, and a plurality of second loss terms are generated based on second registered images and second images corresponding to a plurality of scales.

[0075] Specifically, the first registered image and the first image are calculated at least twice to obtain at least two first loss terms, the first registered image and the first image calculated each time have the same scale, and the first registered image and the first image between at least two calculations have different scales; the second registered image and the second image are calculated at least twice to obtain at least two second loss terms, the second registration parameter and the first image calculated each time have the same scale, and the second registration parameter and the first image between at least two calculations have different scales. The first loss term and the second loss term can be calculated by accumulation, cumulative averaging and weighting.

[0076] Exemplarily, the mean square loss of the first registered image with a size of h×w×d and the first image with a size of h×w×d is calculated, and the first loss term with a size of h×w×d is generated by cumulative averaging; the mean square loss of the first registered image with a size of h / 2×w / 2×d / 2 and the first image with a size of h / 2×w / 2×d / 2 is calculated, and the first loss term with a size of h / 2×w / 2×d / 2 is generated by cumulative averaging; the mean square loss of the first registered image with a size of h / 4×w / 4×d / 4 and the first image with a size of h / 4×w / 4×d / 4 is calculated, and the first loss term with a size of h / 4×w / 4×d / 4 is generated by cumulative averaging. Calculate the mean square loss of the second registered image of size h×w×d and the second image of size h×w×d, and accumulate the average to generate the second loss term of size h×w×d; calculate the mean square loss of the second registered image of size h / 2×w / 2×d / 2 and the second image of size h / 2×w / 2×d / 2, and accumulate the average to generate the second loss term of size h / 2×w / 2×d / 2; calculate the mean square loss of the second registered image of size h / 4×w / 4×d / 4 and the second image of size h / 4×w / 4×d / 4, and accumulate the average to generate the second loss term of size h / 4×w / 4×d / 4.

[0077] Optionally, a target loss function is generated based on multiple first loss terms and multiple second loss terms.

[0078] Specifically, the target loss function includes at least two first loss terms and at least two second loss terms, the first loss terms and the second loss terms have the same number, and can be calculated by accumulation, cumulative average and weighting. For example, the first loss terms of sizes h×w×d, h / 2×w / 2×d / 2 and h / 4×w / 4×d / 4 are accumulated, the second loss terms of sizes h×w×d, h / 2×w / 2×d / 2 and h / 4×w / 4×d / 4 are accumulated, the accumulated first loss terms and the second loss terms are averaged, and then the model parameters are updated according to the target loss function. The target loss function determined by multiple first loss terms and multiple second loss terms adjusts the parameters of the image registration model, improves the accuracy and comprehensiveness of the loss function, and further accelerates the convergence efficiency of the registration model training process.

[0079] The technical solution of this embodiment, by downsampling the first image and the second image at different scales to obtain the first image and the second image at different scales, can better extract data features at different depths in the image; downsampling the first registration parameter and the second registration parameter to obtain the first registration parameter and the second registration parameter at different scales, providing global and local image content of different degrees of refinement, and providing global features and local features of different degrees of refinement for model training, thereby improving the processing speed and efficiency of model registration.

[0080] Embodiment 3

[0081] Figure 5 FIG. 1 is a flow chart of an image registration method provided by Embodiment 3 of the present invention. Figure 5 As shown, the method includes:

[0082] S310, obtaining an image to be processed and a target space image; performing data splicing on the image to be processed and the target space image to obtain spliced ​​data.

[0083] In this embodiment, the target space image refers to the image that is expected to be converted to the target space, which can be obtained by querying the database according to the sample information provided by the user, or it can be an image provided by the user, such as image data collected by different devices for the same sample, or image data collected at different times for the same sample. The image to be processed is the image that needs to be moved during the registration process, which can be an image obtained by an image acquisition device or an image provided by the user.

[0084] S320, inputting the spliced ​​data into a pre-trained image registration model to obtain the registration parameters output by the image registration model.

[0085] The image registration model is trained based on the image registration model training method provided in any embodiment of the present invention.

[0086] S330 , performing registration processing on the image to be processed based on the registration parameters to obtain a registered image corresponding to the image to be processed, wherein the registered image and the target space image are located in the same space.

[0087] The technical solution of this embodiment is to perform data splicing of the image to be processed and the target space image, input the data into a pre-trained image registration model, obtain the registration parameters output by the image registration model, perform registration processing on the image to be processed with the registration parameters, and obtain a registered image corresponding to the image to be processed, so that the image to be processed and the target space image are located in the same space.

[0088] Embodiment 4

[0089] Figure 6 This is a training device for an image registration model provided by the fourth embodiment of the present invention. Figure 6 As shown, the training device comprises:

[0090] A sample acquisition module 410 is used to acquire a sample data set, where the sample data set includes a plurality of sample pairs, where the sample pairs include a first image and a second image located in different spaces;

[0091] The model training module 420 is used to iteratively train the image registration model to be trained based on multiple sample pairs until the training end condition is met to obtain a trained image registration model; wherein each iterative training process includes: for any group of sample pairs, the first image and the second image in the sample pair are spliced ​​to obtain spliced ​​data; the spliced ​​data is input into the image registration model in the training process to obtain a first registration parameter; the second registration parameter is determined based on the first registration parameter, and the second registration parameter is the inverse registration parameter of the first registration parameter; the second image is registered based on the first registration parameter to obtain a first registered image; the first image is registered based on the second registration parameter to obtain a second registered image; a first loss term is generated based on the first registered image and the first image, and a second loss term is generated based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function; the target loss function adjusts the model parameters of the image registration model in the training process.

[0092] The technical solution of this embodiment obtains the input of the image registration model by acquiring the first image and the second image located in different spaces and stitching them together; performs registration processing on the corresponding images using the first registration parameters and the second registration parameters, and adjusts the model parameters of the image registration model by calculating the target loss function, thereby improving the accuracy of image registration.

[0093] Optionally, the sample acquisition module 410 is further used for:

[0094] The second image is obtained by spatially transforming the first image;

[0095] The method also includes: performing image enhancement processing on the second image in each sample pair.

[0096] Optionally, the model training module 420 is further used to:

[0097] The first image and the second image in any sample pair are downsampled respectively to obtain a plurality of first images of different scales and a plurality of second images of different scales; the first image and the second image of the same scale are spliced ​​to obtain spliced ​​data of multiple scales, and the spliced ​​data of multiple scales are input into the image registration model in the training process to obtain the first registration parameter.

[0098] Optionally, the model training module 420 is further used to:

[0099] The first registration parameter is subjected to an inverse transformation process to obtain a second registration parameter.

[0100] Optionally, the model training module 420 is further used to:

[0101] The first registration parameter and the second registration parameter are downsampled to obtain first registration parameters of multiple scales and second registration parameters of multiple scales; the second image is registered based on the first registration parameters of multiple scales to obtain first registered images of multiple scales; the first image is registered based on the second registration parameters of multiple scales to obtain second registered images of multiple scales; a plurality of first loss terms are generated based on the first registered images and the first image corresponding to the multiple scales, and a plurality of second loss terms are generated based on the second registered images and the second image corresponding to the multiple scales; a target loss function is generated based on the multiple first loss terms and the multiple second loss terms.

[0102] The image registration model training device provided in the embodiment of the present invention can execute the image registration model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0103] Embodiment 5

[0104] Figure 7 This is an image registration device provided by Embodiment 5 of the present invention. Figure 6 As shown, the device comprises:

[0105] The image acquisition module 510 is used to acquire the image to be processed and the target space image; perform data splicing on the image to be processed and the target space image to obtain spliced ​​data;

[0106] A registration parameter determination module 520 is used to input the spliced ​​data into a pre-trained image registration model to obtain the registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model provided in any embodiment of the present invention;

[0107] The image registration module 530 is used to perform registration processing on the image to be processed based on the registration parameters to obtain a registered image corresponding to the image to be processed, and the registered image and the target space image are located in the same space.

[0108] The image registration device provided in the embodiment of the present invention can execute the image registration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0109] Embodiment 6

[0110] Figure 8 1 is a schematic diagram of the structure of an electronic device provided in Embodiment 6 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0111] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0112] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0113] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the training method of the image registration model and the image registration method.

[0114] In some embodiments, the training method of the image registration model or the image registration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the image registration model and / or the image registration method described above may be executed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the image registration model and the image registration method by any other appropriate means (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The computer program for implementing the training method of the image registration model or the image registration method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as an independent software package and partially on a remote machine, or completely on a remote machine or server.

[0117] Embodiment 7

[0118] Embodiment 7 of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a training method for an image registration model, the method comprising:

[0119] A sample data set is obtained, wherein the sample data set includes a plurality of sample pairs, wherein the sample pairs include a first image and a second image located in different spaces; an image registration model to be trained is iteratively trained based on the plurality of sample pairs until a training end condition is met, thereby obtaining a trained image registration model, wherein each iterative training process includes:

[0120] For any set of sample pairs, the first image and the second image in the sample pair are spliced ​​to obtain spliced ​​data; the spliced ​​data is input into the image registration model in the training process to obtain a first registration parameter; based on the first registration parameter, a second registration parameter is determined, and the second registration parameter is an inverse registration parameter of the first registration parameter;

[0121] Performing registration processing on the second image based on the first registration parameter to obtain a first registered image; performing registration processing on the first image based on the second registration parameter to obtain a second registered image; generating a first loss term based on the first registered image and the first image, and generating a second loss term based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function;

[0122] The model parameters of the image registration model during training are adjusted based on the target loss function.

[0123] Alternatively, the computer instructions are used to cause a processor to perform an image registration method, the method comprising:

[0124] Acquire the image to be processed and the target space image; perform data splicing on the image to be processed and the target space image to obtain spliced ​​data;

[0125] Input the spliced ​​data into a pre-trained image registration model to obtain registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model provided in any embodiment of the present invention;

[0126] The image to be processed is registered based on the registration parameters to obtain a registered image corresponding to the image to be processed, and the registered image and the target space image are located in the same space.

[0127] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0129] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0130] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0131] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0132] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for an image registration model, characterized in that: include: Acquire a sample data set, wherein the sample data set includes a plurality of sample pairs, wherein the sample pairs include a first image and a second image located in different spaces; The image registration model to be trained is iteratively trained based on the multiple samples until the training end condition is met to obtain a trained image registration model, wherein each iterative training process includes: For any set of sample pairs, the first image and the second image in the sample pair are spliced ​​to obtain spliced ​​data; the spliced ​​data is input into the image registration model in the training process to obtain a first registration parameter; based on the first registration parameter, a second registration parameter is determined, and the second registration parameter is an inverse registration parameter of the first registration parameter; Performing registration processing on the second image based on the first registration parameter to obtain a first registered image; performing registration processing on the first image based on the second registration parameter to obtain a second registered image; generating a first loss term based on the first registered image and the first image, and generating a second loss term based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function; The model parameters of the image registration model in the training process are adjusted based on the target loss function.

2. The method according to claim 1, characterized in that The second image is obtained by performing spatial transformation on the first image; The method further includes: performing image enhancement processing on the second image in each sample pair.

3. The method according to claim 1, characterized in that The determining a second registration parameter based on the first registration parameter comprises: Perform inverse transformation processing on the first registration parameters to obtain second registration parameters.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Downsampling the first image and the second image in any of the sample pairs respectively to obtain a plurality of first images of different scales and a plurality of second images of different scales; The first image and the second image of the same scale are spliced ​​to obtain spliced ​​data of multiple scales, and the spliced ​​data of multiple scales are input into the image registration model in the training process to obtain the first registration parameter.

5. The method according to claim 4, characterized in that The method further comprises: Downsampling the first registration parameter and the second registration parameter to obtain a plurality of first registration parameters of different scales and a plurality of second registration parameters of different scales; Performing registration processing on the second image based on a plurality of first registration parameters of different scales to obtain first registered images of multiple scales; performing registration processing on the first image based on a plurality of second registration parameters of different scales to obtain second registered images of multiple scales; Generate a plurality of first loss terms based on the first registered images and the first image corresponding to a plurality of scales, and generate a plurality of second loss terms based on the second registered images and the second image corresponding to a plurality of scales; A target loss function is generated based on the plurality of first loss terms and the plurality of second loss terms.

6. An image registration method, characterized in that: include: Acquire the image to be processed and the target space image; Performing data splicing on the image to be processed and the target space image to obtain spliced ​​data; Inputting the spliced ​​data into a pre-trained image registration model to obtain the registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model described in any one of claims 1 to 5; The image to be processed is registered based on the registration parameters to obtain a registered image corresponding to the image to be processed, wherein the registered image and the target space image are located in the same space.

7. A training device for an image registration model, characterized in that: include: A sample acquisition module, used to acquire a sample data set, wherein the sample data set includes a plurality of sample pairs, wherein the sample pairs include a first image and a second image located in different spaces; A model training module is used to iteratively train the image registration model to be trained based on the multiple sample pairs until the training end condition is met to obtain a trained image registration model, wherein each iterative training process includes: for any group of sample pairs, the first image and the second image in the sample pair are spliced ​​to obtain spliced ​​data; the spliced ​​data is input into the image registration model in the training process to obtain a first registration parameter; the second registration parameter is determined based on the first registration parameter, and the second registration parameter is the inverse registration parameter of the first registration parameter; the second image is registered based on the first registration parameter to obtain a first registered image; the first image is registered based on the second registration parameter to obtain a second registered image; a first loss term is generated based on the first registered image and the first image, and a second loss term is generated based on the second registered image and the second image, wherein the first loss term and the second loss term form a target loss function; model parameters of the image registration model in the training process are adjusted based on the target loss function.

8. An image registration device, characterized in that: include: An image acquisition module is used to acquire the image to be processed and the target space image; Performing data splicing on the image to be processed and the target space image to obtain spliced ​​data; A registration parameter determination module, used for inputting the stitching data into a pre-trained image registration model to obtain the registration parameters output by the image registration model; wherein the image registration model is trained based on the training method of the image registration model described in any one of claims 1 to 5; The image registration module is used to perform registration processing on the image to be processed based on the registration parameters to obtain a registered image corresponding to the image to be processed, wherein the registered image and the target space image are located in the same space.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the image registration model described in any one of claims 1-5, or the image registration method described in claim 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the image registration model described in any one of claims 1 to 5, or the image registration method described in claim 6 when executed.

Images