Image registration method and apparatus

Abstract

The embodiment of the present application provides a kind of image registration method and device, applied to image processing technical field, obtains the initial rotation parameter and initial translation parameter between X-ray image and CT image;Based on initial rotation parameter and initial translation parameter, CT image is projected to two-dimensional plane, and projection image is obtained;First image is converted from the first image style to the second image style of second image, and third image is obtained;Determine the target image area in fourth image that matches fifth image;According to the position of target image area in fourth image, update the horizontal parameter and longitudinal parameter in initial translation parameter, and obtain the image registration result containing initial rotation parameter, updated initial translation parameter.Application of the scheme provided in the embodiment of the present application can be used for image registration of X-ray image and CT image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image registration method and apparatus. Background Technology

[0002] Image-guided therapy is a common treatment method in medical practice. Doctors can use 2D X-ray images or 3D CT (Computed Tomography) images to identify the location of the disease. Because the principles behind generating X-ray and CT images differ—X-ray images can clearly show tissues such as muscles, while CT images can clearly show tissues such as bones—doctors often need to combine the information from both X-ray and CT images to formulate a treatment plan during image-guided therapy.

[0003] However, X-ray images are 2D images, while CT images are 3D images. The difference in dimensions makes it inconvenient to display both types of images simultaneously to doctors on electronic devices. Therefore, image conversion is necessary to convert X-ray images from 2D to 3D, or CT images from 3D to 2D. The image conversion process first requires image registration of the X-ray and CT images, and then the image conversion is performed based on the registration results.

[0004] Therefore, an image registration scheme is needed. Summary of the Invention

[0005] The purpose of this invention is to provide an image registration method and apparatus for registering X-ray images and CT images. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide an image registration method, the method comprising:

[0007] Obtain the initial rotation and translation parameters between the X-ray and CT images;

[0008] Based on the initial rotation parameters and initial translation parameters, the CT image is projected onto a two-dimensional plane to obtain a projected image;

[0009] The first image is converted from the first image style to the second image style of the second image to obtain the third image, wherein the first image is the X-ray image or the projection image, and the second image is: the X-ray image and the projection image other than the first image;

[0010] Determine the target image region in the fourth image that matches the fifth image, wherein the fourth image is the largest image among the third image and the second image, and the fifth image is the image among the third image and the second image excluding the fourth image;

[0011] Based on the position of the target image region in the fourth image, the horizontal and vertical parameters in the initial translation parameters are updated to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0012] In one embodiment of the present invention, the step of converting the first image from a first image style to a second image style to obtain a third image includes:

[0013] The first image is input into a pre-trained image style transfer model to perform image style transfer on the first image, resulting in a third image with the image style of the second image.

[0014] The image style transfer model is a first transfer model with adjusted network parameters obtained by training the model using a preset training method. The preset training method is a training method that performs generative adversarial training on the first transfer model, the second transfer model, the first discriminant model, and the second discriminant model together.

[0015] The first conversion model is a neural network model for converting the image style of an image to a second image style. The second conversion model is a neural network model for converting the style of an image to a first image style. The first discrimination model is a neural network model for determining whether the image style of an image is the first image style. The second discrimination model is a neural network model for determining whether the image style of an image is the second image style.

[0016] In one embodiment of the present invention, the image style transfer model is trained in the following manner:

[0017] The first sample image is input into the first discrimination model to determine whether the image style of the first sample image is the first image style, and the first discrimination result is obtained, wherein the image style of the first sample image is: the first image style;

[0018] The second sample image is input into the second transformation model, and image style transformation is performed on the second sample image to obtain the first transformed image;

[0019] The first transformed image is input into the first discrimination model, and the image style of the first transformed image is determined to be the first image style, thereby obtaining the second discrimination result;

[0020] Based on the first discrimination result and the second discrimination result, a first loss is calculated, wherein the first loss represents the loss of the first discrimination model in discriminating image style and the loss of the second transformation model in performing image style transformation;

[0021] The second sample image is input into the second discrimination model to determine whether the image style of the second sample image is the second image style, and a third discrimination result is obtained, wherein the image style of the second sample image is: the second image style;

[0022] The first sample image is input into the first conversion model, and image style conversion is performed on the first sample image to obtain the second converted image;

[0023] The second transformed image is input into the second discrimination model to determine whether the image style of the second transformed image is the second image style, and a fourth discrimination result is obtained.

[0024] Based on the third and fourth discrimination results, a second loss is calculated, wherein the second loss represents the loss of the second discrimination model in discriminating image style and the loss of the first transformation model in performing image style transformation;

[0025] The first converted image is input into the first conversion model to obtain the third converted image;

[0026] The second converted image is input into the second conversion model to obtain the fourth converted image;

[0027] A third loss is calculated based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image, wherein the third loss represents the loss of the first transformation model in performing image style transformation and the loss of the second transformation model in performing image style transformation.

[0028] Calculate the total loss based on the first loss, the second loss, and the third loss;

[0029] The parameters of the first conversion model, the second conversion model, the first discriminant model, and the second discriminant model are adjusted according to the total loss. If the convergence condition is not met, the image with the new image style of the first image style is taken as the first sample image, and the image with the new image style of the second image style is taken as the second sample image. The process of inputting the first sample image into the first discriminant model, judging whether the image style of the first sample image is the first image style, and obtaining the first discriminant result is repeated until the preset model convergence condition is met.

[0030] The first conversion model after parameter adjustment is determined as the image style conversion model.

[0031] In one embodiment of the present invention, the initial rotation parameters are obtained in the following manner:

[0032] Extracting image features from X-ray images;

[0033] The image features of the X-ray image are compared with those of historical X-ray images to identify the target historical X-ray image with the highest feature similarity to the X-ray image.

[0034] The historical rotation parameters corresponding to the target historical X-ray image are determined as the initial rotation parameters.

[0035] In one embodiment of the present invention, the height parameter in the initial translation parameters is obtained in the following manner:

[0036] Identify the first object region in the X-ray image and the second object region in the CT image;

[0037] Calculate the size ratio between the first object region and the second object region based on the size of the first object region and the size of the second object region;

[0038] The height parameter in the initial translation parameters is calculated based on the size ratio and the focal length of the device acquiring the X-ray image.

[0039] Secondly, embodiments of the present invention provide an image registration apparatus, the apparatus comprising:

[0040] The parameter acquisition module is used to obtain the initial rotation and initial translation parameters between the X-ray image and the CT image.

[0041] The image projection module is used to project the CT image onto a two-dimensional plane based on the initial rotation parameters and initial translation parameters to obtain a projected image;

[0042] A style conversion module is used to convert a first image from a first image style to a second image style to obtain a third image, wherein the first image is the X-ray image or the projection image, and the second image is: the X-ray image or the projection image other than the first image;

[0043] An image matching module is used to determine a target image region in the fourth image that matches the fifth image, wherein the fourth image is the largest image among the third image and the second image, and the fifth image is the image in the third image and the second image other than the fourth image;

[0044] The parameter update module is used to update the horizontal and vertical parameters in the initial translation parameters according to the position of the target image region in the fourth image, so as to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0045] In one embodiment of the present invention, the style conversion module is specifically used for:

[0046] The first image is input into a pre-trained image style transfer model to perform image style transfer on the first image, resulting in a third image with the image style of the second image.

[0047] The image style transfer model is a first transfer model with adjusted network parameters obtained by training the model using a preset training method. The preset training method is a training method that performs generative adversarial training on the first transfer model, the second transfer model, the first discriminant model, and the second discriminant model together.

[0048] The first conversion model is a neural network model for converting the image style of an image to a second image style. The second conversion model is a neural network model for converting the style of an image to a first image style. The first discrimination model is a neural network model for determining whether the image style of an image is the first image style. The second discrimination model is a neural network model for determining whether the image style of an image is the second image style.

[0049] In one embodiment of the present invention, the image style transfer model is trained by a model training module;

[0050] The model training module includes:

[0051] The first discrimination submodule is used to input the first sample image into the first discrimination model, determine whether the image style of the first sample image is the first image style, and obtain the first discrimination result, wherein the image style of the first sample image is: the first image style;

[0052] The first conversion submodule is used to input the second sample image into the second conversion model, perform image style conversion on the second sample image, and obtain the first converted image;

[0053] The second discrimination submodule is used to input the first transformed image into the first discrimination model, determine whether the image style of the first transformed image is the first image style, and obtain the second discrimination result.

[0054] The first loss calculation submodule is used to calculate a first loss based on the first discrimination result and the second discrimination result, wherein the first loss represents the loss of the first discrimination model in discriminating image style and the loss of the second conversion model in performing image style conversion;

[0055] The third discrimination submodule is used to input the second sample image into the second discrimination model, determine whether the image style of the second sample image is the second image style, and obtain the third discrimination result, wherein the image style of the second sample image is: the second image style;

[0056] The second conversion submodule is used to input the first sample image into the first conversion model, perform image style conversion on the first sample image, and obtain the second converted image.

[0057] The fourth discrimination submodule is used to input the second transformed image into the second discrimination model, determine whether the image style of the second transformed image is the second image style, and obtain the fourth discrimination result;

[0058] The second loss calculation submodule is used to calculate the second loss based on the third and fourth discrimination results, wherein the second loss represents the loss of the second discrimination model in discriminating image style and the loss of the first conversion model in performing image style conversion;

[0059] The third conversion submodule is used to input the first conversion image into the first conversion model to obtain the third conversion image;

[0060] The fourth conversion submodule is used to input the second converted image into the second conversion model to obtain the fourth converted image;

[0061] The third loss calculation submodule is used to calculate the third loss based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image, wherein the third loss represents the loss of the first transformation model performing image style transformation and the second transformation model performing image style transformation.

[0062] The total loss calculation submodule is used to calculate the total loss based on the first loss, the second loss, and the third loss.

[0063] The parameter adjustment submodule is used to adjust the parameters of the first conversion model, the second conversion model, the first discriminant model and the second discriminant model according to the total loss. If the convergence condition is not met, the image with the new image style of the first image style is used as the first sample image, and the image with the new image style of the second image style is used as the second sample image, triggering the execution of the first discriminant submodule until the preset model convergence condition is met.

[0064] The model determination submodule is used to determine the first transformation model after parameter adjustment as the image style transformation model.

[0065] In one embodiment of the present invention, the initial rotation parameters are obtained through the following rotation parameter obtaining submodule;

[0066] The rotation parameter acquisition submodule is specifically used for:

[0067] Extracting image features from X-ray images;

[0068] The image features of the X-ray image are compared with those of historical X-ray images to identify the target historical X-ray image with the highest feature similarity to the X-ray image.

[0069] The historical rotation parameters corresponding to the target historical X-ray image are determined as the initial rotation parameters.

[0070] In one embodiment of the present invention, the height parameter in the initial translation parameters is obtained through the following height parameter obtaining submodule;

[0071] The height parameter acquisition submodule is specifically used for:

[0072] Identify the first object region in the X-ray image and the second object region in the CT image;

[0073] Calculate the size ratio between the first object region and the second object region based on the size of the first object region and the size of the second object region;

[0074] The height parameter in the initial translation parameters is calculated based on the size ratio and the focal length of the device acquiring the X-ray image.

[0075] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0076] Memory, used to store computer programs;

[0077] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0078] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0079] Fifthly, embodiments of the present invention also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the steps described in the first aspect.

[0080] Beneficial effects of the embodiments of the present invention:

[0081] When performing image registration between X-ray and CT images using the scheme provided in this embodiment of the invention, initial rotation and translation parameters are first obtained. Based on these initial rotation and translation parameters, the CT image is projected onto a two-dimensional plane to obtain a projected image. The first image in both the X-ray and projected images is converted from a first image style to a second image style (excluding the first image), resulting in a third image. A target image region matching the fifth image is determined in the fourth image. Based on the position of the target image region in the fourth image, the lateral and longitudinal parameters in the initial translation parameters are calculated and updated. This yields an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0082] As can be seen from the above, the CT image is first projected onto a two-dimensional plane based on the initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if the pixels in the projected CT image and the X-ray image correspond to the same positions in the real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention first converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. Then, the calculated horizontal and vertical parameters are used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration results to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0083] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a flowchart illustrating an image registration method provided in an embodiment of the present invention.

[0086] Figure 2 This is a schematic diagram illustrating the dimensional relationship between a first object region and a second object region, provided in an embodiment of the present invention.

[0087] Figure 3 This is a schematic diagram of the structure of an image registration device provided in an embodiment of the present invention;

[0088] Figure 4 This is a schematic diagram of the structure of a model training module provided in an embodiment of the present invention;

[0089] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0090] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] In order to perform image conversion between X-ray images and CT images, it is necessary to first register the X-ray images and CT images. This invention provides an image registration method and apparatus.

[0092] In one embodiment of the present invention, an image registration method is provided, the method comprising:

[0093] Obtain the initial rotation and translation parameters between the X-ray and CT images.

[0094] Based on the initial rotation and translation parameters mentioned above, the CT image is projected onto a two-dimensional plane to obtain a projected image.

[0095] The first image is converted from a first image style to a second image style to obtain a third image. The first image is either the X-ray image or the projection image, and the second image is any image other than the first image found in either the X-ray image or the projection image.

[0096] Identify the target image region in the fourth image that matches the fifth image. The fourth image is the largest image among the third and second images, and the fifth image is any image in the third and second images other than the fourth image.

[0097] Based on the position of the target image region in the fourth image, the horizontal and vertical parameters in the initial translation parameters are updated to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0098] As can be seen from the above, the CT image is first projected onto a two-dimensional plane based on the initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if the pixels in the projected CT image and the X-ray image correspond to the same positions in the real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention first converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. Then, the calculated horizontal and vertical parameters are used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration results to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0099] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0100] The image registration method and apparatus provided in the embodiments of the present invention will be described below through specific examples.

[0101] See Figure 1 The present invention provides a flowchart of an image registration method, which includes the following steps S101-S105.

[0102] S101: Obtain the initial rotation and translation parameters between the X-ray and CT images.

[0103] Specifically, the initial rotation parameters include the horizontal rotation angle Δθx, the vertical rotation angle Δθy, and the height rotation angle Δθz, while the initial translation parameters include the horizontal parameter Δx, the vertical parameter Δy, and the height parameter Δz.

[0104] Based on the initial rotation and translation parameters mentioned above, the two-dimensional coordinates of pixels in an X-ray image can be converted into three-dimensional coordinates, or the three-dimensional coordinates of pixels in a CT image can be converted into two-dimensional coordinates.

[0105] In one embodiment of the present invention, since the acquisition device for acquiring the X-ray image and the acquisition device for acquiring the CT image are devices with relatively fixed structures, the parameters between them are relatively fixed during the image acquisition process. That is, the initial rotation parameters and initial translation parameters between the X-ray image and the CT image are relatively fixed, and therefore the initial rotation parameters and initial translation parameters can be fixed preset parameters.

[0106] Specifically, the correspondence between initial rotation parameters and the object's posture when acquiring X-ray or CT images can be preset, thereby obtaining the initial rotation parameters based on the object's posture. For example, the lateral decubitus posture corresponds to the first rotation parameter, the supine decubitus posture corresponds to the second rotation parameter, and the prone decubitus posture corresponds to the third rotation parameter, etc.

[0107] Alternatively, the initial rotation parameters described above can also be obtained through steps A-C in the embodiments of the present invention, which will not be detailed here.

[0108] In addition, the height parameter in the above translation parameters can be either a preset parameter or it can be achieved through steps D-F, which will not be detailed here.

[0109] Furthermore, since the lateral and longitudinal parameters in the initial translation parameters will be updated through the following steps S102-S105, the initial lateral and longitudinal parameters can be any preset values, which will be updated to more accurate values ​​later. The initial lateral and longitudinal parameters can be the same or different. For example, the initial lateral and longitudinal parameters can both be 0.

[0110] S102: Based on the above initial rotation parameters and initial translation parameters, the above CT image is projected onto a two-dimensional plane to obtain a projected image.

[0111] Specifically, CT images can be projected onto a two-dimensional plane using the DRR (Digitally Reconstructed Radiograph) algorithm, which converts the three-dimensional coordinates of each pixel in the CT image into two-dimensional coordinates. The initial rotation and translation parameters mentioned above are the parameters used in the DRR algorithm. Alternatively, other existing image conversion algorithms can also be used to project CT images onto a two-dimensional plane, thereby converting the CT image from a three-dimensional image to a two-dimensional image.

[0112] S103: Convert the first image from the first image style to the second image style of the second image to obtain the third image.

[0113] The first image mentioned above is either the X-ray image or the projection image mentioned above.

[0114] When the first image is an X-ray image, the style of the first image is the image style of an X-ray image. Since the muscles, lymph nodes, and other tissues of an object are displayed more clearly in an X-ray image compared to a projection image, the first image style of the X-ray image is one that displays muscles, lymph nodes, and other tissues more clearly.

[0115] When the first image is a projection image, the first image style is the image style of the projection image. Since the bones and other tissues of an object are displayed more clearly in a CT image compared to an X-ray image, the bones and other tissues of the object are displayed more clearly in the projection image of the CT image. Therefore, the first image style of the projection image is a style that displays the bones and other tissues more clearly.

[0116] Specifically, the aforementioned objects can be people, animals, or parts of people or animals, such as a person's legs or an animal's head.

[0117] In addition, the second image mentioned above refers to the image other than the first image mentioned above among the X-ray image and projection image.

[0118] Specifically, if the first image is an X-ray image and the second image is a projection image, then converting the first image from its first image style to the second image style means converting the X-ray image from its X-ray image style to the projection image style. Therefore, the resulting third image will have the same projection image style as the second image, and their image styles will be identical.

[0119] If the first image is a projection image and the second image is an X-ray image, then converting the first image from its first image style to the second image style means converting the projection image from its projection image style to the X-ray image style. Therefore, the resulting third image will have the same X-ray image style as the second image, and their image styles will be identical.

[0120] As can be seen from the above, regardless of whether the first image is an X-ray image or a projection image, the image style of the third image obtained through the above step S103 is the same as that of the second image.

[0121] Furthermore, since X-ray images are typically smaller than the projected images of CT scans, the size differences between different X-ray images are relatively small, while the size differences between different projected images are relatively large. Therefore, using the X-ray image as the first image and converting its image style to that of the projected image can result in smaller size differences and more uniform specifications among the generated third images.

[0122] In one embodiment of the present invention, step S103 can be achieved by the following step G.

[0123] Step G: Input the first image into the pre-trained image style transfer model to perform image style transfer on the first image, and obtain a third image with the image style of the second image style.

[0124] The image style transfer model mentioned above is the first transfer model obtained by training the model using a preset training method and adjusting the network parameters.

[0125] The aforementioned preset training method is a training method that involves performing generative adversarial training on the first conversion model, the second conversion model, the first discriminant model, and the second discriminant model together.

[0126] The first conversion model described above is a neural network model used to convert the image style of an image to a second image style. The second conversion model described above is a neural network model used to convert the style of an image to a first image style.

[0127] Specifically, the first and second transformation models mentioned above can be neural network models with the same structure, or they can be neural network models with different structures. For example, the first and / or second transformation models mentioned above can be a ResNet network containing 9 blocks.

[0128] The first discrimination model mentioned above is a neural network model used to determine whether an image's style belongs to a first image style. The second discrimination model mentioned above is a neural network model used to determine whether an image's style belongs to a second image style.

[0129] The first and second discriminant models mentioned above can be neural network models with the same structure or neural network models with different structures. For example, the first and / or second discriminant models mentioned above can be 70×70 PatchGANs (Patch-based discriminator of GANs, Markov discriminant networks), that is, backtracking based on the output of the first or second discriminant model, and finally corresponding to a 70×70 region in the input image.

[0130] In the case where the above-mentioned preset training method is generative adversarial training, the first conversion model and the second conversion model can be referred to as generators, the trained image style transfer model can be referred to as adversarial network, and the first discriminant model and the second discriminant model can be referred to as discriminators.

[0131] In one embodiment of the present invention, the above-mentioned image style transfer model can be trained through steps H1-H14 in the manner of generating objects for training, which will not be described in detail here.

[0132] Specifically, the aforementioned generative adversarial training can be any training method based on generative adversarial training, such as recurrent adversarial training, in which case the trained image style transfer model can be called a recurrent adversarial network, or other training methods based on generative adversarial training such as contrastive-unpaired-translation.

[0133] In addition to the above-mentioned preset training methods, other model training methods such as deep learning can also be used.

[0134] S104: Determine the target image region in the fourth image that matches the fifth image.

[0135] The fourth image mentioned above is the largest image among the third and second images.

[0136] The fifth image mentioned above refers to the third image and the second image, excluding the fourth image mentioned above.

[0137] Specifically, because CT images are often large, the size of the projected image of a CT image is often larger than that of an X-ray image. Therefore, the fourth image mentioned above is often a projected image, or a third image obtained by performing image style conversion on a projected image, and the fifth image mentioned above is often an X-ray image, or a third image obtained by performing image style conversion on an X-ray image.

[0138] In one embodiment of the present invention, a target image region in the fourth image that matches the fifth image can be determined by template matching. That is, the fifth image is used as a template, and the pixel values ​​of each image region in the fourth image that is the same size as the template are compared with the template in turn to determine the image region with the highest similarity as the target image region.

[0139] S105: Based on the position of the target image region in the fourth image, update the horizontal and vertical parameters in the initial translation parameters to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0140] The position of the target image region in the fourth image can be represented by the coordinates of the edge pixels of the target image region, such as the coordinates of the top left pixel, the bottom right pixel, or the set of coordinates of all edge pixels of the target image region.

[0141] In one embodiment of the present invention, the target horizontal parameter and the target vertical parameter can be determined according to the position of the target image region in the fourth image, and the original horizontal parameter and vertical parameter in the initial translation parameter can be replaced by the determined target horizontal parameter and the target vertical parameter, respectively, so as to realize the update of the horizontal parameter and vertical parameter in the initial translation parameter.

[0142] Specifically, since the third image is often a rectangular image, the target image region is also a rectangular image region. Furthermore, in most cases, when representing the position of pixels in an image using coordinate values, the origin is often the top-left corner of the image. Therefore, the coordinate value of the top-left pixel of the target image region can represent the horizontal relationship between the target image region and the top-left pixel of the fourth image. This is equivalent to the horizontal relationship between the target image region and the fourth image, which is also the horizontal relationship between the fifth image and the fourth image. Since the horizontal and vertical parameters in the initial translation parameters determine the translation of the projected image on the horizontal plane, the horizontal coordinate value of the top-left pixel of the target image region can be used as the horizontal parameter, and the vertical coordinate value as the vertical parameter.

[0143] Alternatively, the initial rotation parameters and the initial translation parameters after updating the horizontal and vertical parameters can be used as the image registration results.

[0144] As can be seen from the above, the CT image is first projected onto a two-dimensional plane based on the initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if the pixels in the projected CT image and the X-ray image correspond to the same positions in the real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention first converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. Then, the calculated horizontal and vertical parameters are used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration results to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0145] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0146] In one embodiment of the present invention, the above-mentioned initial rotation parameters can be obtained through the following steps A-C.

[0147] Step A: Extract image features from the X-ray image.

[0148] In one embodiment of the present invention, image features of X-ray images can be extracted using the LBP (Local Binary Pattern) algorithm, the HOG (Histogram of Oriented Gradient) algorithm, or other existing algorithms. The present invention does not limit this to such methods.

[0149] Step B: Compare the image features of the X-ray image with those of historical X-ray images to identify the target historical X-ray image with the highest feature similarity to the X-ray image.

[0150] Among them, the aforementioned historical X-ray images are X-ray images with known corresponding historical rotation parameters.

[0151] Specifically, feature comparison can be performed using the SIFT (Scale-invariant feature transform) algorithm or other existing algorithms, but this embodiment of the invention does not limit the scope of the comparison.

[0152] Step C: Determine the historical rotation parameters corresponding to the above target historical X-ray image as the above initial rotation parameters.

[0153] In one embodiment of the present invention, the X-ray image can be input into a pre-trained rotation parameter determination model to obtain the initial rotation parameters, thereby realizing steps A-C.

[0154] Specifically, the aforementioned rotation parameter determination model can be a model obtained by training a neural network model using the aforementioned historical X-ray images as sample images, and used to determine the rotation parameters corresponding to the X-ray images.

[0155] The rotation parameter determination model mentioned above can be a convolutional neural network model.

[0156] In another embodiment of the present invention, the height parameter in the initial translation parameters described above can be obtained through the following steps D-F.

[0157] Step D: Identify the first object region in the X-ray image and the second object region in the CT image.

[0158] In one embodiment of the present invention, a pre-trained first object recognition model can be used to identify a first object region in an X-ray image, and a pre-trained second object recognition model can be used to identify a second object region in a CT image.

[0159] The aforementioned first object recognition model can be a model used to identify object regions in X-ray images, obtained by training a neural network model with sample X-ray images of known object regions. The aforementioned neural network model can be a convolutional neural network model.

[0160] The aforementioned second object recognition model can be a model for recognizing object regions in X-ray images, obtained by training a neural network model using sample CT images of known object regions. This neural network model can be a convolutional neural network model.

[0161] Step E: Calculate the size ratio between the first object region and the second object region based on the size of the first object region and the size of the second object region.

[0162] Specifically, the aforementioned dimensions can be length, width, or area. When the dimension is length, the size ratio is the ratio between the length of the first object region and the length of the second object region. When the dimension is width, the size ratio is the ratio between the width of the first object region and the width of the second object region. When the dimension is area, the size ratio is the ratio between the area of ​​the first object region and the area of ​​the second object region.

[0163] Step F: Calculate the height parameter in the initial translation parameters based on the above size ratio and the focal length of the device acquiring the above X-ray image.

[0164] See Figure 2 This is a schematic diagram illustrating the dimensional relationship between a first object region and a second object region, provided in an embodiment of the present invention.

[0165] The diagram contains two similar triangles. The base of the smaller triangle represents the second object region, with a size of x0. The base of the larger triangle represents the first object region, with a size of x1. Point Y represents the location of the X-ray emitting device in the X-ray image acquisition equipment. The dashed line simulates the cone-shaped X-ray beam emitted by the equipment during image acquisition. The vertical distance between point Y and the first object region is the focal length f of the X-ray image acquisition equipment. The vertical distance between the bases of the two triangles is the height parameter z0.

[0166] In one embodiment of the present invention, by Figure 2 It can be seen that, due to Figure 2 The two triangles in the diagram are similar triangles, therefore the height parameters can be calculated using the following formula:

[0167]

[0168] Where L is the aforementioned size ratio, x0 is the size of the aforementioned second object region, x1 is the size of the aforementioned first object region, f is the aforementioned focal length, and z0 is the aforementioned height parameter.

[0169] In one embodiment of the present invention, the above-mentioned image style transfer model can be obtained by training through the following steps H1-H14.

[0170] Step H1: Input the first sample image into the first discrimination model, determine whether the image style of the first sample image is the first image style, and obtain the first discrimination result.

[0171] The image style of the first sample image is: the first image style.

[0172] Specifically, the first sample image can be a sample X-ray image, in which case the first image style can be the image style of an X-ray image; or the first sample image can be a sample projection image, in which case the first image style can be the image style of a projection image.

[0173] When the first sample image is a sample X-ray image, it can be a pre-obtained set of a first predetermined number of X-ray images with the same resolution. That is, the size of the real-world scene corresponding to the pixels in different sample X-ray images is the same.

[0174] In addition, the spacing-to-height ratio of the above sample X-ray images can be the same, and the size of the above sample X-ray images can also be the same.

[0175] For example, the first preset quantity can be 100, 200, etc., and the size of the sample X-ray image can be 256*256 pixels.

[0176] In the case where the first sample image is a sample projection image, the first sample image can be a pre-obtained second preset number of projection images with the same resolution. That is, the size of the area in the real scene corresponding to the pixels in different sample projection images is the same.

[0177] For example, the second preset quantity mentioned above can be 100, 200, etc.

[0178] In addition, the above-mentioned sample projection image can also be obtained by projecting the sample CT image onto a two-dimensional plane.

[0179] Specifically, the DRR algorithm can be used to project the sample CT image onto a two-dimensional plane to obtain the sample projection image.

[0180] In the process of projecting the sample CT image using the DRR algorithm to obtain the sample projection image, the sample rotation parameters used can be any one of a preset set of rotation parameters. Specifically, the sample rotation parameters used can be different for different sample CT images.

[0181] The height parameter in the sample translation parameters used can be a preset value, and the horizontal and vertical parameters in the sample translation parameters used can also be preset values.

[0182] Step H2: Input the second sample image into the second conversion model, perform image style conversion on the second sample image, and obtain the first converted image.

[0183] Specifically, the resulting first transformed image has the same size as the second sample image.

[0184] Step H3: Input the first transformed image into the first discrimination model, determine whether the image style of the first transformed image is the first image style, and obtain the second discrimination result.

[0185] Specifically, if the second discrimination result indicates that the image style of the first transformed image is the first image style, then it means that the second transformation model can transform the image style of the second sample image into the first image style.

[0186] Step H4: Calculate the first loss based on the first and second discrimination results mentioned above.

[0187] The first loss mentioned above represents the loss of the first discrimination model in judging image style and the loss of the second transformation model in performing image style transformation.

[0188] Specifically, since the image style of the first sample image is known to be the first image style, the first discrimination model can determine whether it can accurately identify that the first sample image is the first image style based on the first discrimination result.

[0189] Since the second conversion model is used to convert the image style of an image to the first image style, the second discrimination result can be used to indicate whether the second conversion model has successfully converted the image style of the second sample image to the first image style.

[0190] Therefore, the first loss calculated based on the first and second discrimination results can be used to represent the loss of the first discrimination model in judging image style and the loss of the second transformation model in performing image style transformation.

[0191] Specifically, the first loss mentioned above can be calculated using the following formula:

[0192] LOSS1 = E a-A [logDA (a)]+E b-B [log(1-D A (G BA (b)))]

[0193] Where LOSS1 is the first loss mentioned above, a is the first sample image mentioned above, and D A () represents the first discriminant model mentioned above, D A (a) is the first discrimination result, b is the second sample image mentioned above, G BA () represents the second transformation model mentioned above, G BA (b) is the first transformed image mentioned above, D A (G BA (b) represents the second discrimination result mentioned above, E a-A () and E b-B () indicates the calculation of the expected value of the distribution function, where A is the distribution of the first sample image and B is the distribution of the second sample image.

[0194] Step H5: Input the second sample image into the second discrimination model, determine whether the image style of the second sample image is the second image style, and obtain the third discrimination result.

[0195] The image style of the second sample image is: second image style.

[0196] Specifically, when the first sample image is a sample X-ray image, the second sample image is a sample projection image; when the first sample image is a sample projection image, the second sample image is a sample X-ray image.

[0197] In addition, the resolution of the first sample image is the same as that of the second sample image.

[0198] Step H6: Input the first sample image into the first conversion model, perform image style conversion on the first sample image, and obtain the second converted image.

[0199] Specifically, the resulting second transformed image has the same size as the first sample image.

[0200] Step H7: Input the second transformed image into the second discrimination model, determine whether the image style of the second transformed image is the second image style, and obtain the fourth discrimination result.

[0201] Specifically, if the fourth discrimination result indicates that the image style of the second transformed image is the second image style, it means that the first transformation model can transform the image style of the first sample image into the second image style.

[0202] Step H8: Calculate the second loss based on the third and fourth discrimination results mentioned above.

[0203] The second loss is characterized by the loss of the second discrimination model in judging image style and the loss of the first transformation model in performing image style transformation.

[0204] Specifically, since the image style of the second sample image is known to be the second image style, the second discrimination model can be determined based on the third discrimination result to determine whether the second discrimination model can accurately identify that the second sample image is the second image style.

[0205] Since the first conversion model is used to convert the image style of an image to the second image style, the fourth discrimination result can be used to indicate whether the first conversion model has successfully converted the image style of the first sample image to the second image style.

[0206] Therefore, the second loss calculated based on the third and fourth discrimination results can be used to represent the loss of the second discrimination model in discriminating image style and the first transformation model in performing image style transformation.

[0207] LOSS2 = E b-B [logD B (b)]+E a-A [log(1-D B (G AB (a)))]

[0208] Where LOSS2 is the second loss mentioned above, b is the second sample image mentioned above, and D B () represents the second discriminant model mentioned above, D B (b) is the third discrimination result, and a is the first sample image mentioned above. AB () represents the first transformation model mentioned above, G AB (a) is the second transformed image mentioned above, D B (G AB (a) represents the fourth discrimination result mentioned above, E a-A () and E b-B () indicates the calculation of the expected value of the distribution function, where A is the distribution of the first sample image and B is the distribution of the second sample image.

[0209] Step H9: Input the first transformed image into the first transformed model to obtain the third transformed image.

[0210] Specifically, in theory, the first transformed image is obtained by converting the image style of the second sample image to the first image style. Then, the first transformed image is input into the first transformation model to convert the image style back to the second image style. If both image style conversions are successful, the third transformed image should be converted back to the second image style, similar to the second sample image.

[0211] Step H10: Input the second transformed image into the second transformed model to obtain the fourth transformed image.

[0212] Specifically, in theory, the second transformed image is obtained by converting the image style of the first sample image to a second image style. This second transformed image is then input into the second transformation model to convert the image style back to the first image style. If both image style conversions are successful, the fourth transformed image should be converted back to the first image style, becoming similar to the first sample image.

[0213] Step H11: Calculate the third loss based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image.

[0214] The third loss mentioned above represents the loss of image style transfer performed by the first conversion model and the loss of image style transfer performed by the second conversion model.

[0215] Specifically, if the third transformed image is similar to the second sample image, it means that the second transformation model has successfully performed style transformation on the second sample image, and the first transformation model has also successfully performed style transformation on the first transformed image.

[0216] If the fourth transformed image is similar to the first sample image, it means that the first transformation model successfully performed style transformation on the first sample image, and the second transformation model also successfully performed style transformation on the second transformed image.

[0217] Therefore, based on the third transformed image and the first sample image, and the fourth transformed image and the second sample image, a third loss that can characterize the image style transfer performed by the first transformation model and the image style transfer performed by the second transformation model can be calculated.

[0218] In one embodiment of the present invention, the aforementioned third loss can be calculated according to the following formula:

[0219] LOSS3 = E a-A [||G BA (G AB (a))-a||1]+E b-B [||G AB (G BA (b))-b||1]

[0220] Where LOSS3 is the third loss mentioned above, ||||1 represents the calculation of the L1 norm, a is the first sample image mentioned above, b is the second sample image mentioned above, and G is the third loss mentioned above. AB () represents the first transformation model mentioned above, G AB (a) is the second transformed image mentioned above, G BA() represents the second transformation model described above, G BA (G AB (a) is the fourth transformed image mentioned above, G BA (b) is the second transformed image mentioned above, G AB (G BA (b) is the third transformed image mentioned above, E a-A () and E b-B () indicates the calculation of the expected value of the distribution function, where A is the distribution of the first sample image and B is the distribution of the second sample image.

[0221] Step H12: Calculate the total loss based on the first loss, second loss and third loss mentioned above.

[0222] In one embodiment of the present invention, the sum of the first loss, the second loss, and the third loss can be calculated as the total loss. Alternatively, a weighted sum of the first loss, the second loss, and the third loss can be calculated as the total loss.

[0223] Step H13: Adjust the parameters of the first transformation model, the second transformation model, the first discriminant model, and the second discriminant model according to the total loss. If the convergence condition is not met, take the image with the new image style of the first image style as the first sample image and the image with the new image style of the second image style as the second sample image. Return to step H1 and continue until the preset model convergence condition is met.

[0224] Specifically, the convergence condition can be the number of times steps H1-H13 are executed, or the total loss can be lower than the preset loss.

[0225] Step H14: Determine the first transformation model after parameter adjustment as the above image style transformation model.

[0226] Specifically, since the first conversion model is used to convert the image style of an image into the second image style, during the continuous model training process, the first conversion model continuously makes the image style of the output image approach the real second image style, so that the image output by the first conversion model can pass the test of the second discriminant model more and more. The second discriminant model believes that the image style of the output image is the second image style.

[0227] Meanwhile, the second discriminant model is used to determine whether the image style of an image is the second image style. During the continuous training of the model, the second discriminant model continuously makes the discrimination results more accurate. That is, the image obtained by image style conversion, rather than the real second image style, becomes increasingly difficult to pass the test of the second discriminant model. In other words, the image output by the first conversion model becomes increasingly difficult to pass the test of the second discriminant model.

[0228] As can be seen from the above, the first transformation model needs to ensure that the output image can pass the test of the second discriminant model, while the second discriminant model needs to ensure that the image output by the first transformation model cannot pass the test of the second discriminant model. Therefore, there is an adversarial relationship between the first transformation model and the second discriminant model.

[0229] Similarly, there is also an adversarial relationship between the second transformation model and the first discriminant model.

[0230] Furthermore, after the second transformation model performs image style transfer on the second sample image to obtain the first transformed image, the first transformation model then performs image style transfer on the first transformed image. Similarly, after the first transformation model performs image style transfer on the first sample image to obtain the second transformed image, the second transformation model then performs image style transfer on the second transformed image. This image style transfer process is performed cyclically.

[0231] Since the above model training process is both adversarial and recurrent, the model training process in steps H1-H14 can be called a recurrent adversarial training process, and the trained image style transfer model can be called a recurrent adversarial network.

[0232] As can be seen from the above, during the training process of the aforementioned model, the parameters of the first conversion model are adjusted according to the total loss, and the training satisfies the preset convergence condition. Therefore, the first conversion model is trained to convergence, and the resulting image style transfer model can convert an image from a first image style to a second image style. Furthermore, there is no constraint regarding the correlation between the first and second sample images, making it relatively simple to obtain the required first and second sample images during model training. Moreover, because the method of obtaining the first and second sample images is relatively simple, a large number of first and second sample images can be used to train the first conversion model, the second conversion model, the first discriminant model, and the second discriminant model, resulting in a trained image style transfer model with good generalization performance.

[0233] Corresponding to the aforementioned image registration method, see [link to relevant documentation]. Figure 3 The present invention also provides a schematic diagram of an image registration device, the device comprising:

[0234] The parameter acquisition module 301 is used to obtain the initial rotation parameters and initial translation parameters between the X-ray image and the CT image;

[0235] The image projection module 302 is used to project the CT image onto a two-dimensional plane based on the initial rotation parameters and the initial translation parameters to obtain a projected image;

[0236] Style conversion module 303 is used to convert the first image from the first image style to the second image style of the second image to obtain a third image, wherein the first image is the X-ray image or the projection image, and the second image is: the X-ray image or the projection image other than the first image;

[0237] Image matching module 304 is used to determine the target image region in the fourth image that matches the fifth image, wherein the fourth image is the largest image among the third image and the second image, and the fifth image is the image in the third image and the second image other than the fourth image;

[0238] The parameter update module 305 is used to update the horizontal and vertical parameters in the initial translation parameters according to the position of the target image region in the fourth image, so as to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

[0239] As can be seen from the above, the CT image is first projected onto a two-dimensional plane based on the initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if the pixels in the projected CT image and the X-ray image correspond to the same positions in the real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention first converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. Then, the calculated horizontal and vertical parameters are used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration results to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0240] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0241] In one embodiment of the present invention, the style conversion module 303 is specifically used for:

[0242] The first image is input into a pre-trained image style transfer model to perform image style transfer on the first image, resulting in a third image with the image style of the second image.

[0243] The image style transfer model is a first transfer model with adjusted network parameters obtained by training the model using a preset training method. The preset training method is a training method that performs generative adversarial training on the first transfer model, the second transfer model, the first discriminant model, and the second discriminant model together.

[0244] The first conversion model is a neural network model for converting the image style of an image to a second image style. The second conversion model is a neural network model for converting the style of an image to a first image style. The first discrimination model is a neural network model for determining whether the image style of an image is the first image style. The second discrimination model is a neural network model for determining whether the image style of an image is the second image style.

[0245] See Figure 4 This is a schematic diagram of the structure of a model training module provided in an embodiment of the present invention, wherein the model training module is used to train the above-mentioned image style transfer model.

[0246] Specifically, the above-mentioned model training module includes:

[0247] The first discrimination submodule 401 is used to input the first sample image into the first discrimination model, determine whether the image style of the first sample image is the first image style, and obtain the first discrimination result, wherein the image style of the first sample image is: the first image style;

[0248] The first conversion submodule 402 is used to input the second sample image into the second conversion model, perform image style conversion on the second sample image, and obtain the first converted image;

[0249] The second discrimination submodule 403 is used to input the first transformed image into the first discrimination model, determine whether the image style of the first transformed image is the first image style, and obtain the second discrimination result;

[0250] The first loss calculation submodule 404 is used to calculate a first loss based on the first discrimination result and the second discrimination result, wherein the first loss represents the loss of the first discrimination model in discriminating image style and the loss of the second conversion model in performing image style conversion;

[0251] The third discrimination submodule 405 is used to input the second sample image into the second discrimination model, determine whether the image style of the second sample image is the second image style, and obtain the third discrimination result, wherein the image style of the second sample image is: the second image style;

[0252] The second conversion submodule 406 is used to input the first sample image into the first conversion model, perform image style conversion on the first sample image, and obtain the second converted image.

[0253] The fourth discrimination submodule 407 is used to input the second transformed image into the second discrimination model, determine whether the image style of the second transformed image is the second image style, and obtain the fourth discrimination result;

[0254] The second loss calculation submodule 408 is used to calculate a second loss based on the third discrimination result and the fourth discrimination result, wherein the second loss represents the loss of the second discrimination model in discriminating image style and the first conversion model in performing image style conversion;

[0255] The third conversion submodule 409 is used to input the first conversion image into the first conversion model to obtain the third conversion image;

[0256] The fourth conversion submodule 410 is used to input the second converted image into the second conversion model to obtain the fourth converted image;

[0257] The third loss calculation submodule 411 is used to calculate the third loss based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image, wherein the third loss represents the loss of the first transformation model performing image style transformation and the second transformation model performing image style transformation.

[0258] The total loss calculation submodule 412 is used to calculate the total loss based on the first loss, the second loss and the third loss;

[0259] The parameter adjustment submodule 413 is used to adjust the parameters of the first conversion model, the second conversion model, the first discriminant model and the second discriminant model according to the total loss. If the convergence condition is not met, the image with the new image style of the first image style is used as the first sample image, and the image with the new image style of the second image style is used as the second sample image, triggering the execution of the first discriminant submodule 401 until the preset model convergence condition is met.

[0260] The model determination submodule 414 is used to determine the first transformation model after parameter adjustment as the image style transformation model.

[0261] As can be seen from the above, during the training process of the aforementioned model, the parameters of the first conversion model are adjusted according to the total loss, and the training satisfies the preset convergence condition. Therefore, the first conversion model is trained to convergence, and the resulting image style transfer model can convert an image from a first image style to a second image style. Furthermore, there is no constraint regarding the correlation between the first and second sample images, making it relatively simple to obtain the required first and second sample images during model training. Moreover, because the method of obtaining the first and second sample images is relatively simple, a large number of first and second sample images can be used to train the first conversion model, the second conversion model, the first discriminant model, and the second discriminant model, resulting in a trained image style transfer model with good generalization performance.

[0262] In one embodiment of the present invention, the initial rotation parameters are obtained through the following rotation parameter obtaining submodule;

[0263] The rotation parameter acquisition submodule is specifically used for:

[0264] Extracting image features from X-ray images;

[0265] The image features of the X-ray image are compared with those of historical X-ray images to identify the target historical X-ray image with the highest feature similarity to the X-ray image.

[0266] The historical rotation parameters corresponding to the target historical X-ray image are determined as the initial rotation parameters.

[0267] In one embodiment of the present invention, the height parameter in the initial translation parameters is obtained through the following height parameter obtaining submodule;

[0268] The height parameter acquisition submodule is specifically used for:

[0269] Identify the first object region in the X-ray image and the second object region in the CT image;

[0270] Calculate the size ratio between the first object region and the second object region based on the size of the first object region and the size of the second object region;

[0271] The height parameter in the initial translation parameters is calculated based on the size ratio and the focal length of the device acquiring the X-ray image.

[0272] This invention also provides an electronic device, such as... Figure 5As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0273] Memory 503 is used to store computer programs;

[0274] When the processor 501 executes the program stored in the memory 503, it implements any of the steps described in the above image registration method.

[0275] When performing image registration using the electronic device provided in this embodiment of the invention, the CT image is first projected onto a two-dimensional plane based on initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if pixels in the projected image of the CT image and the X-ray image correspond to the same positions in a real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. The calculated horizontal and vertical parameters are then used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration result to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0276] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0277] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0278] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0279] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0280] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0281] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the steps described in the above image registration method.

[0282] When performing image registration using the computer program stored in the computer-readable storage medium provided in this embodiment of the invention, the CT image is first projected onto a two-dimensional plane based on initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if pixels in the projected image of the CT image and the X-ray image correspond to the same positions in a real scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention first converts the X-ray image to the image style of the projected image, or converts the projected image to the image style of the X-ray image, before performing image matching, so that the image styles of the two are the same. This can improve the accuracy of image matching, and thus improve the accuracy of the calculated horizontal and vertical parameters. Then, the calculated horizontal and vertical parameters are used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration result to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0283] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0284] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the steps described in the above image registration method.

[0285] When performing image registration using the computer program provided in this embodiment of the invention, the CT image is first projected onto a two-dimensional plane based on initial rotation and translation parameters. However, since CT images and X-ray images have different image styles, even if pixels in the projected CT image and X-ray image correspond to the same positions in a real-world scene, the similarity of the pixels is still not high. Therefore, the solution provided in this embodiment of the invention converts the X-ray image to the image style of the projected image, or vice versa, before performing image matching, so that the two have the same image style. This improves the accuracy of image matching, and consequently improves the accuracy of the calculated horizontal and vertical parameters. The calculated horizontal and vertical parameters are then used to update the initial translation parameters, and the initial rotation parameters and the updated initial translation parameters are used as the image registration result to achieve image registration. Furthermore, since the calculated horizontal and vertical parameters have high accuracy, the obtained image registration result is more accurate.

[0286] In addition, since the computational resources required to project CT images onto a two-dimensional plane are relatively small, the computational resources required to perform image style conversion on the images are also relatively small, and the computational resources required to perform image matching on the images are also relatively small, the computational resources required for image registration using the embodiments of the present invention are relatively small, and the time spent on image registration is relatively short.

[0287] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0288] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0289] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for devices, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0290] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An image registration method, characterized in that, The method includes: The initial rotation and translation parameters between the X-ray and CT images are obtained; these initial rotation and translation parameters are fixed preset parameters; or the initial rotation parameters are obtained by: extracting image features from the X-ray image; comparing the image features of the X-ray image with those of historical X-ray images to determine the target historical X-ray image with the highest feature similarity to the X-ray image; and determining the historical rotation parameters corresponding to the target historical X-ray image as the initial rotation parameters; or the height parameter in the initial translation parameters is obtained by: identifying a first object region in the X-ray image and a second object region in the CT image; calculating the size ratio between the first and second object regions based on the size of the first and second object regions; and calculating the height parameter in the initial translation parameters based on the size ratio and the focal length of the device acquiring the X-ray image. Based on the initial rotation parameters and initial translation parameters, the CT image is projected onto a two-dimensional plane to obtain a projected image; The first image is converted from the first image style to the second image style of the second image to obtain the third image, wherein the first image is the X-ray image or the projection image, and the second image is: the X-ray image and the projection image other than the first image; Determine the target image region in the fourth image that matches the fifth image, wherein the fourth image is the largest image among the third image and the second image, and the fifth image is the image among the third image and the second image excluding the fourth image; Based on the position of the target image region in the fourth image, the horizontal and vertical parameters in the initial translation parameters are updated to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

2. The method according to claim 1, characterized in that, The process of converting the first image from a first image style to a second image style to obtain a third image includes: The first image is input into a pre-trained image style transfer model to perform image style transfer on the first image, resulting in a third image with the image style of the second image. The image style transfer model is a first transfer model with adjusted network parameters obtained by training the model using a preset training method. The preset training method is a training method that performs generative adversarial training on the first transfer model, the second transfer model, the first discriminant model, and the second discriminant model together. The first conversion model is a neural network model for converting the image style of an image to a second image style. The second conversion model is a neural network model for converting the style of an image to a first image style. The first discrimination model is a neural network model for determining whether the image style of an image is the first image style. The second discrimination model is a neural network model for determining whether the image style of an image is the second image style.

3. The method according to claim 2, characterized in that, The image style transfer model was trained using the following method: The first sample image is input into the first discrimination model to determine whether the image style of the first sample image is the first image style, and the first discrimination result is obtained, wherein the image style of the first sample image is: the first image style; The second sample image is input into the second transformation model, and image style transformation is performed on the second sample image to obtain the first transformed image; The first transformed image is input into the first discrimination model, and the image style of the first transformed image is determined to be the first image style, thereby obtaining the second discrimination result; Based on the first discrimination result and the second discrimination result, a first loss is calculated, wherein the first loss represents the loss of the first discrimination model in discriminating image style and the loss of the second transformation model in performing image style transformation; The second sample image is input into the second discrimination model to determine whether the image style of the second sample image is the second image style, and a third discrimination result is obtained, wherein the image style of the second sample image is: the second image style; The first sample image is input into the first conversion model, and image style conversion is performed on the first sample image to obtain the second converted image; The second transformed image is input into the second discrimination model to determine whether the image style of the second transformed image is the second image style, and a fourth discrimination result is obtained. Based on the third and fourth discrimination results, a second loss is calculated, wherein the second loss represents the loss of the second discrimination model in discriminating image style and the loss of the first transformation model in performing image style transformation; The first converted image is input into the first conversion model to obtain the third converted image; The second converted image is input into the second conversion model to obtain the fourth converted image; A third loss is calculated based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image, wherein the third loss represents the loss of the first transformation model in performing image style transformation and the loss of the second transformation model in performing image style transformation. Calculate the total loss based on the first loss, the second loss, and the third loss; The parameters of the first conversion model, the second conversion model, the first discriminant model, and the second discriminant model are adjusted according to the total loss. If the convergence condition is not met, the image with the new image style of the first image style is taken as the first sample image, and the image with the new image style of the second image style is taken as the second sample image. The process of inputting the first sample image into the first discriminant model, judging whether the image style of the first sample image is the first image style, and obtaining the first discriminant result is repeated until the preset model convergence condition is met. The first conversion model after parameter adjustment is determined as the image style conversion model.

4. An image registration device, characterized in that, The device includes: A parameter acquisition module is used to obtain initial rotation parameters and initial translation parameters between an X-ray image and a CT image; the initial rotation parameters and initial translation parameters are fixed preset parameters; or, the initial rotation parameters are obtained by: extracting image features from the X-ray image; comparing the image features of the X-ray image with the image features of historical X-ray images to determine the target historical X-ray image with the highest feature similarity to the X-ray image; determining the historical rotation parameters corresponding to the target historical X-ray image as the initial rotation parameters; or, the height parameter in the initial translation parameters is obtained by: identifying a first object region in the X-ray image and a second object region in the CT image; calculating the size ratio between the first object region and the second object region based on the size of the first object region and the size of the second object region; calculating the height parameter in the initial translation parameters based on the size ratio and the focal length of the device acquiring the X-ray image; The image projection module is used to project the CT image onto a two-dimensional plane based on the initial rotation parameters and initial translation parameters to obtain a projected image; A style conversion module is used to convert a first image from a first image style to a second image style to obtain a third image, wherein the first image is the X-ray image or the projection image, and the second image is: the X-ray image or the projection image other than the first image; An image matching module is used to determine a target image region in the fourth image that matches the fifth image, wherein the fourth image is the largest image among the third image and the second image, and the fifth image is the image in the third image and the second image other than the fourth image; The parameter update module is used to update the horizontal and vertical parameters in the initial translation parameters according to the position of the target image region in the fourth image, so as to obtain an image registration result that includes the initial rotation parameters and the updated initial translation parameters.

5. The apparatus according to claim 4, characterized in that, The style conversion module is specifically used for: The first image is input into a pre-trained image style transfer model to perform image style transfer on the first image, resulting in a third image with the image style of the second image. The image style transfer model is a first transfer model with adjusted network parameters obtained by training the model using a preset training method. The preset training method is a training method that performs generative adversarial training on the first transfer model, the second transfer model, the first discriminant model, and the second discriminant model together. The first conversion model is a neural network model for converting the image style of an image to a second image style. The second conversion model is a neural network model for converting the style of an image to a first image style. The first discrimination model is a neural network model for determining whether the image style of an image is the first image style. The second discrimination model is a neural network model for determining whether the image style of an image is the second image style.

6. The apparatus according to claim 5, characterized in that, The image style transfer model is obtained by training the model training module; The model training module includes: The first discrimination submodule is used to input the first sample image into the first discrimination model, determine whether the image style of the first sample image is the first image style, and obtain the first discrimination result, wherein the image style of the first sample image is: the first image style; The first conversion submodule is used to input the second sample image into the second conversion model, perform image style conversion on the second sample image, and obtain the first converted image; The second discrimination submodule is used to input the first transformed image into the first discrimination model, determine whether the image style of the first transformed image is the first image style, and obtain the second discrimination result. The first loss calculation submodule is used to calculate a first loss based on the first discrimination result and the second discrimination result, wherein the first loss represents the loss of the first discrimination model in discriminating image style and the loss of the second conversion model in performing image style conversion; The third discrimination submodule is used to input the second sample image into the second discrimination model, determine whether the image style of the second sample image is the second image style, and obtain the third discrimination result, wherein the image style of the second sample image is: the second image style; The second conversion submodule is used to input the first sample image into the first conversion model, perform image style conversion on the first sample image, and obtain the second converted image. The fourth discrimination submodule is used to input the second transformed image into the second discrimination model, determine whether the image style of the second transformed image is the second image style, and obtain the fourth discrimination result; The second loss calculation submodule is used to calculate the second loss based on the third and fourth discrimination results, wherein the second loss represents the loss of the second discrimination model in discriminating image style and the loss of the first conversion model in performing image style conversion; The third conversion submodule is used to input the first conversion image into the first conversion model to obtain the third conversion image; The fourth conversion submodule is used to input the second converted image into the second conversion model to obtain the fourth converted image; The third loss calculation submodule is used to calculate the third loss based on the third transformed image and the second sample image, and the fourth transformed image and the first sample image, wherein the third loss represents the loss of the first transformation model performing image style transformation and the second transformation model performing image style transformation. The total loss calculation submodule is used to calculate the total loss based on the first loss, the second loss, and the third loss. The parameter adjustment submodule is used to adjust the parameters of the first conversion model, the second conversion model, the first discriminant model and the second discriminant model according to the total loss. If the convergence condition is not met, the image with the new image style of the first image style is used as the first sample image, and the image with the new image style of the second image style is used as the second sample image, triggering the execution of the first discriminant submodule until the preset model convergence condition is met. The model determination submodule is used to determine the first transformation model after parameter adjustment as the image style transformation model.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-3.

Images