Cone beam X-ray perspective deformation correction system and correction method thereof

Abstract

The invention provides an X-ray perspective deformation correction system, which is used for correcting deformation of a detected object caused by X-cone beam ray perspective projection. The X-ray perspective deformation correction system comprises a projection imaging system, a color superposition module and a learning module. The projection imaging system is a cone beam X-ray imaging system including a digital X-ray photography (DR) system and a cone beam computed tomography (CBCT) system. The projection imaging system carries out perspective projection on the detected object from two complementary visual angles so as to obtain a first view and a second view respectively. The first view and the second view are processed by the color superposing module to form a superposing color image. And the superposed color image is input into the learning module and is processed by the learning module to obtain a corrected image.

Description

Technical Field

[0001] This invention relates to a fluoroscopic distortion correction system and method, specifically to a cone-beam X-ray fluoroscopic distortion correction system and method. Background Technology

[0002] X-ray imaging and flat-panel detectors are widely used in disease diagnosis, interventional therapy, and tumor radiotherapy. In cone-beam X-ray imaging, due to the divergence of X-rays, structures at different depths are magnified at different magnifications on the X-ray detector. Therefore, the acquired images suffer geometric distortion, known as perspective distortion. In many practical applications, perspective distortion makes it difficult to perform direct and accurate geometric evaluation of the structure of interest (SOI), such as anatomical marker detection, perspective image stitching, reference marker registration, and bimodal image fusion. Therefore, in many X-ray applications, orthographic projection is more preferred than perspective projection.

[0003] In conventional cone-beam X-ray imaging systems, such as C-arm cone-beam computed tomography (CBCT), fluoroscopy distortion is a challenge in applications such as chest X-ray imaging and cephalometric analysis. To reduce fluoroscopy distortion, specialized imaging devices are designed; for example, chest X-ray machines and cephalometric scanners have long source-to-detector distances. Besides these specialized devices, another approach to address cone-beam distortion is to first reconstruct a three-dimensional image and then digitally reconstruct the X-ray image (DRRs) using a parallel-beam projection. For example, in dental CBCT, DRRs using emmetropic projection are used as synthetic lateral views, providing higher accuracy for cephalometric analysis than fluoroscopic projection. However, obtaining such a three-dimensional CBCT image requires acquiring hundreds of projections for 3D reconstruction, inflicting additional X-ray dose on the patient. Another potential application is dual-modal imaging with magnetic resonance imaging (MRI) and X-rays. MRI can easily obtain two-dimensional parallel-beam projections through frequency domain sampling, while X-rays use cone-beam projections, making registration of the two modal images difficult. Generating DRRs from acquired 3D MRI images is extremely time-consuming, taking approximately 30 minutes per scan. More importantly, pre-acquired 3D MRI images cannot provide accurate information about the daily changes in organs. Therefore, rapidly converting cone-beam X-ray images into parallel-beam images, which can be quickly registered with 2D parallel-beam MRI images, for real-time observation of different structures such as bone and soft tissues has significant clinical value.

[0004] Beyond the medical field, cone-beam X-ray systems are also widely used in the non-destructive testing of industrial devices. In industrial inspection, it is crucial not only to identify internal damage and defects but also to accurately measure the geometric dimensions of critical components, such as the alignment of electrodes and the positional accuracy of weld points inside new energy vehicle batteries. Currently, while inspection methods based on single-image cone-beam X-ray projection can initially identify obvious defects, accurate dimensional measurements are impossible due to perspective distortion during projection. While CBCT 3D reconstruction methods can acquire 3D geometric information, the data acquisition and reconstruction process is time-consuming, making it difficult to meet the real-time efficiency requirements of industrial production lines. Summary of the Invention

[0005] One object of the present invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and correction method thereof, so as to provide a rapid, accurate detection method that does not impose additional X-ray dose on the patient.

[0006] Another object of the present invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and the correction method thereof, wherein a two-dimensional fluoroscopy (cone-beam) projection is directly converted into a two-dimensional parallel projection.

[0007] Another objective of this invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and method thereof, in order to solve the difficulties caused by fluoroscopy distortion in the direct and accurate geometric evaluation of anatomical structures during cone-beam X-ray transmission imaging.

[0008] Another object of the present invention is to provide a cone-beam X-ray perspective distortion correction system and method thereof, wherein the perspective distortion correction problem is solved in a framework using two complementary views (projection angles differing by 180°), wherein the complementary view setting provides an intuitive and practical method for identifying perspective distortion structures by evaluating the deviation between the two views.

[0009] Another objective of this invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and its correction method, wherein, within the calibration accuracy range, the training model has a certain tolerance for geometric errors such as the distance from the X-ray source to the isocenter, rotation angle, detector principal point displacement, and breathing motion, so that a model trained from one CBCT imaging system can be applied to other similar systems.

[0010] Another objective of this invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and method thereof, the effectiveness of which on real human CBCT projection data and its robustness in the presence of metal implants and surgical screws demonstrate its potential for practical application.

[0011] Another object of the present invention is to provide a cone-beam X-ray fluoroscopy distortion correction system and method thereof, wherein a simple complementary view setting is proposed to provide boundary information, thereby reducing the uncertainty and receptive field size required to learn fluoroscopy distortion in X-ray transmission imaging.

[0012] Another object of the present invention is to provide a cone-beam X-ray perspective distortion correction system and method thereof, wherein which structures suffer from perspective distortion is identified by evaluating the deviation between two complementary views, which can be easily observed in the RGB color images of the complementary views.

[0013] Another objective of this invention is to provide a cone-beam X-ray fluoroscopic distortion correction system and its correction method, in order to develop a method based on finite-angle (e.g., dual-view) X-ray projection that can effectively correct fluoroscopic distortion and achieve rapid, high-precision geometric dimension measurement, thereby improving the automation level and production efficiency of industrial non-destructive testing.

[0014] According to one aspect of the present invention, the present invention further provides an X-ray fluoroscopy distortion correction system for correcting distortion caused by X-ray cone-beam fluoroscopic projection of an object under test, comprising:

[0015] A projection imaging system, wherein the projection imaging system performs perspective projection on the object to be detected from two complementary perspectives to obtain a first view and a second view, respectively.

[0016] A color overlay module, wherein the first view and the second view are processed by the color overlay module to form an overlay color image; and

[0017] A learning module, wherein the overlay color image is input into the learning module and processed by the learning module to obtain a corrected image.

[0018] According to one embodiment of the present invention, the projection imaging system includes a cone-beam X-ray imaging module and a driving module, wherein the driving module drives the cone-beam X-ray imaging module to rotate around an isocenter, so that the cone-beam X-ray imaging module performs X-ray fluoroscopic projection on the object under test from two complementary viewpoints, wherein the cone-beam X-ray imaging module includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays towards the object under test from two complementary viewpoints, and the detector images two X-ray fluoroscopic views.

[0019] According to one embodiment of the present invention, the detector can be equivalently transformed into a virtual detector to correct the dimensional distortion of the two X-ray fluoroscopic views caused by geometric magnification, wherein the plane in which the virtual detector is located passes through the isocenter.

[0020] According to one embodiment of the present invention, the projection imaging system includes two cone-beam X-ray imaging modules, wherein the two cone-beam X-ray imaging modules are symmetrical about a common center at complementary angles, so that the cone-beam X-ray imaging modules respectively perform X-ray fluoroscopic projection on the object under test from two complementary viewpoints.

[0021] According to one embodiment of the present invention, each of the cone-beam X-ray imaging modules includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits a cone-beam X-ray toward the object being detected and an X-ray perspective view is obtained by imaging the object on the detector, and the detector can be equivalently transformed into a virtual detector, the plane of which the virtual detector is located passing through the isocenter.

[0022] According to one embodiment of the present invention, the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

[0023] According to one embodiment of the present invention, the learning module includes a coordinate transformation module and a neural network, wherein the overlay color image is transformed from rectangular coordinates to polar coordinates by the coordinate transformation module to obtain a polar coordinate transformed overlay color image, wherein the polar coordinate transformed overlay color image is input into the neural network and processed by the neural network to output an output image, wherein the output image is subjected to an inverse polar coordinate transformation from polar coordinates to rectangular coordinates to obtain the corrected image.

[0024] According to one embodiment of the present invention, the neural network is selected from CNN neural networks, transformer neural networks and diffusion generative models.

[0025] According to one embodiment of the present invention, the neural network is selected from Pix2pixGAN, TransU-Net, DiffusionGAN, and Image-to-Image. Bridge (I2SB).

[0026] According to one embodiment of the present invention, the projection imaging system includes a cone-beam X-ray imaging module and a driving module, wherein the driving module drives the cone-beam X-ray imaging module to rotate around an isocenter, so that the cone-beam X-ray imaging module performs X-ray fluoroscopic projection on the object under test from two complementary viewpoints, wherein the cone-beam X-ray imaging module includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays towards the object under test from two complementary viewpoints, and the detector images two X-ray fluoroscopic views.

[0027] According to one embodiment of the present invention, the detector can be equivalently transformed into a virtual detector to correct the dimensional distortion of the two X-ray fluoroscopic views caused by geometric magnification, wherein the plane in which the virtual detector is located passes through the isocenter.

[0028] According to one embodiment of the present invention, the projection imaging system includes two cone-beam X-ray imaging modules, wherein the two cone-beam X-ray imaging modules are symmetrical about a common center at complementary angles, so that the cone-beam X-ray imaging modules respectively perform X-ray fluoroscopic projection on the object under test from two complementary viewpoints.

[0029] According to one embodiment of the present invention, each of the cone-beam X-ray imaging modules includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits a cone-beam X-ray toward the object being detected and an X-ray perspective view is obtained by imaging the object on the detector, and the detector can be equivalently transformed into a virtual detector, the plane of which the virtual detector is located passing through the isocenter.

[0030] According to one embodiment of the present invention, the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

[0031] According to one embodiment of the present invention, the X-ray fluoroscopic distortion correction system further includes a transport system for sequentially transporting a plurality of objects to be inspected to a detection position, so that the two cone-beam X-ray imaging modules can perform fluoroscopic projection of the objects from two complementary perspectives.

[0032] According to one embodiment of the present invention, the complementary viewing angle range is 175°-185°.

[0033] According to another aspect of the present invention, the present invention further provides a method for correcting X-ray fluoroscopy distortion, comprising the following steps:

[0034] A) Cone-beam X-ray fluoroscopy projection is performed on the same object to be inspected from two complementary angles to form two complementary views, wherein the two complementary views include a first view and a second view;

[0035] B) Reset the two complementary views to a virtual detector to correct the size of the two complementary views;

[0036] C) Convert the two complementary views into color images and overlay them to generate an overlay color image;

[0037] D) The overlay color image is transformed by polar coordinate transformation to obtain a polar coordinate transformed overlay color image;

[0038] E) The polar coordinate transformation overlay color image is input as an input image into a neural network;

[0039] F) The neural network processes the polar coordinate transformation overlaid color image and outputs an output image; and

[0040] G) Inverse polarity transformation of the output image to obtain a corrected image.

[0041] According to an embodiment of the present invention, step A includes the following steps:

[0042] A1) A cone-beam X-ray imaging module performs cone-beam X-ray fluoroscopic projection on the object to be inspected at a first position to obtain the first view;

[0043] A2) Rotate the cone-beam X-ray imaging module 180° around a center to move the cone-beam X-ray imaging module to a second position;

[0044] A3) The cone-beam X-ray imaging module performs cone-beam X-ray fluoroscopic projection on the object to be inspected at the second position to obtain the second view.

[0045] According to one embodiment of the present invention, in step A, the two cone-beam X-ray imaging modules are symmetrical about a common center at complementary angles.

[0046] According to an embodiment of the present invention, the X-ray fluoroscopy distortion correction method further includes the following steps:

[0047] x) Transport an object to be tested to a testing location.

[0048] According to one embodiment of the present invention, the plane in which the virtual detector is located passes through the isocenter.

[0049] According to an embodiment of the present invention, in step C, the first view and the second view are processed by a color overlay module, wherein the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

[0050] According to one embodiment of the present invention, the overlay color image in step D is transformed from rectangular coordinates to polar coordinates to obtain the polar coordinate transformed overlay color image.

[0051] According to one embodiment of the present invention, the output image in step G is subjected to an inverse polar coordinate transformation from polar coordinates to rectangular coordinates to obtain the corrected image.

[0052] According to one embodiment of the present invention, step D involves selecting the neural network from CNN neural networks, transformer neural networks, and diffusion generative models.

[0053] According to one embodiment of the present invention, the neural network is selected from Pix2pixGAN, TransU-Net, DiffusionGAN, and Image-to-Image. Bridge (I2SB). Attached Figure Description

[0054] Figure 1 This is a flowchart of a cone-beam X-ray fluoroscopy distortion correction method according to a first preferred embodiment of the present invention.

[0055] Figure 2A This is a block diagram of a cone-beam X-ray fluoroscopic distortion correction system according to the first preferred embodiment of the present invention.

[0056] Figure 2B The projection imaging system of the cone-beam X-ray fluoroscopic distortion correction system according to the first preferred embodiment of the present invention is explained.

[0057] Figure 3A A CBCT system is described, which is applied to the cone-beam X-ray fluoroscopy distortion correction system of the first preferred embodiment described above.

[0058] Figure 3B The perspective distortion of a single view in the above CBCT system is explained.

[0059] Figure 3C The perspective distortion of the two orthogonal views under the above CBCT system is explained.

[0060] Figure 3D The perspective distortion of the two complementary views under the above CBCT system is explained.

[0061] Figure 4A and Figure 4B It illustrates the perspective distortion from a Cartesian coordinate system to polar coordinates, where the direction and length of the arrows reflect the direction and magnitude of the perspective distortion at the corresponding positions.

[0062] Figure 5A This is a 0° perspective projection image of a small spherical phantom.

[0063] Figure 5B This is a perspective projection image of the small spherical phantom at a 90° angle.

[0064] Figure 5C This is an RGB superposition of perspective projection images of the small spherical phantom at 0° and 90° viewpoints.

[0065] Figure 6A This is a 0° perspective projection image of the small spherical phantom.

[0066] Figure 6B It is the view difference between the 0° perspective projection image of the small spherical phantom and a reference image.

[0067] Figure 6C This is an image of the reference figure, illustrating a 0° orthographic projection.

[0068] Figure 6D This is a 90° perspective projection image of the small spherical phantom.

[0069] Figure 6E Explained Figure 6A and Figure 6D View difference.

[0070] Figure 6F The RGB superposition of perspective projection images of the small spherical phantom at 0° and 90° viewpoints is explained.

[0071] Figure 6G This is a 5° perspective projection image of the small spherical phantom.

[0072] Figure 6H Explained Figure 6A and Figure 6G View difference.

[0073] Figure 6I The RGB superposition of the perspective projection images of the small spherical phantom at 0° and 5° viewpoints is explained.

[0074] Figure 6J This is a 180° perspective projection image of the small spherical phantom.

[0075] Figure 6K Explained Figure 6A and Figure 6J View difference.

[0076] Figure 6L The RGB superposition of the perspective projection images of the small spherical phantom at 0° and 180° viewpoints is explained.

[0077] Figure 7A This is a 0° orthographic projection view of another small spherical phantom.

[0078] Figure 7B This is a cone-beam perspective projection view of the small spherical phantom at a 0° angle.

[0079] Figure 7C yes Figure 7B and Figure 7A View difference.

[0080] Figure 7D This is a 180° perspective projection image of the small spherical phantom.

[0081] Figure 7E It is the difference between the 0° and 180° views of the small spherical phantom.

[0082] Figure 7F This is an RGB superposition of perspective projection images of the small spherical phantom at 0° and 180° viewpoints.

[0083] Figure 8A The point-to-point distance distribution when Dsi = 600 mm is explained.

[0084] Figure 8B The distribution of ratio α is explained in complementary view settings of Dsi = 600 mm and Dsi = 750 mm.

[0085] Figure 9A It is a perspective projection of a reference image in a rectangular coordinate system.

[0086] Figure 9B It is the perspective projection of the reference image in polar coordinates.

[0087] Figure 9C It is the RGB superposition of the 0° and 180° perspective projection images of the reference image in the polar coordinate system.

[0088] Figure 10 and Figure 11 The paper elucidates the quantitative evaluation of small ball phantom data using different methods.

[0089] Figures 12A-12L The Pix2pixGAN predictions in different spaces and views are explained.

[0090] Figures 13A-13O The application of the cone-beam X-ray fluoroscopy distortion correction method and the X-ray fluoroscopy distortion correction system according to the first preferred embodiment of the present invention in human head imaging correction is explained.

[0091] Figures 14A-14C The robustness of Pix2pixGAN to geometric inaccuracies when using 0° and 180° polarity inputs on small spherical phantoms is demonstrated.

[0092] Figures 15A-15C Three DRR perspective projection images used for training are shown.

[0093] Figure 16A It is a 0° perspective projection image of the first knee joint.

[0094] Figure 16BIt is a 0° perspective projection image of a second knee joint, in which two metal implants are located.

[0095] Figure 16C It is a 0° perspective projection image of a third knee joint, which has multiple screws.

[0096] Figure 16D This is a 180° perspective projection image of the first knee joint.

[0097] Figure 16E This is a 180° perspective projection image of the second knee joint.

[0098] Figure 16F This is a 180° perspective projection image of the third knee joint.

[0099] Figure 16G These are perspective projection difference images of the first knee joint from 0° and 180° perspectives.

[0100] Figure 16H These are perspective projection difference images of the second knee joint from 0° and 180° perspectives.

[0101] Figure 16I These are perspective projection difference images of the third knee joint from 0° and 180° perspectives.

[0102] Figure 16J It is a color overlay of perspective projections from the 0° and 180° views of the first knee joint.

[0103] Figure 16K It is a color overlay of perspective projections from the 0° and 180° views of the second knee joint.

[0104] Figure 16L It is a color overlay of perspective projections from the 0° and 180° views of the third knee joint.

[0105] Figure 16M This is a reference image of the first knee joint.

[0106] Figure 16N This is a reference image of the second knee joint.

[0107] Figure 16O This is a reference image of the third knee joint.

[0108] Figure 16P This is a Pix2pixGAN-corrected image of the first knee joint.

[0109] Figure 16Q This is a Pix2pixGAN-corrected image of the second knee joint.

[0110] Figure 16R This is a Pix2pixGAN-corrected image of the third knee joint.

[0111] Figure 17A yes Figure 16A A magnified view of a portion of the image.

[0112] Figure 17B yes Figure 16M A magnified view of a portion of the image.

[0113] Figure 17C yes Figure 16P A magnified view of a portion of the image.

[0114] Figure 18 This is a flowchart of a cone-beam X-ray fluoroscopy distortion correction method according to a second preferred embodiment of the present invention.

[0115] Figure 19 This is a block diagram of a cone-beam X-ray fluoroscopic distortion correction system according to the second preferred embodiment of the present invention.

[0116] Figure 20 A DR system is described in connection with the cone-beam X-ray fluoroscopy distortion correction system according to the second preferred embodiment of the present invention. Detailed Implementation

[0117] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0118] Those skilled in the art should understand that, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention.

[0119] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0120] The instruction manual includes the attached diagram. Figure 1A cone-beam X-ray fluoroscopy distortion correction method according to a first preferred embodiment of the present invention is explained. The X-ray fluoroscopy distortion correction method includes the following steps:

[0121] A) Cone-beam X-ray fluoroscopy projection of the same object under test from two complementary angles to form two complementary views;

[0122] B) Reset the two complementary views to a virtual detector to correct the size of the two complementary views;

[0123] C) Convert the two complementary views into color images and overlay them to generate an overlay color image;

[0124] D) The overlay color image is transformed by polar coordinate transformation to obtain a polar coordinate transformed overlay color image;

[0125] E) The polar coordinate transformation overlay color image is input as an input image into a neural network;

[0126] F) The neural network processes the polar coordinate transformation overlaid color image and outputs an output image; and

[0127] G) Inverse polar coordinate transformation of the output image to obtain a corrected image.

[0128] Step A involves projecting X-ray cones at two complementary angles onto the object under test using perspective projection, resulting in two complementary views: a first view and a second view. To more clearly illustrate the relationship between the first and second views, they are respectively described as a 0° view and a 180° view. (Reference) Figure 2B Step A involves rotating the same cone-beam X-ray imaging module 11 by 180° and then projecting a perspective image onto the object to be inspected to obtain two complementary views: the 0° view and the 180° view. It is worth noting that the angles in the 0° and 180° views describe the relative angular positional relationship of the X-ray cone-beam sources in the two views, without any absolute angle limitation.

[0129] In an alternative embodiment of the first preferred embodiment of the present invention, the two cone-beam X-ray imaging modules 11 may be arranged at 180°, so that the two cone-beam X-ray imaging modules 11 emit X-ray cones with a projection angle of 180° to each other, thereby forming two complementary views. In this alternative embodiment, the two complementary views can be formed simultaneously (e.g., dual-source dual-detector), thereby achieving higher imaging efficiency.

[0130] It is worth mentioning that, in this first preferred embodiment of the present invention, the complementary angle of 180° allows for some error. Optimal correction is achieved when the complementary angle is 180°, and a good correction effect can still be achieved even with a 5° error. In other words, the two complementary angles in step A can be selected from 175° to 185°. Optimally, the two complementary angles are 180°.

[0131] In step C, RGB overlay is used to create a color image of the view, resulting in stronger visualization that is easily recognized by the human eye and neural networks, thus facilitating further correction. The two complementary views, colored in step C, are identified by different colors, allowing them to be distinguished and recognized even after overlay. It is worth noting that overlaying into an RGB image is merely one example of the invention to enhance human visual perception. According to other embodiments of the invention, the two views can also be overlaid into a two-channel image, or other three-channel, four-channel, or multi-channel images.

[0132] Step D performs a polar coordinate transformation on the overlay color image obtained in step C. According to this first preferred embodiment of the present invention, step D performs a polar coordinate transformation on the overlay color image obtained in step C from a rectangular coordinate system to a polar coordinate system.

[0133] The instruction manual includes the attached diagram. Figure 4A and Figure 4B This illustrates perspective distortion from Cartesian coordinates to polar coordinates, where the direction and length of the arrows reflect the direction and magnitude of the perspective distortion at the corresponding positions. The depth is fixed (m>1), and the red 3×3 grid represents a 3×3 convolution kernel. Figure 4A and Figure 4B As can be seen, polar coordinate transformation can convert radial perspective deformation into translational deformation, which makes it easier for convolutional neural networks to learn.

[0134] The polar coordinate transformed overlay color image obtained after the polar coordinate transformation in step D is input as an input image to a neural network in step E. As an example, according to this first preferred embodiment of the invention, the neural network is selected from Pix2pixGAN and TransU-Net.

[0135] The polar coordinate transformation overlay color image is input into the neural network and processed by the neural network to generate the output image.

[0136] Step G further performs an inverse polar coordinate transformation on the output image, that is, a transformation from polar coordinates to rectangular coordinates, to obtain the corrected image.

[0137] It is worth noting that the polar coordinate transformation in step D and the corresponding inverse polar coordinate transformation in step G are not mandatory steps. It is also worth mentioning that adding step D has the following advantages: First, the addition of the polar coordinate transformation step improves the correction effect; secondly, after the polar coordinate transformation, registration between two-dimensional cone-beam X-ray images and two-dimensional parallel-beam MRI images can be achieved through the conversion between fluoroscopy and emphyseal projection, enabling rapid two-dimensional MRI / X-ray hybrid imaging in a single examination without requiring patient repositioning, and its application in interventional procedures and radiotherapy.

[0138] This cone-beam X-ray fluoroscopy distortion correction method also has the potential to accurately measure the cardiothoracic ratio and head imaging.

[0139] The instruction manual includes the attached diagram. Figure 2A A cone-beam X-ray fluoroscopy distortion correction system according to the first preferred embodiment of the present invention is described. The cone-beam X-ray fluoroscopy distortion correction system includes a projection imaging system 10, a color overlay module 20, and a learning module 30, wherein the learning module 30 includes a coordinate transformation module 31 and a neural network 32. According to the first preferred embodiment of the present invention, the projection imaging system 10 includes a cone-beam X-ray imaging module 11, a virtual detector 12, and a driving module 13. The driving module 13 is used to drive the cone-beam X-ray imaging module 11 to rotate about a preset axis, thereby allowing the cone-beam X-ray imaging module 11 to be irradiated with cone-beam X-rays at two different positions at a preset angle, such as 180°, forming two X-ray fluoroscopic views. Specifically, the cone-beam X-ray imaging module 11 includes a cone-beam X-ray source 111 and a detector 112, wherein the cone-beam X-ray source 111 emits cone-beam X-rays towards the object to be detected and images are formed on the detector 112, wherein the detector 112 is a flat panel detector. The virtual detector 12 corrects the dimensional distortion of the two X-ray perspective views caused by geometric magnification. The two X-ray perspective views corrected by the virtual detector 12 are then colored and overlaid by the color overlay module 20 to generate the overlaid color image.

[0140] According to this first preferred embodiment of the invention, the color overlay module 20 colors the X-ray perspective view through RGB three channels.

[0141] It is worth mentioning that, according to other embodiments of the present invention, X-ray perspective views can also be colored using dual-channel or four-channel methods (e.g., RGBA four-channel).

[0142] The overlaid color image is transformed from rectangular coordinates to polar coordinates by the coordinate transformation module 31 to obtain the polar coordinate transformed overlaid color image. The polar coordinate transformed overlaid color image is input into the neural network 32, and after being corrected by the neural network 32, the output image is output. The output image is then transformed from polar coordinates to rectangular coordinates by the coordinate transformation module 31 to obtain the corrected image.

[0143] According to this first preferred embodiment of the invention, the projection imaging system 10 performs X-ray projection imaging of the object under test via a CBCT system, wherein the virtual detector 12 is isocentrically positioned within the CBCT system, and the isocentric point is the intersection of the rotation plane of the cone-beam X-ray source 111 and the rotation center axis. In other words, the virtual detector 12 is perpendicular to the line connecting the cone-beam X-ray source 111 before and after rotating 180°, and passes through the midpoint of the line connecting these two positions.

[0144] Refer to the attached diagram in the instruction manual. Figure 3A In a CBCT system, the distance D from source S (i.e., the cone-beam X-ray source 111) to detector D (i.e., the detector 112) is... sd The distance D between the source S and the isocenter I si The amplification factor caused by the ratio can be eliminated by virtually moving the detector to the isocenter I. (Reference) Figures 3A-3D The virtual detector 12 is placed at the isocenter I of the projection imaging system 10. Upon re-entry into the virtual detector, the structure of the midsagittal plane (where the virtual detector 12 is located) is not magnified. However, the structures of other sagittal planes are magnified or reduced according to their depth, i.e., they suffer perspective distortion. For a point a = (x, y, z, 1) in the homogeneous coordinate system, the perspective projection and orthographic projection of its corresponding point in perspective are respectively represented by... and express,

[0145] in,

[0146] in, and These are the perspective projection matrix and the orthographic projection matrix in the 0° view, respectively. Perspective distortion (PD) is calculated as follows:

[0147]

[0148] When the CBCT system has such Figure 3A When the orientation and position are shown,

[0149]

[0150] In the formula, s is the pixel size of the virtual detector. Given the principal point O on the detector...det =[P u ,P v ,1] T It can be clearly seen that and O det Collinearity indicates that perspective distortion occurs radially. The amount of perspective distortion is...

[0151]

[0152] in, ρ0 is the magnification factor. To the main point O det The distance. This shows that, with a fixed depth z, the farther a point is from the principal ray, the greater its perspective distortion. This positional dependence is achieved through... Figure 4A The different lengths and directions of the middle arrows reflect this.

[0153] To more clearly describe the invention and its advantages, the invention is illustrated by a single-view and three dual-view designs, and the advantages of complementary views in learning perspective distortion correction are demonstrated through numerical uncertainty analysis.

[0154] Refer to the attached diagram in the instruction manual. Figure 3B The perspective distortion of a single view in the aforementioned CBCT system is analyzed, where the red dot represents the perspective projection of the black dot at a 0° viewing angle, and the blue dot represents the orthographic projection of the black dot. The perspective projection matrix is ​​underdetermined. Therefore, the same perspective projection point... Corresponding to non-unique points in the imaged object, these points lie on its back projection line, i.e. in Projection matrix The pseudo-inverse. For example, in Figure 3B In this context, without depth information, a black dot can be located anywhere along the red solid ray. Therefore, all points on the red dashed line, including the blue dots, are candidate points for the orthographic projection of the red dot. Here, the red dashed line is the orthographic projection of l onto the virtual detector (OPBP). Therefore, learning perspective distortion from a single view is very challenging.

[0155] Refer to the attached diagram in the instruction manual. Figure 3C The analysis focuses on the dual-vision projection images (dual orthogonal views) of the aforementioned CBCT system, spaced 90° apart. The red dot represents the perspective projection of the black dot at 0°, the blue dot represents the orthographic projection of the black dot, and the green dot represents the perspective projection of the black dot at 90°. In practice, dual-plane X-ray systems are widely used for depth estimation in interventional surgeries. In dual-detector CBCT systems, an additional view is used, such as... Figure 3C As shown. Given that it is rotated 90° along the y-axis, the perspective projection of a is,

[0156]

[0157] Therefore, under the same detector pixel coordinates, the point-to-point distance between two orthogonal views can be calculated as follows:

[0158] Ideally, two views of the same point of interest can determine its location in three-dimensional space, that is, the intersection of two corresponding light rays. For example... Figure 3C The black dot in the image represents the intersection of the red and green solid lines. This ideal situation is based on the assumption that the point-to-point correspondence between the two views is definite. However, for different points a and d in the imaged object... 90° There is a wide range of variation because it depends not only on the depth z in the 0° view, but also on the depth x in the 90° view. Therefore, and The correspondence is difficult to determine. As an example, Figures 5A-5C The image shows perspective projections of a small spherical phantom from two images that are 90° apart.

[0159] To better compare the perspective projection images of two orthogonal views, such as Figure 5C As shown, an RGB image is formed, where the red and blue channels use an image from a 0° viewing angle, while the green channel uses an image from a 90° viewing angle. The three channels are filled so that pixels with similar intensity from both views appear gray, such as the cylindrical area in the background. In the formed RGB color image, the magenta phantom image from the 0° view and the green phantom image from the 90° view are located in different positions. (Reference) Figure 5C Due to the large number of small spherical phantoms, the point-to-point correspondence between the two views is not direct.

[0160] Let's analyze small-angle double views using 5° as an example. As an alternative to double orthogonal views, in addition to the 0° view, the second view can have a small angular interval (e.g., 5°). If we define the perspective projection of a when the detector rotates by an angle θ as... The distance between two perspective projection points of point a can be defined as When the angular interval is small, such as 5°, and Because the distances are similar, point-to-point correspondences are easier to determine than in biorthogonal views. However, the main difference between the two perspectives lies in the direction of rotation, not radially.

[0161] Figure 3DThe perspective distortion of two 180° apart dual perspective projection images (complementary views) of the above CBCT system is explained. The red dot is the perspective projection of the black dot at the 0° viewing angle, the blue dot is the orthographic projection of the black dot, and the green dot is the perspective projection of the black dot at the 180° viewing angle.

[0162] This invention proposes a complementary view (180°) for learning view distortion. Compared to emphyseal projection DRRs reconstructed by 3D CBCT, complementary view scans provide patients with lower-dose exposure, and since the hundreds of views required for 3D reconstruction are reduced to two, frequent device rotations are no longer necessary. Furthermore, because the virtual detector pixel size of the projected image is typically smaller than the voxel spacing of the 3D reconstruction volume, and the projected image is not affected by the partial volumetric effects of the 3D reconstruction volume, the emphyseal projection images obtained directly from these complementary views have a higher image resolution than DRRs.

[0163] Given a rotation of 180° along the y-axis, followed by a horizontal flip relative to the principal point, the perspective projection of 'a' is:

[0164]

[0165] With formula (3) In comparison, only the sign of z is changed. If defined... This is the magnification factor under a 180° view. The point-to-point distance between two complementary views can be calculated as follows:

[0166]

[0167] In cone-beam X-ray imaging, complementary views provide additional useful information about the location of the target point due to the cone angle. More specifically, under constraints... In this case, one of m and m' is greater than or equal to 1, and the other is less than or equal to 1. Because of this, lie in and Between. Therefore, complementary views directly provide the range in which the orthographic projection of the target point should lie. For example, in Figure 3D In the diagram, the blue dot lies between the red and green dots. We represent this with the ratio α. exist and The relative positions between them

[0168] α= d PD / d 180° (9)

[0169] It is worth mentioning that the condition is met only if z = 0, i.e., m' = m = 1, d PD and d 180°All equal to 0. This provides a basis for the human eye to directly determine which structures have suffered perspective distortion by observing the deviation of the structure of interest between two views. (Reference) Figure 6F and Figure 6L As shown.

[0170] Figures 7A-7F The parallel beam projection image and the cone beam perspective projection image of the same spherical phantom are displayed in the complementary view setting. Figure 7C This explains the cone-beam perspective projection distortion, specifically the cone-beam perspective projection of the 0° view. Figure 7B ) and parallel beam projection ( Figure 7A The difference is ). Figure 7C It can be seen that the magnitude of perspective distortion increases radially (outwards) from the center. A 180° cone-beam perspective projection image is shown below. Figure 7D As shown, the difference between the projection and the 0° cone-beam perspective projection is as follows: Figure 7E As shown. Figure 7E and Figure 7C In the small spherical phantom region, similarities are observed, and the deviation between the complementary views of the cone-beam projection is strongly correlated with cone-beam perspective distortion. To integrate this dual-view information, the perspective projection images are converted from 0° and 180° views to... Figure 7F The image is a 3-channel RGB image. The red and blue channels use the image from the 0° view, while the green channel uses the image from the 180° view. In the RGB image, the colors reveal the intensity differences between the 0° and 180° perspective projection images. The gray areas contain similar intensity values ​​from both views. Conversely, the magenta and green areas represent larger intensity values ​​obtained from the 0° and 180° views, respectively. They correspond to... Figure 7E The positive (bright) and negative (dark) areas in the difference image. It is evident that perspective distortion correction is necessary. Figure 7F In the image, the magenta phantom images and their corresponding green phantom images are close to each other, which allows a network with a limited receptive field to capture point-to-point dependencies.

[0171] Figure 8A and Figure 8B The numerical uncertainty analysis under different view settings is explained. In D si In a CBCT system with a diameter of 600 mm, the point-to-point distance d for an exemplary cylindrical object (diameter 320 mm, height 320 mm) is... PD d 5° d 90° and d 180° The distribution of Figure 8A Numerical analysis was performed. From... Figure 8A It can be seen that perspective distortion d PDPrimarily distributed within the range of [0, 55] mm, this is the minimum receptive field size required for a neural network to learn perspective distortion. For biorthogonal views, d 90° The distribution over a large range of [0, 280] mm necessitates a larger receptive field for the neural network and introduces significant uncertainty into its learning of perspective distortion. For a 5° view, d 5° The distribution is mainly within a narrow range of [0, 25] mm, which is too narrow to reflect different degrees of perspective distortion. In contrast, d 180° Mainly distributed in the range of [0, 9 0] mm, in d PD Within twice the distance. The distribution of A is as follows Figure 8B As shown. When D si When = 600 mm, α is distributed in a narrow range of [0.364, 0.636], where D si Let D be the distance from the ray source to the isocenter. si When the value is increased to 750 mm, the distribution range of α becomes narrower [0.395, 0.605]. The distribution of α indicates that along the radial line, Located near and The location near the midpoint significantly reduces uncertainty. Due to the relatively short distance dependence and lower uncertainty, learning perspective deformation using two complementary views is more effective for neural networks than learning using two orthogonal views.

[0172] Various state-of-the-art and emerging generative neural networks can be applied to the aforementioned perspective distortion correction system to learn perspective distortion. As mentioned earlier, perspective distortion is position-dependent. This positional dependency can be effectively learned by the position encoding layers in Transformer neural networks such as VisionTransformer (ViT). Convolutional neural networks (CNNs) are efficient at learning translational features, but they generally cannot effectively learn rotational features. Rather than designing new network architectures to overcome this limitation, a simple preprocessing step—polar coordinate transformation—allows CNNs to effectively learn perspective distortion correction. For CNNs, learning perspective distortion in polar space is more advantageous than learning it directly in Cartesian space because radial distortion in Cartesian space is transformed into translational distortion in polar space, such as... Figure 4BAs shown. This invention uses Pix2pixGAN as a representative of CNNs and TransU-Net as a representative of Transformer-based neural networks to describe and verify the cone-beam X-ray perspective distortion correction method according to the first preferred embodiment of the present invention. Pix2pixGAN uses U-Net as its generator and a 5-layer CNN as its discriminator. The training loss function includes an adversarial loss function, an L1 loss function, and a perceptual loss function based on pre-trained VGG-16 weights. TransU-Net is a CNN-Transformer hybrid network built on ViT. Like CNNs, it can maintain the high resolution of generated images, and like Transformers, it can handle long-range dependent features. The encoder of TransU-Net uses a ResNet50+ViT-B / 16 configuration, while the decoder is the right half (expansive path) of a regular U-Net. The training loss functions for TransU-Net are the L1 loss function and the perceptual loss function.

[0173] As a proof of concept, this invention first investigated the aforementioned perspective distortion correction method using simulated phantom image data. To evaluate the generalization ability of the proposed complementary view setup, further simulated projection data experiments were conducted using patient head CT datasets and real CBCT projection data. The real data consisted of 20 human knee specimens scanned by a CBCT system. The reference image for the real data was an orthographic projection image derived from iterative 3D reconstruction of the measured CBCT projection data.

[0174] One of the reference images is shown in both rectangular and polar coordinate systems as follows: Figures 9A-9C As shown, the corresponding Pix2pixGAN prediction graphs for different configurations are presented. Figures 12A-12L As shown, where Figure 12A The result image is obtained by using a single view at 0° in a Cartesian coordinate system as network input. Figure 12B The result image is obtained by overlaying 0° and 90° views in a Cartesian coordinate system as network input. Figure 12C The image is a result obtained by overlaying 0° and 180° views in a Cartesian coordinate system as network input. Figure 12D The result image is obtained by using a single view from a 0° perspective in polar coordinates as network input. Figure 12E The result image is obtained by overlaying the 0° and 90° views in polar coordinates as network input. Figure 12F This is the result image obtained by overlaying 0° and 180° views in polar coordinates as network input. Figure 12G yesFigure 12A The error image (compared with the parallel beam projection image), where Figure 12H yes Figure 12B The error image, in which Figure 12I yes Figure 12C The error image, in which Figure 12J yes Figure 12D The error image, in which Figure 12K yes Figure 12E The error image, in which Figure 12L yes Figure 12F Error image. Figure 9C This demonstrates the appearance of a perspective projection image of complementary views in polar coordinates. The neural network only needs to horizontally move a small spherical phantom to the position between the corresponding magenta and green pairs. Figures 12A-12L In the middle, the prediction results are as follows Figures 12A-12F As shown, its error graph is as follows Figures 12G-12L As shown, the error images are displayed in the [-50, 50] window. Among all the error images, it can be seen that... Figure 12L The error is minimized. To compare the overall image quality, the root mean square error (RMSE) and structural similarity index (SSIM) values ​​of the prediction results under different configurations are as follows: Figure 10 As shown.

[0175] exist Figure 10 In addition to Figures 12A-12L In addition to the results, three more results were added for comparison:

[0176] a) Combining 0° and 90° perspective projection images to form Figure 5C As input to the neural network, Figure 10 In Chinese, "0° & 90°" are used to represent this.

[0177] b) Use Figure 7E The difference map between the 0° and 180° perspective projection images shown is used as the third channel of the RGB image, and the 0° perspective projection is no longer used. Figure 10 In Chinese, "0° & 180° +" is used to represent this.

[0178] c) A direct combination of 0°, 90° and 180° perspective projection images as three channels of RGB input, denoted as “0°, 90° and 180°”.

[0179] The experiment of “0° & 180°” was repeated 10 times to avoid the influence of random weight initialization.

[0180] In terms of image space, Figure 10This indicates that learning perspective distortion in polar coordinates significantly improves the image quality of Pix2pixGAN compared to Cartesian coordinates. For the acquired view, the RMSE value is not significantly improved when combining orthogonal views compared to a single view. Using the difference image as a third channel (a comparison between "0° & 180°" and "0° & 180°+") improves image quality in Cartesian coordinates. However, it has little impact on image quality in polar coordinates. Furthermore, combining three views ("0°, 90° & 180°") does not show a significant improvement compared to "0° & 180°". In conclusion, Figure 10 This demonstrates the advantage of Pix2pixGAN in using two complementary views in polar coordinates to correct perspective distortion.

[0181] The quantitative evaluation results of TransU-Net under different spatial coordinate systems and different views are shown below. Figure 10 Regarding spatial coordinate systems, although learning in polar coordinates performs slightly better, TransU-Net performs comparably in learning perspective distortion in both Cartesian and polar coordinate systems. As with Pix2pixGAN, for TransU-Net, using two complementary views is generally better than using two orthogonal views or a single view.

[0182] Figures 13A-13O The results of human head measurement imaging are shown as an example. In a 0° perspective projection image ( Figure 13B In the image, due to perspective distortion, the anatomical structures on the left and right sides do not overlap well, especially the mandible, as shown in the image. Figure 13B As indicated by the red arrow in the middle. This can lead to inaccurate marking of the mandibular angle point in skull detection. Figure 13B Reference Figure 13A Differences such as Figure 13C As shown, where Figure 13C A 2mm scale is shown, where 2mm is a clinically acceptable accuracy for cephalic marker detection. It is clear that in a 0° fluoroscopic projection image, the positional displacement of many anatomical structures is greater than 2mm. The predicted image using a single 0° view in Cartesian coordinates ( Figure 13D In this context, perspective distortion is reduced, such as... Figure 13G As shown. For example, the error is smaller in the mandibular region. However, Figure 13G This also indicates that many bone structures still have deviations greater than 2 mm. The results of complementary view learning in Cartesian and polar coordinate systems are as follows: Figure 13E and Figure 13F As shown. Both images have very small perspective distortion, as... Figure 13H and Figure 13I As shown. However, in Figure 13E In the image, the two arrows represent two dark areas. Figure 13HThe difference plot can better illustrate this. The results of TransU-Net are as follows: Figures 13J-13O As shown. In Figure 13M In the image, structures near some markers are distorted, such as the ear canal indicated by the arrow. The results show that, using dual complementary views, TransU-Net effectively reduces perspective distortion in both Cartesian and polar coordinate systems.

[0183] Figures 14A-14C The robustness of Pix2pixGAN to geometric inaccuracies is demonstrated when using 0° and 180° polarity inputs on small spherical phantoms. Pix2pixGAN is used here as an example because it maintains good performance on the aforementioned datasets. As geometric inaccuracies such as rotation, source-to-isocenter distance Dsi, and detector principal point position increase, the corresponding RMSE curve remains stable within a certain error range, and then increases steadily rather than increasing abruptly.

[0184] Complementary view settings for correcting angular distortion were evaluated on real CBCT projection data. In this evaluation, real CBCT projection data from a knee joint dataset with implanted metal were used for testing, while DRR from a volumetric CT dataset with implanted metal was used for training.

[0185] Figures 15A-15C Three example DRR perspective projection images used for training are shown, in which synthetic metal implants are implanted. Figures 16A-16R As shown, the appearance of the DRR training image differs from that of the real projected image, including image contrast and metallic image resolution. Figures 16A-16R The image shows the results of three knee joints, with and without metal implants. Figures 16A-16C It is a 0° viewing angle projection, in which Figures 16D-16F It is a 180° perspective projection, which performs geometric calibration based on the projections of its principal point and origin, and regenerates the virtual detector. Figures 16G-16I The images show their differences, where the magnitude of the deviation increases from the center outwards, as is the case in DRRs with ideal scan trajectories (e.g., Figure 7E Although the actual projection data is affected by various physical effects, such as beam hardening and Poisson noise.

[0186] Figures 16J-16L A color overlay module displays 0° and 180° perspective projection images. Magenta and green areas represent structures with a significant degree of perspective distortion, such as... Figure 16J The patella in the knee joint, Figure 16K The tops of the two metals in the middle, and Figure 16L Two screws at the bottom. Figures 16M-16OThe reference image is displayed; it is the orthographic projection of the iteratively reconstructed volume from measured CBCT projection data. Five marked locations are selected in the reference image and are indicated by green dots: Figure 16M Mark two locations on the edge of the patella; Figure 16N and Figure 16O In each image, a location on the left edge of the fibula is marked. Additionally, the rectangular frames of the two metals are indicated by dashed green lines, with their width and height indicated by solid green lines, measuring 29.71 mm and 111.99 mm respectively. Figure 16O In the image, the center lines of the two bottom screws are indicated by green lines. The middle screw is 17.14 mm long, and the bottom screw is 19.38 mm long. In the perspective projection image, the corresponding rectangular frames of the two metal parts and the center lines of the screws are also marked in red. Figure 16B In the figure, the width and height of the metal are 32.56 mm and 109.85 mm, respectively, with deviations from the reference values ​​of 2.84 mm and 1.95 mm. Figure 16C In the figures, the centerline lengths are 18.91 mm and 19.54 mm, with deviations of 1.77 mm and 0.16 mm, respectively. Although the length deviation between the bottom screw and the reference is small, the directional deviation between the two screws is significant.

[0187] Results using Pix2pixGAN with 0° and 180° polarity inputs show that, for all markers, the green reference points are accurately located in the Pix2pixGAN image. The rectangular reference frame also accurately covers... Figure 16Q Metals in [the context]. Figure 16R In the image, although the two red center lines do not completely coincide with the green center line, they are very close in length and direction. The last line shows the results used by TransU-Net with 0° and 180° right-angle inputs, comparable to Pix2pixGAN.

[0188] refer to Figure 17A In a true 0° perspective projection, the porous structures and bone edges appear clearly. The presence of Poisson noise can also be visualized to some extent. (Reference) Figure 17B In the DRR reference image, the pore structure and bone edges appear blurred. This is likely due to partial volumetric effects of the intermediate 3D reconstructed volume. (Reference) Figure 17C In the Pix2pixGAN output images, there is a slight smoothing effect. However, overall, the image resolution of most anatomical structures is preserved, such as pore structures.

[0189] like Figure 7F and Figures 16J-16LAs shown, the RGB overlay of two complementary views provides a practical method for identifying which structures suffer from perspective distortion. Note that this is a sufficient condition, but not a necessary one. Colored structures that deviate between the two complementary views must suffer from perspective distortion and be given sufficient geometric calibration. However, due to symmetry relative to the virtual detector, some structures that appear gray may also suffer from perspective distortion. For example, the gray cylindrical background in the spherical digital phantom (… Figure 7F It is also affected by perspective distortion. Nevertheless, neural networks can still correct these areas, such as... Figures 12A-12L As shown. This demonstrates that neural networks not only utilize the geometric features provided by complementary views but also object / task-specific features to learn perspective distortion. Human anatomy is approximately symmetrical with respect to the midsagittal plane of the body. However, in practical applications, the midsagittal plane typically does not overlap with the virtual probe plane. Therefore, structures with perspective distortion are still represented by color. This geometric information can still be utilized by neural networks, as evidenced by the superior performance of complementary view setups on head data. Figures 13A-13O ).

[0190] Since learning perspective distortion using biorthogonal views in a dual-detector X-ray system does not require rotating the detector, it is more convenient in practice than learning using dual complementary views. Therefore, learning perspective distortion from biorthogonal views remains valuable. However, in this invention, the effect of learning perspective distortion using biorthogonal views is not ideal. This is because neural networks exhibit significant uncertainties when learning point-to-point correspondences, such as… Figure 8A and Figure 8B As shown. When the number of phantoms is small, the human visual system can easily determine this correspondence based on size and intensity information. However, learning feature correspondences from two perspectives, such as point-to-point correspondences, remains a challenging task for neural networks. Currently, many studies are dedicated to this topic. However, it remains an open solution. The exploration of correspondence learning and semantic reasoning in interpretation is based on the inappropriate viewpoint offset learning of Rn from dual orthogonal views. Another possible direction is to seek the intermediate 3D reconstruction from two orthogonal views. However, current algorithms are insufficient to generate orthographic projection images from such 3D volumes. Furthermore, the orthographic projection DRR generated by such reconstruction usually has low resolution, such as... Figure 17BAs shown (where 400 projections are used for reconstruction). Further research on how to improve the quality of such few-view reconstructed images is necessary for our application. For tasks requiring long-range dependencies, CNNs exhibit limited performance due to the inherent locality of convolutional operations. With large-scale pre-training, Transformers are able to learn long-range relationships and transfer them to downstream tasks. In this work, Pix2pixGAN and TransU-Net are investigated as representatives of CNNs and Transformers, respectively. Due to the reduction of point-to-point dependencies by using complementary views ( Figure 8A and Figure 8B Pix2pixGAN can effectively learn perspective distortion in polar coordinates. However, TransU-Net learns perspective distortion better in Cartesian space than Pix2pixGAN due to its ability to learn position-dependent features and long-range dependencies. There is often a gap between simulated X-ray images and real X-ray images. In our work, the apparent differences in image contrast and resolution are as follows: Figures 15A-16R As shown. Depending on the specific application, domain adaptation techniques require that deep learning models trained on synthetic data generalize well to real test data. However, for various applications, it has been reported that deep learning models trained solely on synthetic data are directly effective on real data. In synthetic data ( Figures 15A-15C The Pix2pixGAN and TransU-Net models trained on these platforms have a certain degree of universality for the real human data used in this invention, such as... Figures 16A-16R The results show that the locations of landmarks such as the kneecap and patella can be accurately predicted. Although real projection data is affected by various physical effects such as beam hardening and Poisson noise, the magnitude of the deviation between two complementary views in real data increases from the center outwards, similar to the behavior of DRR with an ideal scan trajectory. This is the basis for our model trained from DRR to generalize well to real data.

[0191] Figures 14A-14C The effects of geometric inaccuracies such as source-to-isocenter distance error, rotation angle error, and detector principal point offset were investigated. Results show that, within the range of geometric calibration and mechanical manufacturing accuracy, the model using complementary view settings exhibits a certain robustness to systematic errors. This indicates that a model trained for one CBCT scanner can potentially be generalized to other CBCT scanners of the same type (with the same system configuration but slightly different configuration parameters). Therefore, a model trained using synthetic data on an ideal circular trajectory can still predict satisfactory results for data obtained in our work on real trajectories. Figures 16A-16R To achieve better generalization across different CBCT system settings, further techniques, such as the use of conditional networks, need to be explored.

[0192] Out-of-distribution (OOD) problems are a major obstacle to the practical application of deep learning algorithms. This is a common problem for all data-driven models and is not specific to our application. Current approaches to mitigate this problem involve training a task-specific model on the largest possible training dataset to cover a variety of inference scenarios. Due to our limited access to real-world patient data, a comprehensive study of this problem is not possible at this stage. However, real-world data experiments can provide some clues about the robustness and generality of our method in the presence of surgical instruments, such as inserted metal implants like screws and K-wires. Real-world data experiments demonstrate that our proposed method is effective in the presence of metal implants and surgical screws, such as… Figures 16A-16R As shown.

[0193] This invention investigates the use of a frame with two complementary views in X-ray transmission imaging to learn perspective distortion. The color overlay of complementary views provides a practical method for identifying which structures suffer from perspective distortion. Numerical simulations demonstrate that complementary views outperform orthogonal or single views in learning perspective distortion. Studies of spatial coordinate systems show that Pix2pixGAN, as a CNN, outperforms Cartesian space in polar space. TransU-Net, a transformer-based network, achieves comparable performance in both Cartesian and polar spaces. This invention further investigates the method's tolerance to geometric errors such as source-to-isocenter distance, rotation angle, detector principal point displacement, and respiratory motion within a calibrated accuracy range. This indicates that a model trained from one CBCT scanner is applicable to other scanners of the same type. Experiments with chest and head data demonstrate the potential of our method to accurately measure cardiothoracic ratios and head imaging. Experiments on real-world data show the method's robustness in the presence of metal implants and surgical screws. In conclusion, this invention provides a method for learning perspective distortion through complementary view settings, enabling more future applications for traditional CBCT systems.

[0194] The instruction manual includes the attached diagram. Figure 18 A cone-beam X-ray fluoroscopy distortion correction method according to a second preferred embodiment of the present invention is explained. This X-ray fluoroscopy distortion correction method includes the following steps:

[0195] a) Cone-beam X-ray fluoroscopy projection of the same object under test from two complementary angles to form two complementary views;

[0196] b) Reset the two complementary views to a virtual detector to correct the size of the two complementary views;

[0197] c) Convert the two complementary views into color images and overlay them to generate an overlay color image;

[0198] d) Perform polar coordinate transformation on the overlay color image to obtain a polar coordinate transformed overlay color image;

[0199] e) The polar coordinate transformation overlay color image is input as an input image into a neural network;

[0200] f) The neural network processes the polar coordinate transformation overlaid color image and outputs an output image; and

[0201] g) Perform an inverse polar coordinate transformation on the output image to obtain a corrected image.

[0202] Step a involves projecting X-ray cone beams at two complementary angles onto the object under test to form two complementary views, namely a first view and a second view. To more clearly illustrate the relationship between the first and second views, they are respectively described as a 0° view and a 180° view. According to this second preferred embodiment of the invention, step a involves projecting X-ray cone beams onto the object under test using two cone beam X-ray imaging modules 11 to obtain two complementary views, namely the 0° view and the 180° view. The two cone beam X-ray imaging modules 11 are positioned at 180°, causing them to emit X-ray cone beams with a projection angle of 180° to each other, thus forming two complementary views. It is worth noting that the angles in the 0° and 180° views describe the relative angular positional relationship of the X-ray cone beam sources in the two views, without any limitation on absolute angles.

[0203] It is worth mentioning that, in this second preferred embodiment of the present invention, the complementary angle of 180° allows for some error. The optimal correction effect is achieved when the complementary angle is 180°, and a relatively good correction effect can still be achieved even with a 5° error. In other words, the two complementary angles in step a can be selected from 175° to 185°. Optimally, the two complementary angles are 180°.

[0204] In step c, RGB overlay is used to create a color image of the view, thereby enhancing visualization and making it easier for the human eye and neural networks to recognize, thus facilitating further correction. The two complementary views, colored in step c, are identified by different colors, allowing them to be distinguished and recognized even after overlay. It is worth noting that overlaying into an RGB image is only one example of the present invention. According to other embodiments of the present invention, the two views can also be overlaid into a two-channel image, or other three-channel, four-channel, or multi-channel images.

[0205] Step d involves performing a polar coordinate transformation on the overlay color image obtained in step c. According to this second preferred embodiment of the invention, step d involves performing a polar coordinate transformation on the overlay color image obtained in step c, converting it from a Cartesian coordinate system to a polar coordinate system.

[0206] The polar coordinate transformed overlay color image obtained after polar coordinate transformation in step d is input as an input image to a neural network in step e. As an example, according to this second preferred embodiment of the invention, the neural network is selected from Pix2pixGAN, TransU-Net, DiffusionGAN, and Image-to-Image. Bridge (I2SB).

[0207] The polar coordinate transformation overlay color image is input into the neural network and processed by the neural network to generate the output image.

[0208] Step g further performs an inverse polar coordinate transformation on the output image, that is, after transforming from polar coordinates to rectangular coordinates, the corrected image is obtained.

[0209] It is worth noting that the polar coordinate transformation in step d and the corresponding inverse polar coordinate transformation in step g are not mandatory steps. It is also worth mentioning that adding step d has the following advantages: the added polar coordinate transformation step results in better correction.

[0210] The instruction manual includes the attached diagram. Figure 19 A cone-beam X-ray fluoroscopy distortion correction system according to the second preferred embodiment of the present invention is described. The cone-beam X-ray fluoroscopy distortion correction system includes a projection imaging system 10A, a color overlay module 20, a learning module 30, and a transport module 40, wherein the learning module 30 includes a coordinate transformation module 31 and a neural network 32.

[0211] The projection imaging system 10A includes two cone-beam X-ray imaging modules 11. The two cone-beam X-ray imaging modules 11 are arranged at complementary angles to irradiate the object to be examined with cone-beam X-rays from two different positions at an angle of 180°, thereby forming two X-ray perspective views.

[0212] Specifically, each cone-beam X-ray imaging module 11 includes a cone-beam X-ray source 111 and a detector 112 corresponding to the cone-beam X-ray source 111, wherein the cone-beam X-ray source 111 emits cone-beam X-rays to the object being detected and images are formed in the detector 112.

[0213] The two detectors 112 can be equivalently transformed into a virtual detector 12, which is perpendicular to the line connecting the two cone-beam X-ray sources 111 and passes through the midpoint of the line connecting the two cone-beam X-ray sources 111. It is worth noting that the midpoint of the line connecting the two cone-beam X-ray sources 111 of the projection imaging system 10A is the isocenter of the projection imaging system 10A. The virtual detector 12 corrects the dimensional distortion of the two X-ray perspective views. The two X-ray perspective views corrected by the virtual detector 12 are colored and overlaid by the color overlay module 20 to generate the overlaid color image. The transport module 40 is used to transport the object to be detected along a direction perpendicular to the X-ray projection angle of the two cone-beam X-ray imaging modules 11, so that when the object to be detected is transported to the detection position, the two cone-beam X-ray imaging modules 11 project an image onto the object. In other words, when the object to be inspected is transported to the midpoint of the connecting line between the two cone-beam X-ray imaging modules 11, it is irradiated with cone-beam X-rays from two different positions at a 180° angle, forming two X-ray perspective views. After being irradiated with cone-beam X-rays, the object is transported away from the inspection position, and the next object to be inspected is transported to that inspection position and irradiated with cone-beam X-rays from complementary angles by the two cone-beam X-ray imaging modules 11, thus being inspected. In this way, the cone-beam X-ray perspective distortion correction system can sequentially perform cone-beam X-ray perspective distortion correction on products on an industrial production line.

[0214] According to this second preferred embodiment of the invention, the color overlay module 20 colors the X-ray perspective view through RGB three channels.

[0215] It is worth mentioning that, according to other embodiments of the present invention, X-ray perspective views can also be colored using dual-channel or four-channel methods (e.g., RGBA four-channel).

[0216] The overlaid color image is transformed from rectangular coordinates to polar coordinates by the coordinate transformation module 31 to obtain the polar coordinate transformed overlaid color image. The polar coordinate transformed overlaid color image is input into the neural network 32, and after being corrected by the neural network 32, the output image is output. The output image is then transformed from polar coordinates to rectangular coordinates by the coordinate transformation module 31 to obtain the corrected image.

[0217] According to this second preferred embodiment of the present invention, the cone-beam X-ray fluoroscopy distortion correction method further includes the following steps:

[0218] x) Transport an object to be tested to a testing location.

[0219] According to this second preferred embodiment of the invention, the projection imaging system 10A performs X-ray projection imaging of the object under inspection using a digital X-ray radiography (DR) system. (See reference...) Figure 20 Two cone-beam X-ray imaging modules 11 use two cone-beam X-ray sources 111 as X-ray source 1 and X-ray source 2, respectively, to project the object being inspected, which has been transported to the detection position, with cone-beam X-ray fluoroscopic projection from complementary angles. Two detectors 112 serve as flat panel detectors 1 and 2, respectively, corresponding to X-ray source 1 and X-ray source 2, to form the two complementary views, namely a 0° view and a 180° view. In step c, the views are overlaid using RGB to form a color image. In step d, the overlaid color image obtained in step c is transformed into polar coordinates. The polar coordinate transformed overlaid color image obtained in step d is input into a neural network as an input image in step e. After being input into and processed by the neural network, the polar coordinate transformed overlaid color image is generated. In step g, the output image is further transformed into polar coordinates inversely, that is, transformed from polar coordinates to rectangular coordinates, to obtain the corrected image.

[0220] The objects to be inspected on the production line pass through the inspection positions sequentially and are successively subjected to X-ray distortion correction using the method described above. This allows the X-ray distortion correction method to be applied in industrial production line operations. For example, this X-ray distortion correction method can be used for the inspection of solar cells, etc.

Claims

1. An X-ray fluoroscopy distortion correction system for correcting distortion caused by X-ray cone-beam fluoroscopic projection of an object under inspection, including: A projection imaging system, wherein the projection imaging system performs perspective projection on the object to be detected from two complementary perspectives to obtain a first view and a second view, respectively. A color overlay module, wherein the first view and the second view are processed by the color overlay module to form an overlay color image; and A learning module, wherein the overlay color image is input into the learning module and processed by the learning module to obtain a corrected image.

2. The X-ray fluoroscopic distortion correction system according to claim 1, wherein the projection imaging system comprises a cone-beam X-ray imaging module and a driving module, wherein the driving module drives the cone-beam X-ray imaging module to rotate around an isocenter, so that the cone-beam X-ray imaging module performs X-ray fluoroscopic projection on the object under test from two complementary viewpoints, wherein the cone-beam X-ray imaging module comprises a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays towards the object under test from two complementary viewpoints, and the detector images two X-ray fluoroscopic views.

3. The X-ray fluoroscopy distortion correction system according to claim 2, wherein the detector can be equivalently transformed into a virtual detector to correct the dimensional distortion of the two X-ray fluoroscopic views caused by geometric magnification, wherein the plane in which the virtual detector is located passes through the isocenter.

4. The X-ray fluoroscopic distortion correction system according to claim 1, wherein the projection imaging system comprises two cone-beam X-ray imaging modules, wherein the two cone-beam X-ray imaging modules are symmetrical at complementary angles with a common center, so that the cone-beam X-ray imaging modules respectively perform X-ray fluoroscopic projection on the object under test from two complementary viewpoints.

5. The X-ray fluoroscopic distortion correction system according to claim 4, wherein each of the cone-beam X-ray imaging modules includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays toward the object being inspected and images an X-ray fluoroscopic view on the detector, and the detector can be equivalently transformed into a virtual detector, the plane of which the virtual detector is located passing through the isocenter.

6. The X-ray fluoroscopy distortion correction system according to claim 1, wherein the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

7. The X-ray fluoroscopy distortion correction system according to claim 1, wherein the learning module includes a coordinate transformation module and a neural network, wherein the overlay color image is transformed from rectangular coordinates to polar coordinates by the coordinate transformation module to obtain a polar coordinate transformed overlay color image, wherein the polar coordinate transformed overlay color image is input into the neural network and processed by the neural network to output an output image, wherein the output image is subjected to an inverse polar coordinate transformation from polar coordinates to rectangular coordinates to obtain the corrected image.

8. The X-ray fluoroscopy distortion correction system according to claim 7, wherein the neural network is selected from CNN neural network, transformer neural network and diffusion generation model.

9. The X-ray fluoroscopy distortion correction system according to claim 7, wherein the neural network is selected from Pix2pixGAN, TransU-Net, DiffusionGAN, and Image-to-Image. Bridge (I2SB).

10. The X-ray fluoroscopic distortion correction system according to claim 7, wherein the projection imaging system comprises a cone-beam X-ray imaging module and a driving module, wherein the driving module drives the cone-beam X-ray imaging module to rotate around an isocenter, so that the cone-beam X-ray imaging module performs X-ray fluoroscopic projection on the object under test from two complementary viewpoints, wherein the cone-beam X-ray imaging module comprises a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays onto the object under test from two complementary viewpoints, and the detector images two X-ray fluoroscopic views.

11. The X-ray fluoroscopy distortion correction system according to claim 10, wherein the detector can be equivalently transformed into a virtual detector to correct the dimensional distortion of the two X-ray fluoroscopic views caused by geometric magnification, wherein the plane in which the virtual detector is located passes through the isocenter.

12. The X-ray fluoroscopic distortion correction system according to claim 7, wherein the projection imaging system comprises two cone-beam X-ray imaging modules, wherein the two cone-beam X-ray imaging modules are symmetrical about a common center at complementary angles, so that the cone-beam X-ray imaging modules respectively perform X-ray fluoroscopic projection on the object under test from two complementary viewpoints.

13. The X-ray fluoroscopic distortion correction system according to claim 12, wherein each of the cone-beam X-ray imaging modules includes a cone-beam X-ray source and a detector, wherein the cone-beam X-ray source emits cone-beam X-rays toward the object being examined and images an X-ray fluoroscopic view on the detector, and the detector can be equivalently transformed into a virtual detector, the plane of which the virtual detector is located passing through the isocenter.

14. The X-ray fluoroscopy distortion correction system according to claim 11 or 13, wherein the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

15. The X-ray fluoroscopic distortion correction system according to claim 13, further comprising a transport system for sequentially transporting a plurality of objects to be inspected to a detection position, so that the two cone-beam X-ray imaging modules can project the objects to be inspected from two complementary perspectives.

16. The X-ray fluoroscopy distortion correction system according to claim 14, wherein the complementary viewing angle range is 175°-185°.

17. A method for correcting X-ray fluoroscopic distortion, characterized in that, Includes the following steps: A) Cone-beam X-ray fluoroscopy projection is performed on the same object to be inspected from two complementary angles to form two complementary views, wherein the two complementary views include a first view and a second view; B) Reset the two complementary views to a virtual detector to correct the size of the two complementary views; C) Convert the two complementary views into color images and overlay them to generate an overlay color image; D) The overlay color image is transformed by polar coordinate transformation to obtain a polar coordinate transformed overlay color image; E) The polar coordinate transformation overlay color image is input as an input image into a neural network; F) The neural network processes the polar coordinate transformation overlaid color image and outputs an output image; and G) Inverse polarity transformation of the output image to obtain a corrected image.

18. The X-ray fluoroscopy distortion correction method according to claim 17, wherein step A includes the following steps: A1) A cone-beam X-ray imaging module performs cone-beam X-ray fluoroscopic projection on the object to be inspected at a first position to obtain the first view; A2) Rotate the cone-beam X-ray imaging module 180° around a center to move the cone-beam X-ray imaging module to a second position; A3) The cone-beam X-ray imaging module performs cone-beam X-ray fluoroscopic projection on the object to be inspected at the second position to obtain the second view.

19. The X-ray fluoroscopy distortion correction method according to claim 17, wherein in step A, the two cone-beam X-ray imaging modules are symmetrical about a common center at complementary angles.

20. The X-ray fluoroscopy distortion correction method according to claim 19 further includes the following steps: x) Transport an object to be tested to a testing location.

21. The X-ray fluoroscopy distortion correction method according to claim 19, wherein the plane in which the virtual detector is located passes through the isocenter.

22. The X-ray fluoroscopy distortion correction method according to any one of claims 17 to 21, wherein in step C, the first view and the second view are processed by a color overlay module, wherein the color overlay module includes a red channel, a blue channel and a green channel, wherein the red channel and the blue channel use the first view, and wherein the green channel uses the second view.

23. The X-ray fluoroscopy distortion correction method according to claim 22, wherein the overlay color image in step D is transformed from rectangular coordinates to polar coordinates to obtain the polar coordinate transformed overlay color image.

24. The X-ray fluoroscopy distortion correction method according to claim 23, wherein the output image in step G is subjected to an inverse polar coordinate transformation from polar coordinates to rectangular coordinates to obtain the corrected image.

25. The X-ray fluoroscopy distortion correction method according to claim 24, wherein step D involves selecting the neural network from CNN neural networks, transformer neural networks, and diffusion generative models.

26. The X-ray fluoroscopy distortion correction method according to claim 25, wherein the neural network is selected from Pix2pixGAN, TransU-Net, DiffusionGAN, and Image-to-Image. Bridge (I2SB).

Images