Method for converting panorama x-ray images by using three-dimensional reconstruction network architecture

Abstract

The present invention relates to a method for converting panorama X-ray images by using a three-dimensional reconstruction network architecture. More particularly, the method comprises the steps of: receiving panorama X-ray image information and cone-beam computed tomography volume image information; and obtaining, by using an SF3D generator algorithm, synthesized flattened three-dimensional volume image information and two-dimensional synthetic panorama X-ray image information from the received panorama X-ray image information and cone-beam computed tomography volume image information.

Description

A method for converting panoramic X-ray images using a three-dimensional reconstruction network structure.

[0001] The present invention relates to a method for converting a panoramic X-ray image using a three-dimensional (3D) reconstruction network architecture, and more specifically, to an image conversion method including a step of receiving panoramic X-ray image information and cone-beam computed tomography (CBCT) volumetric image information, and a step of obtaining synthesized flattened 3D volume paired dataset (SF3D) and planar synthetic panoramic X-ray (2D synthetic PX-Ray, 2DSCT) image information from the received panoramic X-ray image information and cone-beam computed tomography (CBCT) volumetric image information using an SF3D Generator Algorithm.

[0002] As of now, panoramic X-ray image information is frequently used for medical purposes. In particular, panoramic X-ray image information used in dentistry presents challenges in assessing a patient's oral structure based on a single image.

[0003] First, because panoramic X-ray image information provides flat image information, there are parts that are difficult to confirm, such as when wisdom teeth overlap.

[0004] In particular, the trajectory of the dental arch, a frequently photographed area in the dental field, has significant differences in the physical structure of each patient, and when confirmed after photographing, it forms a significantly different trajectory for each person.

[0005] Due to the above difficulties, the inconvenience of having to measure additional oral photographs arises.

[0006] However, even these additional methods have low accuracy in determining the patient's dental arch trajectory.

[0007] To solve the above problems, cone beam computed tomography is often performed in conjunction.

[0008] Cone beam computed tomography (CBT) can be performed more quickly and exposes patients to less radiation than computed tomography (CT).

[0009] However, the image information obtained through cone beam computed tomography is also low in accuracy because it only provides a portion of the information that can be obtained from the patient being photographed.

[0010] Therefore, in medical practice, there are many uncertainties in judging a patient's condition using only panoramic X-ray image information and cone beam computed tomography image information.

[0011] The present invention was created to solve the above-described problems, and the purpose of the present invention is to provide a method for converting planar image information into three-dimensional image information having the same form as reality by using a three-dimensional reconstruction network structure from panoramic X-ray image information and cone beam computed tomography volumetric image information.

[0012] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the art from the description of the present invention.

[0013] In order to achieve the above-described purpose, a method for converting a panoramic X-ray image using a three-dimensional reconstruction network structure according to an embodiment of the present invention may include a step of receiving panoramic X-ray image information and cone beam computed tomography volumetric image information, and a step of obtaining planarized three-dimensional volumetric image information and planar synthesized panoramic X-ray image information synthesized using an SF3D Generator Algorithm from the received panoramic X-ray image information and cone beam computed tomography volumetric image information.

[0014] Preferably, the SF3D Generator Algorithm may include a step of obtaining normalized cone beam computed tomography volumetric image information by intensity normalization and resizing from the received panoramic X-ray image and cone beam computed tomography volumetric image information, and a step of obtaining flattened three-dimensional volumetric image information synthesized by projecting the normalized cone beam computed tomography volumetric image information onto a normal curve estimated maximum intensity.

[0015] Preferably, the SF3D Generator Algorithm may include a step of obtaining binarized stereoscopic cone beam computed tomography image information by binarizing the received cone beam computed tomography volumetric image information, a step of obtaining image information by maximum intensity projecting the image information on an axial plane and then obtaining image information by approximating a Bezier curve, a step of obtaining flattened stereoscopic volumetric image information by finding a normal line in the image information and performing line-trace sampling, and a step of obtaining flattened composite panoramic X-ray image information by maximum intensity projecting the image information.

[0016] Preferably, the method may include a step of obtaining planar synthetic panoramic X-ray image information using a pixel-to-pixel mapping network from the received panoramic X-ray image information.

[0017] It may preferably include a step of obtaining flattened three-dimensional volumetric image information synthesized using generative adversarial networks from the received flat synthetic panoramic X-ray image information.

[0018] Preferably, the adversarial neural network may include a multi-scale vision transformer encoder.

[0019] Preferably, the multi-scale vision transformer encoder may be characterized by including a convolution mechanism and an attention mechanism.

[0020] Preferably, the adversarial neural network may include a 2D to 3D Transformer Decoder that transforms from a plane to a three-dimensional plane.

[0021] Preferably, the adversarial neural network may include a convolutional neural network (CNN).

[0022] Preferably, the method may include a step of obtaining realistic 3D volume image information using a voxel to voxel module from the received cone beam computed tomography volume image information.

[0023] Through the image conversion method using the three-dimensional reconstruction network structure provided by the present invention, in the dental field, which is a medical field, three-dimensional image information in a form most similar to the actual oral cavity of a patient can be obtained.

[0024] More specifically, the automated stereoscopic reconstruction network architecture can be used to acquire stereoscopic image information in a cost-effective manner without requiring additional image acquisition processes. This will enable medical professionals to conveniently develop diagnosis and treatment plans.

[0025] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0026] Figure 1 is a flowchart for explaining an image conversion method using a three-dimensional reconstruction network structure according to the present invention.

[0027] Figure 2 is a flowchart illustrating an image conversion method using the SF3D Generator Algorithm.

[0028] Figure 3 is a flowchart for explaining an image transformation method using an adversarial neural network structure.

[0029] Figure 4 is a flowchart illustrating an image conversion method using a multi-scale vision transformer encoder.

[0030] Figure 5 is a flowchart illustrating an image conversion method using a multi-scale vision transformer decoder.

[0031] Figure 6 is cone beam computed tomography volume image information given according to an embodiment of the present invention.

[0032] Fig. 7 is image information obtained by using an image conversion method using an adversarial neural network structure for the image information of Fig. 6 according to the above embodiment.

[0033] Fig. 8 is image information converted from the image information of Fig. 7 according to the above embodiment using stereoscopic software.

[0034] Figure 9 is a photograph that overall represents an image conversion method using a voxel-to-voxel module according to the above embodiment.

[0035] The terms used in this specification are selected from the most widely used general terms possible while taking into account the functions of the present invention, but these may vary depending on the intention of engineers working in the relevant field, precedents, the emergence of new technologies, etc.

[0036] Additionally, in certain cases, there are terms arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant invention.

[0037] In this regard, first, FIG. 1 is a flowchart illustrating an image conversion method using a stereoscopic reconstruction network structure according to the present invention, FIG. 2 is a flowchart illustrating an image conversion method using an SF3D Generator Algorithm, FIG. 3 is a flowchart illustrating an image conversion method using an adversarial neural network structure, FIG. 4 is a flowchart illustrating an image conversion method using a multi-scale vision transformer encoder, FIG. 5 is a flowchart illustrating an image conversion method using a multi-scale vision transformer decoder, FIG. 6 is a cone beam computed tomography volumetric image information given according to an embodiment of the present invention, FIG. 7 is image information obtained by converting the image information of FIG. 6 using an adversarial neural network structure according to the embodiment, FIG. 8 is image information obtained by converting the image information of FIG. 7 using stereoscopic software according to the embodiment, and FIG. 9 is a photograph generally expressing an image conversion method using a voxel-to-voxel module according to the embodiment.

[0038] Hereinafter, a method for converting a panoramic X-ray image using a three-dimensional reconstruction network structure according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 9.

[0039] First, the three-dimensional reconstruction network structure (100) according to the above embodiment of the present invention can be largely composed of a pixel-to-pixel mapping network (102), an adversarial neural network (120), and a voxel-to-voxel module (111), as illustrated in FIG. 1.

[0040] More specifically, referring to FIG. 1, the image conversion method using the three-dimensional reconstruction network structure (100) can perform image conversion, including the steps of receiving panoramic X-ray image information (101) and cone beam computed tomography volumetric image information (110), and the steps of obtaining planarized three-dimensional volumetric image information (121) and planar synthesized panoramic X-ray image information (103) synthesized using an SF3D Generator Algorithm from the received panoramic X-ray image information (101) and cone beam computed tomography volumetric image information (110).

[0041] More specifically, the SF3D Generator Algorithm may include a step of obtaining normalized cone beam computed tomography volume image information by intensity normalization and resizing from the received panoramic X-ray image (101) and cone beam computed tomography volume image information (110), and a step of obtaining flattened three-dimensional volume image information synthesized by projecting the normalized cone beam computed tomography volume image information onto a normal curve estimated maximum intensity.

[0042] In more detail, the image conversion method using the SF3D Generator Algorithm (200) can be explained with reference to FIG. 2.

[0043] As illustrated in FIG. 2, the SF3D Generator Algorithm (200) according to the embodiment of the present invention may include a step of obtaining binarized stereoscopic cone beam computed tomography image information by binarizing the received cone beam computed tomography volumetric image information (201), a step of obtaining image information (202) by performing maximum intensity projection (Normal Curve Estimated MIP Projection) on the image information in the axial plane, and then obtaining image information (203) by performing Bezier Curve Approximation, a step of obtaining flattened stereoscopic volumetric image information (121) synthesized by performing line-trace sampling on image information (204) whose normal is found in the image information (203), and a step of obtaining flat synthetic panoramic X-ray image information (103) by performing maximum intensity projection on the image information (121).

[0044] The reason for including the SF3D Generator Algorithm (200) in the image conversion method according to the above embodiment of the present invention is to obtain a plurality of image information that can be used in the pixel-to-pixel mapping network (102), adversarial neural network (120), and voxel-to-voxel module (111) that can be included in the three-dimensional reconstruction network structure (100).

[0045] Specifically, the image information can be acquired through the SF3D Generator Algorithm (200) and the image conversion process using the three-dimensional reconstruction network structure (100) through supervised learning can be automated.

[0046] Meanwhile, the image conversion method using the pixel-to-pixel mapping network (102) in the three-dimensional reconstruction network structure (100) according to an embodiment of the present invention may include a step of obtaining planar synthetic panoramic X-ray image information (103) from the received panoramic X-ray image information (101).

[0047] More specifically, looking at an embodiment of the present invention, the image conversion method using the pixel-to-pixel mapping network (102) may include a step of receiving the panoramic X-ray image information (101) and cone beam computed tomography volumetric image information (110).

[0048] Next, it may include a step of obtaining the planar composite panoramic X-ray image information (103) from the received cone beam computed tomography volume image information (110) using the SF3D Generator Algorithm.

[0049] Next, it may include a step of pairing the image information and transmitting it to the discriminator of the adversarial neural network (120).

[0050] Next, the discriminator of the adversarial neural network (120) may include a step of arriving at image information that is as similar as possible to actual image information by using the panoramic X-ray image information (101) and the planar synthetic panoramic X-ray image information (103).

[0051] The reason why the image conversion method using the pixel-to-pixel mapping network (102) is included in the embodiment of the present invention is that when the planar synthetic panoramic X-ray image information (103) is obtained, the accuracy of the final result image information of the image conversion in the three-dimensional reconstruction network structure (100) increases.

[0052] More specifically, since the planar synthetic panoramic X-ray image information (103) is highly aligned with the synthesized planarized stereoscopic volume image information (121) obtained through the SF3D Generator Algorithm (200), the accuracy of the discriminator in the adversarial neural network (120) can be increased in the unsupervised learning process.

[0053] Meanwhile, the image conversion method using the adversarial neural network (120) in the three-dimensional reconstruction network structure (100) according to the above embodiment of the present invention may include a step of obtaining flat three-dimensional volume image information (121) synthesized using the adversarial neural network (120) from the received flat synthetic panoramic X-ray image information (101).

[0054] In more detail, the flow of the image conversion method using the adversarial neural network (120) can be confirmed with reference to the above-described FIG. 3.

[0055] As illustrated in the above FIG. 3, the adversarial neural network (120) may include a multi-scale vision transformer encoder.

[0056] In more detail, the structure of the multi-scale vision transformer encoder can be confirmed with reference to the above-described FIG. 4.

[0057] As illustrated in FIG. 4, the multi-scale vision transformer encoder may include a convolution mechanism and an attention mechanism.

[0058] The reason for including the convolution mechanism and the attention mechanism in the image conversion process through the image conversion method according to the above embodiment of the present invention is that a richer image expression can be obtained as a result of unsupervised learning by obtaining the value of the most central image information from the actual object, which is the original of the image to be obtained in the above embodiment.

[0059] More specifically, referring to the above-described FIG. 3, it can be confirmed that the image transformation method using the above-described adversarial neural network (120) can include a transformer decoder that transforms from a plane to a three-dimensional image.

[0060] The reason why the transformer decoder for converting from the plane to a three-dimensional image is included in the image conversion process through the image conversion method according to the above embodiment of the present invention is that the rich image expression converted using the convolution mechanism and the attention mechanism can be converted into image information in voxel units, so that the image information that was plane can be converted into three-dimensional image information.

[0061] More specifically, referring to the above-described FIG. 3, it can be confirmed that the image transformation method using the above-described adversarial neural network (120) may include a convolutional neural network.

[0062] The method of converting an image from a plane to a three-dimensional image using a transformer decoder and a convolutional neural network can be confirmed with reference to Fig. 5.

[0063] In this way, as illustrated in the above-described FIG. 3, the reason for including the decoder and encoder in the adversarial neural network (120) in the embodiment of the present invention is that the image information obtained through image conversion can have a three-dimensional volume that is as similar to reality as possible.

[0064] Additionally, the adversarial neural network (120) in the embodiment of the present invention may be characterized by including about 30 million learnable parameters.

[0065] Meanwhile, the image conversion method using the adversarial neural network (120) in the embodiment of the present invention may include the contents of Table 1 below among the parameters.

[0066] Table 1Corresponding parameter values ​​(Embedding Dimension =512)idt_lamda=10, proj_lamda=10, gan_lambda=0.1idt_lamda=50, proj_lamda=20, gan_lambda=0.1idt_lamda=50, proj_lamda=20, gan_lambda=0.05, weighted patch_embed fusionidt_lamda=50, proj_lamda=35, gan_lambda=0.07, weighted patch_embed fusionidt_lamda=50, proj_lamda=35, gan_lambda=0.04, weighted patch_embed fusion

[0067] In the embodiment of the present invention, unsupervised learning was attempted for an adversarial neural network (120) including the above parameters using the above panoramic X-ray image information (101) of a patient in a dental clinic. In the embodiment of the present invention, unsupervised learning was attempted to convert image information into flattened three-dimensional volumetric image information (121) synthesized from the above panoramic X-ray image information (101) through unsupervised learning based on the above adversarial generative neural network (120).

[0068] A learning method for converting a planar image into a stereoscopic image through unsupervised learning based on an adversarial neural network (120) according to an embodiment of the present invention may include a step of obtaining planar synthetic panoramic X-ray image information (101) from the panoramic X-ray image information (101), a step of generating image information corresponding to a stereoscopic area from the received planar synthetic panoramic X-ray image information (101), a step of generating synthesized flat stereoscopic volume image information (121) as stereoscopic image information based on the generated image information, a step of having a discriminator discriminate the generated stereoscopic image information, a step of learning a discriminant operation to increase the discriminant accuracy, and a step of autonomously learning each of the discriminant operations.

[0069] In particular, the loss function generated in the step where the above adversarial generative neural network (121) learns image information through unsupervised learning,

[0070]

[0071] and

[0072] may include calculating as, where L A is a loss function that occurs when learning image information, D is the discriminator, and G is a generator by the SF3D Generator Algorithm.

[0073] In addition, the loss function that occurs in the step of generating the synthesized flat stereoscopic volume image information (121) which is the stereoscopic image information,

[0074]

[0075] may include calculating as, where L r is the loss function that occurs in the step of generating stereoscopic image information, and || ||2 is the L2 norm operator.

[0076] Also, the loss function for Projection Loss is

[0077]

[0078] may include,

[0079] Whole body cannibalism

[0080]

[0081] is configured to autonomously learn the stereoscopic image information generation and the discrimination operation, respectively, so that the derived value of the overall loss function is minimized, and here, λ1, λ2 and λ3 are parameters reflecting the importance of the corresponding loss function, and in the embodiment of the present invention, parameters as shown in Table 1 were used.

[0082] Table 2MEAMSECosine SimilarityPSNR-3DSSIM-1SSIM-2SSIM-3SSIM187.28196767.690.8019.830.680.680.680.68171.76163351.750.8320.200.700.700.700.70 121.1192208.610.9222.820.740.740.740.74154.70131285.820.882 1.150.730.730.730.73157.95137522.470.8620.920.720.720.720.72

[0083] As a result, as shown in Table 2 above, it can be seen that the image conversion was achieved with a mean squared error (MSE) of 92208.61, a peak signal-to-noise ratio (PSNR) of 22.82, and a structural similarity index measure (SSIM) of 0.74.

[0084] More specifically, the closer the resulting numerical value of the structural similarity index measurement is to 1, the higher the accuracy of the image transformation method.

[0085] Meanwhile, the image conversion method using the voxel-to-voxel (111) in the stereoscopic reconstruction network structure (100) according to an embodiment of the present invention may include a step of obtaining actual stereoscopic volumetric image information (122) from the received cone beam computed tomography volumetric image information (110).

[0086] More specifically, the image conversion method using the voxel-to-voxel (111) according to the embodiment of the present invention may include a 3D to Flattened-3D network (112) and a Flattened-3D to 3D network (113) as shown in FIG. 1.

[0087] More specifically, through the above-described FIG. 9, it is possible to confirm the realistic 3D volume image information (122) for the image conversion method using the voxel-to-voxel (111) in the three-dimensional reconstruction network structure (100) according to the embodiment of the present invention.

[0088] The present invention relates to a method for converting a panoramic X-ray image using a three-dimensional (3D) reconstruction network architecture, and more specifically, to an image conversion method including a step of receiving panoramic X-ray image information and cone-beam computed tomography (CBCT) volumetric image information, and a step of obtaining synthesized flattened 3D volume paired dataset (SF3D) and planar synthetic panoramic X-ray (2D synthetic PX-Ray, 2DSCT) image information from the received panoramic X-ray image information and cone-beam computed tomography volumetric image information using an SF3D Generator Algorithm, and has industrial applicability.

Claims

1. A step of receiving panoramic X-ray image information and cone beam computed tomography volumetric image information; and An image conversion method comprising a step of obtaining planarized three-dimensional volumetric image information and planar synthesized panoramic X-ray image information using an SF3D Generator Algorithm from the received panoramic X-ray image information and cone beam computed tomography volumetric image information.

2. In paragraph 1, The above SF3D Generator Algorithm comprises a step of obtaining normalized cone beam computed tomography volume image information by intensity normalization and resizing from the received panoramic X-ray image and cone beam computed tomography volume image information; and An image conversion method comprising a step of obtaining flattened stereoscopic volumetric image information synthesized by projecting the normalized cone beam computed tomography volumetric image information onto a normal curve estimate maximum intensity.

3. In paragraph 1, The above SF3D Generator Algorithm is A step of binarizing the received cone beam computed tomography volumetric image information to obtain binarized stereoscopic cone beam computed tomography image information; A step of obtaining image information by projecting the image information on the axial plane to obtain maximum intensity and then obtaining image information by approximating a Bezier curve; A step of obtaining planarized three-dimensional volume image information synthesized by finding a normal line in the above image information and performing line-trace sampling; and An image conversion method comprising a step of obtaining planar composite panoramic X-ray image information by projecting the image information at maximum intensity.

4. In paragraph 1, An image conversion method comprising a step of obtaining planar synthetic panoramic X-ray image information using a pixel-to-pixel mapping network from the received panoramic X-ray image information.

5. In paragraph 1, An image conversion method comprising a step of obtaining flattened three-dimensional volumetric image information synthesized using an adversarial neural network from the received flat synthetic panoramic X-ray image information.

6. In paragraph 5, The above adversarial neural network is an image transformation method including a multi-scale vision transformer encoder.

7. In paragraph 6, An image transformation method, characterized in that the above multi-scale vision transformer encoder includes a convolution mechanism and an attention mechanism.

8. In paragraph 5, The above adversarial neural network is an image transformation method including a transformer decoder that transforms from a plane to a three-dimensional plane.

9. In paragraph 5, The above adversarial neural network is an image transformation method including a convolutional neural network.

10. In paragraph 1, An image conversion method comprising a step of obtaining actual stereoscopic volumetric image information using a voxel-to-voxel module from the received cone beam computed tomography volumetric image information.

Images