Method and apparatus for supporting surgery using dual-source x-ray images captured during surgery

Abstract

The present invention provides a surgical support method using dual-source X-ray images during surgery, performed by a computing device comprising at least one processor according to one embodiment of the present disclosure, comprising: a step of inputting dual-source X-ray images of a surgical site taken during surgery into a neural network model to output 2D landmarks of each X-ray image; and, in the case where there are no CT images of the surgical site taken prior to surgery, a step of generating a 3D bone model by optimizing SSM parameters extracted from a State-Space Model (SSM) and a rigid body pose to align with the 2D landmarks.

Description

Technology Field

[0001] The present disclosure relates to a surgical support method and apparatus using dual-source X-ray images during surgery. Background Technology

[0002] X-ray scanners, ultrasound diagnostic devices, computed tomography (CT) scanners, and magnetic resonance imaging (MRI) scanners are utilized to obtain internal body information for observing and diagnosing the inside of a patient's body.

[0003] Among these, CT and X-ray are mainly used in orthopedic surgery, but CT has the disadvantage of being high-risk because it uses a much higher dose of radiation than simple X-ray, and X-ray has the disadvantage of not being able to visualize the surgical site in 3D.

[0004] To overcome these drawbacks, research is being conducted on techniques for generating 3D images using 2D X-ray images. Among these, active research is being conducted on methods that estimate 3D structures using deep learning models by receiving two or more X-ray images as input, or methods that generate virtual 2D X-rays from CT or 3D models and compare them with actual captured images to estimate poses and perform 3D alignment. Prior art literature

[65535] Korean Patent Publication No. 10-2023-0149097 (October 26, 2023) The problem to be solved

[0005] A surgical support method and apparatus using dual-source X-ray images taken during surgery according to one embodiment of the present disclosure aims to perform navigation by providing real-time 3D images of the surgical site using a neural network model with X-ray images taken with dual sources during surgery. means of solving the problem

[0006] The present invention provides a surgical support method using dual-source X-ray images during surgery, performed by a computing device including at least one processor according to one embodiment of the present disclosure for realizing the aforementioned problem, comprising: a step of inputting dual-source X-ray images of a surgical site taken during surgery into a neural network model to output 2D landmarks of each X-ray image; and a step of generating a 3D bone model by optimizing SSM parameters extracted from a State-Space Model (SSM) and a rigid body pose to align with the 2D landmarks.

[0007] Alternatively, if a CT image of the surgical site taken prior to surgery is available, the present invention provides a method for generating a rigidly aligned 3D bone model by optimizing a rigid pose so that the 3D landmarks extracted from the CT image and the 2D landmarks are aligned.

[0008] Alternatively, the present invention provides a method in which the above neural network model is trained using CT images and DRRs matched according to the surgical site or type of surgery as training data.

[0009] Alternatively, the present invention provides a method in which the dual-source X-ray image is captured through a plurality of X-ray emitters arranged to emit X-rays at an angle between 10 degrees and 30 degrees.

[0010] Alternatively, the present invention provides a method in which the aforementioned landmarks are pre-set according to the surgical site or type of surgery, and multiple landmarks are provided with different loss function weights applied to each other.

[0011] Alternatively, if the above CT image is available, we intend to provide a method for extracting the 3D landmark by generating a bone-specific 3D mesh and a dual-source DRR from the above CT image.

[0012] Alternatively, SSM aims to provide a method in which multiple 3D meshes are aligned to form a statistical shape model based on principal component analysis (PCA).

[0013] Alternatively, we intend to provide a method that applies a perspective depth loss function during the above optimization process and separates and blocks the gradient component for the rotation parameter.

[0014] In a dual-source X-ray imaging device during surgery according to another embodiment of the present disclosure, the device comprises: a C-arm; a first X-ray emitter disposed at one end of the C-arm; a second X-ray emitter disposed at the other end of the C-arm; a first detector disposed adjacent to the second X-ray emitter and detecting X-rays emitted from the first X-ray emitter; and

[0015] A device comprising a second detector disposed adjacent to the first X-ray emitting unit and detecting X-rays generated from the second X-ray emitting unit. Effects of the invention

[0016] A surgical support method and apparatus using dual-source X-ray images during surgery according to one embodiment of the present disclosure can perform navigation by providing real-time 3D images of the surgical site using a neural network model with dual-source X-ray images taken during surgery. Brief explanation of the drawing

[0017] FIG. 1 is a block diagram of a surgical support computing device using dual-source X-ray images according to one embodiment of the present disclosure. FIG. 2 is a flowchart showing a surgical support method using dual-source X-ray images according to one embodiment of the present disclosure. FIG. 3 is a drawing showing an X-ray image, an X-ray image with 2D landmarks displayed, and a 3D image with 3D landmarks displayed, according to one embodiment of the present disclosure. FIG. 4 is a drawing for illustrating a rigid body pose optimization image according to one embodiment of the present disclosure. FIG. 5 is a drawing showing a display UI that outputs an optimized 3D bone model image according to one embodiment of the present disclosure. FIG. 6 is a drawing showing a display UI for modifying a landmark according to one embodiment of the present disclosure. FIG. 7 is a drawing showing a dual-source X-ray imaging device according to another embodiment of the present disclosure. Specific details for implementing the invention

[0018] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.

[0019] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.

[0020] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.

[0021] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.

[0022] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0023] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”

[0024] The term "the N (N is a natural number)" used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but need to be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.

[0025] The term “acquisition” as used in this disclosure can be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0026] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0027] As used in this disclosure, the term "model" may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to an overall system implemented as a neural network that possesses problem-solving capabilities through learning. In this case, the neural network may possess problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks composed of multiple neural networks.

[0028] The term “data” as used in this disclosure may include images, signals, etc. The term “image” as used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” may be understood as a term referring to a digital representation of an object visible to the human eye. For example, “image” may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. “Image” may refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0029] For example, the image may include, but is not limited to, a medical image acquired by a medical imaging device such as a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, an ultrasound scanner, or an X-ray scanner.

[0030] As used in this disclosure, the term "medical image" is a general concept encompassing all forms of images that include medical knowledge, and may include images acquired through various modalities such as visible light cameras, IR cameras, ultrasound, X-ray, CT, MRI, PET, etc.

[0031] As used in this disclosure, the term "picture archiving and communication system (PACS)" may refer to a system that stores, processes, and transmits medical images in accordance with the Digital Imaging and Communications in Medicine (DICOM) standard. For example, the "picture archiving and communication system" may be linked with digital medical imaging scanning equipment to store medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT) images, in accordance with the Digital Imaging and Communications in Medicine standard. The "picture archiving and communication system" may transmit medical images to terminals inside or outside the hospital via a communication network. At this time, meta information, such as interpretation results and medical records, may be added to the medical images.

[0032] As used in this disclosure, the term "object" refers to a subject of scanning and may include a person, an animal, or a part thereof. For example, an object may include a part of the body (such as an organ or tissue) or a phantom. A phantom refers to a substance having a volume that is very close to the density and effective atomic number of a living organism, and may include a spherical phantom having properties similar to those of the body.

[0033] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.

[0034] FIG. 1 is a block diagram of a surgical support computing device using dual-source X-ray images taken during surgery according to one embodiment of the present disclosure.

[0035] Referring to FIG. 1, a surgical support computing device (100) according to one embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (100) may be a cloud system that enables multiple servers and clients to interact to comprehensively process data. Since the above description is merely one example regarding the type of computing device (100), the type of computing device (100) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure. As an example, the computing device (100) may include a smartphone, tablet PC, PC, smart TV, micro server, cloud server, etc., that process CT and X-ray images or perform processing functions. As another example, the computing device (100) may be a device that directly scans X-ray images. The computing device (100) may be a dual-source X-ray imaging device according to another embodiment of the present disclosure, or a control unit responsible for the control function of the dual-source X-ray imaging device.

[0036] A computing device (100) may include at least one processor (110 or less, processor), a communication interface (120), and a memory (130). However, since FIG. 1 is merely an example, the computing device (100) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (100).

[0037] A processor (110) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program and perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described type of processor (110) is merely an example, the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0038] The processor (110) is electrically connected to other components of the computing device (100) (i.e., communication interface (120), and memory (130)) to control the overall operation of the computing device (100).

[0039] A memory (130) according to one embodiment of the present disclosure may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed by a computing device (100). That is, the memory (130) may store data of any form generated or determined by a processor (110) and data of any form received by a communication interface (120). For example, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk. Additionally, the memory (130) may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory (130) is merely an example, the type of memory (130) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0040] The memory (130) can structure and organize data, combinations of data, and program code executable by the processor (110) that are necessary for the processor (110) to perform calculations. For example, the memory (130) may store a neural network model, a training data set, and a set of setting values ​​for X-ray image scanning. Additionally, the memory (130) may store program code that operates the neural network model to perform training based on the training data set and the set of setting values ​​for X-ray image scanning, program code that operates the neural network model to receive X-ray images and perform inference according to the purpose of use of the computing device (100), and processed data generated as the program code is executed.

[0041] A communication interface (120) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data through any known wired or wireless communication system. For example, the communication interface (120) may perform data transmission and reception using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultrawide-band communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the communication interface (120) may be applied in various ways other than those described above.

[0042] The communication interface (120) can receive data necessary for the processor (110) to perform calculations through wired or wireless communication with any system or any client. Additionally, the communication interface (120) can transmit data generated through the calculations of the processor (110) through wired or wireless communication with any system or any client. For example, the communication interface (120) can receive medical data through communication with a database within a hospital environment, a cloud server performing tasks such as standardization of medical data, or a computing device (100). The communication interface (120) can transmit output data of the neural network model (10), and intermediate data and processed data derived during the calculation process of the processor (110), through communication with the aforementioned database, server, or computing device (100).

[0043] FIG. 2 is a flowchart showing a surgical support method using dual-source X-ray images taken during surgery according to one embodiment of the present disclosure.

[0044] Referring to FIG. 2, a surgical support method using dual-source X-ray images taken during surgery according to one embodiment of the present disclosure may perform the step (S210) of inputting dual-source X-ray images of the surgical site taken during surgery into a neural network model to output 2D landmarks of each X-ray image.

[0045] Here, the neural network model may include at least one neural network. The neural network may include, but is not limited to, network models such as deep neural networks (DNN), recurrent neural networks (RNN), bidirectional recurrent deep neural networks (BRDNN), multilayer perceptrons (MLP), and convolutional neural networks (CNN). For example, the neural network model may be a deep learning model. The neural network model may be a neural network model trained using X-ray and CT images of an unspecified number of patients as a training dataset. For example, the neural network model may be trained on 3D meshes of bones extracted from CT images and dual-source X-ray images or digitally reconstructed radiographs (DRR). Additionally, the neural network model may be trained on CT images with landmarks marked, X-ray images, or DRR images with landmarks marked. Furthermore, the neural network model may be trained on CT images and DRR training data matched according to the surgical site or type of surgery. Types of surgery may include spinal surgery, hip surgery, knee surgery, shoulder and upper limb surgery, ankle and wrist surgery, etc. In particular, the neural network model may include a model trained on training data regarding surgeries involving the insertion of screws or artifacts during the procedure.

[0046] Simply generated DRRs produce images that are too ideal and clean, which can result in significant differences from actual X-rays. To address this, intensity and geometric transformations can be applied to make the images resemble real X-rays. Intensity augmentation methods utilize Poisson noise (λ=10⁻⁵⁰) to reproduce shot noise occurring in low-dose X-rays. Gaussian noise (σ=0.01–0.05) can be used to account for sensor noise. Additionally, contrast adjustments (±30%) can be utilized to account for the use of contrast agents and differences in exposure intensity.

[0047] Geometric augmentation methods can be used to reflect motion and focus blur effects by utilizing random blur (3-7 pixel kernels). X-ray detector lens effects (distortion) can be reflected by utilizing barrel / pincushion distortion. Metal scattering and overlapping shadows, which frequently occur in actual equipment, can be simulated by utilizing simulations of C-arm-specific artifacts. Through these augmentations, DRR becomes visually and statistically similar to actual clinical X-rays, enabling the trained model to function well in real-world environments.

[0048] FIG. 3 is a drawing showing an X-ray image, an X-ray image with 2D landmarks displayed, and a 3D image with 3D landmarks displayed, according to one embodiment of the present disclosure.

[0049] Referring to FIG. 3, landmarks may refer to specific points and information on an image of a surgical site that are necessary to generate and output information required for surgery on the image of the surgical site. 2D landmarks (see FIG. 3 (b)) refer to landmarks displayed on a 2D image (see FIG. 3 (a)), and 3D landmarks (see FIG. 3 (c)) refer to landmarks displayed on a 3D image.

[0050] Specifically, landmarks must satisfy the condition of robust feature points that are consistently detected regardless of changes in patient posture or imaging angle (C-arm angle). In addition, landmarks must have a mathematical one-to-one correspondence between vertices on the 3D SSM model and projection points on the 2D X-ray. Furthermore, landmarks must have sub-pixel resolution to meet the error tolerance of the surgical robot (e.g., within 1 mm).

[0051] For example, if the surgical site is the spine, landmarks may include those related to the pedicles, vertebral bodies, and posterior structures. When the surgical site is the spine, landmarks may include reference points that determine the geometric center and rotation axis of the vertebral body, such as the tips of the superior and inferior articular processes, the center point of the pedicle, and the edges of the vertebral body. Additionally, landmarks related to the pedicles may include reference points to prevent nerve damage during surgery. Landmarks related to the vertebral bodies may include reference points for calculating the alignment angle of the spine. Landmarks related to the posterior structures of the spine may include reference points for recognizing the position and determining the rotation of the spine.

[0052] When the surgical site is the hip joint, it may include points that serve as the basis for alignment and offset calculations during total hip replacement, such as the center of the femoral head, the superior aspect of the greater trochanter, and the outline of the acetabulum. Additionally, in the case of knee surgery, it may include kinematic indicators that define lower limb alignment, such as the medial and lateral epicondyles of the femur and the center of the tibial plane.

[0053] When using a neural network model, the loss function weights can be set differently for each landmark. The importance of each landmark in surgery may vary; for example, in spinal surgery, the center point of the pedicle is more important than other landmarks because it serves as the criterion for screw insertion, so its weight may be set relatively higher.

[0054] In addition, the neural network model can be trained by applying a differential weighting map based on the anatomical importance of the surgical site. In particular, weights are increased for key landmarks that determine the trajectory of instrument insertion to ensure precision within the clinical tolerance, and geometric correlations between adjacent landmarks are reflected in the loss function to suppress abnormal shape deformation.

[0055] Meanwhile, if a CT image of the surgical site taken prior to surgery is available, a step of generating a rigidly aligned 3D bone model by optimizing the rigid pose so that 3D landmarks extracted from the CT image and 2D landmarks of the X-ray image are aligned can be performed.

[0056] Referring to FIG. 2, according to one embodiment of the present disclosure, a processor (110) may acquire a CT image of the surgical site of a surgical subject prior to surgery. Here, the CT image may be acquired by the computing device (100) directly taking a picture of the subject (e.g., a patient), or it may be acquired from a plurality of other electronic devices that acquire CT images through a communication interface (120). For example, the computing device (100) may acquire CT images acquired by each of the plurality of other electronic devices.

[0057] Additionally, the processor (110) may receive DICOM data from another electronic device to acquire CT images. Specifically, the processor (110) may extract and acquire CT images included in the DICOM data. Additionally, the processor (110) may instruct other electronic devices, such as CT image scanning devices, to input various setting values ​​applied during scanning, and CT image scanning may proceed in a state set by these setting values.

[0058] If a CT image is obtained using the method described above, a 3D mesh for each bone can be extracted from the CT image, and a dual-source DRR can be generated at the same shooting angle (0 degrees, 15 degrees) as the dual-source X-ray taken during surgery to extract 3D landmarks.

[0059] Optimizing the rigid body pose so that 3D landmarks extracted from CT images and 2D landmarks from X-ray images are aligned can be performed through rigid body alignment-based techniques and / or landmark-based alignment-based techniques.

[0060] Rigid body matching-based techniques are methods that project 3D CT image data into a virtual space to find the position that is most similar to a 2D X-ray. First, a DRR 2D image can be created by projecting CT image data from a virtual X-ray source. Then, a similarity measurement can be performed by overlaying the actual captured X-ray with the virtual DRR image to calculate how similar the two images are (correlation, etc.). Subsequently, optimization can be performed to find the 6-degrees-of-freedom (6-DOF) coordinates that maximize similarity by rotating and moving the virtual 3D model.

[0061] For example, one can set only the 6-degrees-of-freedom rigid body pose as a variable, fix the CT image 3D mesh, and perform rigid body registration. Meanwhile, here, as the loss, L_img = Σ w i* ||proj_i - target_i||²(L_img : image loss, w i : weights set per landmark, proj_i : projected image coordinates, target_i : actual X-ray image coordinates) may occur, but this can be resolved by performing manual correction based on the display UI provided to the user (doctor, etc.) as described below.

[0062] Landmark-based registration is a method of extracting the coordinates of anatomical feature points (e.g., ends of pedicles, joint processes, etc.) from the volumetric data of a 3D CT image, finding the same points in a 2D X-ray image, and calculating a projection matrix that matches the actual 2D landmark positions when the 3D points are projected onto a 2D plane.

[0063] In this process, since depth information is lost when projecting 3D onto 2D, accuracy may be reduced using only a single frontal X-ray image. To solve this problem, dual-source X-rays can be utilized to perform high-accuracy rigid body pose optimization.

[0064] Alternatively, if there is no CT image of the surgical site taken prior to surgery, a step (S230) of generating a 3D bone model by optimizing the SSM parameters extracted from the State-Space Model (SSM) and the rigid body pose to align with the 2D landmarks of the X-ray image may be performed.

[0065] SSM can be constructed as a statistical shape model based on Principal Component Analysis (PCA) by aligning a number of 3D spine meshes (preferably 1,000 or more).

[0066] First, the 3D model(S) to be created can be expressed by the following [Equation 1].

[0067] [Mathematical Formula 1]

[0068]

[0069] The SSM parameter (α) is the average shape ( principal component vector( It is a shape factor that determines how much to apply. The rigid body pose (θ,t) is a 6-degrees-of-freedom (6-DOF) parameter representing rotation (θ) and translation (t) in 3D space.

[0070] The goal of rigid body pose optimization is to find α, θ, and t that minimize the distance between a 3D landmark projected onto a 2D plane and a 2D landmark detected in an X-ray image. Regarding data preparation, the 3D side can select anatomical 3D landmarks defined in the SSM. The positions of these points change according to a. The 2D side can detect corresponding 2D landmarks in the X-ray image using a neural network model.

[0071] The objective function for optimization can be constructed as shown in [Equation 2] below.

[0072] [Mathematical Formula 2]

[0073]

[0074] ε is the reprojection error, which is the Euclidean distance between the point projected from the 3D landmark and the actual 2D landmark. As a constraint, it can regulate the SSM parameter α to prevent it from forming an overly bizarre shape. In general, the Mahalanobis distance can be used to restrict it to exist within a statistical probability distribution.

[0075] Next, iterative optimization can be performed. Non-linear least-squares algorithms, such as Levenberg-Marquardt (LM) or Gauss-Newton, can be used. Rigid pose-first optimization can fix the geometry (α) initially and first align the position and angle (θ, t). Non-rigid registration can optimize α together once the position (θ) is aligned to some extent, allowing the structure to be deformed to fit the patient's detailed bone shape.

[0076] Meanwhile, to achieve high accuracy, this embodiment utilizes dual-source X-rays to prevent a decrease in accuracy in the depth direction. By using dual-source X-ray images to simultaneously calculate projection errors in two directions, accuracy can be dramatically improved.

[0077] It may be difficult to capture the overall volume of the bone using landmarks alone. In this case, a silhouette-based term can be used as a cost function to extract the bone's outline using a contour matching method and match it with the silhouette of the 3D model.

[0078] Once optimization is complete, the final calculated α value can be substituted into the SSM equation to generate a high-resolution 3D mesh that reflects the patient's anatomical structure. This model can be used as a surgical guide image for the patient during surgery without the need for CT images. Furthermore, this method can be effectively utilized to reduce CT scanning costs and radiation exposure, particularly in scoliosis analysis, hip replacement planning, and knee osteoarthritis diagnosis.

[0079] Here, Image / Reprojection Loss may occur, but this can be resolved by performing manual correction based on the display UI provided to the user (doctor, etc.) as described later. Collision Loss may occur, but this can be resolved by preventing overlap between the vertebra meshes. Gap Loss may occur, but this can be resolved by constraining the disc spacing to maintain a physiological range (e.g., 3-10 mm).

[0080] Rotation / SmoothRotation Loss may occur, but it can be resolved by preventing abrupt angle changes of adjacent bones. Regularization Loss may occur, but it can be resolved by constraining α to the Mahalanobis distance based on the eigenvalue (±3σ).

[0081] When using multiple X-ray images captured in a narrow baseline environment (e.g., about 15°), projective geometric ambiguity may occur in such a narrow-angle viewing environment, where displacement in the depth direction (e.g., Y-axis) does not cause significant pixel changes in the 2D projected image. Consequently, existing 2D and 3D alignment optimization algorithms converge to minimize only the error on the 2D projection plane, and as a result, there is a limitation in reaching a local minimum while leaving an error of tens of mm (e.g., about 29 mm) in the depth direction. To solve this problem, a method according to one embodiment of the present disclosure introduces a perspective-based depth constraint.

[0082] The method according to the present embodiment can perform perspective optimization using the correlation between the anatomical specifications of a target object (e.g., a spinal segment) and the size change of a projected shape.

[0083] Meanwhile, the Perspective DepthLoss function can calculate the ratio between the actual physical size of an anatomical feature defined from the SSM (e.g., pedicle width) and the projected width of that feature in the 2D projected image. By designing a loss function that induces the width of the projected feature to match the actual width of the target image, displacement in the depth direction can be accurately estimated.

[0084] During the optimization process using the perspective depth loss function, the gradient component of the rotation parameter can be isolated and blocked. This allows the loss function to be configured to contribute independently only to the translation parameter in the depth direction, without affecting the object's rotation values. This is intended to prevent optimization instability caused by interference from rotation parameters and to maximize the convergence speed of the depth axis.

[0085] As a result of experiments according to the method of this embodiment, significant improvement in accuracy was confirmed, with the final residual error decreasing to within approximately 1.4 mm from an initial depth direction error of approximately 32 mm. The depth direction error, which had stagnated at a level of approximately 29.5 mm in the conventional method, could be brought closer to the correct position by forcing a movement of approximately 30.6 mm or more through a perspective optimization mechanism.

[0086] Next, a step (S240) of outputting the generated 3D bone model can be performed.

[0087] FIG. 4 is a drawing for illustrating a rigid body pose optimization image according to one embodiment of the present disclosure.

[0088] Referring to FIG. 4, images (a) before rigid body pose optimization and images (b) after optimization can be seen in frontal (0-degree) images ((a), (b)) of the surgical site. Additionally, images (c) before rigid body pose optimization and images (d) after optimization can be seen in 15-degree angle images ((c), (d)) of the surgical site.

[0089] FIG. 5 is a drawing showing a display UI that outputs an optimized 3D bone model image according to one embodiment of the present disclosure.

[0090] Referring to Fig. 5, dual-source X-ray images and optimized 3D bone model images can be provided to the user. Additionally, transverse, sagittal, and coronal images can also be provided. By referring to these, the user can check images of the patient's surgical site during surgery and perform accurate surgery.

[0091] FIG. 6 is a drawing showing a display UI for modifying a landmark according to one embodiment of the present disclosure.

[0092] Referring to Fig. 6, a 3D bone model image with landmarks marked can be output to the user along with a 2D image with landmarks marked. Based on this, the user can modify the landmarks. This enables the generation of an optimized 3D bone model image with higher accuracy. Additionally, the user can verify whether rigid body pose optimization has been performed correctly and review whether there are any errors in the landmarks.

[0093] Hereinafter, a dual source X-ray imaging device (60) according to another embodiment of the present disclosure will be described.

[0094] FIG. 7 is a drawing showing a dual-source X-ray imaging device according to another embodiment of the present disclosure.

[0095] Referring to FIG. 7, a dual source X-ray imaging device (60) according to another embodiment of the present disclosure may include a C-arm (610), a first X-ray emitter (620), a second X-ray emitter (630), a first detector (640), and a second detector (650).

[0096] Additionally, the dual-source X-ray imaging device (60) may further include a control unit, a display, etc., for performing processes and communication necessary for X-ray imaging. Here, the control unit may be replaced by the aforementioned computing device (100).

[0097] The C-arm (610) is an arm member formed in a C-shape, and may be a part that facilitates X-ray imaging of a patient on an operating table by having an X-ray emitter (620, 630) and a detector (640, 650) placed on each end side. The C-arm (610) can be rotated to allow imaging of various parts of the patient.

[0098] The first X-ray emitter (620) may be positioned on one end of the C-arm (610). The X-ray emitter may be a device that generates X-rays and emits them toward the patient's body. The first X-ray emitter (620) may be a part that is positioned adjacent to one end of the C-arm (610) and emits X-rays toward the other end of the C-arm (610). The first X-ray emitter (620) may be positioned at the end of the C-arm (610). Additionally, when the second detector (650), which will be described later, is positioned at the end of the C-arm (610), the first X-ray emitter (620) may be positioned inside the C-arm (610).

[0099] The second X-ray emitting unit (630) may be positioned on the other end side of the C-arm (610). The second X-ray emitting unit (630) may be a part that emits X-rays toward the one end side of the C-arm (610) by being positioned adjacent to the other end of the C-arm (610). The second X-ray emitting unit (630) may be positioned at the end of the C-arm (610). Additionally, when the first detector (640), which will be described later, is positioned at the end of the C-arm (610), the second X-ray emitting unit (630) may be positioned inside the C-arm (610).

[0100] The first detector (640) may be positioned adjacent to the second X-ray emitter (630) and may detect X-rays emitted from the first X-ray emitter (620). The first detector (640) may be positioned slightly away from the end side of the C-arm (610). Here, the second X-ray emitter (630) may be positioned closer to the end side of the C-arm (610).

[0101] The second detector (650) may be positioned adjacent to the first X-ray emitter (620) and may detect X-rays emitted from the second X-ray emitter (630). The second detector (650) may be positioned slightly away from the end side of the C-arm (610). Here, the first X-ray emitter (620) may be positioned closer to the end side of the C-arm (610).

[0102] The first X-ray emitter (620) and the second X-ray emitter (630) can emit X-rays while forming a certain angle greater than 0 degrees and less than 90 degrees. In order to generate a 3D image from two X-ray images, the two X-ray emitters must emit X-rays while forming a predetermined angle without facing each other. Preferably, the X-rays emitted by the two X-ray emitters can form an angle (β) between 10 degrees and 30 degrees.

[0103] The surgical support method and device using dual-source X-ray images taken during surgery described above can generate and output a 3D image of the surgical site during surgery to perform surgical navigation.

[0104] A surgical support method and device using dual-source X-ray images taken during surgery. When a pre-operative CT image is available, a highly accurate 3D surgical site image can be provided during surgery by utilizing the CT image, DRR, and dual-source X-ray image.

[0105] A surgical support method and device using dual-source X-ray images taken during surgery can provide highly accurate 3D images of the surgical site during surgery by utilizing SSM and dual-source X-ray images even when pre-operative CT images are unavailable.

[0106] A surgical support method and device using dual-source X-ray images taken during surgery can automatically input landmarks into dual-source X-ray images taken during surgery using a neural network model.

[0107] The various embodiments of the present disclosure described above may be combined with additional embodiments and modified to the extent understandable to those skilled in the art in light of the detailed description above. The embodiments of the present disclosure are illustrative in all respects and should be understood as not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form. Accordingly, all modifications or variations derived from the meaning, scope, and equivalents of the claims of the present disclosure should be interpreted as being included within the scope of the present disclosure. Explanation of the symbols

[0108] 100: Computing device 110: Processor 120: Communication interface 130: Memory 60: Dual-source X-ray imaging device 610: Siam 620: 1st X-ray Emission Unit 630: 2nd X-ray Emission Unit 640: 1st Detector 650: Second detector

Claims

Claim 1 A surgical support method using dual-source X-ray images taken during surgery, performed by a computing device comprising at least one processor, comprising: a step of inputting dual-source X-ray images of a surgical site taken during surgery into a neural network model to output 2D landmarks of each X-ray image; and a step of generating a 3D bone model by optimizing SSM parameters extracted from a State-Space Model (SSM) and a rigid body pose to align with the 2D landmarks, wherein the step of generating the 3D bone model includes applying a perspective depth loss function during the optimization process and separating and blocking the gradient component for the rotation parameter. Claim 2 A method according to claim 1, wherein, if a CT image of the surgical site taken prior to surgery is available, a rigid body pose is optimized so that a 3D landmark extracted from the CT image and the 2D landmark are aligned to generate a rigidly aligned 3D bone model. Claim 3 A method according to claim 1, wherein the neural network model is trained using CT images and DRRs matched according to the surgical site or type of surgery as training data. Claim 4 A method according to claim 1, wherein the dual-source X-ray image is captured through a plurality of X-ray emitters arranged to emit X-rays at an angle between 10 degrees and 30 degrees. Claim 5 A method according to claim 1, wherein the landmark is pre-set according to the surgical site or type of surgery, and a plurality of landmarks are provided with different loss function weights applied to each other. Claim 6 In paragraph 2, a method for extracting 3D landmarks by generating a bone-specific 3D mesh and a dual-source DRR from the CT image when the CT image is available. Claim 7 The method of claim 1, wherein the SSM is constructed as a Principal Component Analysis (PCA)-based statistical shape model by aligning a plurality of 3D meshes. Claim 8 delete Claim 9 A dual-source X-ray imaging device for surgery comprises: a C-arm; a first X-ray emitter disposed at one end of the C-arm; a second X-ray emitter disposed at the other end of the C-arm; a first detector disposed adjacent to the second X-ray emitter and detecting an X-ray emitted from the first X-ray emitter; a second detector disposed adjacent to the first X-ray emitter and detecting an X-ray generated from the second X-ray emitter; and at least one processor, wherein the processor inputs a dual-source X-ray image of a surgical site captured during surgery into a neural network model to output 2D landmarks of each X-ray image, generates a 3D bone model by optimizing SSM parameters extracted from a State-Space Model (SSM) and a rigid body pose to align with the 2D landmarks, applies a perspective depth loss function during the optimization process, and separates and blocks the gradient component of the rotation parameter.

Images