Method and apparatus for surgical assistance using intra-operative x-ray imaging

Abstract

The present invention provides a surgical support method using an X-ray image taken during surgery, performed by a computing device comprising at least one processor according to one embodiment of the present disclosure, comprising: a step of inputting an X-ray image of a surgical site taken during surgery into a neural network model; a step of the neural network model estimating six degrees of freedom pose parameters of the bone of the surgical site based on the input X-ray image; and a step of generating a surgical guide image by aligning the estimated six degrees of freedom pose parameters with a differentiable DRR, wherein the neural network model has learned pre-operative CT image data of the surgical site.

Description

Technology Field

[0001] The present disclosure relates to a surgical support method and apparatus using X-ray images. Background Technology

[0002] X-ray scanners, ultrasound diagnostic devices, computed tomography (CT) scanners, and magnetic resonance imaging (MRI) scanners are utilized to obtain internal body information for observing and diagnosing the inside of a patient's body.

[0003] Among these, CT and X-ray are mainly used in orthopedic surgery, but CT has the disadvantage of being high-risk because it uses a much higher dose of radiation than simple X-ray, and X-ray has the disadvantage of not being able to visualize the surgical site in 3D.

[0004] To overcome these drawbacks, research is being conducted on techniques for generating 3D images using 2D X-ray images. Among these, active research is being conducted on methods that estimate 3D structures using deep learning models by receiving two or more X-ray images as input, or methods that generate virtual 2D X-rays from CT or 3D models and compare them with actual captured images to estimate poses and perform 3D alignment. Prior art literature

[65535] U.S. Patent Application Publication US2021 / 0366135 (Nov. 25, 2021) The problem to be solved

[0005] A surgical support method and apparatus using X-ray images according to one embodiment of the present disclosure aims to perform navigation by providing real-time 3D images of the surgical site using a neural network model with CT images taken before surgery and X-ray images taken during surgery. means of solving the problem

[0006] The present invention provides a surgical support method using an X-ray image taken during surgery, which is performed by a computing device including at least one processor according to one embodiment of the present disclosure for realizing the aforementioned problem, comprising: a step of inputting an X-ray image of a surgical site taken during surgery into a neural network model; a step of the neural network model estimating six degrees of freedom pose parameters of the bone of the surgical site based on the input X-ray image; and a step of generating a surgical guide image by aligning the estimated six degrees of freedom pose parameters with a differentiable DRR using the estimated six degrees of freedom pose parameters; wherein the neural network model has learned pre-operative CT image data of the surgical site.

[0007] Alternatively, the present invention provides a method in which the neural network model is trained to estimate the 6 degrees of freedom pose parameters of the rigid body based on the preoperative CT image data and the preoperative CT image data, wherein the neural network model is a rigid body model of the surgical site bone extracted from the preoperative CT image data.

[0008] Alternatively, we intend to provide a method in which a rigid body transformation is applied during the step in which the neural network model estimates the 6 degrees of freedom pose parameters.

[0009] Alternatively, we intend to provide a method characterized by the application of Adam-series optimization in the step where the neural network model estimates the 6 degrees of freedom pose parameters.

[0010] Alternatively, we intend to provide a method characterized by applying brightness-based Multiscale Normalized Cross-Correlation (MNCC) and boundary-based Gradient Normalized Cross-Correlation (GNCC) in the step of generating the surgical guide image.

[0011] In a device for supporting surgery using X-ray images during surgery according to another embodiment of the present disclosure, the device comprises: a processor; and a memory in which a surgical support program is stored. As the processor executes the program, an X-ray image of a surgical site taken during surgery is input into a neural network model. Based on the input X-ray image, the neural network model estimates six degrees of freedom pose parameters of the bone of the surgical site. The estimated six degrees of freedom pose parameters are matched with a differentiable DRR to generate a surgical guide image. The neural network model has learned pre-operative CT image data of the surgical site. Effects of the invention

[0012] A surgical support method and apparatus using X-ray images according to one embodiment of the present disclosure can perform navigation by providing real-time 3D images of the surgical site using a neural network model with CT images taken before surgery and X-ray images taken during surgery. Brief explanation of the drawing

[0013] FIG. 1 is a block diagram of a surgical support computing device using X-ray images according to one embodiment of the present disclosure. FIG. 2 is a flowchart showing a surgical support method using an X-ray image according to one embodiment of the present disclosure. FIG. 3 is an exemplary diagram of a surgical support computing device using X-ray images according to one embodiment of the present disclosure. FIG. 4 is a drawing showing a preoperative CT image according to one embodiment of the present disclosure. FIG. 5 is a drawing showing an X-ray image during surgery according to one embodiment of the present disclosure. FIG. 6 is a drawing showing a 3D image of a surgical site according to one embodiment of the present disclosure. FIG. 7 is a drawing for explaining an initial pose prediction method according to one embodiment of the present disclosure. Specific details for implementing the invention

[0014] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.

[0015] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.

[0016] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.

[0017] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.

[0018] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.

[0019] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”

[0020] The term "the N (N is a natural number)" used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but need to be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.

[0021] The term “acquisition” as used in this disclosure may be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0022] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concepts of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.

[0023] As used in this disclosure, the term "model" may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to an overall system implemented as a neural network that possesses problem-solving capabilities through learning. In this case, the neural network may possess problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks composed of multiple neural networks.

[0024] The term “data” as used in this disclosure may include images, signals, etc. The term “image” as used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” may be understood as a term referring to a digital representation of an object visible to the human eye. For example, “image” may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. “Image” may refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0025] For example, the image may include, but is not limited to, a medical image acquired by a medical imaging device such as a magnetic resonance imaging (MRI) scanner, a computed tomography (CT) scanner, an ultrasound scanner, or an X-ray scanner.

[0026] As used in this disclosure, the term "medical image" is a general concept encompassing all forms of images that include medical knowledge, and may include images acquired through various modalities such as visible light cameras, IR cameras, ultrasound, X-ray, CT, MRI, PET, etc.

[0027] As used in this disclosure, the term "picture archiving and communication system (PACS)" may refer to a system that stores, processes, and transmits medical images in accordance with the Digital Imaging and Communications in Medicine (DICOM) standard. For example, the "picture archiving and communication system" may be linked with digital medical imaging scanning equipment to store medical images, such as magnetic resonance imaging (MRI) and computed tomography (CT) images, in accordance with the Digital Imaging and Communications in Medicine standard. The "picture archiving and communication system" may transmit medical images to terminals inside or outside the hospital via a communication network. At this time, meta information, such as interpretation results and medical records, may be added to the medical images.

[0028] As used in this disclosure, the term "object" refers to a subject of scanning and may include a person, an animal, or a part thereof. For example, an object may include a part of the body (such as an organ or tissue) or a phantom. A phantom refers to a substance having a volume that is very close to the density and effective atomic number of a living organism, and may include a spherical phantom having properties similar to those of the body.

[0029] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.

[0030] FIG. 1 is a block diagram of a surgical support computing device using an X-ray image taken during surgery according to one embodiment of the present disclosure.

[0031] Referring to FIG. 1, a surgical support computing device (100) according to one embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (100) may be a cloud system that enables multiple servers and clients to interact to comprehensively process data. Since the above description is merely one example regarding the type of computing device (100), the type of computing device (100) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure. As an example, the computing device (100) may include a smartphone, tablet PC, PC, smart TV, micro server, cloud server, etc., that process CT and X-ray images or perform processing functions. As another example, the computing device (100) may be a device that directly scans X-ray images.

[0032] A computing device (100) may include at least one processor (110 or less, processor), a communication interface (120), and a memory (130). However, since FIG. 1 is merely an example, the computing device (100) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (100).

[0033] A processor (110) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program and perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described type of processor (110) is merely an example, the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0034] The processor (110) is electrically connected to other components of the computing device (100) (i.e., communication interface (120), and memory (130)) to control the overall operation of the computing device (100).

[0035] A memory (130) according to one embodiment of the present disclosure may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed by a computing device (100). That is, the memory (130) may store data of any form generated or determined by a processor (110) and data of any form received by a communication interface (120). For example, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, and an optical disk. Additionally, the memory (130) may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory (130) is merely an example, the type of memory (130) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0036] The memory (130) can structure and organize data, combinations of data, and program code executable by the processor (110) that are necessary for the processor (110) to perform calculations. For example, the memory (130) may store a neural network model, a training data set, and a set of setting values ​​for X-ray image scanning. Additionally, the memory (130) may store program code that operates the neural network model to perform training based on the training data set and the set of setting values ​​for X-ray image scanning, program code that operates the neural network model to receive X-ray images and perform inference according to the purpose of use of the computing device (100), and processed data generated as the program code is executed.

[0037] A communication interface (120) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data through any known wired or wireless communication system. For example, the communication interface (120) may perform data transmission and reception using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultrawide-band communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the communication interface (120) may be applied in various ways other than those described above.

[0038] The communication interface (120) can receive data necessary for the processor (110) to perform calculations through wired or wireless communication with any system or any client. Additionally, the communication interface (120) can transmit data generated through the calculations of the processor (110) through wired or wireless communication with any system or any client. For example, the communication interface (120) can receive medical data through communication with a database within a hospital environment, a cloud server performing tasks such as standardization of medical data, or a computing device (100). The communication interface (120) can transmit output data of the neural network model (10), and intermediate data and processed data derived during the calculation process of the processor (110), through communication with the aforementioned database, server, or computing device (100).

[0039] FIG. 2 is a flowchart showing a surgical support method using an X-ray image taken during surgery according to one embodiment of the present disclosure.

[0040] Referring to FIG. 2, according to one embodiment of the present disclosure, a processor (110) may acquire a CT image of the surgical site of a surgical subject prior to surgery. Here, the CT image may be acquired by the computing device (100) directly taking a picture of the subject (e.g., a patient), or it may be acquired from a plurality of other electronic devices that acquire CT images through a communication interface (120). For example, the computing device (100) may acquire CT images acquired by each of the plurality of other electronic devices.

[0041] Additionally, the processor (110) may receive DICOM data from another electronic device to acquire CT images. Specifically, the processor (110) may extract and acquire CT images included in the DICOM data. Additionally, the processor (110) may instruct other electronic devices, such as CT image scanning devices, to input various setting values ​​applied during scanning, and CT image scanning may proceed in a state set by these setting values.

[0042] FIG. 3 is a diagram illustrating a neural network model according to one embodiment of the present disclosure. FIG. 4 is a diagram showing a preoperative CT image according to one embodiment of the present disclosure. FIG. 5 is a diagram showing an intraoperative X-ray image according to one embodiment of the present disclosure. FIG. 6 is a diagram showing a 3D image of the surgical site according to one embodiment of the present disclosure.

[0043] Referring to FIGS. 3 through 6, the neural network model (10) may include at least one neural network. The neural network may include network models such as a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), a multilayer perceptron (MLP), and a convolutional neural network (CNN), but is not limited thereto. The neural network model (10) may be, for example, a deep learning model. The neural network model (10) may be trained using X-ray images and CT images of an unspecified number of patients as a training data set in a surgical support method using X-ray according to one embodiment of the present disclosure.

[0044] In addition, the neural network model may be trained using input CT images and images with anatomical structures labeled in the CT images as a training dataset. For example, the input data is a CT image of a surgical subject (see Fig. 4), and the nnU-Net deep learning-based neural network model can be used to automatically label anatomical structures in the CT images.

[0045] In the case of the vertebrae, the cervical (C1-C7), thoracic (T1-T12), lumbar (L1-L5), and sacrum can be divided individually. For the pelvic bones, the ilium, ischium, pubis, and femoral head can be distinguished. Furthermore, it is possible to finely subdivide even the joints of the limbs and toes.

[0046] Meanwhile, this data allows users to view and review the segmented results in 3D through a 3D interactive viewer. Regarding fracture surgery, each bone fragment can be labeled to verify whether each fragment has been properly separated. Additionally, the target surgical location can be set. Furthermore, partial marking can be performed on each bone fragment.

[0047] In relation to spinal surgery, it is possible to check whether the spinal regions are properly labeled, to indicate the insertion locations of screws, and to mark lesion sites such as herniated discs and spinal stenosis.

[0048] In the consistency check for quality verification of this neural network model (10), a volume consistency check can be performed to check whether the volume of the same structure is abnormally small or large. This allows for the filtering out of incorrect segmentation (omission, over-segmentation, etc.).

[0049] Meanwhile, training data can be generated through a synthetic DRR (Data-driven Digitally Reconstructed Radiograph) generation process using CT images of the surgical site of the patient taken prior to surgery. The synthetic DRR generation process may mean generating a large number of various X-ray images that can be obtained during actual surgery in advance through simulation to create a dataset necessary for training a neural network model (10).

[0050] The input data can be segmented CT volumes (0.5–1.0 mm slices). By thinly slicing the CT image to obtain high-resolution 3D structures, segmented regions of interest such as bones and organs can be used. This allows for the creation of X-ray images that accurately reflect structural features when generating DRR.

[0051] Pose sampling allows X-rays to be captured from various angles and positions, similar to actual C-arm imaging, rather than from a single fixed viewpoint, thereby enabling the projection of a virtual CT volume from multiple locations. For example, a large amount of training data can be obtained by generating multiple pairs of DRR images. The movement range can be set to ±100mm (Gaussian distribution, σ=30mm) to reflect changes in camera and / or patient position. The rotation range can be set to ±45° (uniform distribution) to reflect various shooting angles.

[0052] Simply generated DRR images are too ideal and clean, which can result in significant differences from actual X-rays. To address this, intensity / geometric transformations can be applied to make them similar to actual X-rays. Intensity augmentation methods include Poisson noise ( Shot noise generated from low-dose X-rays can be reproduced by utilizing (σ = 10-50). Sensor noise can be reflected by utilizing Gaussian noise (σ = 0.01-0.05). Additionally, the use of contrast agents and differences in exposure intensity can be reflected by utilizing contrast change (±30%).

[0053] Geometric augmentation methods can be used to reflect motion and focus blur effects by utilizing random blur (3-7 pixel kernels). X-ray detector lens effects (distortion) can be reflected by utilizing barrel / pincushion distortion. Metal scattering and overlapping shadows, which frequently occur in actual equipment, can be simulated by utilizing simulations of C-arm-specific artifacts. Through these augmentations, DRR becomes visually and statistically similar to actual clinical X-rays, enabling the trained model to function well in real-world environments.

[0054] In this way, surgical navigation can be performed by using a neural network model pre-trained with CT images of the surgical subject before surgery to generate and output a 3D image of the surgical site (see Fig. 6) using X-ray images taken during surgery (see Fig. 5) as input. Specifically, by registering a patient and using CT images taken before surgery and X-ray images taken during surgery, a ResNet18-based neural network model pre-trained with clinical data can be fine-tuned quickly by about 20 epochs with patient-specific CT-DRR data to provide a structure capable of estimating the 6-degree-of-freedom (rotation and translation) position of the bone at the surgical site with minimal error.

[0055] First, the patient is registered, and pre-operative CT images are provided to train and optimize the neural network model. Specifically, a rigid body model of the bone at the surgical site can be obtained using the CT images taken before surgery (S210). Next, a neural network model for estimating the 6-degree-of-freedom pose parameters of the rigid body can be trained based on the CT image data (S220).

[0056] Referring to FIG. 4, the next step is to input an X-ray image of the surgical site taken during surgery into a neural network model (S230).

[0057] FIG. 7 is a diagram illustrating an initial pose prediction method according to an embodiment of the present disclosure. Referring to FIG. 7, an initial pose prediction can be performed based on an X-ray image (see FIG. 4) taken during surgery. When the X-ray image is input into a pre-trained neural network model, inference can be performed in real time to predict the initial pose. For example, the initial pose may refer to the position of the beam source (BG: Beam generator) during X-ray imaging.

[0058] The neural network model can estimate 6-degree-of-freedom pose parameters of the surgical site bone based on the input X-ray image (S240). For example, through various CNN backbones (ResNet, EfficientNet, ConvNeXt), it can provide a stable and continuous 6-degree-of-freedom representation using an SE (3) log map and provide images that comply with medical imaging standards using the RAS+ coordinate system. Here, the SE (3) log map is used for parameterization, and SE (3) is a set representing 3D rigid body transformations (rotation + translation). By using the log map, the discrete rotation representation can be converted into a continuous 6-dimensional vector. Through this, the learning of the 6-degree-of-freedom continuous representation is stable, gradient vanishing / bounceback can be prevented during backpropagation, and the problem of rotation singularity that occurs in Euler angle-based representations can be eliminated.

[0059] Regarding inference speed, hardware optimization through TensorRT-based lightweighting and memory optimization enables the provision of speeds suitable for clinical application on hospital workstations and even edge devices. In terms of TensorRT optimization, FP16 inference utilizes 16-bit (FP16) operations instead of 32-bit (FP32), reducing memory usage and doubling computation speed while providing less than 1% loss in accuracy. Kernel Fusion merges multiple sequential operations (e.g., Convolution -> BatchNorm -> ReLU) into a single kernel, reducing computational call overhead and improving GPU processing speed.

[0060] In relation to memory optimization, utilizing a unified memory pool minimizes layer-specific memory allocation and deallocation, thereby reducing GPU memory waste and increasing cache efficiency.

[0061] FIG. 5 is a diagram illustrating 6-degrees-of-freedom optimization by anatomical structure according to one embodiment of the present disclosure.

[0062] The following algorithm can be used for optimization for each anatomical structure. SE(3) represents a set of 3D rigid body transformations (rotation + translation) and is expressed in matrix form.

[0063] [Mathematical Formula 1]

[0064]

[0065] se(3) can represent the space of all possible infinitesimal rigid body motions as the tangent space (vector space) of SE(3).

[0066] [Mathematical Formula 2]

[0067]

[0068] Here,

[0069] A(x) represents the space after each rigid body transformation, and can be optimized by applying the generated DRR and the similarity loss from the actual X-ray.

[0070] The same preprocessing is applied to actual X-ray images to calculate NCC series similarity, and You can perform gradient descent (Adam).

[0071] Additionally, AdamW optimization can be performed. Specifically, through structure-specific learning rate scheduling, momentum, clipping, line search optimization, and early termination criteria, AdamW optimization can reduce the number of iterations to less than half of the existing model and provide a model with improved stability, speed, and clinical applicability. Regarding the Base Optimizer (AdamW), AdamW is a variant of the Adam optimizer that applies weight decay decoupled from momentum and offers superior generalization performance compared to standard Adam.

[0072] Regarding the Adaptive Learning Rate Schedule, different learning rate policies can be applied to rotation (poses_rot) and translation (poses_xyz) for each PolyPose structure.

[0073] With the Convergence Acceleration strategy, exploration can be performed with low momentum in the early stages, and convergence can be accelerated with high momentum in the later stages. Gradient explosion can be prevented through Gradient Clipping (max_norm=1.0). Unnecessary overshoot can be prevented by automatically adjusting the step size through Adaptive Step Size (Armijo line search).

[0074] As for early stopping criteria, regarding the loss plateau, the condition |△L| < 1e-5 can be applied for three consecutive cycles to stop the process when there is almost no change in loss. Regarding the gradient magnitude, ||▽L|| < 1e-6 can be applied to stop the process when the gradient magnitude becomes so small that training can no longer proceed. Regarding parameter change, ||△θ|| < 0.01mm / 0.001rad can be applied to stop the process when the actual parameter change is clinically insignificant.

[0075] Regarding iteration reduction, the achievement rate can be reduced by 40–60% from the existing 153 iterations (3-step coarse-to-fine alignment, 51 iterations each) to the target 60–90 iterations (20–30 iterations x 3 steps), allowing for real-time application during surgery through rapid convergence.

[0076] Next, the rigid body model and the estimated 6 degrees of freedom parameters can be matched to a differentiable DRR (S250). Specifically, through the calculation of real-time similarity between the synthetic DRR and the X-ray taken during surgery via a hybrid metric system, the accuracy of the pose estimation of the neural network model can be learned and optimized.

[0077] The basic structure of the neural network model (Image Loss) is that the input is synthetic DRR vs. real X-ray, the goal is to minimize the difference by simultaneously reflecting the shape, intensity, and edge information of the two images, and the method uses the combined loss of MNCC (Multiscale Normalized Cross-Correlation) and GNCC (Gradient Normalized Cross-Correlation).

[0078] MNCC considers global alignment (entire bone location) and local alignment (fine alignment) simultaneously using multi-scale (Global + Local) methods, sets weights with equal weight for global and local alignment, calculates correlation coefficients in small window units for patch size, and enables real-time computation through acceleration with GPU kernels. Using MNCC, image intensity pattern (brightness / contrast)-based alignment can be performed.

[0079] GNCC was configured to appropriately set the patch size, adjust the degree of gradient smoothing, align based on the bone contour (gradient edge) using edge-aware alignment, and be less affected by intensity variations (exposure, noise) using intensity-invariant. By performing contour-based alignment using GNCC, it is possible to evaluate whether the bone boundaries in the DRR image match those in the X-ray image.

[0080] By balancing the intensity-based of MNCC and the boundary-based of GNCC, stable matching performance is ensured even in the presence of intensity distortion or noise.

[0081] Next, a surgical guide image can be generated as a result of the alignment (S260). Specifically, fast ray tracing can be implemented through CUDA optimization using a Differentiable Rendering Pipeline and an X-ray simulation DRR generation algorithm of Siddon's algorithm (CUDA optimized). By utilizing Trilinear for interpolation, CT voxel data can be smoothly interpolated to improve the quality of the DRR image. Detector scaling can be set to 0.125 to reduce resolution and ensure memory efficiency. Bone attenuation can be set to a 2.0X multiplier to emphasize the bone absorption coefficient similar to that of an actual X-ray image.

[0082] In summary, the neural network model can be designed to provide stable matching loss by combining strength-based (MNCC) and contour-based (GNCC) similarity, support backpropagation in a form that enables network training through differentiable DRR rendering, and allow for real-time processing during surgery through GPU optimization.

[0083] Next, it will be explained that the surgical support method according to the present embodiment is applied to fracture reduction surgery.

[0084] Fragment-Specific PolyPose (fracture tracking architecture) models each fragment of a fractured bone as an independent rigid body. By tracking each fragment with 6 degrees of freedom (3 rotational axes + 3 translational axes), it enables the individual estimation of the position and orientation of each fragment. Furthermore, by applying mass (m_k) and distance (d_k)-based weight fields to each bone fragment, weights are increased for closer points and decreased for further points, thereby ensuring stable tracking. Additionally, connectivity constraints (adjacent surface matching) can be applied to surface alignment constraints between adjacent fracture surfaces. By maintaining and adjusting for actual anatomical connectivity, unrealistic separation or overlap can be prevented. In other words, while each fracture fragment moves independently, constraints are applied to surfaces that must be physically attached, enabling stable multi-fragment tracking.

[0085] A real-time surgical support method and apparatus according to one embodiment of the present disclosure can provide an augmented reality overlay (AR Overlay Display). It can project a target location onto a surgical image, display the movement path of the current piece to the target location, and provide a warning when there is a risk of collision between bone pieces. The surgeon can intuitively compare the current state of the bone at the surgical site with the target location.

[0086] A deviation heatmap can be provided to indicate areas where the goal has been achieved and no further adjustment is needed in green (<1mm), areas requiring fine-tuning in yellow (1-3mm), and areas requiring additional correction measures in red (>3mm).

[0087] Clinical metrics can be provided to indicate the gap distance (<2mm), angular alignment tolerance (<5°), and length restoration tolerance (±5mm).

[0088] A surgical support method according to one embodiment of the present disclosure can help the surgeon reduce the bone quickly and accurately within a clinically acceptable range by independently tracking each bone fragment in multi-piece fracture surgery and providing real-time guidance based on AR and heatmaps.

[0089] Next, we will explain how the surgical support method according to the present embodiment is applied to spinal deformity correction surgery.

[0090] Through the Hierarchical PolyPose structure, tracking can be divided into a hierarchical structure because the spine is a complex structure composed of multiple connected segments rather than a single large bone. By applying spine alignment (C7-S1 axis) at the global level, the alignment status of the entire spine can be tracked, allowing for the overall monitoring of the spinal alignment extending from C7 to S1.

[0091] Regarding the Regional Level, Cervical, Thoracic, and Lumbar curves are applied to track the curvature of each segment of the cervical, thoracic, and lumbar spine. Regarding the Local Level, Individual vertebra (6 DoF each) is applied to track the 6 degrees of freedom (rotation + translation) of each individual vertebra, allowing for detailed verification of local deformations and misalignments. In other words, the spine can be tracked at a multi-scale by dividing it into the whole (alignment), parts (curvature), and details (individual bones).

[0092] Regarding surgical guidance features, pedicle screw placement guidance can be performed. For example, by setting the screw insertion starting point error to within 1 mm and the screw insertion path angle error to within 2 degrees, immediate notification can be provided if the depth is exceeded, and a warning can be issued if the screw penetrates the vertebral cortex. This enables real-time feedback on the safety of pedicle screw placement, which is clinically critical.

[0093] Regarding spinal deformity correction guidance, it provides real-time measurement capabilities to assess the degree of scoliosis and kyphosis, measure pelvic-spine balance and anterior-posterior balance indices, predict the machining of corrective metal rods, calculate patient-specific optimal curvature, predict load distribution during correction, and provide result prediction functions through simulation. In other words, it can quantitatively display the degree of spinal deformity during corrective surgery and provide real-time simulations on how to machine the rods.

[0094] The various embodiments of the present disclosure described above may be combined with additional embodiments and modified to the extent understandable to those skilled in the art in light of the detailed description above. The embodiments of the present disclosure are illustrative in all respects and should be understood as not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form. Accordingly, all modifications or variations derived from the meaning, scope, and equivalents of the claims of the present disclosure should be interpreted as being included within the scope of the present disclosure. Explanation of the symbols

[0095] 100: Computing device 110: Processor 120: Communication interface 130: Memory

Claims

Claim 1 A surgical support method using an intraoperative X-ray image performed by a computing device comprising at least one processor, comprising: a step of inputting an X-ray image of a surgical site taken during surgery into a neural network model; and a step of the neural network model estimating six-degree-of-freedom pose parameters of the bone of the surgical site based on the input X-ray image. The method comprises the step of generating a surgical guide image by aligning the estimated 6-degrees-of-freedom pose parameters using the estimated 6-degrees-of-freedom pose parameters and a differentiable DRR; wherein the neural network model is trained on pre-operative CT image data of the surgical site, and in the step of generating the surgical guide image, brightness-based alignment MNCC (Multiscale Normalized Cross-Correlation) and boundary-based alignment GNCC (Gradient Normalized Cross-Correlation) are applied, wherein the MNCC sets weights with equal weight for global alignment and local alignment in a multiscale manner and accelerates with a GPU kernel, and wherein the GNCC adjusts the degree of gradient smoothing and applies intensity invariant characteristics. Claim 2 A method according to claim 1, wherein the neural network model is trained to estimate the 6-degree-of-freedom pose parameters of the rigid body based on the preoperative CT image data and the preoperative CT image data, wherein the neural network model is a rigid body model of the surgical site bone extracted from the preoperative CT image data. Claim 3 A method according to claim 1, wherein, in the step of the neural network model estimating the 6-degree-of-freedom pose parameters, a rigid body transformation is applied. Claim 4 A method according to claim 1, wherein Adam series optimization is applied in the step of the neural network model estimating the 6-degree-of-freedom pose parameters. Claim 5 delete Claim 6 In a device that supports surgery using X-ray images during surgery, a processor; The system includes memory storing a surgical support program, and as the processor executes the program, an X-ray image of the surgical site captured during surgery is input into a neural network model; based on the input X-ray image, the neural network model estimates 6-degrees-of-freedom pose parameters of the bone at the surgical site; and generates a surgical guide image by aligning the estimated 6-degrees-of-freedom pose parameters using a differentiable DRR with the estimated 6-degrees-of-freedom pose parameters; the neural network model is trained on pre-operative CT image data of the surgical site; and generating the surgical guide image applies brightness-based alignment MNCC (Multiscale Normalized Cross-Correlation) and boundary-based alignment GNCC (Gradient Normalized Cross-Correlation); the MNCC sets weights with equal weight for global alignment and local alignment at a multiscale and accelerates with a GPU kernel; and the GNCC adjusts the degree of gradient smoothing and applies intensity invariant characteristics. A device with features.

Images