A surgical navigation system and method using a surface registration model supporting anatomical landmark prediction

Abstract

A surgical navigation system using a surface registration model according to an embodiment of the present invention comprises: a medical image generation unit that generates medical image spatial data based on point cloud data based on a medical image taken before surgery; a patient image generation unit that generates patient surface point cloud data based on a patient image taken during surgery and generates patient image spatial data based on the point cloud data on the patient surface; an adjustment registration unit that performs adjustment registration by matching major anatomical landmarks in the medical image space and the patient image space based on the generated medical image spatial data and the patient image spatial data; a surface registration model that predicts internal anatomical landmarks on the surface by training an artificial intelligence based on the generated medical image spatial data; and a weighted ICP precision registration unit that performs final precision registration using a weighted iterative nearest point algorithm based on the predicted internal anatomical landmarks and the initial registration result of the adjustment registration unit, wherein the predicted internal anatomical landmarks are provided between the adjustment registration unit and the weighted ICP precision registration unit.

Description

Technology Field

[0001] The present invention relates to a surgical navigation system and method using a surface registration model that supports anatomical landmark prediction, and more specifically, to a surgical navigation system and method using a surface registration model that supports anatomical landmark prediction capable of providing improved surface registration performance by generating various types of patient spatial point cluster data from preoperative medical images, training a deep learning model between the point cluster data and the location of internal anatomical structures, and integrating the location of internal anatomical structures estimated from the trained model with facial surface coordinate points. Background Technology

[0002] With the advancement of medical technology, new equipment is being introduced into operating rooms to improve patient treatment outcomes, and minimally invasive surgery is an effective method that minimizes patient burden, reduces bleeding, and shortens recovery time by performing surgery without large incisions. A surgical navigation system, which is essential for such minimally invasive surgery, enables precise surgical execution by visualizing surgical instruments in real-time on the patient's 3D medical image.

[0003] The core of a surgical navigation system is the accuracy of tracking surgical tools, and point registration and surface registration methods are used to accurately match 3D medical image data with the patient's anatomical spatial coordinates.

[0004] In conventional surgical navigation systems, surface registration aligns point cloud data acquired from preoperative medical images with point cloud data acquired from the patient during surgery, and this alignment was essential for the real-time accurate guidance of surgical tools based on the patient's anatomical structure.

[0005] Conventional surface registration consists of initial registration for initial alignment and precision registration for precise adjustment; in precision registration, the Iterative Closest Point (ICP) algorithm is primarily used to align surface data. The ICP algorithm matches adjacent points between surface point clouds and registers the point clouds by calculating a transformation matrix.

[0006] Conventional technology performs registration using only surface data, resulting in registration errors in deep anatomical structures. There is a problem in that registration errors are amplified as internal structures move further away from the surface due to the lever arm effect.

[0007] Accordingly, the surface registration process is divided into an initial registration stage and a precision registration stage, and if the accuracy of the initial registration is low, it affects the overall registration result.

[0008] Generally, landmark-based registration techniques attempt to improve registration accuracy by using specific anatomical landmarks, but the process of manually setting these landmarks is cumbersome.

[0009] In addition, existing surface registration methods require iterative and computationally intensive processes to minimize registration errors.

[0010] Conventional technology has proposed various algorithms to improve the accuracy of surface registration, but they have limitations in terms of automation and efficiency.

[0011] To solve these problems, it is necessary to develop a new technology that combines deep learning-based internal landmark prediction and weight-based ICP. Prior art literature

[0012] U.S. Patent No. 10217217 (Feb. 26, 2019) Korean Patent No. 10-2722709 (Oct. 23, 2024) Korean Published Patent No. 10-2024-0163849 (Nov. 19, 2024) The problem to be solved

[0013] The technical problem that the present invention aims to solve is to provide a surgical navigation system and method using a surface registration model that supports anatomical landmark prediction capable of improving registration accuracy by combining surface point cloud data and internal landmarks using a deep learning-based surface registration model.

[0014] The technical problem to be solved by the present invention is to provide a surgical navigation system and a method that can provide highly reliable alignment results by using Top-K bins features and Chamfer distance to improve reliability and consistency during the alignment process, and to secure stable and consistent alignment performance through a multi-stage alignment process including initial alignment, correction, and refinement.

[0015] Furthermore, the technical objective of the present invention is to provide a surgical navigation system and a method thereof that can collect patient surface data in real time and immediately provide alignment results to support a surgeon in performing surgery in real time, and can shorten surgery time and increase efficiency through a data-based system with high alignment accuracy.

[0016] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0017] To achieve the above technical objectives, a surgical navigation system using a surface registration model according to an embodiment of the present invention comprises: a medical image generation unit that generates medical image spatial data based on point cloud data based on a medical image taken before surgery; a patient image generation unit that generates patient surface point cloud data based on a patient image taken during surgery and generates patient image spatial data based on the point cloud data on the patient surface; an adjustment registration unit that performs adjustment registration by matching major anatomical landmarks in the medical image space and the patient image space based on the generated medical image spatial data and the patient image spatial data; a surface registration model that predicts internal anatomical landmarks on the surface by training an artificial intelligence based on the generated medical image spatial data; and a weighted ICP precision registration unit that performs final precision registration using a weighted iterative nearest point algorithm based on the predicted internal anatomical landmarks and the initial registration result of the adjustment registration unit, wherein the predicted internal anatomical landmarks are provided between the adjustment registration unit and the weighted ICP precision registration unit.

[0018] The surface registration model described above may include: a training data generation unit that generates training data including point cloud data of a patient's face surface in an initially registered state with a medical image-based face surface model based on the medical image spatial data; a bounding box generation unit that generates a bounding box including face surface data extracted from a medical image based on the generated training data; an anatomical landmark selection unit that selects anatomical landmarks to initially register curve data of each patient's face surface generated with the medical image-based face surface model; an initial bounding box registration module that performs alignment based on the generated bounding box and the selected anatomical landmarks to perform initial registration; a bounding box refinement module that refines the bounding box by combining the selected internal anatomical landmarks and surface point cloud data; a loss function calculation unit that calculates a composite loss function to optimize the accuracy of the initial registration and the precision of the refined bounding box in a balanced manner; a hyperparameter setting unit that sets hyperparameters to control the batch size and learning rate of the surface registration model; and a landmark prediction unit that predicts the internal anatomical landmarks for the patient's surface point cloud data.

[0019] The above anatomical landmark selection unit can initially align curve data of each patient's face surface with a fixed reference landmark of a medical image-based face surface model when the landmark is selected, thereby performing initial alignment in the medical image space.

[0020] The above initial bounding box matching module can extract spatial features of a point cloud by utilizing multi-scale set abstraction, generate multiple initial candidates for the rotation and transformation of the bounding box based on learned features, and generate candidates for the initial bounding box transformation based on multiple hypotheses after refining the feature vectors.

[0021] The bounding box refinement module can select the top K bins with high confidence for each candidate bounding box, calculate the confidence of the predicted transformation for each selected bin, assign a greater weight to the bins with high confidence, merge the features extracted from the initial bounding box matching module with the high-dimensional features of the point cloud data, and process the merged features using a fully connected layer.

[0022] The above loss function calculation unit can use bin-based loss to give greater weight to candidates with higher confidence, and calculate using Chamfer distance and mean squared error (MSE) as loss functions.

[0023] In addition, for achieving the aforementioned technical objectives, a surface registration method for a surgical navigation system using a surface registration model according to an embodiment of the present invention comprises: a step of capturing a medical image before surgery; a step in which the surgical navigation system generates medical image spatial data based on point cloud data based on the captured medical image; a step in which the surgical navigation system tracks the surface of a patient image during surgery based on a patient image captured during surgery; a step in which the surgical navigation system generates point cloud data of the patient's surface based on the tracked patient image surface data during surgery and generates patient image spatial data based on point cloud data on the patient surface; a step in which the surgical navigation system performs adjustment registration by aligning major anatomical landmarks in the medical image space and the patient image space based on the generated medical image spatial data and the patient image spatial data; a step in which the surgical navigation system inputs the generated medical spatial data into a surface registration model to track and train a medical image-based face surface model; and a step in which the surgical navigation system trains data using a surface registration model based on the trained medical image tracking data. The surgical navigation system includes the step of predicting internal anatomical landmarks based on the learned data; and the step of performing final precise alignment using a weighted iterative nearest point (ICP) algorithm based on the predicted internal anatomical landmarks and the adjusted initial alignment result.

[0024] The step of training data using the surface registration model described above may include: generating training data including point cloud data of a patient's face surface that is initially registered with a medical image-based face surface model based on the medical image spatial data; generating bounding boxes including face surface data extracted from a medical image based on the generated training data; selecting anatomical landmarks to initially register curve data of each patient's face surface generated with the medical image-based face surface model; registering the initial bounding boxes by performing alignment based on the generated bounding boxes and the selected anatomical landmarks; refining the bounding boxes by combining the selected internal anatomical landmarks and the surface point cloud data; calculating a composite loss function to optimize the accuracy of the initial registration and the precision of the refined bounding boxes in a balanced manner; setting hyperparameters to control the batch size and learning rate of the surface registration model; and predicting the internal anatomical landmarks for the patient's surface point cloud data.

[0025] The step of selecting the above anatomical landmarks allows for initial registration in the medical image space by aligning the curve data of each patient's face surface with fixed reference landmarks of the medical image-based face surface model once the landmarks are selected.

[0026] The step of matching the initial bounding box above can utilize multi-scale set abstraction to extract spatial features of the point cloud, generate multiple initial candidates for the rotation and transformation of the bounding box based on the learned features, and after refining the feature vector, generate candidates for the initial bounding box transformation based on multiple hypotheses.

[0027] The step of refining the bounding box above may include selecting the top K bins with high confidence for each candidate of the bounding box, calculating the confidence of the predicted transformation for each selected bin and assigning a greater weight to the bins with high confidence, merging the features extracted from the initial bounding box matching module with the high-dimensional features of the point cloud data, and processing the merged features using a fully connected layer.

[0028] The step of calculating the above loss function can be performed by using a bin-based loss to give greater weight to candidates with higher confidence, and using Chamfer distance and mean squared error (MSE) as the loss function. Effects of the invention

[0029] According to an embodiment of the present invention, the accuracy of the matching can be improved by combining surface point cloud data and internal landmarks using a deep learning-based surface matching model.

[0030] According to an embodiment of the present invention, highly reliable matching results are provided by using Top-K bins features and Chamfer distance to improve reliability and consistency during the matching process, and stable and consistent matching performance can be secured through a multi-stage matching process including initial matching, correction, and refinement.

[0031] In addition, according to an embodiment of the present invention, patient surface data can be collected in real time and alignment results can be provided immediately to support a surgeon in performing surgery in real time, and surgery time can be shortened and efficiency increased through a data-based system with high alignment accuracy.

[0032] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the composition of the invention described in the description or claims of the present invention. Brief explanation of the drawing

[0033] FIG. 1 is a block diagram illustrating the overall configuration of a surgical navigation system using a surface registration model that supports anatomical landmark prediction according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the detailed configuration of a surface matching model according to an embodiment of the present invention. FIG. 3 is an example diagram showing the initial alignment of a medical image and a converted patient surface bounding box according to an embodiment of the present invention. FIG. 4 is an exemplary diagram illustrating the bounding box generation step and the bounding box refinement step in a surface matching model according to an embodiment of the present invention. FIG. 5 is an example diagram showing the correlation between the target rotation angle, the predicted rotation angle, and the conversion value according to an embodiment of the present invention. Figure 6 is a graph showing the results of comparing the matching accuracy between the conventional method and the present invention for four phantoms according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a surface registration method of a surgical navigation system using a surface registration model that supports anatomical landmark prediction according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating a detailed learning method of a surface matching model according to an embodiment of the present invention. Specific details for implementing the invention

[0034] The present invention will be described below with reference to the attached drawings. However, the present invention may be implemented in various different forms and is therefore not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification have been given similar reference numerals.

[0035] Throughout the specification, when it is stated that a part is "connected (connected, in contact, combined)" with another part, this includes not only cases where they are "directly connected," but also cases where they are "indirectly connected" with other members interposed between them. Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for the inclusion of additional components.

[0036] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] Embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0038] FIG. 1 is a block diagram illustrating the overall configuration of a surgical navigation system using a surface registration model that supports anatomical landmark prediction according to an embodiment of the present invention.

[0039] As illustrated in FIG. 1, a surgical navigation system (100) using a surface registration model that supports anatomical landmark prediction may include a medical image generation unit (110), a patient image generation unit (120), an adjustment registration unit (130), a surface registration model (140), and a weighted ICP precision registration unit (150).

[0040] A surgical navigation system (100) using a surface registration model that supports anatomical landmark prediction can track the position of a surgical instrument and visually superimpose it onto a 3D medical image of the patient to reliably provide the position of the instrument during surgery through accurate registration.

[0041] First, the medical image generation unit (110) can receive medical images of a patient taken using medical equipment such as CT, MRI, etc., before surgery and generate medical image spatial data based on point cloud data.

[0042] Medical images are CT scan data acquired as 2D cross-sections, which can be reconstructed into a 3D iso-surface using the Marching Cube algorithm.

[0043] Point cloud data can be generated by extracting points representing the surface of the patient's face from this 3D model.

[0044] These medical imaging spatial data include patient anatomical structures and landmark information to perform initial registration and can be used as reference data for the surface registration model described later.

[0045] The patient image generation unit (120) receives an image of the patient taken using medical equipment (e.g., CT, MRI, etc.) during surgery, generates surface point cloud data of the patient, and can generate patient image space data based on the point cloud data on the patient surface.

[0046] Patient image spatial data contains information (e.g., coordinates) necessary to perform real-time registration during surgery.

[0047] Patient surface point cloud data can be acquired in real time using optical cameras, 3D scanners, infrared-based tracking systems, etc.

[0048] For example, in an operating room, point cloud data corresponding to the patient's anatomical space can be acquired through a navigation system using an optical camera and a passive probe. The camera tracks infrared markers attached to the probe to record the precise 3D coordinates of each marker and calculates the position of the probe tip to generate point cloud data of the patient's face surface.

[0049] The coarse registration unit (130) receives the generated medical image space data and patient image space data and performs coarse registration, i.e., initial registration, by matching major anatomical landmarks in the medical image space and patient image space to achieve approximate alignment.

[0050] For example, the transformation of the initial match can be calculated as in Equation 1.

[0051] [Mathematical Formula 1]

[0052]

[0053] Here, pi represents the coordinates of the i-th landmark in medical image space, and pi aligned represents the corresponding landmark coordinates in patient space after alignment. R is a 3x3 rotation matrix, and t is a 3x1 transformation vector. This transformation estimates the rotation and translation required to roughly align the two point cloud datasets. R and t are calculated using Singular Value Decomposition (SVD) for pairs of points. This process proceeds in the order of calculating the center points of both point cloud datasets, centering the points relative to the origin, calculating the cross-covariance matrix, and applying SVD; finally, the rotation matrix R and transformation vector t are obtained from the SVD results. Through this initial transformation, the point cloud data can be roughly aligned to establish initial conditions for precise alignment.

[0054] The surface registration model (140) can receive medical imaging spatial data as input and train artificial intelligence to predict internal anatomical landmarks so as to reduce the lever arm effect from the surface to the internal target area.

[0055] The surface registration model (140) can receive medical spatial data generated by the medical image generation unit, track a CT-based face surface model, and train it.

[0056] The surface matching model (140) is an artificial intelligence learning model, and various artificial intelligence learning models including deep learning can be used.

[0057] The above-mentioned predicted internal anatomical landmarks can be utilized in the surface registration process to improve the accuracy of surface registration.

[0058] Additionally, the surface registration model (140) can provide predicted landmarks for customized surface registration for each individual patient through model learning.

[0059] The landmark results predicted through this surface matching model (140) can be provided between the adjustment matching unit (130) and the weighted ICP precision matching unit (150) described later with reference to FIG. 1, and can affect the weighted ICP precision matching unit (150).

[0060] The specific configuration of the surface matching model (140) will be described later with reference to FIG. 2.

[0061] The weighted ICP precision matching unit (150) receives internal anatomical landmarks predicted by the surface matching model (140) and the initial matching result of the adjustment matching unit (130) as input, and uses a weighted iterative nearest point (ICP) algorithm to perform final precision matching.

[0062] This weight can increase the influence of internal anatomical landmarks during the registration process, thereby improving accuracy in deep regions.

[0063] The weighted ICP algorithm can be calculated as shown in Equation 2 according to double term optimization.

[0064] [Mathematical Formula 2]

[0065]

[0066] Here, R and t represent the rotation matrix and translation vector, and and is a surface point correspondence pair in patient space and medical imaging space. Also, and represent internal landmark correspondence pairs in patient space and medical image space, respectively. Weights It emphasizes the importance of internal landmarks over surface points, allowing internal landmarks to have a greater influence during the optimization process. The weighted optimization process minimizes matching errors by precisely adjusting transformations between point clouds, and can achieve high matching accuracy by integrating surface points and internal landmarks.

[0067] The surgical navigation system (100) using a surface registration model that supports anatomical landmark prediction according to the present invention provides higher accuracy and reliability than conventional registration methods by predicting internal landmarks and performing surface registration using artificial intelligence learning, improves the limitations of surface registration to reduce the registration error of lesions, and can improve surgical navigation systems by increasing precision, thereby shortening surgery time and improving safety.

[0068] FIG. 2 is a block diagram illustrating the detailed configuration of a surface matching model according to an embodiment of the present invention.

[0069] FIG. 3 is an example diagram showing the initial alignment of a medical image and a converted patient surface bounding box according to an embodiment of the present invention.

[0070] FIG. 4 is an exemplary diagram illustrating the bounding box generation step and the bounding box refinement step in a surface matching model according to an embodiment of the present invention.

[0071] As illustrated in FIG. 2, the surface matching model (140) may include a training data generation unit (141), a bounding box generation unit (142), an anatomical landmark selection unit (143), an initial bounding box matching module (144), a bounding box refinement module (145), a loss function calculation unit (146), a hyperparameter setting unit (147), and a landmark prediction unit (148).

[0072] The learning data generation unit (141) can receive medical image spatial data from the medical image generation unit and generate realistic learning data including point cloud data of the patient's face surface that is initially aligned with the medical image-based face surface model.

[0073] Through this, the surface matching model (140) can learn relative geometric information necessary for accurate prediction of internal anatomical landmarks.

[0074] The bounding box generation unit (142) can generate a bounding box that completely includes face surface data extracted from a medical image based on the training data generated by the training data generation unit (141).

[0075] A bounding box can be generated by calculating a convex hull of points on the surface of a medical image (e.g., CT). A convex hull is the smallest convex polygon that contains all given points and is the minimum boundary that encloses the entire facial structure by connecting the outermost points of the face. The bounding box calculated in this way can then be used as a consistent spatial reference frame to be aligned with curve data of the patient's face surface during the initial registration process.

[0076] In addition, the bounding box generating unit (142) can generate patient surface curve data.

[0077] This can generate a two-dimensional patient surface pattern from a CT-based facial surface model to simulate partial scans typically acquired in patient space using a surgical navigation system.

[0078] These patterns can be based on Fourier series adjusted to generate various shapes. Fourier series are an effective tool for analyzing periodic signals, and by treating the shape of the facial surface as a periodic signal, controlled variations can be applied to the Fourier series to represent various partial surface contours.

[0079] The generated 2D Fourier-based pattern can be placed near a CT-based facial surface model to approximate the visible area seen in a partial scan of the patient's face.

[0080] Afterward, the bounding box generating unit (142) can place a two-dimensional pattern near the CT surface and then perform additional processing to generate a realistic three-dimensional contour.

[0081] Each 2D pattern can be adjusted to fit the 3D surface curvature of a CT-based face surface model using the k-Nearest Neighbors (KNN) search algorithm. The KNN algorithm is effective for classifying new data points based on surrounding data points, and in this case, a 3D curve can be generated by mapping each point of the 2D pattern to the nearest point on the CT surface.

[0082] Thus, the bounding box generation unit (142) can generate curve data of a three-dimensional patient face surface that follows the actual contour of the face structure. Each curve generated through this process can realistically represent a partial patient surface with various geometric characteristics that can be seen in an actual partial scan.

[0083] The generation of the above patient surface curve data can be calculated as shown in Equation 3.

[0084] [Mathematical Formula 3]

[0085]

[0086] Referring to FIG. 2, the anatomical landmark selection unit (143) can select anatomical landmarks to initially align curve data of each patient's face surface generated with the CT-based face surface model.

[0087] First, the reference landmark selected on the CT mask can be fixed during the data generation process. Landmarks for the curve data of each patient's facial surface can be selected within a defined range of landmark selections around the reference landmark.

[0088] For example, the landmark selection range can be set between 1mm and 6mm. This reflects errors that may occur when manually selecting landmarks, and changes in the simulated landmark position may represent various errors that may occur in real-world situations.

[0089] Additionally, the anatomical landmark selection unit (143) can perform initial registration of patient surface curve data into CT space after anatomical landmark selection.

[0090] For example, once a landmark is set, as shown in Fig. 3 (b), curve data of each patient's face surface can be initially aligned in CT space by aligning it with a fixed reference landmark of the CT-based face surface model.

[0091] During the initial registration process, a transformation matrix including rotation and transformation components can be calculated. At this time, the same transformation matrix can be applied to the bounding box to generate a scene in which both the curve data of the patient's face surface and the bounding box are aligned with the CT reference frame.

[0092] This transformation can simulate the initial alignment phase in clinical scenarios where a partial scan is aligned with the entire CT reference frame based on anatomical landmarks.

[0093] Figure 3 (a) shows the tracing of the face surface coordinate plane (blue outline) in medical image space as a red line, and the internal feature points as blue stars. Figure 3 (b) shows the results of simulating the initial registration, where the red box represents the transformed patient surface and bounding box, and the landmark (red star) predicted through the internal feature points (blue stars) extracted by tracing the face surface coordinate plane (blue outline) in medical image space.

[0094] Additionally, the anatomical landmark selection unit (143) can configure a training dataset and set target values ​​after performing initial registration of patient surface curve data into CT space.

[0095] For example, the training dataset may consist of pairs of an initially matched partial patient surface and its corresponding transformed bounding box.

[0096] The forward transformation matrix calculated during the initial registration process is the target output. When a new patient surface is input into the trained surface registration model, the model can predict the forward transformation features necessary to accurately align the bounding box with the patient surface.

[0097] Since the relative positions of the bounding box corners and internal anatomical landmarks in the CT reference frame are known, internal anatomical landmarks on the patient surface can be inferred using the features of the transformed bounding box.

[0098] In this way, the surface registration model can be generalized to new patient surfaces by predicting transformations for accurately placing reference bounding boxes, and can specify accurate internal landmark locations based on the transformed bounding boxes.

[0099] The surface registration model (140) of the present invention solves the problem of predicting internal anatomical landmarks in a partial facial point cloud and can influence the initial registration module, which is an initial bounding box registration module and a weighted ICP precision registration unit. The initial bounding box registration module and the weighted ICP precision registration unit operate sequentially to align bounding boxes around the partial point cloud and can provide a stable reference point for predicting accurate anatomical landmark locations.

[0100] Referring to FIG. 2, the initial bounding box alignment module (144) can set the initial direction and position of the bounding box by performing approximate alignment before precise alignment.

[0101] The initial bounding box matching module (144) can generate various candidates for bounding box transformations by utilizing a multi-hypothesis-based approach to set and predict the initial direction and position of the bounding box.

[0102] First, the initial bounding box matching module (144) can extract spatial features of the point cloud through a multi-scale set abstraction technique inspired by PointNet++ by referring to the bounding box generation step of FIG. 4. This technique can learn local and global information by setting various radii (e.g., 0.05, 0.1, 0.15) to capture details of the partial point cloud at various spatial resolutions.

[0103] Through this process, both the regional detailed structure and the overall geometric pattern of the facial surface can be encompassed.

[0104] Next, the initial bounding box matching module (144) can generate multiple initial candidates for the rotation (R) and transformation (t) of the bounding box based on learned features. Each candidate includes various potential transformation values ​​of the bounding box and can be evaluated in the precise matching step.

[0105] Afterwards, the initial bounding box matching module (144) can further refine the feature vector through an additional set of abstraction layers including high-dimensional channel configurations.

[0106] For example, the extracted features can learn complex spatial information required for initial registration using larger radii (e.g., 0.1, 0.2) and high-dimensional channels. Through this, the model learns complex geometric characteristics of the face surface at local and global levels and can build a comprehensive feature representation to generate reliable initial bounding box registrations.

[0107] Next, after feature extraction is completed, the initial bounding box matching module (144) can generate multiple prediction values ​​including Euler angles for rotation of the bounding box and transformations along the x, y, and z axes.

[0108] At this time, the initial bounding box matching module (144) can generate various candidates for the initial bounding box transformation based on a multiple hypotheses-based approach.

[0109] This approach allows the model to output multiple potential bounding box transformations by providing various initial matching candidates to explore various possibilities, rather than relying on a single solution in the initial matching.

[0110] And, the extracted features can be processed into feature vectors using fully connected layers of 2048, 1024, 512, and 256 dimensions.

[0111] At this time, the initial bounding box matching module (144) can increase stability during the learning process and ensure efficient backpropagation by applying batch normalization and a Rectified Linear Unit (ReLU) activation function after each layer.

[0112] In addition, a dropout probability of 0.3 can be applied to the key layer to prevent overfitting.

[0113] Through this process, the initial bounding box matching module (144) effectively analyzes and compresses spatial features to generate nine bounding box matching hypotheses, which can provide the maximum pool of reliable candidates in the precise matching stage.

[0114] Referring to FIG. 3, the bounding box refinement module (145) can precisely refine the bounding box by combining the internal anatomical landmarks and the surface point cloud data.

[0115] This refinement process is intended to obtain a bounding box that accurately captures the facial surface, serving as a stable reference point for estimating the locations of anatomical landmarks.

[0116] Additionally, the bounding box refinement module (145) can integrate and utilize a feature set including Euler angles and transformation values ​​predicted by the initial bounding box matching module (144) and high-dimensional spatial features extracted from the point cloud to improve matching accuracy.

[0117] The bounding box refinement module (145) can precisely adjust the position and orientation of the initially created bounding box.

[0118] The bounding box refinement module can first receive bounding box candidates generated by the initial bounding box matching module (144), high-dimensional features of point cloud data, and internal landmark information, and calculate an optimal bounding box.

[0119] For example, the bounding box refinement module (145) can receive semantic features and Top-K Bins features as shown in FIG. 4 above.

[0120] Semantic features combine spatial features extracted from point cloud data with internal anatomical landmark data to learn the relationship between surface information and internal structure, thereby increasing the reliability of the registration process.

[0121] On the other hand, the Top-K Bins feature selects the top K bins with the highest confidence for each bounding box candidate, where each bin represents the confidence of the predicted transformation (e.g., rotation and translation values), and assigns greater weight to bins with higher confidence.

[0122] Subsequently, the bounding box refinement module (145) can precisely adjust the direction and position of the bounding box by selecting a highly reliable transformation value based on semantic features and Top-K Bins features as shown in FIG. 4 and utilizing the spatial relationship between the point cloud data and internal landmarks.

[0123] Subsequently, the bounding box refinement module (145) merges the Top-K Bins features extracted from the initial bounding box matching module (144) and the high-dimensional features of the point cloud data, which are semantic features, as shown in FIG. 4, and processes the merged features to finally select the bounding box in which the most reliable prediction value is prioritized.

[0124] These merged features can be used as basic data to accurately adjust the rotation (R) and transformation (t) of the bounding box, and can include the spatial context of the point cloud to utilize not only the initial bounding box configuration but also detailed geometric information of the face surface.

[0125] In addition, merged features are processed through fully connected layers with dropout of 256, 128, and 64 dimensions, and batch normalization and activation functions are applied to each layer to improve the stability and convergence efficiency of the learning process.

[0126] The final output of the bounding box refinement module (145) is the refined Euler angle (R) and transformation value (t) from the refined bounding box features shown in FIG. 4, which can provide the direction and position of the bounding box for the partial face point group.

[0127] These refined bounding boxes form reference points for consistently and reliably estimating anatomical landmarks and can improve registration accuracy.

[0128] Again, referring to FIG. 2, the loss function calculation unit (146) can calculate a composite loss function so that the surface matching model learning optimizes the initial matching accuracy and the precision of the refined bounding box in a balanced manner.

[0129] In the initial bounding box matching module (144) above, a bin-based loss may be used to evaluate the accuracy of each bounding box prediction. This loss function calculates the Smooth L1 loss between the predicted Euler angle and transform value of each bin and the reference transform value, and the loss of each bin is weighted according to the probability value of that bin. This approach can provide a suitable initial match in the refinement stage by prioritizing adjustments for bins with high confidence.

[0130] The loss function calculation unit (146) represents the Mean Squared Error (MSE) loss and can be calculated as in Equation 4 to measure the error between the model's predicted value and the true value, and represents the Chamfer Distance (CD) and can be calculated as in Equation 5 to be used to calculate the spatial discrepancy between two point groups.

[0131] [Mathematical Formula 4]

[0132]

[0134] [Mathematical Formula 5]

[0135]

[0136] In the refinement step, Chamfer distance and mean squared error (MSE) loss may be used.

[0137] Chamfer distance minimizes alignment errors by calculating the spatial mismatch between predicted bounding box corners and reference corners, and can effectively reflect complex spatial differences. Additionally, MSE loss is applied individually to refined Euler angles and transformation values, which can minimize errors in rotation and transformation.

[0138] This allows the bounding box to have an accurate orientation and position relative to the facial structure. These refinement losses work complementarily to precisely adjust the bounding box, providing registration results that accurately reflect the shape and location of the partial facial point cloud.

[0139] The total loss function used for model training consists of a weighted sum of bin-based loss, Chamfer distance, and mean squared error (MSE) loss, and can be designed to maintain a balance between the accuracy of the initial prediction and the refined match. This loss structure can optimize the contribution of each module to refine multiple bin candidates in the initial prediction stage and to fine-tune the match by reflecting the geometry of the point cloud in the refinement stage.

[0140] Referring to FIG. 2, the hyperparameter setting unit (147) can control the batch size and learning rate to support effective model training.

[0141] The hyperparameter setting unit (147) can control the speed and stability of the learning process by controlling the model's learning speed, data processing method, normalization method, etc., and can determine the complexity of the model required for problem solving by defining the structural characteristics of the model.

[0142] For example, to support effective learning, the batch size can be set to 20 to balance memory efficiency and gradient stability.

[0143] Learning is performed using the Adam optimizer with an initial learning rate of 0.001, ensuring stable convergence throughout the learning process.

[0144] The model is trained for 320 epochs, which can provide enough iterations to learn complex spatial relationships within a sub-point cloud and achieve robust and stable bounding box matching.

[0145] Referring to FIG. 2, the landmark prediction unit (148) can predict internal anatomical landmarks for patient surface point cloud data through the components of the surface matching model described above.

[0146] These predicted internal anatomical landmarks can be used for registration between the patient's anatomical structures and pre-operative medical images to improve the accuracy of the registration.

[0147] In addition, the predicted internal anatomical landmark data is provided to the weighted ICP precision alignment unit to enable final precision alignment.

[0148] FIG. 5 is an example diagram showing the correlation between the target rotation angle, the predicted rotation angle, and the conversion value according to an embodiment of the present invention.

[0149] Verification was performed to evaluate the surface matching performance using the surface matching model of the present invention by comparing it with conventional methods.

[0150] For example, a total of four head phantoms were used in the experiment and classified into two groups based on material properties and the presence or absence of a silicone mask. Phantom #1 (hard female) and Phantom #2 (hard male) were included in the hard phantom group without a silicone mask, while Phantom #3 (soft male) and Phantom #4 (soft female) were included in the soft phantom group covered with a silicone mask.

[0151] A target frame simulating an internal lesion is inserted into every phantom. The target frame is fabricated using a 3D printer and designed in a grid shape of 3 columns and 5 rows. The spacing between centerlines is set to 35 mm, and 10 target points are marked in each row at 10 mm intervals. Patient point cloud data is acquired using an optical tracking system.

[0152] This system consists of four optical cameras and a passive surgical probe and is used to accurately and reliably acquire point cloud data in a surgical environment. Ten independent experiments are performed for each phantom, and in each experiment, different patient surface point clouds and patient landmark selections are applied to increase the reliability of the evaluation.

[0153] Statistical analysis is used to compare the target match error (TRE) and surface match error (SRE) between the surgical navigation system using the surface match model of the present invention and a conventional surgical navigation system. Statistical significance is evaluated through a paired-samples t-test, and all analyses are performed using PASW Statistics version 18. The significance levels are set to p < 0.05 and p < 0.01.

[0154] The Surface Registration Error (SRE) between the above-mentioned conventional surgical navigations can be calculated as shown in Equation 6, and the Target Registration Error (TRE) between the above-mentioned conventional surgical navigations can be calculated as shown in Equation 7.

[0155] [Mathematical Formula 6]

[0156]

[0158] [Mathematical Formula 7]

[0159]

[0160] FIG. 5 shows rotation (to verify how high the agreement of the model matching result of the present invention is with the target data) The correlation was plotted by comparing the accuracy of , θ, ψ) and the transformation (x, y, z), respectively.

[0161] This displays the correlation between the actual value (reference value) and the model prediction value of the present invention in the form of a graph based on each data point, and rotation ( Independent correlations were visually shown for each of , θ, ψ) and transformation (x, y, z).

[0162] First, the rotation angle ( The result for , θ, ψ is the correlation coefficient R between the predicted rotation value and the target rotation value, as shown in Fig. 5. 2 All values ​​were very high at 0.99 and showed a strong linear correlation.

[0163] Accordingly, it can be confirmed that the surface matching model of the present invention predicts the target rotation angle very accurately.

[0164] Next, the result for the transformation values ​​(x, y, z) is R for each axis (x, y, z) as shown in FIG. 5. 2 The values ​​were found to be 0.96, 0.97, and 0.98, and the average R² 2 The value was calculated as 0.98, confirming that the model of the present invention predicted the target position transformation with high accuracy.

[0165] Therefore, the surface registration model of the present invention provides a very high level of prediction accuracy for target values ​​and demonstrates that it is particularly effective in minimizing registration errors of rotation and transformation elements, confirming that it is an important component in achieving accurate registration of anatomical structures in surgical navigation systems.

[0166] Figure 6 is a graph showing the results of comparing the matching accuracy between the conventional method and the present invention for four phantoms according to an embodiment of the present invention.

[0167] As a result of comparing the alignment accuracy of the conventional method and the method of the present invention for the four phantoms described in Fig. 5, Fig. 6 visually shows experimental results demonstrating how much the surface alignment model of the present invention provides improved alignment accuracy compared to the conventional method.

[0168] The conventional method is to perform target registration using existing point cloud registration methods (e.g., ICP).

[0169] The conventional method and the method of the present invention can compare matching accuracy based on the Target Registration Error (TRE).

[0170] Referring to Fig. 6, for Phantom #1, the median target registration error (TRE) of the conventional method (left graph) was found to be 2.05 ± 1.21 mm, and the method of the present invention (right graph) was measured to have a TRE of 1.25 ± 0.59 mm (p < 0.01).

[0171] In Phantom #2, the TRE of the conventional method (left graph) is measured to be 1.65 ± 0.98 mm, and the TRE of the method of the present invention (right graph) is measured to be 1.15 ± 0.45 mm (p < 0.01).

[0172] In Phantom #3, the conventional method (left graph) showed a TRE of 2.25 ± 1.42 mm, and the method of the present invention (right graph) was measured to have a TRE of 1.18 ± 0.89 mm (p < 0.01).

[0173] In Phantom #4, the conventional method (left graph) showed a TRE of 2.75 ± 0.89 mm, while the method of the present invention (right graph) was measured to have a TRE of 1.58 ± 0.52 mm (p < 0.01). The interquartile range of Phantom #4 decreased from 1.82 mm in the conventional method (left graph) to 0.95 mm in the method of the present invention (right graph). The data range (excluding outliers) was 1.45 to 4.65 mm for the conventional method, and was evaluated to be 0.25 to 2.55 mm for the method of the present invention. Similar reduction in the data range was observed in all phantoms, and statistical significance of p < 0.01 was confirmed.

[0174] This confirms that the present invention demonstrates consistent alignment performance by reducing data variance across all fandoms and significantly improves alignment accuracy and consistency by showing a more uniform error distribution.

[0175] FIG. 7 is a flowchart illustrating a surface registration method of a surgical navigation system using a surface registration model that supports anatomical landmark prediction according to an embodiment of the present invention.

[0176] In step (S110), the patient's preoperative medical images are prepared by scanning them using medical equipment such as CT or MRI, and the prepared data can be provided to a surgical navigation system.

[0177] In a subsequent step (S120), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can input the provided data into a medical image generation unit to generate medical image spatial data based on point cloud data.

[0178] First, medical images are CT scan data acquired as 2D cross-sections, which can be reconstructed into 3D iso-surfaces using the Marching Cube algorithm.

[0179] Point cloud data can be generated by extracting points representing the surface of the patient's face from this 3D model.

[0180] These medical imaging spatial data include patient anatomical structures and landmark information to perform initial registration and can be used as reference data for the surface registration model described later.

[0181] In step (S130), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can track the surface of the patient image during surgery based on the patient image captured during surgery.

[0182] In step (S140), a surgical navigation system using a surface registration model that supports anatomical landmark prediction inputs the tracked intraoperative patient image surface data into a patient image generation unit to generate patient surface point cloud data, and can generate patient image spatial data based on the point cloud data on the patient surface.

[0183] Patient image spatial data contains information (e.g., coordinates) necessary to perform real-time registration during surgery.

[0184] Patient surface point cloud data can be acquired in real time using optical cameras, 3D scanners, infrared-based tracking systems, etc.

[0185] For example, in an operating room, point cloud data corresponding to the patient's anatomical space can be acquired through a navigation system using an optical camera and a passive probe. The camera tracks infrared markers attached to the probe to record the precise 3D coordinates of each marker and calculates the position of the probe tip to generate point cloud data of the patient's face surface.

[0186] In step (S150), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can perform coarse registration, i.e., initial registration, by inputting the generated medical image space data and patient image space data into a coarse registration unit to match major anatomical landmarks in the medical image space and the patient image space to achieve approximate alignment.

[0187] For example, the transformation of the initial match can be calculated as in the aforementioned mathematical formula 1.

[0188] In step (S160), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can receive medical spatial data generated by the medical image generation unit into the surface registration model, track a CT-based face surface model, and train it.

[0189] These surface matching models are artificial intelligence learning models, and various artificial intelligence learning models, including deep learning, can be used.

[0190] In step (S170), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can learn data to predict internal anatomical landmarks using the surface registration model based on the medical image tracking data learned with reference to FIG. 7.

[0191] The specific training method of the surface matching model will be described later with reference to Fig. 8.

[0192] In step (S180), a surgical navigation system using a surface registration model that supports anatomical landmark prediction can predict internal anatomical landmarks based on the learned data.

[0193] The surface registration model receives medical imaging spatial data as input and trains artificial intelligence to predict internal anatomical landmarks, thereby reducing the lever arm effect from the surface to the internal target region.

[0194] Predicted internal landmarks can be utilized in the surface registration process to improve the accuracy of surface registration.

[0195] In addition, the surface registration model can provide predicted landmarks for personalized surface registration for each individual patient through model training.

[0196] Step (S190) A surgical navigation system using a surface registration model that supports anatomical landmark prediction can input the predicted internal anatomical landmark data and the initial registration result of the adjustment registration unit into a weighted ICP precision registration unit and perform final precision registration using an iterative nearest point (ICP) algorithm.

[0197] This weight can increase the influence of internal anatomical landmarks during the registration process, thereby improving accuracy in deep regions.

[0198] In addition, the weighted ICP algorithm can be calculated as shown in Equation 2 above according to double term optimization.

[0199] Therefore, the surgical navigation system using the surface registration model that supports anatomical landmark prediction according to the present invention provides higher accuracy and reliability than conventional registration methods by predicting internal landmarks and performing surface registration using artificial intelligence learning. It also has the advantage of improving the limitations of surface registration to reduce lesion registration errors and increasing precision in the surgical navigation system, thereby shortening surgery time and enhancing safety.

[0200] FIG. 8 is a flowchart illustrating a detailed learning method of a surface matching model according to an embodiment of the present invention.

[0201] Although the surface registration model is not illustrated in FIG. 8, it can first receive medical image spatial data from the medical image generation unit, input it into the training data generation unit, and generate realistic training data including point cloud data of the patient's face surface that is initially registered with the medical image-based face surface model.

[0202] Through this, the surface registration model can learn the relative geometric information necessary for accurate prediction of internal anatomical landmarks.

[0203] In step (S171), the surface matching model inputs the training data generated by the training data generation unit into the bounding box generation unit to generate a bounding box that completely includes the face surface data extracted from the medical image.

[0204] A bounding box can be generated by calculating a convex hull of points on the surface of a medical image (e.g., CT). A convex hull is the smallest convex polygon that contains all given points and is the minimum boundary that encloses the entire facial structure by connecting the outermost points of the face. The bounding box calculated in this way can then be used as a consistent spatial reference frame to be aligned with curve data of the patient's face surface during the initial registration process.

[0205] In addition, the bounding box generation unit can generate patient surface curve data.

[0206] This can generate a two-dimensional patient surface pattern from a CT-based facial surface model to simulate partial scans typically acquired in patient space using a surgical navigation system.

[0207] These patterns can be based on Fourier series adjusted to generate various shapes. Fourier series are an effective tool for analyzing periodic signals, and by treating the shape of the facial surface as a periodic signal, controlled variations can be applied to the Fourier series to represent various partial surface contours.

[0208] The generated 2D Fourier-based pattern can be placed near a CT-based facial surface model to approximate the visible area seen in a partial scan of the patient's face.

[0209] Afterwards, the bounding box generation unit can place a 2D pattern near the CT surface and then perform additional processing to generate a realistic 3D contour.

[0210] Each 2D pattern can be adjusted to fit the 3D surface curvature of a CT-based face surface model using the k-Nearest Neighbors (KNN) search algorithm. The KNN algorithm is effective for classifying new data points based on surrounding data points, and in this case, a 3D curve can be generated by mapping each point of the 2D pattern to the nearest point on the CT surface.

[0211] This bounding box generation step can generate curve data of the 3D patient face surface that follows the actual contours of the facial structure. Each curve generated through this process can realistically represent the partial patient surface with various geometric characteristics visible in actual partial scans.

[0212] The generation of the above patient surface curve data can be calculated as in the aforementioned mathematical formula 3.

[0213] In step (S172), the surface registration model can select anatomical landmarks to initially register the curve data of each patient's face surface generated with the CT-based face surface model in the anatomical landmark selection section with reference to FIG. 8.

[0214] First, the reference landmark selected on the CT mask can be fixed during the data generation process. Landmarks for the curve data of each patient's facial surface can be selected within a defined range of landmark selections around the reference landmark.

[0215] For example, the landmark selection range can be set between 1mm and 6mm. This reflects errors that may occur when manually selecting landmarks, and changes in the simulated landmark position may represent various errors that may occur in real-world situations.

[0216] The anatomical landmark selection unit can perform initial registration of patient surface curve data into CT space after anatomical landmark selection.

[0217] For example, once a landmark is set, as illustrated in the aforementioned Fig. 3, curve data of each patient's face surface can be initially aligned with fixed reference landmarks of the CT-based face surface model to achieve initial alignment in CT space.

[0218] During the initial registration process, a transformation matrix including rotation and transformation components can be calculated. At this time, the same transformation matrix can be applied to the bounding box to generate a scene in which both the curve data of the patient's face surface and the bounding box are aligned with the CT reference frame.

[0219] This transformation can simulate the initial alignment phase in clinical scenarios where a partial scan is aligned with the entire CT reference frame based on anatomical landmarks.

[0220] For the initial registration of patient surface curve data into CT space, refer to the aforementioned Fig. 3. As shown in Fig. 3 (a), tracing can be indicated by a red line on the face surface coordinate plane (blue outline) in medical image space, and internal feature points can be indicated by blue stars. Additionally, as shown in Fig. 3 (b), the result of simulating the initial registration is displayed. The red box represents the transformed patient surface and bounding box, and the landmark (red star) predicted through the internal feature points (blue stars) extracted by tracing the face surface coordinate plane (blue outline) in medical image space is indicated.

[0221] In addition, the anatomical landmark selection unit can configure a training dataset and set target values ​​after performing initial registration of patient surface curve data into CT space.

[0222] For example, the training dataset may consist of pairs of an initially matched partial patient surface and its corresponding transformed bounding box.

[0223] The forward transformation matrix calculated during the initial registration process is the target output. When a new patient surface is input into the trained surface registration model, the model can predict the forward transformation features necessary to accurately align the bounding box with the patient surface.

[0224] Since the relative positions of the bounding box corners and internal anatomical landmarks in the CT reference frame are known, internal anatomical landmarks on the patient surface can be inferred using the features of the transformed bounding box.

[0225] In this way, the surface registration model can be generalized to new patient surfaces by predicting transformations for accurately placing reference bounding boxes, and can specify accurate internal landmark locations based on the transformed bounding boxes.

[0226] Thus, the surface registration model of the present invention solves the problem of predicting internal anatomical landmarks in a partial facial point cloud and can influence the initial registration module, which is the initial bounding box registration module and the weighted ICP precision registration unit. The initial bounding box registration module and the weighted ICP precision registration unit operate sequentially to align bounding boxes around the partial point cloud and can provide a stable reference point for predicting accurate anatomical landmark locations.

[0227] In step (S173), the surface alignment model can perform an initial bounding box alignment by performing a rough alignment before precise alignment in the initial bounding box alignment module with reference to FIG. 8, thereby setting the initial direction and position of the bounding box.

[0228] This initial bounding box matching module can generate various candidates for bounding box transformations by utilizing a multi-hypothesis-based approach to set and predict the initial orientation and position of the bounding box.

[0229] First, the initial bounding box matching module can extract spatial features of a point cloud through a multi-scale set abstraction technique inspired by PointNet++ by referring to the bounding box generation step of Fig. 4 described above. This technique can learn local and global information by setting various radii (e.g., 0.05, 0.1, 0.15) to capture details of the partial point cloud at various spatial resolutions.

[0230] Through this process, both the regional detailed structure and the overall geometric pattern of the facial surface can be encompassed.

[0231] Next, the initial bounding box matching module can generate multiple initial candidates for the rotation (R) and transformation (t) of the bounding box based on learned features. Each candidate includes various potential transformation values ​​of the bounding box and can be evaluated during the precise matching step.

[0232] Subsequently, the initial bounding box matching module can further refine the feature vectors through an additional set abstraction layer that includes high-dimensional channel configurations.

[0233] For example, the extracted features can learn complex spatial information required for initial registration using larger radii (e.g., 0.1, 0.2) and high-dimensional channels. Through this, the model learns complex geometric characteristics of the face surface at local and global levels and can build a comprehensive feature representation to generate reliable initial bounding box registrations.

[0234] Next, after feature extraction is completed, the initial bounding box matching module can generate multiple prediction values ​​including Euler angles for rotating the bounding box and transformations along the x, y, and z axes.

[0235] At this time, the initial bounding box matching module can generate various candidates for the initial bounding box transformation based on a multiple hypotheses-based approach.

[0236] This approach allows the model to output multiple potential bounding box transformations by providing various initial matching candidates to explore various possibilities, rather than relying on a single solution in the initial matching.

[0237] And, the extracted features can be processed into feature vectors using fully connected layers of 2048, 1024, 512, and 256 dimensions.

[0238] At this time, the initial bounding box matching module can apply batch normalization and the ReLU activation function after each layer to increase stability during the training process and ensure efficient backpropagation.

[0239] In addition, a dropout probability of 0.3 can be applied to the key layer to prevent overfitting.

[0240] Through this process, the initial bounding box matching module effectively analyzes and compresses spatial features to generate nine bounding box matching hypotheses, which can provide the maximum pool of reliable candidates during the precise matching stage.

[0241] In step (S174), the surface matching model can precisely refine the bounding box by combining the internal anatomical landmarks and the surface point cloud data in the bounding box refinement module with reference to FIG. 8.

[0242] This refinement process is intended to obtain a bounding box that accurately captures the facial surface, serving as a stable reference point for estimating the locations of anatomical landmarks.

[0243] In addition, to improve matching accuracy, the bounding box refinement module can integrate and utilize high-dimensional spatial features extracted from point clouds with a feature set including Euler angles and transform values ​​predicted by the initial bounding box matching module.

[0244] The bounding box refinement module can precisely adjust the position and orientation of the initially generated bounding box.

[0245] The bounding box refinement module can first receive bounding box candidates generated by the initial bounding box matching module, high-dimensional features of point cloud data, and internal landmark information as input to calculate the optimal bounding box.

[0246] For example, the bounding box refinement module can receive semantic features and Top-K Bins features as input, as shown in the aforementioned Fig. 4.

[0247] Semantic features combine spatial features extracted from point cloud data with internal anatomical landmark data to learn the relationship between surface information and internal structure, thereby increasing the reliability of the registration process.

[0248] On the other hand, the Top-K Bins feature selects the top K bins with the highest confidence for each bounding box candidate, where each bin represents the confidence of the predicted transformation (e.g., rotation and translation values), and assigns greater weight to bins with higher confidence.

[0249] Subsequently, the bounding box refinement module can precisely adjust the direction and position of the bounding box by selecting a highly reliable transformation value based on semantic features and Top-K Bins features as described in Figure 4 above, and utilizing the spatial relationship between the point cloud data and internal landmarks.

[0250] Subsequently, the bounding box refinement module merges the Top-K Bins features extracted from the initial bounding box matching module and the high-dimensional features of the point cloud data, which are semantic features, as illustrated in Figure 4 above, and processes the merged features to finally select the bounding box in which the most reliable prediction value is prioritized.

[0251] These merged features can be used as basic data to accurately adjust the rotation (R) and transformation (t) of the bounding box, and can include the spatial context of the point cloud to utilize not only the initial bounding box configuration but also detailed geometric information of the face surface.

[0252] Merged features are processed through fully connected layers with dropout of 256, 128, and 64 dimensions, and batch normalization and activation functions are applied to each layer to improve the stability and convergence efficiency of the learning process.

[0253] The final output of the bounding box refinement module is the refined Euler angle (R) and transformation value (t) from the refined bounding box features shown in FIG. 4 above, which can provide the direction and position of the bounding box for the partial face point cloud.

[0254] These refined bounding boxes form reference points for consistently and reliably estimating anatomical landmarks and can improve registration accuracy.

[0255] In step (S175), the surface matching model can calculate a composite loss function in the loss function calculation unit with reference to FIG. 8 so that the surface matching model learning optimizes the initial matching accuracy and the precision of the refined bounding box in a balanced manner.

[0256] In the initial bounding box matching step described above, a bin-based loss can be used to evaluate the accuracy of each bounding box prediction. This loss function calculates the Smooth L1 loss between the predicted Euler angles and transform values ​​of each bin and the reference transform value, and the loss for each bin is weighted according to the probability value of that bin. This approach can provide a suitable initial match in the refinement step by prioritizing adjustments for bins with high confidence.

[0257] In addition, the loss function calculation unit represents the Mean Squared Error (MSE) loss and can be calculated as in the aforementioned Equation 4 to measure the error between the model's predicted value and the true value, and represents the Chamfer Distance (CD) and can be calculated as in the aforementioned Equation 5 to be used to calculate the spatial discrepancy between two point clouds.

[0258] This allows the bounding box to have an accurate orientation and position relative to the facial structure. These refinement losses work complementarily to precisely adjust the bounding box, providing registration results that accurately reflect the shape and location of the partial facial point cloud.

[0259] In addition, the total loss function used for model training consists of a weighted sum of bin-based loss, Chamfer distance, and mean squared error (MSE) loss, and can be designed to maintain a balance between the accuracy of the initial prediction and the refined match. This loss structure can optimize the contribution of each module to refine multiple bin candidates in the initial prediction stage and to fine-tune the match by reflecting the geometry of the point cloud in the refinement stage.

[0260] In step (S176), the surface matching model can set hyperparameters in the hyperparameter setting unit to support effective model training with reference to FIG. 8.

[0261] The hyperparameter setting section can control the batch size and learning rate to support effective model training.

[0262] The hyperparameter settings section can control the speed and stability of the learning process by controlling the model's learning speed, data processing method, and normalization method, and can determine the complexity of the model required for problem solving by defining the structural characteristics of the model.

[0263] For example, to support effective learning, the batch size can be set to 20 to balance memory efficiency and gradient stability.

[0264] Learning is performed using the Adam optimizer with an initial learning rate of 0.001, ensuring stable convergence throughout the learning process.

[0265] The model is trained for 320 epochs, which can provide enough iterations to learn complex spatial relationships within a sub-point cloud and achieve robust and stable bounding box matching.

[0266] In step (S177), the surface registration model can predict internal anatomical landmarks for the patient surface point cloud data through the components of the surface registration model described above in the landmark prediction unit.

[0267] These predicted internal anatomical landmarks can be used for registration between the patient's anatomical structures and pre-operative medical images to improve the accuracy of the registration.

[0268] In addition, the predicted internal anatomical landmark data can be provided to the weighted ICP precision registration step to achieve final precision registration.

[0269] Therefore, by utilizing the surface registration model learning of the present invention to precisely register point cloud data and medical image data through the initial bounding box generation and refinement, and by presenting a process that maximizes registration accuracy from the generation of training data to the optimization of the loss function, the systematic nature of the registration process and the learning efficiency of the model are emphasized, thereby offering the advantage of enhancing the accuracy and reliability of the surgical navigation system.

[0270] In particular, through internal landmark-based alignment correction and real-time alignment, it can ensure precision and safety in surgical procedures, increase surgical success rates, contribute to shortening the patient recovery process, and can be utilized in various medical environments.

[0271] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0272] The scope of the present invention is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present invention.

Claims

Claim 1 A surgical navigation system using a surface registration model comprises: a medical image generation unit that generates medical image spatial data based on point cloud data based on a medical image taken before surgery; a patient image generation unit that generates patient surface point cloud data based on a patient image taken during surgery and generates patient image spatial data based on the point cloud data on the patient surface; an adjustment registration unit that performs adjustment registration by matching major anatomical landmarks in the medical image space and the patient image space based on the generated medical image spatial data and the patient image spatial data; a surface registration model that predicts internal anatomical landmarks on the surface by training an artificial intelligence based on the generated medical image spatial data; and a weighted ICP precision registration unit that performs final precision registration using a weighted iterative nearest point algorithm based on the predicted internal anatomical landmarks and the initial registration result of the adjustment registration unit, wherein the predicted internal anatomical landmarks are provided between the adjustment registration unit and the weighted ICP precision registration unit, thereby supporting anatomical landmark prediction. Claim 2 In claim 1, the surface registration model comprises: a training data generation unit that generates training data including point cloud data of a patient's face surface in an initially registered state with a medical image-based face surface model based on the medical image spatial data; a bounding box generation unit that generates a bounding box including face surface data extracted from a medical image based on the generated training data; an anatomical landmark selection unit that selects anatomical landmarks to initially register curve data of each patient's face surface generated with the medical image-based face surface model; an initial bounding box registration module that performs alignment based on the generated bounding box and the selected anatomical landmarks to perform initial registration; a bounding box registration module that registers the bounding box by combining the selected internal anatomical landmarks and surface point cloud data; a loss function calculation unit that calculates a composite loss function to optimize the accuracy of the initial registration and the precision of the registered bounding box in a balanced manner; and a hyperparameter setting unit that sets hyperparameters to control the batch size and learning rate of the surface registration model. A surgical navigation system using a surface registration model that supports anatomical landmark prediction, comprising a landmark prediction unit that predicts internal anatomical landmarks for patient surface point cloud data. Claim 3 In paragraph 2, the anatomical landmark selection unit, when the landmark is selected, aligns the curve data of each patient's face surface with a fixed reference landmark of a medical image-based face surface model to perform initial registration in the medical image space, thereby supporting a surgical navigation system using a surface registration model that supports anatomical landmark prediction. Claim 4 In paragraph 2, the initial bounding box matching module utilizes multi-scale set abstraction to extract spatial features of a point cloud, generates multiple initial candidates for the rotation and transformation of bounding boxes based on learned features, and, after refining the feature vectors, generates candidates for the initial bounding box transformation based on multiple hypotheses, thereby providing a surgical navigation system using a surface matching model that supports anatomical landmark prediction. Claim 5 In paragraph 2, the bounding box refinement module selects the top K bins with high confidence for each candidate bounding box, calculates the confidence of the predicted transformation for each selected bin, assigns a greater weight to the bins with high confidence, merges the features extracted from the initial bounding box matching module with the high-dimensional features of the point cloud data, and processes the merged features using a fully connected layer, thereby providing a surgical navigation system using a surface matching model that supports anatomical landmark prediction. Claim 6 In paragraph 2, the loss function calculation unit uses a bin-based loss to assign greater weight to candidates with higher confidence, and calculates using Chamfer distance and mean squared error (MSE) as loss functions, thereby providing a surgical navigation system using a surface registration model that supports anatomical landmark prediction. Claim 7 A surface registration method for a surgical navigation system using a surface registration model comprises: a step of capturing a medical image prior to surgery; a step in which the surgical navigation system generates medical image spatial data based on point cloud data based on the captured medical image; a step in which the surgical navigation system tracks the surface of a patient image during surgery based on a patient image captured during surgery; a step in which the surgical navigation system generates point cloud data of the patient's surface based on the tracked patient image surface data during surgery and generates patient image spatial data based on point cloud data on the patient surface; a step in which the surgical navigation system performs adjustment registration by aligning major anatomical landmarks in the medical image space and the patient image space based on the generated medical image spatial data and the patient image spatial data; and a step in which the surgical navigation system inputs the generated medical spatial data into a surface registration model to track and train a medical image-based face surface model. A surface registration method of a surgical navigation system using a surface registration model that supports anatomical landmark prediction, comprising: a step of the surgical navigation system learning data using a surface registration model based on the medical image tracking data learned above; a step of the surgical navigation system predicting internal anatomical landmarks based on the learned data; and a step of the surgical navigation system performing final precise registration using a weighted iterative nearest point (ICP) algorithm based on the predicted internal anatomical landmarks and the initial registration result adjusted by the surgical navigation system. Claim 8 In claim 7, the step of training data using the surface registration model comprises: generating training data including point cloud data of a patient face surface that is initially registered with a medical image-based face surface model based on the medical image spatial data; generating a bounding box including face surface data extracted from a medical image based on the generated training data; selecting anatomical landmarks to initially register curve data of each patient face surface generated with the medical image-based face surface model; registering the initial bounding box by performing alignment based on the generated bounding box and the selected anatomical landmarks; refining the bounding box by combining the selected internal anatomical landmarks and the surface point cloud data; calculating a composite loss function to optimize the accuracy of the initial registration and the precision of the refined bounding box in a balanced manner; setting hyperparameters to control the batch size and learning rate of the surface registration model; and predicting the internal anatomical landmarks for the patient surface point cloud data. Claim 9 In claim 8, the step of selecting the anatomical landmark involves, when the landmark is selected, aligning the curve data of each patient's face surface with a fixed reference landmark of a medical image-based face surface model to perform initial registration in the medical image space, thereby a surface registration method of a surgical navigation system using a surface registration model that supports anatomical landmark prediction. Claim 10 In claim 8, the step of matching the initial bounding box involves extracting spatial features of a point cloud using multi-scale set abstraction, generating multiple initial candidates for the rotation and transformation of the bounding box based on learned features, and, after refining the feature vector, generating candidates for the initial bounding box transformation based on multiple hypotheses, thereby forming a surface matching method for a surgical navigation system using a surface matching model that supports anatomical landmark prediction. Claim 11 In claim 8, the step of refining the bounding box comprises selecting the top K bins with high confidence for each candidate of the bounding box, calculating the confidence of the predicted transformation for each selected bin, giving greater weight to the bins with high confidence, merging the features extracted from the initial bounding box matching module with the high-dimensional features of the point cloud data, and processing the merged features using a fully connected layer, thereby a surface matching method for a surgical navigation system using a surface matching model that supports anatomical landmark prediction. Claim 12 In claim 8, the step of calculating the loss function is to use a bin-based loss to give greater weight to candidates with higher confidence, and to calculate using Chamfer distance and mean squared error (MSE) as the loss function. Surface registration method of a surgical navigation system using a surface registration model that supports anatomical landmark prediction.

Images