2D image-based automated surgical planning method and system

Abstract

A two-dimensional (2D) image-based surgical planning method and system therefor is disclosed, which includes the steps of acquiring a first image and a second image of a spinal surgical site, defining a virtual three-dimensional (3D) surgical space and aligning the first image with the second image, extracting a first vertebra region and a first pedicle region from the first image, extracting a second vertebra region and a second pedicle region within the second vertebra region from the second image, setting a first screw path that passes through the center or near the center of the first pedicle region in the first image, and setting an entry point and a target point on a second screw path that corresponds to the first screw path in the second pedicle region of the second image based on the first screw path.

Description

[Technical Field]

[0001] The present invention relates to an automated surgical planning method and system for a surgical robot system, and more particularly to a 2D image-based automated surgical planning method and system that can generate a surgical planning path in a 3D (three-dimensional) image space using multiple 2D (two-dimensional) images. [Background technology]

[0002] A spinal surgical procedure, such as a pedicle screw fixation surgery using a surgical robot system, requires a 3D image capable of an axial view. Therefore, a 3D image capable of an axial view, such as a computed tomography (CT) image, is used as the axial view image, rather than a 2D image.

[0003] Typically, during intra-operation, a 2D image acquisition device, such as a C-arm, is used to verify in real time whether the surgery is proceeding according to the pre-operative plan. To navigate surgical tools according to the pre-planned plan using the acquired 2D images, the 2D images acquired during surgery must be registered with the 3D images acquired pre-operatively. However, the process of registering the 2D images with the 3D images can cause problems such as delays in the surgery and registration errors due to imperfect registration.

[0004] Furthermore, surgical planning based solely on 3D images acquired before surgery without the aid of 2D images during surgery cannot avoid problems such as radiation exposure while acquiring 3D images, e.g., CT images, and differences in the patient's condition before surgery and during surgery.

[0005] However, if a surgical plan is established based on 2D X-ray images acquired during surgery, it would be possible to flexibly change the surgical plan as needed during surgery without the need to match the images acquired before surgery. However, this process requires the surgeon to estimate the surgical location in 3D space while viewing the 2D images. As a result, such a surgical plan requires the surgeon to have a high level of surgical experience. Even if this were possible, the image quality of 2D images is inferior to that of 3D CT images, and the lack of visual information from three-dimensional space makes it difficult to expect a successful surgery unless the surgeon is highly experienced.

[0006] Considering these points, it is highly desirable to develop a 2D image-based surgical planning method and system that can reduce the cost burden in terms of system configuration while reducing the problem of radiation exposure and maintaining performance comparable to that of existing 3D image-based surgical planning systems. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure presents a method and system for 2D image-based surgical planning.

[0008] The present disclosure provides a 2D image-based surgical planning method and system that can estimate the three-dimensional position, posture, and shape of a surgical target vertebra body based on 2D images without 3D images.

[0009] The present disclosure also provides a method and system for establishing a surgical plan based on 2D images acquired in real time during surgery without prior imaging, and for performing surgery based on the plan. [Means for solving the problem]

[0010] The 2D image-based surgical planning method according to the present disclosure includes: acquiring, by an image acquisition device, a first image in a first direction related to a spinal surgical site and a second image in a second direction different from the first direction; a space matching unit defining a virtual 3D surgical space corresponding to the surgical site, and matching the first image and the second image to the virtual 3D surgical space; an object separating unit extracting a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracting a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; a path generating unit setting a first screw path that passes through a center or near a center of a first pedicle region in the first image and is perpendicular to a mid-boundary line that passes between the first pedicle region and a first vertebral body region; The coordinate determination unit sets an entry point and a target point on a second screw path corresponding to the first screw path in the second pedicle region of the second image based on the first screw path, and determines 3D coordinates for inserting surgical screws in the surgical space.

[0011] According to one or more embodiments, A screw entry point and a screw target point are set on the first screw path, and two normals extending from the entry point and the target point in the first image may pass through the second pedicle region in the second image.

[0012] According to one or more embodiments, The first pedicle region may have a medial edge facing the first vertebral body region and a lateral edge opposite the medial edge, the entry point may be located on or adjacent to the lateral edge of the first pedicle region, and the target point may be located within the first vertebral body region.

[0013] According to one or more embodiments, The entry and target points within the second vertebral body region can be located adjacent to or on the medial edge of the second pedicle region located within the second vertebral body region.

[0014] According to one or more embodiments, The entry point may be located on one edge of the second pedicle region facing the outside of the second vertebral body region, or adjacent to the inner edge thereof, and the target point may be located on the other edge opposite the one side, or adjacent to the inner edge thereof.

[0015] According to one or more embodiments, The step of acquiring a first image in the first direction and a second image in a second direction different from the first direction may further include a step of detecting the orientation of the image acquisition unit using an OTS (optical tracking system) and calculating the angle of the second direction relative to the first direction.

[0016] According to one or more embodiments, In the aligning step, the angle may be reflected to align the first image and the second image in the surgical space.

[0017] The 2D image-based surgical planning system according to the present disclosure comprises: an image acquisition device for acquiring a first image in a first direction related to a spinal surgical site and a second image in a second direction different from the first direction; a space matching unit that defines a virtual 3D surgical space corresponding to the surgical site and matches the first image and the second image to the virtual 3D surgical space; an object separation unit that extracts a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; a path generating unit that sets a first screw path that passes through a center or near a center of a first pedicle region in the first image and is perpendicular to a mid-boundary line that passes between the first pedicle region and a first vertebral body region; and a coordinate determination unit that sets a step of setting an entry point and a target point of a screw corresponding to the first screw path in the second pedicle region of the second image based on the first screw path and determining 3D coordinates for screw insertion in the surgical space.

[0018] According to one or more embodiments, The path generation unit A screw entry point and a screw target point are set on the first screw path so that two normals extending from the entry point and the target point in the first image pass through the second pedicle region in the second image.

[0019] According to one or more embodiments, the first pedicle region has a medial edge facing the first vertebral body region and an opposite lateral edge; The path generation unit The entry point is located on or medially adjacent the lateral border of the first pedicle region, and the target point is located within the first vertebral body region.

[0020] According to one or more embodiments, The path generation unit The entry and target points within the second vertebral body region can be located adjacent to or on the medial edge of the second pedicle region located within the second vertebral body region.

[0021] According to one or more embodiments, The coordinate determination unit The entry point may be located on or adjacent to the medial edge of the second pedicle region toward the lateral side of the second vertebral body region, and the target point may be located on or adjacent to the opposite medial edge.

[0022] According to one or more embodiments, In the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction, the device also includes an OTS (optical tracking system) that detects the orientation of the image acquisition device and calculates the angle of the second direction relative to the first direction. According to one or more embodiments, The alignment unit may reflect the angle and align the first image and the second image in the surgical space. [Effects of the Invention]

[0023] O-arm devices, 3D C-arm devices, and mobile 3D imaging devices allow for three-dimensional patient imaging, enabling surgical planning by viewing areas not visible in two dimensions. However, they have drawbacks, such as exposing patients and medical staff to a large amount of radiation, requiring long imaging times, and being expensive, making them unavailable in many hospitals. 2D imaging devices, on the other hand, are widely used in many hospitals and are inexpensive and easy to use, but they also have drawbacks, such as a lack of patient image information due to the two-dimensional images, increased radiation exposure due to multiple imaging sessions, and differences in surgical planning and surgical results depending on the level of expertise.

[0024] The method and system disclosed herein are a system that automatically generates a surgical plan based solely on two-dimensional images, without using CT or a mobile 3D imaging device. This systematically resolves the difficulties of surgical planning due to two-dimensional image information, which is inferior to three-dimensional information, and can reduce errors due to skill level and radiation exposure. By automatically generating a surgical plan, the overall surgical time can be shortened. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a flowchart of the time-sequential processing steps of a 2D image-based surgical planning method in accordance with one or more embodiments of the present disclosure. [Figure 2]10A-10C are diagrams illustrating the alignment of a first image and a second image to a virtual 3D surgical space and the definition of 3D coordinates, in accordance with one or more embodiments of the present disclosure. [Figure 3] FIG. 2 illustrates an output image in which a vertebra body and a pedicle are segmented from an LL (lateral-lateral) image by applying a first image segmentation model according to one or more embodiments of the present disclosure. [Figure 4] FIG. 10 illustrates an output image in which vertebral bodies and pedicles are segmented from an anterior-posterior (AP) view using a second image segmentation model according to one or more embodiments of the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating a process for generating a deep learning model that may be applied by one or more embodiments according to the present disclosure. [Figure 6] FIG. 10 illustrates multiple training first images labeled with vertebral bodies and pedicles according to one or more embodiments of the present disclosure. [Figure 7] FIG. 10 illustrates multiple training second images labeled with vertebral bodies and pedicles according to one or more embodiments of the present disclosure. [Figure 8] 10A-10C illustrate the results of labeling a target segmented into a first image and a second image according to one or more embodiments of the present disclosure. [Figure 9] FIG. 10 illustrates an example of a previously selected L1 target according to one or more embodiments of the present disclosure. [Figure 10] 1 illustrates a vertebral body region (A) and a pedicle region (B) set in a first view according to one or more embodiments of the present disclosure. FIG. [Figure 11] 1 is a graph of the yz plane showing the vertex coordinates of the vertebral body region (A) and pedicle region (B) set in a first image, and the midline between those regions, according to one or more embodiments of the present disclosure. [Figure 12]12 is a graph in the yz plane showing an extraordinary line rotated 90 degrees from the midline of FIG. 11 in accordance with one or more embodiments of the present disclosure. [Figure 13] 1 is a yz-plane graph showing a first screw path D arranged parallel to a temporary straight line D′, according to one or more embodiments of the present disclosure. [Figure 14] 1 is a yz-plane graph illustrating the set coordinates of a target point T or end point of a surgical screw on a screw path in accordance with one or more embodiments of the present disclosure. [Figure 15] 1 is an image showing a second vertebral body region (a) and a second pedicle region (b, b') extracted or separated via a first image segmentation model in one or more embodiments of the present disclosure. [Figure 16] In one or more embodiments of the present disclosure, this is an xz plane graph showing an elliptical second pedicle region (b, b1) within a second vertebral body region (a) consisting of vertebrae a1, a2, a3, and a4, with a dotted line. [Figure 17] FIG. 1 illustrates a schematic configuration of a surgical robot system in accordance with one or more embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] Preferred embodiments of the inventive concept will now be described in detail with reference to the accompanying drawings. However, the embodiments of the inventive concept may be modified in various other forms, and the scope of the inventive concept should not be construed as being limited by the examples detailed below. It is desirable that the embodiments of the inventive concept be construed as being provided to more completely explain the inventive concept to those with average knowledge in the art. The same reference numerals refer to the same elements throughout. Furthermore, various elements and regions in the drawings are depicted schematically. Therefore, the inventive concept should not be limited by the relative sizes and spacings depicted in the accompanying drawings.

[0027] Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used only to distinguish one component from another. For example, a first component may be designated a "second component" and conversely, the second component may be designated the "first component" without departing from the scope of the inventive concept.

[0028] The terms used in this application are merely used to describe specific embodiments and are not intended to limit the concept of the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, words such as "comprise" or "have" specify the presence of a feature, number, step, operation, element, component, or combination thereof described in the specification, and should be understood not to preclude the possibility of the presence or addition of one or more other features, numbers, operations, elements, components, or combinations thereof.

[0029] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the concept of the present invention belongs. Furthermore, commonly used and predefined terms should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and unless explicitly defined herein, they should not be interpreted in an overly formal sense.

[0030] If an embodiment can be implemented differently, the order of certain steps may be different from that described. For example, two steps described in succession may be performed substantially simultaneously or in the reverse order from that described.

[0031] DETAILED DESCRIPTION OF THE INVENTION In the following, a two-dimensional (2D) image-based surgical planning method and system according to one or more embodiments will be described in detail.

[0032] FIG. 1 is a time-series flowchart of a 2D video-based surgical plan according to the present disclosure.

[0033] The process of FIG. 1 includes steps S5 to S10 that are repeatedly performed on a plurality of target vertebral bodies. When there is one vertebral body, a surgical plan for one spine is established in one flow.

[0034] Describing FIG. 1 step by step, it is as follows.

[0035] The method according to the present disclosure generally includes a preparatory work process, an LL (lateral-lateral) point generation process, an AP (anterior-posterior) point generation process, and an optimization and automation process.

[0036] <Step S1> Step S1 is a step of acquiring a 2D video in two directions, a first video and a second video, for the affected part of the patient. As the first video, for example, an LL video (lateral-lateral image), and as the second video, for example, an AP video (anterior-posterior image) are acquired. The two directions for acquiring such a first video and a second video are orthogonal to each other, intersect, or intersect while maintaining an arbitrary angle close to orthogonal. For the acquisition of such a first video and a second video, a video acquisition device having an X-ray source in the form of a point source on one side and an X-ray detector on the other side opposite thereto, for example, a so-called C-arm device, is applied around the affected part of the patient, that is, the spinal region to be surgically treated. According to another embodiment of the present disclosure, a commercially available OTS (optical tracking system) equipped with markers attached to a part of the patient's body, markers of a calibrator capable of estimating the positions of the source and the detector, and a sensor for detecting the three-dimensional position thereof is used to measure the angles of the first direction and the second direction for acquiring the first video and the second video.

[0037] <S2 stage> In the S2 stage, the first video and the second video are aligned in a virtual three-dimensional (3D) surgical space, and coordinates related to the pixels of these videos are determined or defined as the coordinates of the first video and the second video in the 3D space. FIG. 2 schematically shows the alignment of the first video and the second video obtained by X-rays in a cone-shaped or conical beam form with respect to a virtual 3D surgical space, and the definition of 3D coordinates. As shown in the figure, the source and the detector have a perspective projection relationship by a geometrically cone-shaped X-ray beam.

[0038] The virtual surgical space, that is, the virtual space has axes in the X, Y, and Z directions. In the AP video, the vertical direction (head-foot) is the Z axis, the horizontal direction (left arm-right arm) is the X axis, and in the LL video, the horizontal direction (abdomen-back) is the Y axis, and the vertical direction (head-foot) is still the Z axis. Therefore, the first video, for example, the LL video, is parallel to the YZ plane, and the second video, for example, the AP plane, is parallel to the XZ plane.

[0039] In the present embodiment, the first video of the surgical site, for example, the AP image and the LL image, is acquired through a 2D video acquisition device such as a C-arm equipment, and the video acquisition device such as a C-arm equipment is aligned with a coordinate system based on a marker provided on a part of the patient's body or a marker of a calibrator provided in the surgical space. The applicant's registered patent KR2203544 discloses a technique for aligning 2D videos in a 3D space, and this application includes the registered patent KR2203544 in its entirety by reference. However, this disclosure is not technically limited to a specific alignment technique.

[0040] <S3 stage> In the S3 stage, extraction or segmentation of the vertebral body and pedicle as target objects proceeds from the first video and the second video obtained in the above process and aligned in a virtual three-dimensional surgical space.

[0041] The segmentation of the object can be performed by applying an LL segmentation model and an AP segmentation model trained by deep learning. FIG. 3 shows an output image in which the vertebral body and the pedicle are segmented from the LL image by applying the first video segmentation model, and FIG. 4 shows an output image in which the vertebral body and the pedicle are segmented from the AP image by applying the second video segmentation model.

[0042] As shown in FIG. 5, the above-mentioned segmentation model can be based on the DeepLabv3+ model. This model is one of the deep learning models used in the field of computer vision and is used to perform extraction or segmentation operations. This model is particularly based on a deep convolutional neural network (CNN) architecture and has an encoder structure and a decoder structure, which is suitable for image segmentation.

[0043] The training of the model includes the same process as general deep learning training. For the training of such a model, various spine-related video materials collected for training are required. The training videos include a first training video taken in the first direction and a second training video taken in the second direction. As shown in FIGS. 4 and 5, pixel-level labeling is performed on the first video and the second video, and then, by performing the learning process, the target first video segmentation model and second video segmentation model can be obtained. FIGS. 6 and 7 described above show images in which the vertebral body and the pedicle are labeled for each of a large number of first and second learning videos.

[0044] <Step S4> In this step, it is a step of labeling the target segmented in the previous step. FIG. 8 shows the results of labeling the targets segmented in the AP video and the LL video.

[0045] As shown in FIG. 8, for the divided targets, they are sequentially labeled as L1, L2, L3, L3, L4, and L5 from top to bottom, and for the same vertebral body, the same label is attached.

[0046] <S5 stage> In this stage, prior to the process of extracting the LL point, as a stage of selecting one target from among the plurality of targets, for example, as shown in FIG. 9, the L1 target is first selected and will go through the subsequent process.

[0047] <S6 stage> Extract the LL parameters related to the selected labeled target, and determine the first screw path in the LL image plane. In the path planning, the pedicle region and the vertebral body region must be set so as not to deviate. Here, if only the inclination of either one of the vertebral body region or the pedicle region is reflected, the path will deviate in other regions, and in order to prevent the path planning failure due to the detection error generated during the object segmentation, the screw path is set to the average inclination of the inclinations of the two regions.

[0048] For this purpose, first, ROI (region of interest) regions for the pedicle and the vertebral body are set. In FIG. 10, (A) is the vertebral body region and (B) is the pedicle region. Such ROIs can be obtained by the aforementioned first image segmentation model.

[0049] As shown in FIG. 11, the vertebral body region (A) is a rectangular region formed by four vertices A1, A2, A3, and A4, and the pedicle region (B) is a rectangular region formed by four vertices B1, B2, B3, and B4, and these two regions are adjacent. An intermediate line C passes between the adjacent sides of the two adjacent regions.

[0050] The intermediate line C passes through the middle between the two adjacent sides (B2-B3, A1-A4) of both areas (A, B), and intersections C1 and C2 are located on the intermediate line C. Intersection C1 is located midway between the line segment connecting vertex B2 and vertex A1 of both areas, and intersection C2 is located midway between the line segment connecting vertex B3 and vertex A4. Therefore, the y coordinate of C1 (C1.y) is (A1.y + B2.y) / 2, and the z coordinate of C1 (C1.z) is (A1.z + B2.z) / 2. The y coordinate of C2 (C2.y) is (A4.y + B3.y) / 2, and the z coordinate of C2 (C2.z) is (A4.z + B3.z) / 2.

[0051] As a result, the intermediate line C has an average gradient (Ca) for the slopes of both regions in the LL image plane coordinates, i.e., the yz plane. Such an intermediate line C becomes a reference for setting the first screw path in the LL image plane, and in the yz plane coordinate system, the coordinates C1(y,z) and C2(y,z) of the intersection points C1 and C2 and the gradient Ca of the intermediate line C are expressed by the following formula:

[0052]

number

[0053] FIG. 12 is a graph showing a temporary point C3 in the middle of the process of setting the first screw path from the midline C between the pedicle region (B) and the vertebral body region (A) in the yz plane.

[0054] The coordinates of the temporary point C3 coincide with the coordinates of the intersection point C1 on the intermediate line C rotated 90° around the intersection point C2. That is, if the coordinates of C1 are (y1, z1) in the yz plane coordinate system, the coordinates of C3 become (z1, -y1). As a result, a temporary straight line D' connecting the intersection point C2 and the temporary point C3 is a normal line perpendicular to the intermediate line C, and the equation of the normal line is D':Z = Da(Y - C2.y) + C2.z.

[0055] Figure 13 illustrates a first screw arrangement line, i.e., a first screw path D, arranged parallel to the temporary straight line D'. The first screw path D is orthogonal to the intermediate line C and parallel to the temporary straight line D'. The temporary straight line D' is translated parallel by a predetermined distance so as to pass through the center (B5) or the vicinity of the center of the pedicle region (B) having four vertices B1, B2, B3, and B4, and the first screw path D is determined in the yz plane.

[0056] The y coordinate B5.y and the z coordinate B5.z of B5 are expressed as follows. B5.y = (B1.y + B2.y + B3.y + B4.y) / 4 B5.z = (B1.z + B2.z + B3.z + B4.z) / 4

[0057] The linear equation of the first screw path D having a slope Da is expressed as follows. D: Z = Da(Y - C2.y) + C2.z Da = (C3.z - C2.z) / (C3.y - C2.y)

[0058] The first screw path D expressed by the linear equation (D: z) as described above passes through the outer sides (BS, B1 - B4) of the pedicle, and an intersection point E is generated. This intersection point E will be the entry point of the surgical screw to be described later.

[0059] The outer side BS of the pedicle is expressed by the following formula.

[0060] BS: Z = Ba(Y - B1.y) + B1.z Ba = (B4.z - B1.z) / (B4.y - B1.y)

[0061] <Step S7> In this step, the coordinates of the entry point of the screw located on the first screw path D and the target point or the terminating point that the end of the surgical screw reaches are calculated.

[0062] The coordinates (Ey, Ez) of the entry point E located on the outer edge of the pedicle can be calculated as the y coordinate (Ez) and z coordinate (Ez) of the entry point by applying a linear equation (D:Z) of the first screw path and a linear equation (BS:Z) of the outer edge BS of the pedicle using a linear system of equations. D:Z=Da(Y-C2.y)+C2.z BS:Z=Ba(Y-B1.y)+B1.z

[0063] First, the two lines have the same Y and Z in (Ey, Ez), so if we calculate as follows, we get the following: Da(Y-C2.y)+C2.z=Ba(Y-B1.y)+B1.z

[0064] The Ey value can be obtained by collecting the terms related to the above-mentioned equation Y on one side and separating Y. Ey=Y=(B1.z-C2.z+Ba*B1.y-Da*C2.y) / (Da-Ba)

[0065] By substituting Ey obtained from the above formula into the linear equation (D:Z) of the screw path, the z-coordinate Ez of the entry point can be obtained. Ez=Z=Da(Ey-C2.y)+C2.z

[0066] The above calculation process is one of various calculation methods for determining the coordinates of the entry point E, and other methods can also be used for calculation, but it goes without saying that the specific calculation method does not limit the technical scope of the present disclosure.

[0067] FIG. 14 illustrates the set coordinates of the target point T or the end point of the surgical screw on the screw path on the yz plane. The coordinates of the target point or the end point are determined by the total length Ls of the screw used. In one embodiment according to the present disclosure, the total length Ls of the screw is set so as not to exceed the length of the side (A1 - A2) of the vertebral body region in the direction in which the surgical screw advances, and its diameter Ds is set so as not to be larger than the length of the outer side (B1 - B4) where the screw entry point is located.

[0068]

Number

[0069] Here, the coordinates T(y, z) of the target point are expressed by the following mathematical formula. T.y = E.y + Ls * Cos(arctan(Da)) T.z = E.z + Ls * Sin(arctan(Da))

[0070] <S8 Step> This step is a parameter extraction step for determining a second screw path on a second imaging plane, that is, the xz plane of the AP image, from the screw path obtained in the yz plane.

[0071] FIG. 15 shows a state in which a second vertebral body region (a) and the left and right second pedicle regions (b, b') inside thereof are extracted or separated through an AP segmentation model.

[0072] In FIG. 15, two straight lines L T , L E cross the second vertebral body region (A), and at this time, pass through the left and right second pedicle regions (b, b') inside the vertebral body region (A).

[0073] Of the above two straight lines, "L T " is the target point setting line, and "L E" is the entry point setting line. As described above, for the LL plane of the first video, since the AP plane of the second video is aligned, the entry point and the target point of the LL plane of the first video are on the AP plane of the second video, and the target point setting line L T and the entry point setting line L E can be formed.

[0074] According to an embodiment of the present disclosure, the entry point setting line L E is a normal line extending in the x direction, which is perpendicular to the yz plane, from the entry point in the yz plane, and the target point setting line L T is a normal line extending in the x direction, which is perpendicular to the yz plane, from the target point in the yz plane.

[0075] <S9 step> This step is to set the entry point and the target point in the xz plane on the entry point setting line and the target point setting line.

[0076] The entry point and the target point in the xz plane determine or form the second screw path on the xz plane and are formed on the boundary line of the pedicle. It is for setting the most secure entry path of the screw. However, for the second screw path for both sides of the screw, the entry point and the target point are set so that the second screw path goes from the outside of the vertebral body to the center side of the vertebral body.

[0077] FIG. 16 shows, by dotted lines, the elliptical second pedicle regions (b, b1) within the second vertebral body region (a) formed by the vertices a1, a2, a3, a4 on the xz plane.

[0078] As shown in FIG. 16 and as described above, the target point setting line L T and the entry point setting line L E form intersections on the boundary line of the second pedicle region (b, b1) while passing through the two second pedicle regions (b, b1). There are a total of four intersections, including two intersections formed by the target point setting line L T and two intersections formed by the entry point setting line L E .

[0079] At the aforementioned intersection points, among the four intersection points formed by the entry point setting line L E , two intersection points located outside the second vertebral body region (a) are selected as screw entry points E L , E R , and among the four intersection points formed by the target point setting line L T , two intersection points located in the central region of the second vertebral body region (a) are selected as screw target points T L , T R . Therefore, the left and right entry points E L , E R and the left and right target points T L , T R in each pedicle region (b, b1) are respectively connected to form the left and right straight lines D” which are the second screw paths.

[0080] <S10 step> In this step, 3D coordinates in the surgical space are obtained through the coordinates of the screw insertion points and screw target points in the yz plane and the coordinates of the screw insertion points and screw target points in the xz plane obtained by the method as described above.

[0081] In this step, using the respective coordinates E L , E R and target points T L , T R in the xz plane, i.e., E L (x, z), E R (x, z), T L (x, z), T R (x, z), and the coordinates of the entry point E.y and the target point E.z in the yz plane, the coordinates of the screw entry point E(x, y, z) and the target point T(x, y, z) in the three-dimensional surgical space can be obtained. Such acquisition of coordinates in the 3D surgical space is possible because, as described above, the first video and the second video are aligned in the virtual 3D surgical space.

[0082] The above process is performed for one target. However, if there are multiple targets, i.e., multiple vertebrae to be operated on, and it is determined that the final target has not yet been reached in the next step, S11, the process returns to S5 and repeats the process from S5 to S10, and then ends.

[0083] The above-described embodiments of the present disclosure can successfully automatically plan screw paths in a 3D surgical space using only 2D images during surgery. The method and system of the present disclosure can be implemented at low cost because it uses a 2D imaging device. The method and system of the present disclosure automatically generate a surgical plan based solely on 2D images, without using a CT scan or a mobile 3D imaging device. This system systematically overcomes the difficulties of surgical planning using 2D image information, which is inferior to 3D information, and can reduce errors due to differences in skill level and radiation exposure. Automatically generating a surgical plan can also shorten the overall surgical time.

[0084] Embodiments of the present disclosure may be embodied in a computer program medium storing software for performing an image matching method on a computer, or in an image matching device including a processor, memory, display, etc., for performing the image matching method.

[0085] The present invention may also be embodied in a surgical robot system based on the image matching method described above.

[0086] 17, a surgical robot system 1 according to one or more embodiments of the present disclosure employs a 2D image capturing device 100 as a main image capturing device. The robot system 1 also includes a surgical robot 200, a position sensor 300, and a navigation system 400. The surgical robot 200 includes a main body 201, a robot arm 203 equipped with an end effector 203a, and a robot controller 205.

[0087] In an embodiment according to the present disclosure, except for the image acquisition device, the spatial matching unit, object separation unit, path generation unit, and coordinate determination unit are functionally included in the above-mentioned devices, and the spatial matching unit may be included in the navigation system 400. The object separation unit, path generation unit, and coordinate determination unit are implemented by surgical planning software, and in particular, the object separation unit applies an LL segmentation model and an AP segmentation model trained by deep learning to extract or separate vertebral bodies and pedicles from the first and second 2D images.

[0088] The 2D image may be acquired by a two-dimensional image acquisition device, for example, a so-called C-arm device having an X-ray source and a detector arranged facing each other on both sides of a patient undergoing surgery placed in the middle. That is, in an embodiment according to the present disclosure, a C-arm image acquisition device may be applied as the 2D image acquisition device 100.

[0089] During surgery, the 2D image acquisition device acquires a first 2D image, e.g., an LL image, and a second 2D image, e.g., an AP image, of the patient's surgical site to obtain a 3D screw path. The robot arm 203 is fixedly mounted on the robot body 201 and has an end effector 203a at its end, to which a surgical tool can be attached or detached. The position sensor 300 is implemented as an optical tracking system (OTS) that tracks the real-time position of the surgical tool or end effector 203a through marker recognition. The controller 205 is mounted on the robot body 201 and controls the robot arm 203 according to a surgical plan determined during surgery and control software in accordance with the present disclosure. The navigation system 400 performs the image matching method described above and can display surgical plan information related to surgical tools or implants as 2D images acquired during surgery, or the real-time position of the surgical tool or implant as 2D images or, in some cases, as 3D images acquired before surgery, to assist the surgeon in performing the surgical procedure. To this end, a display that allows the surgeon to visually compare the real-time positions of surgical tools and the like with the surgical plan and current surgical situation during surgery may be connected to the navigation system 400.

[0090] Although various embodiments of the present invention have been described in detail above, those skilled in the art will recognize that the present invention may be modified in various ways without departing from the spirit and scope of the present invention as defined in the appended claims. Therefore, any modifications to the embodiments of the present invention will not be deemed to be outside the scope of the present invention.

Claims

1. acquiring a first image in a first direction of a spinal surgical site and a second image in a second direction different from the first direction; defining a virtual three-dimensional (3D) surgical space corresponding to the surgical site, and aligning the first image and the second image with the surgical space; extracting a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracting a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; setting a first screw path in the first image, the first screw path passing through a center or near a center of a first pedicle region, the first screw path being perpendicular to a mid-boundary line passing between the first pedicle region and a first vertebral body region; and setting an entry point and a target point on a second screw path corresponding to the first screw path in a second pedicle region of the second image based on the first screw path, and determining 3D coordinates for inserting surgical screws in the surgical space.

2. 2. The 2D image-based surgical planning method of claim 1, wherein a screw entry point and a screw target point are set on the first screw path, and two normals extending from the entry point and the target point in the first image pass through the second pedicle region in the second image.

3. 2. The 2D image-based surgical planning method of claim 1, wherein the first pedicle region has a medial edge facing the first vertebral body region and a lateral edge opposite the medial edge, the entry point is located on or adjacent to the lateral edge of the first pedicle region, and the target point is located within the first vertebral body region.

4. 4. The 2D image-based surgical planning method of claim 3, wherein the entry point and the target point in the second vertebral body region are set on or adjacent to the edge of the second pedicle region located in the second vertebral body region on the medial side thereof.

5. 4. The 2D image-based surgical planning method of claim 3, wherein the entry point is located on one side edge of the second pedicle region facing the outside of the second vertebral body region or adjacent to the inner edge thereof, and the target point is located on the other side edge opposite the one side or adjacent to the inner edge thereof.

6. 6. The 2D image-based surgical planning method of claim 1, further comprising: detecting a posture of an image acquisition unit using an optical tracking system (OTS) and calculating an angle of the second direction relative to the first direction, in the step of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

7. The 2D image-based surgical planning method of claim 6, wherein the alignment step reflects the angle and aligns the first image and the second image in the surgical space.

8. an image acquisition device for acquiring a first image in a first direction related to a spinal surgical site and a second image in a second direction different from the first direction; a space matching unit that defines a virtual 3D surgical space corresponding to the surgical site and matches the first image and the second image to the virtual 3D surgical space; an object separation unit that extracts a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; a path generating unit that sets a first screw path that passes through a center or near a center of a first pedicle region in the first image and is perpendicular to a mid-boundary line that passes between the first pedicle region and a first vertebral body region; and a coordinate determination unit that sets an entry point and a target point of a screw corresponding to the first screw path in a second pedicle region of the second image based on the first screw path, and sets a step of determining 3D coordinates for screw insertion in the surgical space.

9. The path generation unit 9. The 2D image-based surgical planning system of claim 8, wherein a screw entry point and a screw target point are set on the first screw path so that two normals extending from the entry point and the target point in the first image pass through the second pedicle region in the second image.

10. the first pedicle region has a medial edge facing the first vertebral body region and an opposite lateral edge; The path generation unit 9. The 2D image-based surgical planning system of claim 8, wherein the entry point is located on the lateral edge of the first pedicle region or adjacent to the lateral edge on the medial side thereof, and the target point is located within the first vertebral body region.

11. The path generation unit The 2D image-based surgical planning system of claim 10, wherein the entry point and target point within the second vertebral body region are set on or adjacent to the edge of the second pedicle region located within the second vertebral body region on the medial side thereof.

12. The coordinate determination unit 12. The 2D image-based surgical planning system of claim 8, wherein the entry point is located on one side edge of the second pedicle region toward the outside of the second vertebral body region or adjacent to the inner edge thereof, and the target point is located on the other side edge opposite the one side or adjacent to the inner edge thereof.

13. 13. The 2D image-based surgical planning system of claim 12, further comprising an optical tracking system (OTS) that detects the orientation of the image acquisition device and calculates the angle of the second direction relative to the first direction during the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

14. The 2D image-based surgical planning system of claim 13 , wherein the alignment unit reflects the angle and aligns the first image and the second image in the surgical space.

15. 12. The 2D image-based surgical planning system of claim 8, further comprising an optical tracking system (OTS) that detects the orientation of the image acquisition device and calculates the angle of the second direction relative to the first direction during the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

16. The 2D image-based surgical planning system of claim 13 , wherein the alignment unit reflects the angle and aligns the first image and the second image in the surgical space.

Images