2d image-based automatic surgery planning method and system

Abstract

Disclosed are a 2D image-based surgery planning method and system. The planning method comprises the steps of: obtaining a first image and a second image from a spinal surgical site; defining a virtual 3D surgery space and registering the first image and the second image; extracting a first vertebral region and a first pedicle region from the first image, and extracting a second vertebral body region and a second pedicle region in the second vertebral body region from the second image; setting a first screw path passing through the center of the first pedicle region or the vicinity of the center of the first pedicle region in the first image; and setting an entry point and an end point, on a second screw path corresponding to the first screw path, in the second pedicle region of the second image on the basis of the first screw path.

Description

2D image-based automatic surgical planning method and system

[0001] The present disclosure relates to an automatic surgical planning method and system in a surgical robot system, and more particularly, to a 2D image-based automatic surgical planning method and system capable of generating a surgical planning path in a 3D image space using a plurality of 2D images.

[0002] Spinal surgical procedures, such as pedicle screw fixation using a surgical robot, require 3D images capable of an axial view. Therefore, 3D images capable of an axial view, such as computed tomography (CT) images, rather than 2D images, are used as these axial view images.

[0003] Typically, intraoperatively, a 2D image acquisition device, such as a C-arm, is used to verify in real time whether the surgery is proceeding as planned. To navigate surgical tools as planned using these acquired 2D images, the intraoperative 2D images must be registered with the preoperative 3D images. However, the process of registering 2D images to 3D images can lead to delays in surgery and registration errors due to incomplete registration.

[0004] Meanwhile, surgical planning based only on 3D images acquired before surgery without the help of 2D images during surgery cannot avoid problems that arise due to radiation exposure caused by acquiring 3D images, such as CT images, and differences in the patient's condition before surgery and during surgery.

[0005] However, if a surgical plan is established based on 2D X-ray images acquired during surgery, there is no need for registration with images acquired before surgery, and the flexibility to change the surgical plan as needed can be expected even during surgery. However, this process requires the operating surgeon to estimate the location of the desired 3D space in 3D space while looking at the 2D image. Consequently, this surgical plan requires a high level of surgical experience from the operating surgeon. Even if this is realized, it is difficult to expect a successful surgery unless the surgery is performed by an experienced operating surgeon, because 2D images lack image information in terms of image quality and 3D spatial information compared to 3D CT images.

[0006] Considering these points, it is highly desirable to develop a 2D image-based surgical planning method and system that can reduce the cost burden of system configuration while maintaining performance comparable to that of existing 3D image-based surgical planning systems and reducing radiation exposure issues.

[0007] The present disclosure presents a surgical planning method and system based on 2D images.

[0008] The present disclosure proposes a 2D image-based surgical planning method and system capable of estimating the 3D position, posture, and shape of a surgical target vertebra body based on a 2D image without a 3D image.

[0009] In addition, the present disclosure proposes a method and system for establishing a technical plan based on 2D images acquired in real time during surgery without prior shooting, and performing surgery based on the same.

[0010] 2D image-based surgical planning method according to the present disclosure:

[0011] A step of an image acquisition device acquiring a first image in a first direction with respect to a spinal surgical site and a second image in a second direction different from the first direction;

[0012] A step of defining a virtual 3D surgical space corresponding to the surgical site by a spatial alignment unit and aligning the first image and the second image to the surgical space;

[0013] A step of the object separation unit extracting a first vertebra region and a first pedicle region corresponding to the vertebrae and pedicle from the first image, and extracting a second vertebral body region and a second pedicle region within the second vertebral body region from the second image;

[0014] A step of the path generation unit setting a first screw pass that passes through the center or vicinity of the center of the first pedicle region in the first image, the first screw pass being orthogonal to the middle boundary line passing between the first pedicle region and the first vertebral body region; and

[0015] The coordinate determination unit includes a step of determining 3D coordinates for insertion of a surgical screw in the surgical space by setting an entry point and an end point on a second screw pass corresponding to the first screw pass in a second pedicle region of the second image based on the first screw pass.

[0016] According to one or more embodiments,

[0017] A screw entry point and an end point are set on the first screw pass, and two normal lines extending from each of the entry point and the end point in the first image can pass through the second pedicle region of the second image.

[0018] According to one or more embodiments,

[0019] The first vertebral body region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, the entry point may be located on or adjacent to the outer edge of the first vertebral body region, and the end point may be located within the first vertebral body region.

[0020] According to one or more embodiments,

[0021] The entry and end points within the second vertebral body region may be set adjacent to the edge of the second pedicle region located within the second vertebral body region or within the second pedicle region.

[0022] According to one or more embodiments,

[0023] The entry point may be positioned on one edge of the second vertebral region facing outward from the second vertebral body region or adjacent to the edge from the inside thereof, and the end point may be positioned on the other edge opposite to the one side of the vertebral body region or adjacent to the edge from the inside thereof.

[0024] According to one or more embodiments,

[0025] In the step of acquiring a first image in the first direction and a second image in a second direction different from the first direction, the method may further include a step of detecting the posture of an image acquisition unit by an optical tracking system (OTS) and calculating an angle between the second direction and the first direction.

[0026] According to one or more embodiments,

[0027] In the above alignment step, the first image and the second image can be aligned in the surgical space by reflecting the angle between them.

[0028] 2D image-based surgical planning system according to the present disclosure:

[0029] An image acquisition device for acquiring a first image in a first direction with respect to a spinal surgical site and a second image in a second direction different from the first direction;

[0030] A space alignment unit defining a virtual 3D surgical space corresponding to the surgical site and aligning the first image and the second image to the surgical space;

[0031] An object separation unit that extracts a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image;

[0032] A path generation unit for setting a first screw pass that passes through the center or near the center of the first pedicle region in the first image, the first screw pass being orthogonal to the middle boundary line passing between the first pedicle region and the first vertebral body region; and

[0033] A step of determining 3D coordinates for insertion of a screw in the surgical space by setting an entry point and an end point of a screw corresponding to the first screw pass in a second pedicle region of the second image based on the first screw pass; including a coordinate determination unit;

[0034] According to one or more embodiments,

[0035] The above path generation part:

[0036] A screw entry point and an end point are set on the first screw pass, and two normal lines extending from each of the entry point and the end point in the first image can pass through the second pedicle region in the second image.

[0037] According to one or more embodiments,

[0038] The first vertebral body region has an inner edge facing the first vertebral body region and an outer edge opposite thereto,

[0039] The above path generation unit:

[0040] The entry point may be positioned adjacent to the outer edge of the first vertebral body region or on the inner side thereof, and the end point may be positioned within the first vertebral body region.

[0041] According to one or more embodiments,

[0042] The above path generation unit:

[0043] The entry and end points within the second vertebral body region may be set adjacent to the edge of the second pedicle region located within the second vertebral body region or within the second pedicle region.

[0044] According to one or more embodiments,

[0045] The above coordinate determination unit;

[0046] The entry point may be positioned on or adjacent to the edge of the second pedicle region facing outward from the second vertebral body region, and the end point may be positioned on or adjacent to the edge on the opposite side from the edge.

[0047] According to one or more embodiments,

[0048] The method may include an optical tracking system (OTS) that detects the attitude of the image acquisition device in the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction and calculates an angle between the second direction and the first direction.

[0049] According to one or more embodiments,

[0050] The above alignment unit can align the first image and the second image in the surgical space by reflecting the angle between the first and second images.

[0051] O-arm or 3D C-arm devices and mobile 3D imaging devices can view patient images in 3D, so they can see areas that are invisible in 2D and enable surgical planning, but they cause a lot of radiation exposure to patients and medical staff, take a long time to scan, and are expensive imaging devices, so they cannot be used in many hospitals. However, 2D imaging devices are widely used in many hospitals and are inexpensive and easy to use, but they have the disadvantage of lacking patient image information in 2D images, increasing radiation exposure through multiple scans, and differences in surgical planning and effectiveness depending on skill level.

[0052] The method and system according to the present disclosure are systems that automatically generate surgical plans based only on two-dimensional images without using CT or a mobile 3D imaging device, and systematically resolve difficulties in surgical planning due to two-dimensional image information inferior to three-dimensional information, reduce errors and radiation exposure according to skill level, and shorten the overall surgical time by automatically generating a surgical plan.

[0053] FIG. 1 is a flowchart of sequential processing steps of a 2D image-based surgical planning method according to one or more embodiments of the present disclosure.

[0054] FIG. 2 schematically illustrates the alignment and definition of 3D coordinates for a virtual 3D surgical space of a first image and a second image, according to one or more embodiments of the present disclosure.

[0055] FIG. 3 shows an output image in which a vertebra body and a pedicle are segmented from an LL image by applying a first image segmentation model according to one or more embodiments of the present disclosure.

[0056] FIG. 4 shows an output image in which a vertebra body and a pedicle are segmented from an AP image by applying a second image segmentation model according to one or more embodiments of the present disclosure.

[0057] FIG. 5 schematically illustrates a process for generating a deep learning model applied to one or more embodiments according to the present disclosure.

[0058] FIG. 6 shows an image of a plurality of first learning images labeled with respect to a vertebra body and a pedicle according to one or more embodiments of the present disclosure.

[0059] FIG. 7 shows an image of a plurality of second learning images labeled with respect to the vertebra body and pedicle according to one or more embodiments of the present disclosure.

[0060] FIG. 8 shows the results of labeling segmented targets in a first image and a second image according to one or more embodiments of the present disclosure.

[0061] FIG. 9 illustrates a first selected L1 target according to one or more embodiments of the present disclosure.

[0062] FIG. 10 illustrates a vertebral body region (A) and a pedicle region (B) set in a first image, according to one or more embodiments of the present disclosure.

[0063] FIG. 11 is a graph of a yz plane showing the coordinates of vertices of a vertebral body region (A) and a pedicle region (B) set in a first image and the midline between these regions, according to one or more embodiments of the present disclosure.

[0064] FIG. 12 is a graph of the yz plane showing a temporary straight line obtained by rotating the midline of FIG. 11 by 90 degrees, according to one or more embodiments according to the present disclosure.

[0065] FIG. 13 is a yz-plane graph showing a first screw pass D arranged parallel to a temporary straight line D' according to one or more embodiments of the present disclosure.

[0066] FIG. 14 is a yz plane graph showing the set coordinates of a target point (T) or an end point of a surgical screw on a screw pass according to one or more embodiments of the present disclosure.

[0067] FIG. 15 is an image showing a state in which a second vertebral body region (a) and a second pedicle region (b, b') are extracted or separated through a first image segmentation model in one or more embodiments of the present disclosure.

[0068] FIG. 16 is an xz plane graph showing an elliptical second vertebral region (b, b1) within a second vertebral region (a) having vertices a1, a2, a3, a4, in one or more embodiments of the present disclosure, as a dotted line.

[0069] FIG. 17 shows a schematic configuration of a surgical robot system according to one or more embodiments of the present disclosure.

[0070] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the embodiments described below. It is preferable to interpret that the embodiments of the present invention are provided to more completely explain the present invention to a person having average knowledge in the art. Like reference numerals denote like elements throughout. Furthermore, various elements and areas in the drawings are schematically drawn. Therefore, the present invention is not limited by the relative sizes or intervals drawn in the accompanying drawings.

[0071] While terms like "first" and "second" may be used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and vice versa, without departing from the scope of the present invention.

[0072] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the concept of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the expressions “comprises” or “has” are intended to specify the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

[0073] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, it is to be understood that commonly used terms, such as those defined in dictionaries, should be interpreted to have a meaning consistent with their meaning within the relevant technical context, and should not be interpreted in an overly formal sense unless explicitly defined herein.

[0074] In some embodiments, where the implementation is otherwise feasible, a particular process sequence may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0075] Hereinafter, a 2D image-based surgical planning method and system are described in detail according to one or more embodiments.

[0076] Figure 1 is a sequential flowchart of a 2D image-based surgical plan according to the present disclosure.

[0077] The process of Fig. 1 includes steps S5 to S10 that are repeatedly performed for multiple target vertebral bodies, and in the case of one vertebral body, a surgical plan for one vertebra is established in one flow.

[0078] A step-by-step explanation of Figure 1 is as follows.

[0079] The method according to the present disclosure generally includes a preparatory work process, an LL point generation process, an AP point generation process, and an optimization and automation process.

[0080] < Stage S1 >

[0081] Step S1 is a step of acquiring a first image and a second image from two directions for the affected area of ​​the patient. For example, an LL image (Lateral-Lateral Image) is acquired as the first image, and for example, an AP image (Anterior-Posterior Image) is acquired as the second image. The two directions in which the first and second images are acquired may be orthogonal to each other, intersect, or intersect while maintaining an arbitrary angle close to orthogonality. For acquiring the first and second images, an image acquisition device, such as a so-called C-arm device, having an X-ray source in the form of a point source on one side and an X-ray detector on the other side centered on the affected area of ​​the patient, i.e., the target spine area for surgery, is applied. According to another embodiment of the present disclosure, a commercially available OTS having a marker attached to a part of a patient's body and a sensor for detecting the three-dimensional position of the marker and the calibrator capable of estimating the position of the Source and Detector can be used to measure the angle between the first direction and the second direction in which the first image and the second image are acquired.

[0082] < S2 stage >

[0083] Step S2 aligns the first and second images in a virtual 3D surgical space, and determines or defines coordinates in the 3D space for pixels of these images. Fig. 2 schematically shows the alignment of the first and second images obtained by X-rays in the form of a cone-shaped or conical beam in a virtual 3D surgical space and the definition of 3D coordinates. As illustrated, the source and the detector have a geometrical perspective projection relationship by the cone-shaped X-ray beam.

[0084] The virtual surgical space, or virtual space, has axes in the X, Y, and Z directions. In the AP image, the up-down direction (head-to-foot) is the Z-axis, and the left-right direction (left arm-right arm) is the X-axis. In the LL image, the left-right direction (abdomen-back) is the Y-axis, and the up-down direction (head-to-foot) is also the Z-axis. Therefore, the first image, for example, the LL image, is aligned with the YZ plane, and the second image, for example, the AP plane, is aligned with the XZ plane.

[0085] In this embodiment, a first image of a surgical site, for example, an AP image and an LL image, is acquired through a 2D image acquisition device such as a C-arm device, and the image acquisition device such as the C-arm device is aligned with a coordinate system based on a marker installed on a part of a patient's body or markers of calibrators installed in other surgical spaces. The applicant's registered patent No. KR2203544 discloses a technology for aligning a 2D image to a 3D space, and the present application incorporates No. 544' in its entirety by reference, but the present disclosure is not technically limited to a specific alignment technology.

[0086] < S3 stage >

[0087] In step S3, the vertebral body and pedicle as target objects are extracted or segmented from the first and second images acquired in the aforementioned process and aligned to the virtual three-dimensional surgical space.

[0088] Object segmentation can be performed by applying the LL segmentation model and the AP segmentation model trained by deep learning. Fig. 3 shows an output image in which the vertebra body and pedicle are segmented from the LL image by applying the first image segmentation model, and Fig. 4 shows an output image in which the vertebra body and pedicle are segmented from the AP image by applying the second image segmentation model.

[0089] The above segmentation models can be based on the DeepLabv3+ model, as illustrated in Figure 5. This model is a deep learning model used in computer vision and is used to perform extraction or segmentation tasks. Based on a deep convolutional neural network (CNN) architecture, this model features an encoder and decoder structure, making it particularly well-suited for image segmentation.

[0090] Training the above model involves the same process as general deep learning training. Training such a model requires various spine-related image data collected for training. The training images include a first training image captured from a first direction and a second training image captured from a second direction. As shown in FIGS. 4 and 5, pixel-level labeling is performed on the first and second images, and then a learning process is performed to obtain the desired first and second image segmentation models. FIGS. 6 and 7 described above show images in which the vertebrae body and pedicle are labeled in each of a plurality of first and second training images, respectively.

[0091] < Stage S4 >

[0092] This step involves labeling the targets segmented in the previous step. Figure 8 shows the results of labeling the targets segmented in the AP and LL images.

[0093] As shown in Figure 8, the segmented targets were sequentially labeled from top to bottom as L1, L2, L3, L3, L4, and L5, and the same label was given to the same vertebral body.

[0094] < S5 stage >

[0095] In this step, one of multiple targets is selected prior to the process of extracting LL points. For example, as shown in Fig. 9, the L1 target may be selected first and then subjected to subsequent processes.

[0096] < S6 stage >

[0097] The first screw pass in the LL image plane is determined by extracting the LL parameters for the target of the selected label. When planning the pass, it must be set so as not to exceed the pedicle and vertebral body regions. Here, if the slope of only one region, either the vertebral body region or the pedicle region, is reflected, the pass may deviate from the other region. In addition, to prevent pass planning failure due to detection errors occurring during object segmentation, the screw pass is set to the average slope of the two regions.

[0098] To this end, regions of interest (ROIs) are first established for the spinal column and vertebral body. In Fig. 10, (A) represents the vertebral body region, and (B) represents the spinal column region. These ROIs can be obtained using the first image segmentation model described above.

[0099] As illustrated in Figure 11, the vertebral body region (A) is a rectangular region defined by four vertices A1, A2, A3, and A4, and the pedicle region (B) is a rectangular region defined by four vertices B1, B2, B3, and B4, and these two regions are adjacent. A midline (C) passes between the adjacent edges of the two adjacent regions.

[0100] The above midline C passes through the exact center between adjacent sides (B2-B3, A1-A4) of both areas (A, B), and the intersections C1 and C2 are located on the midline C. The intersection C1 is located in the middle of the line segment connecting the vertices B2 and A1 of both areas, and C2 is located in the middle of the line segment connecting the vertices B3 and A4. Therefore, the y-coordinate of C1 (C1.y) is (A1.y+B2.y) / 2, and the z-coordinate of C1 (C1.z) is (A1.z+B2.z) / 2. In addition, the y-coordinate of C2 (C2.y) is (A4.y+B3.y) / 2, and the z-coordinate of C2 (C2.z) is (A4.z+B3.z) / 2.

[0101] Thus, the midline C has the average slope (Ca) for the slopes of both regions in the plane coordinates of the LL image, i.e., the yz plane. This midline C becomes the reference for setting the first screw pass in the LL image plane, and the coordinates C1(y, z), C2(y, z) of the intersection points C1 and C2 in the yz plane coordinate system and the slope Ca of the midline C are expressed by the following formulas.

[0102]

[0103] Figure 12 is a graph showing a temporary point C3 in the intermediate process of setting the first screw pass from the midline C between the pedicle region (B) and the vertebral body region (A) in the yz plane.

[0104] The coordinates of the above temporary point C3 correspond to the coordinates of the intersection point C1 rotated 90 degrees around the intersection point C2 on the above midline C. That is, if the coordinates of C1 in the yz plane coordinate system are (y1, z1), the coordinates of C3 become (z1, -y1). Consequently, the temporary straight line D' connecting the intersection point C2 and the temporary point C3 is a normal line orthogonal to the above midline C, and the straight line equation of this normal line is D':Z = Da(Y - C2.y) + C2.z.

[0105] Fig. 13 illustrates a first screw placement line, i.e., a first screw pass D, which is arranged parallel to the temporary straight line D'. The first screw pass D is perpendicular to the center line C and parallel to the temporary straight line D'. The temporary straight line D' is moved parallel to a predetermined distance so that it passes through the center (B5) or the vicinity of the center of the pedicle region (B) having four vertices B1, B2, B3, and B4, thereby determining the first screw pass D in the y-z plane.

[0106] The y-coordinate B5.y and z-coordinate B5.z of the above B5 are expressed as follows.

[0107] B5.y = (B1.y + B2.y + B3.y + B4.y) / 4

[0108] B5.z = (B1.z + B2.z + B3.z + B4.z) / 4

[0109] The equation of the straight line of the first screw pass (D) with slope Da is expressed as follows.

[0110] D:Z = Da (Y - C2.y) + C2.z

[0111] Da = (C3.z - C2.z) / (C3.y - C2.y)

[0112] The first screw pass D, expressed by the straight line equation (D:z) as above, passes through the outer side of the pedicle (BS, B1-B4) and creates an intersection point E, which becomes the entry point of the surgical screw to be described later.

[0113] The outer side (BS) of the above pedicle is expressed as follows.

[0114] BS: Z = Ba(Y - B1.y) + B1.z

[0115] Ba = (B4.z - B1.z) / (B4.y - B1.y)

[0116] < S7 stage >

[0117] In this step, the coordinates of the entry point of the screw located on the first screw pass D and the target point or terminating point reached by the tip of the surgical screw are calculated.

[0118] The coordinates (Ey, Ez) of the entry point E located on the outer side of the pedicle can be calculated by a first-order simultaneous equation that applies the straight line equation (D:Z) of the first screw pass and the straight line equation (BS:Z) of the outer side BS of the pedicle. The y-coordinate (Ez) and the z-coordinate (Ez) of the entry point can be calculated by a first-order simultaneous equation that applies the straight line equation (D:Z) of the first screw pass and the straight line equation (BS:Z) of the outer side BS of the pedicle.

[0119] D:Z = Da (Y - C2.y) + C2.z

[0120] BS: Z = Ba(Y - B1.y) + B1.z

[0121] First, the two lines are the same at (Ey, Ez), and since Y and Z of the two straight lines are the same, the calculation is as follows.

[0122] Da (Y - C2.y) + C2.z = Ba(Y - B1.y) + B1.z

[0123] The Ey value can be obtained by grouping the terms related to Y in the above equation into one side and separating Y from the other.

[0124] Ey = Y = (B1.z - C2.z + Ba * B1.y - Da * C2.y) / (Da - Ba)

[0125] By substituting Ey obtained from the above equation into the straight line equation (D:Z) of the screw pass, the z-direction coordinate Ez of the entry point can be obtained.

[0126] Ez = Z = Da (Ey - C2.y) + C2.z

[0127] The above calculation process is one of several calculation methods for obtaining the coordinates of the entry point (E), and can also be calculated by other methods, and it is obvious that a specific calculation method does not limit the technical scope of the present disclosure.

[0128] Fig. 14 illustrates the coordinates of the target point (T) or the end point of the surgical screw on the screw path on the yz plane. The coordinates of the target point or the end point are determined by the total length (Ls) of the screw to be used. In one embodiment according to the present disclosure, the total length (Ls) of the screw is set so as not to exceed the length of the side (A1-A2) of the vertebral body region in the direction in which the surgical screw advances, and the diameter (Ds) thereof is set so as not to be larger than the length of the outer side (B1-B4) where the screw entry point is located.

[0129]

[0130] Here, the coordinates of the endpoint T(y, z) are expressed by the equation below.

[0131] Ty = Ey + Ls * Cos(arctan(Da))

[0132] Tz = Ez + Ls * Sin(arctan(Da))

[0133] < S8 stage >

[0134] This step is a parameter extraction step for determining a second screw pass on a second image plane, i.e., the xz plane of the AP image, from the screw pass obtained on the yz plane.

[0135] Figure 15 shows the state in which the second vertebral body region (a) and the left and right second pedicle regions (b, b') inside it are extracted or separated through the AP segmentation model.

[0136] Two straight lines L tilted to the right in Fig. 15 T , L EIt traverses the second vertebral body region (A), and at this time passes through the left and right second pedicle regions (b, b') inside the vertebral body region (A).

[0137] Among the two lines above, “L T " is the end point setting line, and "L E ” is the entry point setting line. As described above, since the AP plane of the second image is aligned with the LL plane of the first image, the entry point and the end point (target point) of the LL plane of the first image are set on the end point setting line (L) on the AP plane of the second image. T ) and the entry point setting line (L E ) can be formed.

[0138] According to one embodiment of the present disclosure, the entry point setting line (L E ) is a normal line extending in the x direction perpendicular to the yz plane from the entry point in the yz plane, and the terminal point setting line (L T ) is a normal line extending in the x direction, which is perpendicular to the yz plane, from the endpoint in the yz plane.

[0139] <S9 단계>

[0140] This step is a step of setting the entry point and the end point in the xz plane on the above entry point setting line and end point setting property.

[0141] The entry and end points in the xz plane are formed on the boundary of the pedicle to determine or form the second screw path in the xz plane. This is to set the safest screw entry path, and the entry and end points are set so that the second screw path for both screws is directed from the outside of the vertebral body toward the center of the vertebral body.

[0142] Figure 16 shows the elliptical second vertebral region (b, b1) within the second vertebral region (a) with vertices a1, a2, a3, and a4 on the xz plane as a dotted line.

[0143] As shown in Fig. 16, the terminal point setting line (L) on the above T ) and the entry point setting line (L E ) passes through the second pedicle area (b, b1) and forms an intersection on the boundary line of the second pedicle area (b, b1). There are four intersections in total, and the end point setting line (L T ) by two intersections, entry point setting line (L E ) form two intersections.

[0144] Among the above intersections, the entry point setting line (L E ), the two intersections located on the outer side of the second vertebral body area (a) are the screw entry points (E L, E R ) is selected, and then, the end point setting line (L T ), the two intersections located in the central region of the second vertebral body region (a) are the screw end points (T L, T R ) is selected. Therefore, the left and right entry points (E) in each pedicle region (b, b1) L , E R ) and left and right endpoints (T L , T R ) are the left and right straight lines (D”) connecting each other, which are the second screw passes.

[0145] <S10 단계>

[0146] In this step, 3D coordinates in the surgical space are obtained through the coordinates of the screw insertion point and end point in the yz plane and the coordinates of the screw insertion point and end point in the xz plane, which are obtained in the same manner as described above.

[0147] At this stage, the screw entry point (E) in the xz plane L, E R ) and the endpoint (T L, T R ) each coordinate E L (x , z), ER (x , z), T L (x , z) T R (x , z) and the coordinates Ez of the entry point Ey and the end point in the yz plane can be used to obtain the coordinates E(x,y,z) of the entry point of the screw in the 3D surgical space and the coordinates T(x,y,z) of the end point. This acquisition of coordinates in the 3D surgical space is possible because the first image and the second image are aligned in the virtual 3D surgical space as described above.

[0148] The above processes were performed for one target, but if there are multiple target targets, that is, if there are multiple target spines and it is not the last target yet in step S11, the process returns to step S5, repeats the process from step S5 to step S10, and then ends.

[0149] The embodiment according to the present disclosure as described above can successfully automatically plan a screw path in a 3D surgical space using only 2D images during surgery. The method and system of the embodiment according to the present disclosure can be implemented at a low cost because a 2D imaging device is applied. The method and system according to the present disclosure is a system that automatically generates a surgical plan based only on 2D images without using a CT or a mobile 3D imaging device, and systematically solves difficulties in surgical planning with 2D image information inferior to 3D, and can reduce errors and radiation exposure due to differences in skill levels, and can shorten the overall surgical time by automatically generating a surgical plan.

[0150] An embodiment according to the present disclosure may be implemented as a computer program medium storing software for performing an image alignment method on a computer. In addition, the present invention may be implemented through an image alignment device that may include a processor, memory, display, etc. that performs the image alignment method described above.

[0151] Additionally, the present invention can also be implemented by a surgical robot system based on the image alignment method described above.

[0152] Referring to FIG. 17, a surgical robot system (1) according to one or more embodiments of the present disclosure applies a 2D image acquisition device (100) as a main image acquisition device. In addition, the robot system (1) is configured to include a surgical robot (200), a position sensor (300), and a navigation system (400), and the surgical robot (200) is configured to include a main body (201), a robot arm (203) equipped with an end effector (203a), and a robot controller (205).

[0153] In an embodiment according to the present disclosure, except for the image acquisition device described above, the spatial alignment unit, the object separation unit, the path generation unit, and the coordinate determination unit may be functionally included in the devices mentioned above, and the spatial alignment unit, etc. may be included in the navigation system (400). In addition, the object separation unit, the path generation unit, the coordinate determination unit, etc. may be implemented by surgical planning software, and in particular, the object separation unit performs extraction or separation of the vertebral body and the pedicle from the first and second 2D images by applying the LL segmentation model and the AP segmentation model trained by deep learning.

[0154] The above 2D image can be acquired by a 2D image acquisition device, for example, a so-called C-arm device having an X-ray source and a detector positioned facing each other on both sides with a patient undergoing surgery in the middle. That is, in the embodiment according to the present disclosure, a C-arm imaging device can be applied as the 2D image acquisition device (100).

[0155] This 2D image acquisition device acquires a 2D first image, for example, an LL image, and a 2D second image, for example, an AP image, of the patient's surgical site to obtain a 3D screw path during surgery. The robot arm (203) is fixedly installed on the robot body (201), and an end effector (203a) from which a surgical tool can be detachably attached is provided at the end. The position sensor (300) is implemented as an OTS that tracks the real-time position of the surgical tool or the end effector (203a) through marker recognition. The controller (205) is provided on the robot body (201) and controls the robot arm (203) according to a surgical plan and control software determined during surgery according to the present disclosure. The navigation system (400) can perform the aforementioned image alignment method to display surgical planning information for surgical tools or implants on a 2D image acquired during the surgery, or to display the real-time location of surgical tools or implants on a 2D image or, in some cases, a 3D image acquired before the surgery, thereby assisting the surgeon's surgical actions. To this end, a display that allows the surgeon to visually compare the real-time location of surgical tools, etc. with the surgical plan and surgical status during the surgery can be connected to the navigation system (400).

[0156] While various embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, modifications to future embodiments of the present invention will not depart from the scope of the present invention.

Claims

1. A step of acquiring a first image in a first direction for a spinal surgical site and a second image in a second direction different from the first direction; A step of defining a virtual 3D surgical space corresponding to the surgical site and aligning the first image and the second image to the surgical space; A step of extracting a first vertebral region and a first pedicle region corresponding to the vertebrae and pedicle from the first image, and extracting a second vertebral region and a second pedicle region within the second vertebral region from the second image; A step of setting a first screw pass that passes through the center or vicinity of the center of the first pedicle region in the first image, the first screw pass being orthogonal to the medial boundary line passing between the first pedicle region and the first vertebral body region; and A 2D image-based surgical planning method, comprising: a step of determining 3D coordinates for insertion of a surgical screw in the surgical space by setting an entry point and an end point on a second screw pass corresponding to the first screw pass in a second pedicle region of the second image based on the first screw pass; 2. In paragraph 1, A 2D image-based surgical planning method, wherein a screw entry point and an end point are set on the first screw pass, and two normal lines extending from each of the entry point and the end point in the first image pass through the second pedicle region in the second image.

3. In paragraph 1, A 2D image-based surgical planning method, wherein the first pedicle region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, the entry point is located on or adjacent to the outer edge of the first pedicle region, and the end point is located within the first vertebral body region.

4. In paragraph 3, A 2D image-based surgical planning method, wherein the entry and end points within the second vertebral body region are set adjacent to the edge of the second pedicle region located within the second vertebral body region or within the second pedicle region.

5. In paragraph 3, A 2D image-based surgical planning method, wherein the entry point is positioned on or adjacent to the edge of one side of the second pedicle region facing the outside of the second vertebral body region, and the end point is positioned on or adjacent to the edge of the other side opposite to the one side.

6. In any one of paragraphs 1 to 5, A 2D image-based surgical planning method, further comprising: a step of detecting the posture of an image acquisition unit by an optical tracking system (OTS) in a step of acquiring a first image in the first direction and a second image in a second direction different from the first direction, and calculating an angle between the second direction and the first direction.

7. In paragraph 6, A 2D image-based surgical planning method, wherein, in the above alignment step, the first image and the second image are aligned in the surgical space by reflecting the angle between them.

8. An image acquisition device for acquiring a first image in a first direction for a spinal surgical site and a second image in a second direction different from the first direction; A space alignment unit defining a virtual 3D surgical space corresponding to the surgical site and aligning the first image and the second image to the surgical space; An object separation unit that extracts a first vertebra region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; A path generation unit for setting a first screw pass that passes through the center or near the center of the first pedicle region in the first image, the first screw pass being orthogonal to the middle boundary line passing between the first pedicle region and the first vertebral body region; and A 2D image-based surgical planning system, comprising: a coordinate determination unit for determining 3D coordinates for screw insertion in the surgical space by setting the entry and end points of the screw corresponding to the first screw pass in the second pedicle region of the second image based on the first screw pass; 9. In paragraph 8, The above path generation part: A 2D image-based surgical planning system that sets a screw entry point and an end point on the first screw pass, and allows two normal lines extending from each of the entry point and the end point in the first image to pass through the second pedicle region in the second image.

10. In paragraph 8, The first vertebral body region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, The above path generation unit: A 2D image-based surgical planning system, wherein the entry point is located on or adjacent to the outer edge of the first vertebral body region, and the end point is located within the first vertebral body region.

11. In paragraph 10, The above path generation unit, A 2D image-based surgical planning system, wherein the entry and end points within the second vertebral body region are set adjacent to the edge of the second pedicle region located within the second vertebral body region or within the second pedicle region.

12. In any one of paragraphs 8 to 11, The above coordinate determination unit; A 2D image-based surgical planning system, wherein the entry point is positioned on or adjacent to the edge of one side of the second pedicle region facing the outside of the second vertebral body region, and the end point is positioned on or adjacent to the edge of the other side opposite to the one side.

13. In paragraph 12, A 2D image-based surgical planning system, comprising an optical tracking system (OTS) that detects the posture of the image acquisition device in the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction, and calculates an angle between the second direction and the first direction.

14. In paragraph 13, The above alignment unit: is a 2D image-based surgical planning system that aligns the first image and the second image in the surgical space by reflecting the angle between the first and second images.

15. In any one of paragraphs 8 to 11, A 2D image-based surgical planning system, comprising an optical tracking system (OTS) that detects the posture of the image acquisition device in the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction, and calculates an angle between the second direction and the first direction.

16. In paragraph 13, The above alignment unit: is a 2D image-based surgical planning system that aligns the first image and the second image in the surgical space by reflecting the angle between the first and second images.

Images