Automatic operation planning method and system based on two-dimensional image

Abstract

A two-dimensional image-based surgical planning method and system are disclosed. The planning method comprises: a step of obtaining a first image and a second image from a spinal surgical site; defining a virtual three-dimensional surgical space and registering the first image and the second image; a step of extracting a first vertebra region and a first pedicle region from the first image, and extracting a second vertebral body region and a second pedicle region within the second vertebral body region from the second image; setting, in the first image, a first screw path passing through the center of the first pedicle region or near the center; and a step of setting an entry point and an end point on a second screw path corresponding to the first screw path in a second pedicle region of the second image on the basis of the first screw path.

Description

Technical Field

[0001] The present disclosure relates to an automatic surgical planning method and system in a surgical robot system, and more particularly, to an automatic surgical planning method and system based on two-dimensional images, wherein a surgical planning path can be generated in a three-dimensional image space through a plurality of 2D images. Background Art

[0002] In spinal surgery such as pedicle screw fixation using a surgical robot system, a three-dimensional image that can be viewed in the axial direction is required. Instead of using 3D images that enable axial viewing, such as computed tomography (CT), A CT (Computed Tomography) image is used as the axial view image.

[0003] Typically, during intraoperative surgery, a two-dimensional image acquisition device, such as a so-called C-arm, is used to verify in real time whether the surgical procedure is being performed according to the preoperative planning. In order to use these acquired two-dimensional images to navigate surgical tools according to the pre-established plan, it is necessary to register the three-dimensional images acquired before surgery with the two-dimensional images acquired during surgery. However, during this registration process, problems can arise, such as delayed surgery and incomplete registration, leading to registration errors.

[0004] For surgical planning based solely on three-dimensional images obtained before surgery without the assistance of two-dimensional images during surgery, exposure to radiation when obtaining three-dimensional images such as CT images and problems arising from the difference between the patient's condition before surgery and the patient's condition during surgery are inevitable.

[0005] However, if the surgical plan is established based on the two-dimensional X-ray images obtained during surgery, there is no need to register with the images obtained before surgery, and the surgical plan can be changed flexibly as needed even during surgery. However, this procedure requires the surgeon to estimate the surgical location in three dimensions while viewing the two-dimensional images. Consequently, this type of surgical planning requires the surgeon to have experience with highly complex surgeries. Even if this surgical plan is implemented, two-dimensional images lack the clarity and three-dimensional spatial information of three-dimensional CT images. Therefore, unless the surgeon is experienced, success is unlikely.

[0006] In view of these problems, the development of surgical planning methods and systems based on two-dimensional images is highly preferred, which reduce the problem of exposure to radiation, have performance that is not inferior to that of existing surgical planning systems based on three-dimensional images, and can reduce the cost burden on the system structure. Summary of the Invention Technical issues

[0007] The present disclosure provides a two-dimensional image-based surgical planning method and system.

[0008] The present disclosure provides a two-dimensional image-based surgical planning method and system, wherein the three-dimensional position, posture, and shape of a surgical target vertebra can be estimated based on the two-dimensional image without using a three-dimensional image.

[0009] Furthermore, the present disclosure provides a method and system that can establish a technical plan based on two-dimensional images obtained in real time during surgery without prior imaging and perform surgery according to the technical plan. Technical Solution

[0010] The two-dimensional image-based surgical planning method according to the present disclosure includes:

[0011] The step of obtaining a first image in a first direction and a second image in a second direction of a spinal surgical site by an image obtaining device, wherein the second direction is different from the first direction;

[0012] A step of defining, by a spatial registration unit, a virtual three-dimensional surgical space corresponding to the surgical site and registering the first image and the second image to the virtual three-dimensional surgical space;

[0013] The object segmentation unit extracts a first vertebra region and a first pedicle region corresponding to the vertebra and the pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image;

[0014] a step of setting, by a pathway generating unit, a first screw path in the first image through a center or near a center of a first pedicle region, wherein the first screw path is perpendicular to a boundary line passing midway between the first pedicle region and a first vertebral body region; and

[0015] A step of determining three-dimensional coordinates by a coordinate determination unit based on the first screw path by setting an entry point and an end point on a second screw path corresponding to the first screw path in a second pedicle area of ​​the second image, wherein the three-dimensional coordinates are used to insert a surgical screw in the surgical space.

[0016] According to one or more embodiments,

[0017] A screw entry point and an end point may be set on the first screw path, and two normal lines respectively extending from the entry point and the end point in the first image may pass through the second pedicle region in the second image.

[0018] According to one or more embodiments,

[0019] The first pedicle region may have a medial edge opposite the first vertebral body region and a lateral edge opposite the medial edge, the entry point may be located on or near the lateral edge of the first pedicle region, and the terminal point may be located within the first vertebral body region.

[0020] According to one or more embodiments,

[0021] The entry point and the end point in the second vertebral body region may be set on an edge of the second pedicle region in the second vertebral body region or near an edge inside the second pedicle region.

[0022] According to one or more embodiments,

[0023] The entry point may be located on an edge of one side of the second pedicle region facing the outside of the second vertebral body region or near the edge within the second pedicle region, and the terminal point may be located on an edge of the other side opposite to the one side or near the edge inside the other side.

[0024] According to one or more embodiments,

[0025] The step of obtaining the first image in the first direction and the second image in a second direction different from the first direction may further include obtaining the first image in the first direction and the second image in the second direction by an optical tracking system. The method further comprises the step of detecting the posture of the image acquisition unit by using an optical transmission system (OTS) to calculate the angle between the second direction and the first direction.

[0026] According to one or more embodiments,

[0027] In the registration step, the first image and the second image may be registered to the surgical space by reflecting the angle.

[0028] The two-dimensional image-based surgical planning system according to the present disclosure includes:

[0029] An image acquisition device that acquires a first image in a first direction and a second image in a second direction of a spinal surgical site, wherein the second direction is different from the first direction;

[0030] a spatial registration unit defining a virtual three-dimensional surgical space corresponding to the spinal surgical site and registering the first image and the second image to the virtual three-dimensional surgical space;

[0031] an object segmentation unit that extracts a first vertebra region and a first pedicle region corresponding to the vertebra and the pedicle from the first image, and extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image;

[0032] a pathway generating unit that sets a first screw pathway through a center or near a center of a first pedicle region in the first image, wherein the first screw pathway is perpendicular to a boundary line passing through a middle portion between the first pedicle region and a first vertebral body region; and

[0033] A coordinate determining unit is configured to determine three-dimensional coordinates for inserting a screw into the surgical space by setting an entry point and an end point of a screw corresponding to the first screw path in a second pedicle region of the second image based on the first screw path.

[0034] According to one or more embodiments,

[0035] The pathway generation unit

[0036] A screw entry point and an end point are set on the first screw path, and two normal lines respectively extending from the entry point and the end point in the first image are allowed to pass through the second pedicle region in the second image.

[0037] According to one or more embodiments,

[0038] The first pedicle region may have a medial edge opposite the first vertebral body region and a lateral edge opposite the medial edge, and,

[0039] The pathway generation unit can

[0040] The entry point is located on or near the outer edge of the first pedicle region, and the termination point is located within the first vertebral body region.

[0041] According to one or more embodiments,

[0042] The pathway generation unit

[0043] The entry point and the end point in the second vertebral body region may be set on an edge of the second pedicle region in the second vertebral body region or near an edge inside the second pedicle region.

[0044] According to one or more embodiments,

[0045] The coordinate determination unit

[0046] The entry point may be arranged to be located on the edge of the second pedicle region facing the outside of the second vertebral body region or close to the edge inside the second pedicle region, and the terminal point may be arranged to be located on the edge on the opposite side of the edge of the second pedicle region or close to the edge inside the opposite side of the edge of the second pedicle region.

[0047] According to one or more embodiments,

[0048] An optical tracking system (OTS) may be included, which calculates the angle between the second direction and the first direction by detecting the posture of the image acquisition device during the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

[0049] According to one or more embodiments,

[0050] The registration unit may register the first image and the second image in the surgical space by reflecting the angle. Beneficial effects

[0051] Using O-arms, 3D C-arms, and mobile 3D imaging devices allows viewing of patient images in 3D, allowing surgical planning to be done by viewing areas that cannot be viewed in 2D. However, these devices expose patients and medical personnel to large amounts of radiation, take a long time to image, and are expensive, making them unavailable in many hospitals. However, 2D imaging devices have the disadvantage that, even though they are widely used in many hospitals, they are still expensive. It is cheap and easy to use, but the two-dimensional images do not provide sufficient information about the patient, and multiple radiographs are required to expose more radiation. This may also cause differences in surgical planning and surgical results depending on proficiency.

[0052] According to the method and system disclosed herein, surgical plans are automatically generated based on two-dimensional images only without using CT or mobile three-dimensional imaging devices. The system can solve the difficulties in surgical planning caused by two-dimensional image information being inferior to three-dimensional image information and reduce errors caused by proficiency and exposure to radiation. In addition, since the surgical plan is automatically generated, the entire operation time can be shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The present invention is a flowchart of the sequential processing steps in the two-dimensional image-based surgical planning method according to one or more embodiments of the present disclosure.

[0054] Figure 2The following schematically illustrates the registration of a first image and a second image to a virtual three-dimensional surgical space and the definition of three-dimensional coordinates according to one or more embodiments of the present disclosure.

[0055] Figure 3 An output image obtained by applying a first image segmentation model according to one or more embodiments of the present disclosure to segment a vertebral body and a pedicle from a lateral-lateral (LL) image is shown.

[0056] Figure 4 An output image obtained by applying a second image segmentation model according to one or more embodiments of the present disclosure to segment a vertebral body and a pedicle from an anterior-posterior (AP) image is shown.

[0057] Figure 5 A schematic diagram of a process for generating a deep learning model according to one or more embodiments of the present disclosure.

[0058] Figure 6 The vertebral body is shown in a plurality of first learning images according to one or more embodiments of the present disclosure. This is an image showing the state where the vertebra body and pedicle are labeled.

[0059] Figure 7 The vertebral body is shown in a plurality of second learning images according to one or more embodiments of the present disclosure. This is an image showing the state where the vertebra body and pedicle are labeled.

[0060] Figure 8 Results of labeling objects segmented in a first image and a second image according to one or more embodiments of the present disclosure are shown.

[0061] Figure 9 An example of an L1 target being selected first according to one or more embodiments of the present disclosure is shown.

[0062] Figure 10 A vertebral body region A and a pedicle region B set in a first image according to one or more embodiments of the present disclosure are shown.

[0063] Figure 11 1 is a graph showing the yz plane of the vertex coordinates of the vertebral body region A and the pedicle region B and the middle line between the vertebral body region A and the pedicle region B set in the first image according to one or more embodiments of the present disclosure.

[0064] Figure 12To illustrate the use of one or more embodiments according to the present disclosure Figure 11 Graph in the yz plane of a temporary straight line formed by rotating the median line by 90°.

[0065] Figure 13 FIG. 2 is a yz plane diagram illustrating first screw paths D arranged in parallel on a provisional straight line D′ according to one or more embodiments of the present disclosure.

[0066] Figure 14 FIG. 1 is a yz plane diagram illustrating set coordinates of a target point T or terminal point of a surgical screw on a screw path according to one or more embodiments of the present disclosure.

[0067] Figure 15 1 and 2 are images showing a state in which the second vertebral body region a and the second pedicle regions b and b′ are extracted or separated by the first image segmentation model in one or more embodiments of the present disclosure.

[0068] Figure 16 FIG. 1 is an xz plane diagram illustrating an elliptical second pedicle region b and b1 within a second vertebral body region a formed by vertices a1 , a2 , a3 , and a4 in one or more embodiments of the present disclosure using dotted lines.

[0069] Figure 17 The figure schematically illustrates the structure of a surgical robot system according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0070] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified into various other forms, and the scope of the present invention should not be construed as being limited to the embodiments described in detail below. The embodiments of the present invention are preferably interpreted as being provided for the purpose of more completely explaining the present invention to those skilled in the art. The same symbols represent the same elements throughout. However, various elements and regions in the drawings are schematically illustrated, and therefore, the concept of the present invention is not limited to the relative sizes or intervals illustrated in the drawings.

[0071] Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used only to distinguish one component from other components. For example, without departing from the scope of the present invention, the first component may be named the second component, and conversely, the second component may be named the first component.

[0072] The terms used in this application are only used to illustrate specific embodiments and are not intended to limit the concept of the present invention. Unless otherwise apparent from the context, expressions in the singular include expressions in the plural. In this context, expressions such as "comprising" or "having" should be understood as specifying the presence of the features, quantities, steps, operations, elements, components or combinations thereof described in the specification, and not as precluding the presence or additional possibility of one or more other features, quantities, operations, elements, components or combinations thereof.

[0073] Unless otherwise defined, the terms used herein include technical and scientific terms and have the same meaning as commonly understood by those of ordinary skill in the technical field to which the concept of the present invention pertains. Also, it should be understood that terms commonly used and predefined as such should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an overly formal sense unless explicitly defined herein.

[0074] In cases where some embodiments are implemented differently, a specific process order may be carried out in an order different from the described order. For example, two consecutively described processes may be carried out substantially simultaneously or in an order opposite to the described order.

[0075] Hereinafter, based on one or more embodiments, a method and system for surgical planning based on two-dimensional images will be described in detail.

[0076] Figure 1 is a timing flowchart for surgical planning based on two-dimensional images according to the present disclosure.

[0077] Figure 1 The process shown includes steps S5 to S10 repeatedly performed on a plurality of vertebral bodies as the target. When the number of vertebral bodies is one, a surgical plan for one vertebra is established in one flow.

[0078] According to the step description Figure 1 as follows.

[0079] The method according to the present disclosure generally includes a preparatory work process, an LL point generation process, an AP point generation process, and an optimization and automation process.

[0080] <Step S1>

[0081] Step S1 is the step of obtaining two-dimensional images, i.e., a first image and a second image, of the affected part of the patient in two directions Steps. For example, an LL image (Lateral-Lateral Image) is obtained as the first image, and for example, an AP image (Anterior-Posterior Image) is obtained as the second image. The two directions for obtaining these first and second images can be orthogonal to each other, intersecting, or can intersect at any angle while maintaining an almost orthogonal state. To obtain these first and second images, an X-ray source in the form of a point source is applied on one side centered on the affected part of the patient, i.e., the spinal region to be operated on, and an image acquisition device with an X-ray detector, such as a so-called C-arm device, is applied on the other side opposite thereto. According to another embodiment of the present disclosure, markers attached to a part of the patient's body, markers of a calibrator capable of estimating the positions of the source and the detector (Detector), and a commercial OTS with a sensor for detecting the three-dimensional position of the marker are used to measure the angle between the first direction and the second direction for obtaining the first and second images.

[0082] <Step S2>

[0083] In Step S2, the first and second images are registered to a virtual three-dimensional surgical viewing space, and the coordinates of the pixels of these images in the three-dimensional space are determined or defined. Figure 2 Schematic diagram of the registration of the first and second images obtained by X-rays in the form of a conical or conical beam to the virtual three-dimensional surgical space and the definition of the three-dimensional coordinates. As shown, the source and the detector geometrically have a perspective projection relationship formed by the conical X-ray beam.

[0084] The virtual surgical space, i.e., the virtual space, has axes in the X, Y, and Z directions. In the AP image, the up-down direction (head- foot) is the Z axis, and the left-right direction (left arm - right arm) is the X axis. In the LL image, the left-right direction (abdomen - back part) is the Y axis, and the up-down direction (head - foot) is also the Z axis. Therefore, the first image, such as the LL image, is parallel to the Y-Z plane, and the second image, such as the AP plane, is parallel to the X-Z plane.

[0085] In this embodiment, a first image of the surgical site, such as an AP image and an LL image, is obtained by a two-dimensional image acquisition device such as a C-arm device, and the image acquisition device such as the C-arm device is registered to a coordinate system, the coordinate system being based on markers provided on a part of the patient's body or multiple calibrators provided in other surgical spaces Based on the marker of the (Calibrator). The applicant's registered patent No. KR2203544 discloses a technique for registering a two-dimensional image into a three-dimensional space, and the entire content of the '544 is incorporated into this application by reference, but this disclosure is not limited by the technical limitations of a specific registration technique.

[0086] <Step S3>

[0087] In Step S3, the vertebral body and pedicle as the target object are extracted or segmented from the first image and the second image obtained and registered in the virtual three-dimensional surgical space in the above process. (segmentation).

[0088] The segmentation of the object can be performed by applying the LL segmentation model and the AP segmentation model trained by deep learning. Figure 3 Shows the output image obtained by segmenting the vertebral body and pedicle from the LL image by applying the first image segmentation model. body) and pedicle, Figure 4 Shows the output image obtained by segmenting the vertebral body and pedicle from the AP image by applying the second image segmentation model.

[0089] As Figure 5 shown, the segmentation model as described above can be based on the DeepLabv3+ model. This model is one of the deep learning models used in the field of computer vision and is used to perform extraction or segmentation work. This model is based on the structure of a deep convolutional neural network (Deep Convolutional Neural Network, CNN), and in particular has an encoder and a decoder structure, so it is suitable for image segmentation.

[0090] The training of the model includes processes such as in ordinary deep learning training. To train such a model, various spine-related image materials collected for training are required. The training images include the first training image photographed in the first direction and the second training image photographed in the second direction, where, as Figure 4 and Figure 5 shown, the first image and the second image are labeled at the pixel level, and then the learning process is performed, whereby the desired first image segmentation model and second image segmentation model can be obtained. As described above, Figure 6 and Figure 7 respectively show the vertebral bodies in each of the multiple first learning images and second learning images. An image of the state of labeling the (vertebra body) and the pedicle.

[0091] <Step S4>

[0092] This step is to perform labeling on the target segmented in the previous step. Figure 8 Shows the results of labeling the targets segmented in the AP image and the LL image.

[0093] As Figure 8 shown, the segmented targets are labeled as L1, L2, L3, L3, L4, and L5 in sequence from top, and the same label is given to the same vertebra body.

[0094] <Step S5>

[0095] This step is to select one target from multiple targets before the process of extracting LL points. One can first select the L1 target and perform subsequent processes, as Figure 9 shown.

[0096] <Step S6>

[0097] Determine the first screw path in the LL image plane by extracting LL parameters for the selected labeled target. When planning the path, it should be set so as not to deviate from the pedicle and vertebra body regions. Here, if only the inclination of either the vertebra body region or the pedicle region is reflected, the path may deviate from the other region. Also, in order to prevent path planning failure caused by detection errors occurring during the segmentation of the object, the screw path is set based on the average inclination of the inclinations of the two regions.

[0098] To this end, first set the regions of interest (ROI, region of interest) for the pedicle and the vertebra body. In Figure 10 , (A) is the vertebra body region, and (B) is the pedicle region. Such ROIs can be obtained through the above-mentioned first image segmentation model.

[0099] As Figure 11 shown, the vertebra body region A is a quadrilateral region composed of four vertices A1, A2, A3, and A4, and the pedicle region B is a quadrilateral region composed of four vertices B1, B2, B3, and B4. The two regions are adjacent. The midline C passes between the adjacent sides of the two adjacent regions.

[0100] The center line C passes through the center between the two adjacent sides B2-B3 and A1-A4 of the two areas A and B, and the intersection points C1 and C2 are located on the center line C. The intersection point C1 is located in the middle of the line connecting the vertices B2 and A1 of the two areas, and C2 is located in the middle of the line connecting the vertices B3 and A4. Therefore, the y coordinate of C1 (C1.y) is (A1.y+B2.y) / 2, the z coordinate of C1 (C1.z) is (A1.z+B2.z) / 2. And, the y coordinate of C2 (C2.y) is (A4.y+B3.y) / 2, the z coordinate of C2 (C2.z) is (A4.z+B3.z)2.

[0101] Thus, the center line C has plane coordinates in the LL image, namely, the average tilt Ca of the two regions in the yz plane. This center line C serves as the reference for setting the first screw path in the LL image plane. In the yz plane coordinate system, the coordinates C1(y,z) and C2(y,z) of the intersection points C1 and C2, and the tilt Ca of the center line C are expressed as follows.

[0102]

[0103] Figure 12 1 is a diagram showing a temporary point C3 in the middle of setting the first screw path from the middle line C between the pedicle region B and the vertebral body region A in the yz plane.

[0104] The coordinates of the temporary point C3 match those obtained by rotating the intersection point C1 90° around the intersection point C2 on the center line C. That is, if the coordinates of C1 in the yz plane coordinate system are (y1, z1), then the coordinates of C3 are (z1, -y1). As a result, the temporary straight line D' connecting the intersection point C2 and the temporary point C3 is a normal line perpendicular to the center line C. The equation of this normal line is D': Z = Da(Y - C2.y) + C2.z.

[0105] Figure 13 The figure shows a first screw placement line arranged parallel to the provisional straight line D', i.e., a first screw path D. The first screw path D is orthogonal to the center line C and parallel to the provisional straight line D'. The provisional straight line D' is moved parallel to the center B5 of the pedicle region B having four vertices, B1, B2, B3, and B4, or near the center thereof, thereby defining the first screw path D on the yz plane.

[0106] The y-coordinate B5.y and the z-coordinate B5.z of B5 are expressed as follows.

[0107] B5.y = (B1.y + B2.y + B3.y + B4.y) / 4

[0108] B5.z = (B1.z + B2.z + B3.z + B4.z) / 4

[0109] The expression of the linear equation of the first screw path D with an inclination Da is as follows.

[0110] D: Z = Da(Y - C2.y) + C2.z

[0111] Da = (C3.z - C2.z) / (C3.y - C2.y)

[0112] The first screw path D expressed as the above linear equation (D: z) passes through the outer side BS, B1 - B4 of the pedicle to generate an intersection point E, which serves as the entry point of the surgical screw to be described later.

[0113] The following equation represents the outer side BS of the pedicle.

[0114] BS: Z = Ba(Y - B1.y) + B1.z

[0115] Ba = (B4.z - B1.z) / (B4.y - B1.y)

[0116] <Step S7>

[0117] In this step, the coordinates of the entry point of the screw located on the first screw path D and the target point or terminating point of the surgical screw are calculated. point).

[0118] For the coordinates (E.y, E.z) of the entry point E located on the outer side of the pedicle, the y - coordinate (E.z) and z - coordinate (E.z) of the entry point can be calculated by applying the linear equation of the first screw path (D: Z) and the linear equation of the outer side BS of the pedicle (BS: Z) in a system of linear equations.

[0119] D: Z = Da(Y - C2.y) + C2.z

[0120] BS: Z = Ba(Y - B1.y) + B1.z

[0121] First, at (E.y, E.z), the Y and Z of the two lines are the same, so the calculation method is as follows.

[0122] Da(Y - C2.y) + C2.z = Ba(Y - B1.y) + B1.z

[0123] In the above formula, the terms related to Y can be grouped on one side, and Y can be separated from each to obtain the value of E.y.

[0124] E.y = Y = (B1.z - C2.z + Ba * B1.y - Da * C2.y) / (Da - Ba)

[0125] By substituting E.y obtained from the above formula into the linear equation (D: Z) of the screw path, the coordinate E.z of the entry point in the z direction is obtained.

[0126] E.z = Z = Da(E.y - C2.y) + C2.z

[0127] The above calculation process is one of the many calculation methods for calculating the coordinates of the entry point E, and it is also possible to calculate the coordinates of the entry point E by other methods. It is obvious that the technical scope of this disclosure is not limited to a specific calculation method.

[0128] Figure 14 Illustrate the set coordinates of the target point T or the end point of the surgical screw on the screw path in the y - z plane. The coordinates of the target point or the end point depend on the total length Ls of the screw to be used. In one embodiment according to this disclosure, the total length Ls of the screw is set not to exceed the length of the side A1 - A2 of the vertebral body region in the direction of the surgical screw, and its diameter Ds is set not to be greater than the length of the outer side B1 - B4 where the screw entry point is located.

[0129]

[0130] Here, the coordinates of the end point T(y, z) are expressed using the following formula.

[0131] T.y = E.y + Ls * Cos(arctan(Da))

[0132] T.z = E.z + Ls * Sin(arctan(Da))

[0133] <Step S8>

[0134] This step is a parameter extraction step, which is used to determine the second screw path on the x - z plane of the second image plane, that is, the AP image, from the screw path obtained in the y - z plane.

[0135] Figure 15 Show the state of extracting or separating the second vertebral body region a and the left and right second pedicle regions b and b' inside the second vertebral body a through the AP segmentation model.

[0136] In Figure 15 it, two straight lines L T and L E cross the second vertebral body region A, at this time, passing through the left and right second pedicle regions b and b' inside the vertebral body region A.

[0137] Among the two straight lines, "L T " is the endpoint setting line, "L E " is the entry point setting line. As described above, The AP plane of the second image is registered to the LL plane of the first image. Therefore, an endpoint setting line L T and an entry point setting line L E can be formed on the AP plane of the second image based on the entry point (Entry Point) and the endpoint (Target Point) of the LL image of the first image.

[0138] According to an embodiment of the present disclosure, the entry point setting line L E is a normal line extending in the x direction perpendicular to the y - z plane from the entry point in the y - z plane, and the endpoint setting line L T is a normal line extending in the x direction perpendicular to the y - z plane from the endpoint in the y - z plane.

[0139] <Step S9>

[0140] This step is to set the entry point and the endpoint on the x - z plane on the entry point setting line and the endpoint setting line.

[0141] The entry point and the endpoint on the x - z plane determine or form a second screw path on the x - z plane and are formed on the boundary line of the pedicle. This is to set the screw entry path most safely, and the entry point and the endpoint are set so that the second screw paths for the two - side screws are from the outer direction of the vertebral body towards the central direction of the vertebral body.

[0142] Figure 16 The second pedicle regions b and b1 in the form of an ellipse within the second vertebral body region a formed by the vertices a1, a2, a3, a4 on the x - z plane are shown in dashed lines.

[0143] As Figure 16 shown, on the x - z plane, the endpoint setting line L T and the entry point setting line L E form an intersection point on the boundary line of the two second pedicle regions b and b1 while passing through the two second pedicle regions b and b1. The total number of intersection points is four, formed by the endpoint setting line L TTwo intersection points are formed, and line L is set by the entry point E Another two intersection points are formed.

[0144] Among the said intersection points, line L is set by the entry point E Among the four intersection points, two intersection points located outside the second vertebral body region a are selected as screw entry points E L and E R , and among the four intersection points formed by line L set by the terminal point T , two intersection points located in the central region of the second vertebral body region a are selected as screw terminal points T L and T R . Therefore, the left and right entry points E L and E R in each pedicle region b and b1 and the left and right terminal points T L and T R are connected to each other by the left and right straight lines D” as the second screw path.

[0145] <S10 step>

[0146] This step is to obtain the three-dimensional coordinates in the surgical space by using the coordinates of the screw insertion points and terminal points in the y-z plane and the coordinates of the screw insertion points and terminal points in the x-z plane obtained by the method described above.

[0147] In this step, the coordinates E L and E R of the screw entry points and the terminal points T L and T R in the x-z plane can be used to obtain the coordinates E(x, y, z) of the screw entry point and the coordinates T(x, y, z) of the terminal point in the three-dimensional surgical space. The coordinates in this three-dimensional surgical space can be obtained because the first image and the second image are registered to the virtual three-dimensional surgical space as described above.

[0148] The above processes are all performed on one target. However, in the case of multiple target objects, that is, in the case of multiple surgical target vertebrae and not reaching the last target in the S11 step, return to the S5 step and repeatedly execute the processes from the S5 step to the S10 step, and then end.

[0149] ​​​​​​​​According to the embodiment of the present disclosure as described above, it is possible to successfully automatically plan the screw path in the three-dimensional surgical space using only the two-dimensional images during surgery. The method and system according to the embodiment of the present disclosure uses a two-dimensional imaging device, so it can be implemented with lower components. The method and system according to the present disclosure does not use CT or a mobile three-dimensional imaging device, but only automatically generates a surgical plan based on two-dimensional images. The system can solve the difficulties in surgical planning caused by two-dimensional image information being inferior to three-dimensional image information and reduce errors and radiation exposure caused by differences in proficiency. In addition, the automatic generation of surgical plans can shorten the entire operation time.

[0150] The embodiments of the present disclosure may be embodied as a computer program medium configured to store software for executing the image registration method on a computer. Furthermore, the present invention may also be implemented by an image registration device, which may include a processor, a memory, a display, etc., for executing the above-described image registration method.

[0151] Furthermore, the present invention can also be implemented by a surgical robot system based on the above-mentioned image registration method.

[0152] Reference Figure 17 In one or more embodiments of the present disclosure, the surgical robot system 1 uses a two-dimensional image acquisition device 100 as the main image acquisition device. In addition, the present robot system 1 is configured to include a surgical robot. 200 , a position sensor 300 and a navigation system 400 , the surgical robot 200 is configured to include a body 201 , a robot arm 203 including an end effector 203 a , and a robot controller 205 .

[0153] In an embodiment of the present disclosure, in addition to the aforementioned image acquisition device, the spatial registration unit, the object isolation unit, the pathway generation unit, and the coordinate determination unit may be functionally included in the aforementioned devices, and the spatial registration unit, etc. may be included in the navigation system 400. Furthermore, the object isolation unit, the pathway generation unit, the coordinate determination unit, etc. may be implemented by surgical planning software. In particular, the object isolation unit utilizes the LL segmentation model and the AP segmentation model trained through deep learning to extract or separate the vertebral body and pedicle from the two-dimensional first and second images.

[0154] The two-dimensional image can be obtained by a two-dimensional image acquisition device, such as a so-called C-arm device, which has an X-ray source and a detector disposed on opposite sides of the patient undergoing surgery, with the patient being interposed therebetween. Specifically, in embodiments of the present disclosure, a C-arm imaging device can be used as the two-dimensional image acquisition device 100.

[0155] This two-dimensional image acquisition device acquires a two-dimensional first image, such as an LL image, and a two-dimensional second image, such as an AP image, of the patient's surgical site to determine a three-dimensional screw path during surgery. The robotic arm 203 is fixed to the robot body 201, and an end effector 203a is disposed at the end of the robotic arm 203, capable of attaching and detaching surgical tools. The position sensor 300 is embodied as an OTS that tracks the real-time position of the surgical tool or end effector 203a by identifying markers. The controller 205 is disposed on the robot body 201 and controls the robotic arm 203 according to the surgical plan determined during surgery and control software according to the present disclosure. The navigation system 400 can assist the surgeon in performing the surgical procedure by performing the aforementioned image registration method, by displaying surgical plan information regarding surgical tools or implants on a two-dimensional image obtained during surgery, or by displaying the real-time position of the surgical tools or implants on the two-dimensional image, or in some cases, on a three-dimensional image obtained before surgery. To this end, a display can be connected to the navigation system 400, allowing the surgeon to visually compare and view the surgical plan and the real-time position of surgical tools, etc., as part of the live surgical process.

[0156] Although various embodiments of the present invention have been described in detail above, it should be possible for a person skilled in the art to which the present invention pertains to various modifications without departing from the spirit and scope of the present invention as defined by the appended claims. Therefore, modifications to the embodiments of the present invention made later will not depart from the technical scope of the present invention.

Claims

1. A surgical planning method based on two-dimensional images, comprising: The step of obtaining a first image in a first direction and a second image in a second direction of a spinal surgical site, wherein the second direction is different from the first direction; A step of defining a virtual three-dimensional surgical space corresponding to the spinal surgical site and registering the surgical space with the first image and the second image; The steps of extracting a first vertebra region and a first pedicle region corresponding to the vertebra and the pedicle from the first image, and extracting the second vertebral body region and a second pedicle region within the second vertebral body region from the second image; The step of defining a first screw path through a center or near a center of a first pedicle region in the first image, wherein the first screw path is orthogonal to a boundary line passing midway between the first pedicle region and a first vertebral body region; and The step of determining three-dimensional coordinates for inserting a surgical screw into the surgical space by setting an entry point and an end point on a second screw path corresponding to the first screw path in a second pedicle region of the second image based on the first screw path.

2. The two-dimensional image-based surgical planning method according to claim 1, wherein: A screw entry point and an end point are set on the first screw path, and two normal lines respectively extending from the entry point and the end point of the first image pass through the second pedicle region of the second image.

3. The two-dimensional image-based surgical planning method according to claim 1, wherein: The first pedicle region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, the entry point is located on or near the outer edge of the first pedicle region, and the terminal point is located in the first vertebral body region.

4. The two-dimensional image-based surgical planning method according to claim 3, wherein: The entry point and the terminal point in the second vertebral body region are set on the edge of the second pedicle region in the second vertebral body region or close to the edge inside the second pedicle.

5. The two-dimensional image-based surgical planning method according to claim 3, wherein: The entry point is arranged on one side edge of the second pedicle region facing the outside of the second vertebral body region or close to the edge inside the second pedicle region, and the terminal point is arranged on the edge of the other side opposite to the one side or close to the edge inside the other side.

6. The two-dimensional image-based surgical planning method according to any one of claims 1 to 5, further comprising: In the step of obtaining a first image in the first direction and a second image in a second direction different from the first direction, a step of calculating an angle between the second direction and the first direction by detecting a posture of an image obtaining unit through an optical tracking system.

7. The two-dimensional image-based surgical planning method according to claim 6, wherein: In the registration step, the first image and the second image are matched in the surgical space by reflecting the angle.

8. A two-dimensional image-based surgical planning system, comprising: An image acquisition device that acquires a first image in a first direction and a second image in a second direction of a spinal surgical site, wherein the second direction is different from the first direction; a spatial registration unit, which defines a virtual three-dimensional surgical space corresponding to the spinal surgical site and registers the first image and the second image with the surgical space; An object separation unit extracts a first vertebra region and a first pedicle region corresponding to the vertebra and the pedicle from the first image and extracts a second vertebra body region and a second pedicle region within the second vertebra body region from the second image; a pathway generating unit that sets a first screw path through a center or near a center of a first pedicle region in the first image, wherein the first screw path is perpendicular to a boundary line passing through an intermediate portion between the first pedicle region and a first vertebral body region; and A coordinate determining unit sets a step of determining three-dimensional coordinates for inserting a screw into the surgical space by setting an entry point and an end point of a screw corresponding to a first screw path in a second pedicle region of the second image based on the first screw path.

9. The two-dimensional image-based surgical planning system according to claim 8, wherein: The pathway generation unit sets a screw entry point and an end point on the first screw path and enables two normal lines respectively extending from the entry point and the end point in the first image to pass through the second pedicle region in the second image.

10. The two-dimensional image-based surgical planning system according to claim 8, wherein: The first pedicle region has a medial edge opposite the first vertebral body region and a lateral edge opposite thereto, and, The pathway generation unit arranges the entry point to be located on an outer edge of a first pedicle region or inside the first pedicle region near an outer edge, and arranges the terminal point to be located within the first vertebral body region.

11. The two-dimensional image-based surgical planning system according to claim 10, wherein: The pathway generation unit sets the entry point and the terminal point in the second vertebral body region to be on an edge of the second pedicle region in the second vertebral body region or to be close to the edge inside the second pedicle region.

12. The two-dimensional image-based surgical planning system according to any one of claims 8 to 11, wherein: The coordinate determination unit arranges the entry point to be located on an edge of one side of the second pedicle region facing the outside of the second vertebral body region or close to an edge inside the second pedicle region, and arranges the terminal point to be located on an edge of the other side opposite to the one side or close to an edge inside the other side.

13. The two-dimensional image-based surgical planning system according to claim 12, comprising: An optical tracking system calculates an angle of the second direction relative to the first direction by detecting a posture of the image acquisition device during the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

14. The two-dimensional image-based surgical planning system according to claim 13, wherein: The registration unit registers the first image and the second image in the surgical space by reflecting the angle.

15. The two-dimensional image-based surgical planning system according to any one of claims 8 to 11, wherein: An optical tracking system calculates an angle of the second direction relative to the first direction by detecting a posture of the image acquisition device during the process of acquiring a first image in the first direction and a second image in a second direction different from the first direction.

16. The two-dimensional image-based surgical planning system according to claim 13, wherein The registration unit registers the first image and the second image in the surgical space by reflecting the angle.

Images