Image-based 3D automatic surgery planning method and system

Abstract

An image-based 3D surgery planning method and system are disclosed. The planning method comprises steps of: obtaining a first image and a second image from an image of a spinal surgery site; obtaining first and second reference lines from the first image and the second image; and obtaining a third image from the spinal surgery site, on the basis of the first and second reference lines, and determining a 3D screw path by setting a proximal setting coordinate and a distal setting coordinate in the central region of at least one of the left and right third pedicles, the proximal setting coordinate being for determining a screw path, the distal setting coordinate being set at or near the center of a vertebral body region.

Description

Image-based 3D automatic surgical planning method and system

[0001] The present disclosure relates to an automatic surgical planning method and system in a surgical robot system, and more particularly, to an automatic surgical planning method and system capable of generating a surgical planning path in a 3D image space.

[0002] Spinal surgical procedures, such as pedicle screw fixation using a surgical robot, require 3D images capable of an axial view. Therefore, 3D images capable of an axial view, such as computed tomography (CT) images, rather than 2D images, are used as these axial view images.

[0003] Typically, intraoperatively, a 2D image acquisition device, such as a C-arm, is used to verify in real time whether the surgery is proceeding as planned. To navigate surgical tools as planned using these acquired 2D images, the intraoperative 2D images must be registered with the preoperative 3D images. However, the process of registering 2D images to 3D images can lead to delays in surgery and registration errors due to incomplete registration.

[0004] Meanwhile, surgical planning based only on 3D images acquired before surgery without the help of 2D images during surgery cannot avoid problems that arise due to radiation exposure caused by acquiring 3D images, such as CT images, and differences in the patient's condition before surgery and during surgery.

[0005] However, if a surgical plan is established based on 2D X-ray images acquired during surgery, there is no need for registration with images acquired before surgery, and the flexibility to change the surgical plan as needed can be expected even during surgery. However, this process requires the operating surgeon to estimate the location of the desired 3D space in 3D space while looking at the 2D image. Consequently, this surgical plan requires a high level of surgical experience from the operating surgeon. Even if this is realized, it is difficult to expect a successful surgery unless the surgery is performed by an experienced surgeon, because 2D images lack image information in terms of image quality and 3D spatial information compared to 3D CT images.

[0006] Considering these points, the development of an image-based surgical planning method and system based on an image system rather than a subjective surgical planning method that relies on the surgeon's experience is desirable in many respects.

[0007] <Prior Art Literature>

[0008] US8,335,553

[0009] CN107157579

[0010] US 7,787,932 B2

[0011] KR101264198 B1

[0012] US0659599 B2

[0013] US6226548 B1

[0014] US20070270866 A1

[0015] US20050192575 A1

[0016] The present disclosure proposes a method and system for automatically establishing an image-based 3D surgical plan based on multiple images taken in different directions and performing surgery based on the plan.

[0017] The present disclosure proposes a method and a system for applying the same, which can automatically establish a 3D surgical plan using three or more images from different directions acquired before and / or during surgery, and perform surgery based on the plan.

[0018] Image-based 3D surgical planning method according to the present disclosure:

[0019] A step of one or more image acquisition devices acquiring one or more surgical site images including image information in multiple directions including different first directions, second directions, and third directions with respect to a spinal surgical site of a patient;

[0020] A step of the image processing unit extracting a first image in a first direction and a second image in a second direction from an image of a surgical site;

[0021] A step of the object separation unit extracting a first vertebral region and a first pedicle region from the first image, and extracting a second vertebral region and left and right second pedicle regions within the second vertebral region from the second image;

[0022] A step of setting a reference line, wherein a first reference line passing through a first coordinate set in the central region of a first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through both a third coordinate and a fourth coordinate set in the central region of the left and right vertebral regions in the second image;

[0023] A step of the image processing unit extracting a third image in the third direction from the surgical site image based on the first reference line and the second reference line;

[0024] A step of extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and

[0025] The path setting unit sets a screw path passing through at least one central region of the left and right third pedicles and the third vertebral body region, and sets proximal setting coordinates on the second pedicle, and sets distal setting coordinates on a third reference line passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

[0026] According to one or more embodiments,

[0027] The above object extraction unit can extract the vertebral body region and the pedicle region by applying a model generated by machine learning.

[0028] According to one or more embodiments,

[0029] The first reference line may be set to be perpendicular or at an arbitrary angle to the midline passing through the area between the first pedicle region and the first vertebral body region, but may be set to pass through the central area of ​​the first pedicle region or be parallel thereto at an arbitrary set distance therefrom.

[0030] According to one or more embodiments,

[0031] The central region may include the center and / or vicinity of the corresponding vertebral body region and / or pedicle region.

[0032] According to one or more embodiments,

[0033] The third image can be obtained from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0034] According to one or more embodiments,

[0035] The third image is obtained from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line can pass through the central region of each of the left and right pedicles in the second image.

[0036] According to one or more embodiments,

[0037] The second reference line may be set parallel to any straight line connecting the midpoint between the center of one side of the spinal cord region and the coordinates of one corner of the third image close thereto, and the midpoint between the center of the other side of the spinal cord region and the coordinates of the other corner of the third image close thereto.

[0038] According to one or more embodiments,

[0039] The above object separation unit extracts the neural foramen region between the left and right third pedicle regions from the third image,

[0040] The above path setting section:

[0041] In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and

[0042] The distal setting coordinates may be set at a point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other or at a location near the point, or at an intersection where the third reference line and the screw path reference line intersect outside the vertebral body.

[0043] According to one or more embodiments,

[0044] The above path setting unit can be set so that the screw path reference line on which the screw path is placed passes through the above distal setting coordinates.

[0045] According to one or more embodiments,

[0046] A screw starting point for the pedicle and a target point within the vertebral body are set on the first reference line, and the distance between the screw starting point and the target point can be set as the length of the screw used in the surgery.

[0047] According to one or more embodiments,

[0048] The first vertebral body region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, the starting point is located on or adjacent to the outer edge of the first vertebral body region, and the target point can be located within the first vertebral body region.

[0049] According to one or more embodiments,

[0050] The starting point may be positioned on one side of the second vertebral body region facing outward from the second vertebral body region or adjacent to the edge from the inside thereof, and the target point may be positioned on the other side opposite to the one side or adjacent to the edge from the inside thereof.

[0051] According to one or more embodiments,

[0052] The surgical site image including the above multi-directional image information can be acquired by a 3D image acquisition device.

[0053] Image-based 3D surgical planning system according to the present disclosure:

[0054] One or more image acquisition devices for acquiring one or more images of the affected area having image components in multiple directions with respect to the spinal surgical site;

[0055] An image processing unit responsible for image processing related to surgery, including extracting a first image in a first direction from one or more of the above-mentioned images, a second image in a second direction different from the first direction, and a third image in a third direction different from the first and second directions;

[0056] An object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a pedicle region from the third image;

[0057] A reference line setting unit that sets a first reference line passing through a first coordinate set in the central region of the first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through the third coordinate and the fourth coordinate set in the central region of the left and right vertebral regions in the second image; and

[0058] A screw path setting unit is provided for determining a 3D screw path by setting a screw path passing through at least one central region of the left and right third pedicles and the third vertebral body region, setting a proximal setting coordinate on the second pedicle, and setting a distal setting coordinate through which a reference line of the screw path passes on a third reference line passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

[0059] According to one or more embodiments, in the image-based 3D surgical planning system, the reference line setting unit may set the first reference line to orthogonally cross a midline passing through an area between the first pedicle region and the first vertebral body region.

[0060] According to one or more embodiments, in the image-based 3D surgical planning system, the image processing unit can obtain a third image from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0061] According to one or more embodiments, in the image-based 3D surgical planning system, the image processing unit obtains the third image from the surgical site image based on coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line can pass through the central region of each of the left and right pedicles in the second image.

[0062] According to one or more embodiments, in the image-based 3D surgical planning system, the second reference line may be set parallel to an arbitrary straight line connecting a midpoint between the center of one of the spinal pedicle regions and a coordinate of a corner of the third image close thereto, and a midpoint between the center of the other spinal pedicle region and a coordinate of the corner of the other image close thereto.

[0063] According to one or more embodiments, in the image-based 3D surgical planning system, the object segmentation unit extracts a neural canal region between the left and right third pedicle regions from the third image,

[0064] The above path setting section:

[0065] In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and

[0066] The distal setting coordinates may be set at a point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other or at a location near the point, or at an intersection point where the third reference line and the screw pass reference line intersect outside the vertebral body.

[0067] According to one or more embodiments, in the image-based 3D surgical planning system, the path setting unit can set the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

[0068] According to one or more embodiments, in an image-based 3D surgical planning system, a screw starting point for a pedicle and a target point within a vertebral body are set on the first reference line, and the distance between the screw starting point and the target point (or end point) can be set as the length of a screw used in the surgery.

[0069] The present invention proposes a method and system capable of automatically establishing a 3D surgical plan using multiple image data. This invention addresses the limitations inherent in surgical methods, such as the limitations of existing surgical plans where the outcome is limited by the surgeon's experience, thereby enabling efficient surgical planning and execution. According to the present invention, by automatically generating a surgical plan, the overall surgical time can be shortened, while maintaining high-quality results.

[0070] FIG. 1 is a flowchart of sequential processing steps of an image-based 3D surgical planning method according to one or more embodiments of the present disclosure.

[0071] FIG. 2 schematically illustrates the definition of 3D coordinates for a virtual 3D surgical space of a first image and a second image, according to one or more embodiments of the present disclosure.

[0072] FIG. 3 shows an output image in which a vertebra body and a pedicle are extracted, segmented, or separated from an LL image by applying a first image segmentation model according to one or more embodiments of the present disclosure.

[0073] FIG. 4 shows an output image in which a vertebra body and a pedicle are segmented from an AP image by applying a second image segmentation model according to one or more embodiments of the present disclosure.

[0074] FIG. 5 schematically illustrates a process for generating a deep learning model applied to one or more embodiments according to the present disclosure.

[0075] FIG. 6 shows an image of a plurality of first learning images labeled with respect to a vertebra body and a pedicle according to one or more embodiments of the present disclosure.

[0076] FIG. 7 shows an image of a plurality of second learning images labeled with respect to the vertebra body and pedicle according to one or more embodiments of the present disclosure.

[0077] FIG. 8 shows an image of a plurality of third learning images labeled with respect to a vertebra body, a pedicle, and a foramen according to one or more embodiments of the present disclosure.

[0078] FIG. 9 shows the results of labeling targets segmented in a first image and a second image according to one or more embodiments of the present disclosure.

[0079] FIG. 10 illustrates a first selected L1 target according to one or more embodiments of the present disclosure.

[0080] FIG. 11 illustrates a vertebral body region (A) and a pedicle region (B) set in a first image, according to one or more embodiments of the present disclosure.

[0081] FIG. 12 is a graph of a yz plane showing the coordinates of vertices of a vertebral body region (A) and a pedicle region (B) set in a first image and the midline between these regions, according to one or more embodiments of the present disclosure.

[0082] FIG. 13 is a graph of the yz plane showing a temporary straight line obtained by rotating the midline of FIG. 11 by 90 degrees, according to one or more embodiments according to the present disclosure.

[0083] FIG. 14a is a yz-plane graph showing an initial first baseline (D1) arranged parallel to a temporary straight line D', according to one or more embodiments of the present disclosure.

[0084] FIG. 14b is a yz-plane graph showing a modified first baseline (D1a) that is aligned with and replaces the first baseline D1 of FIG. 14a, according to one or more embodiments of the present disclosure.

[0085] FIG. 15a is a yz plane graph showing the set coordinates of a target point (T) or an end point of a surgical screw on the initial first reference line of FIG. 14a, according to one or more embodiments of the present disclosure.

[0086] FIG. 15b is a yz plane graph showing the set coordinates of a target point (T) or an end point of a surgical screw on the modified first reference line of FIG. 14b according to one or more embodiments of the present disclosure.

[0087] FIG. 16 is an image showing a state in which a second vertebral body region (a) and a second pedicle region (b, b') are extracted or separated using a first image segmentation model in one or more embodiments of the present disclosure.

[0088] FIG. 17 is an xz plane graph showing an elliptical second vertebral region (b, b') within a second vertebral region (a) having vertices a1, a2, a3, a4, in one or more embodiments of the present disclosure, represented by a dotted line.

[0089] FIG. 18 illustrates a state in which a first reference line and a second reference line are set in a first image and a second image acquired according to one or more embodiments.

[0090] FIG. 19(A) illustrates a third image acquired along an axial slice path according to one or more embodiments, and (B) illustrates a vertebral body region, a pedicle region, and a neural canal region as ROIs extracted from the third image.

[0091] FIG. 20a illustrates a process for setting a screw path baseline for an ROI of a third image, according to one or more embodiments.

[0092] FIG. 20b shows another example of a process for setting an alternative screw path baseline for an ROI of a third image, according to one or more embodiments.

[0093] FIG. 21a illustrates the ROI and the third image with a screw path reference line set for the ROI of the third image according to the embodiment of FIG. 20a.

[0094] FIG. 21b illustrates the ROI and the third image with the screw path reference line set for the ROI of the three images according to the embodiment of FIG. 20b.

[0095] FIG. 22a illustrates a state in which the entry point and target point (or end point) of the screw pass of the total length are set on the screw pass reference line of FIG. 21a according to the embodiment of FIG. 20.

[0096] FIG. 22b illustrates a state in which the entry point and target point (or end point) of the screw pass of the total length are set on the screw pass reference line of FIG. 21b according to the embodiment of FIG. 20a.

[0097] FIG. 23 shows a schematic configuration of a surgical robot system according to one or more embodiments of the present disclosure.

[0098] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the embodiments described below. It is preferable to interpret that the embodiments of the present invention are provided to more completely explain the present invention to a person having average knowledge in the art. Like reference numerals denote like elements throughout. Furthermore, various elements and areas in the drawings are schematically drawn. Therefore, the present invention is not limited by the relative sizes or intervals drawn in the accompanying drawings.

[0099] While terms like "first" and "second" may be used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and vice versa, without departing from the scope of the present invention.

[0100] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the concept of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the expressions "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

[0101] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, it is to be understood that commonly used terms, such as those defined in dictionaries, should be interpreted to have a meaning consistent with their meaning within the relevant technical context, and should not be interpreted in an overly formal sense unless explicitly defined herein.

[0102] In some embodiments, where the implementation is otherwise feasible, a particular process sequence may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0103] Hereinafter, an image-based 3D surgical planning method and system are described in detail according to one or more embodiments.

[0104] FIG. 1 is a sequential flowchart of an image-based 3D surgical planning method according to the present disclosure.

[0105] The process of Figure 1 includes steps S6 to S14 that are repeatedly performed for multiple target vertebral bodies, and in the case of one vertebral body, a surgical plan for one vertebra is established in one flow.

[0106] One or more embodiments described below are described based on a 3D image obtained by a 3D image acquisition device, but it should be noted that the embodiments of the present disclosure are not proposed by a 3D image obtained by a 3D image acquisition device. The present disclosure includes any device capable of producing an image obtained by one or more image acquisition devices capable of capturing image information from three different directions, particularly a combined or mutually aligned three-directional image.

[0107] A step-by-step explanation of Figure 1 is as follows.

[0108] The method according to the present disclosure generally includes a preparatory work process, an LL point generation process, an AP point generation process, an axial (AX) point generation process, and an optimization and automation process.

[0109] <S1 단계>

[0110] Step S1 is a step of capturing multiple images of the patient's affected area from various directions. This step utilizes 3D images obtained by a 3D image acquisition device according to the present embodiment.

[0111] FIG. 2 schematically illustrates the definition of 3D coordinates for a virtual 3D surgical space of a first image and a second image, according to one or more embodiments of the present disclosure.

[0112]

[0113] <S2 단계>

[0114] Step S2 is the step of loading a 3D image with axes in the X, Y, and Z directions into the image processing unit for processing. In this step, 3D images and coordinate information are loaded.

[0115] <S3 단계>

[0116] Step S3 involves processing the loaded 3D image from two directions to obtain the first and second images as DRR (Digitally Reconstructed Radiograph) images. In this step, the first image, for example, a Lateral-Lateral Image (LL image), is obtained, and the second image, for example, an Anterior-Posterior Image (AP image), is obtained. These images can be obtained by orthogonal projection in the first and second directions on the 3D image.

[0117] In the above LL image, i.e., the first image, the left-right direction (back to stomach) is the Y-axis, and the up-down direction (head to feet) is also the Z-axis. And in the above AP image, i.e., the second image, the up-down direction (head to feet) is the Z-axis, and the left-right direction (left arm to right arm) is the X-axis, so the first image, e.g., the LL image, is parallel to the YZ plane, and the second image, e.g., the AP plane, is parallel to the XZ plane.

[0118] <S4 단계>

[0119] In step S4, the vertebral body and pedicle are extracted or segmented or separated as target objects from the first and second images obtained in the aforementioned process.

[0120] Object segmentation can be performed by applying the LL segmentation model and AP segmentation model trained by deep learning.

[0121] Figure 3 shows an output image in which the vertebra body and pedicle are segmented from the LL image by applying the first image segmentation model.

[0122] Figure 4 shows an output image in which the vertebra body and pedicle are segmented from the AP image by applying the second image segmentation model.

[0123] As illustrated in FIG. 5, the above segmentation models may be based on the DeepLabv3+ model, and it is understood that other models may be applied according to other embodiments of the present disclosure.

[0124] The DeepLavb3+ model is a deep learning model used in computer vision, used for extraction and segmentation tasks. Based on a deep convolutional neural network (CNN) architecture, it features an encoder and decoder structure, making it particularly well-suited for image segmentation.

[0125] Training the above model involves the same process as typical deep learning training. Training this model requires a variety of spine-related image data collected for training purposes. The training images include a first training image captured from a first direction and a second training image captured from a second direction. As illustrated in Figures 4 and 5, pixel-level labeling is performed on the first and second images, followed by a learning process to obtain the desired first and second image segmentation models.

[0126] Figures 6 and 7 show images of a plurality of first and second learning images, respectively, with the vertebra body and pedicle labeled.

[0127] FIG. 8 shows an image of a plurality of third learning images labeled with respect to a vertebra body, a pedicle, and a foramen according to one or more embodiments of the present disclosure.

[0128] < S5 stage >

[0129] This step involves labeling the targets segmented in the previous step. Figure 9 shows the results of labeling the targets segmented in the AP and LL images.

[0130] As shown in Figure 9, the segmented targets were sequentially labeled from top to bottom as L1, L2, L3, L3, L4, and L5, and the same label was given to the same vertebral body.

[0131] < S6 stage >

[0132] As illustrated in Fig. 10, this step is a step of selecting one target among multiple targets prior to the process of extracting LL points. For example, the L1 target may be selected first and then subjected to subsequent processes.

[0133] <S7-S8 단계>

[0134] The extraction of LL parameters for the target of the selected label is performed, and the first reference line in the LL image plane is determined through this. This reference line is the first axial slice path (axial slice path (1)) for extracting the aforementioned third image, which corresponds to one axis of the third-direction image plane to be performed later. When planning the first reference line, it must be set so as not to go beyond the pedicle and vertebral body regions. Here, if the slope of only one region of the vertebral body region or the pedicle region is reflected, the path may go outside the other region, and in order to prevent a path planning failure due to a detection error that occurs during object segmentation, the first reference line is set to the average slope of the slopes of the two regions.

[0135] For this purpose, first, a region of interest (ROI) is set for the spinal cord and vertebral body.

[0136] In Figure 11, (A) is the vertebral body region, and (B) is the pedicle region. These ROIs can be obtained by the first image segmentation model described above.

[0137] As illustrated in Figure 12, the vertebral body region (A) is a rectangular region defined by four vertices A1, A2, A3, and A4, and the pedicle region (B) is a rectangular region defined by four vertices B1, B2, B3, and B4, and these two regions are adjacent. A midline (C) passes between the adjacent edges of the two adjacent regions.

[0138] The above midline C passes through the exact center between adjacent sides (B2-B3, A1-A4) of both areas (A, B), and the intersections C1 and C2 are located on the midline C. The intersection C1 is located in the middle of the line segment connecting the vertices B2 and A1 of both areas, and C2 is located in the middle of the line segment connecting the vertices B3 and A4. Therefore, the y-coordinate of C1 (C1.y) is (A1.y+B2.y) / 2, and the z-coordinate of C1 (C1.z) is (A1.z+B2.z) / 2. In addition, the y-coordinate of C2 (C2.y) is (A4.y+B3.y) / 2, and the z-coordinate of C2 (C2.z) is (A4.z+B3.z) / 2.

[0139] Thus, the midline C has the average slope (Ca) for the slopes of both regions in the plane coordinates of the LL image, i.e., the yz plane. This midline C serves as a reference for setting the first reference line in the LL image plane, and the coordinates C1(y, z), C2(y, z) of the intersection points C1 and C2 in the yz plane coordinate system and the slope Ca of the midline C are expressed by the following formulas.

[0140]

[0141] Figure 13 shows a temporary point C3 in the intermediate process of establishing a first reference line from the midline C between the pedicle region (B) and the vertebral body region (A) in the yz plane.

[0142] The coordinates of the above temporary point C3 correspond to the coordinates of the intersection C1 rotated 90 degrees around the intersection C2 on the above midline C. That is, if the coordinates of C1 in the yz plane coordinate system are (y1, z1), the coordinates of C3 become (z1, -y1). Consequently, the temporary straight line D'0 connecting the intersection C2 and the temporary point C3 is a normal line orthogonal to the above midline C, and the straight line equation of the normal line is D0:Z = Da(Y - C2.y) + C2.z.

[0143] Fig. 14a illustrates an initial first baseline (D1) arranged parallel to the temporary straight line D'. The initial first baseline D1 intersects or is perpendicular to the midline C at a predetermined angle and is parallel to the temporary straight line D'. The temporary straight line D' is moved parallel to a predetermined distance so that it passes through the center area (B) of the pedicle region (B) having four vertices, such as B1, B2, B3, and B4, that is, the center (B5) or the vicinity of the center, thereby determining the initial first baseline D1 in the y-z plane.

[0144] The y-coordinate B5.y and z-coordinate B5.z of the above B5 are expressed as follows.

[0145] B5.y = (B1.y + B2.y + B3.y + B4.y) / 4

[0146] B5.z = (B1.z + B2.z + B3.z + B4.z) / 4

[0147]

[0148] The equation of the straight line of the first reference line (D1) having slope Da is expressed as follows.

[0149] D1:Z = Da (Y - C2.y) + C2.z

[0150] Da = (C3.z - C2.z) / (C3.y - C2.y)

[0151]

[0152] The first reference line D1, expressed by the straight line equation (D1:z) as above, passes through the outer side of the pedicle (BS, B1-B4) and creates an intersection point E1. This intersection point E1 becomes the start point or entry point for determining the length of the surgical screw to be described later.

[0153] The outer side (BS) of the above pedicle is expressed as follows.

[0154] BS: Z = Ba(Y - B1.y) + B1.z

[0155] Ba = (B4.z - B1.z) / (B4.y - B1.y)

[0156]

[0157] Next, the coordinates of the start point located on the initial first reference line D1 and the target point, which is the terminating point reached by the end of the surgical screw, are calculated.

[0158] The coordinates (Ey, Ez) of the starting point E located on the outer side of the pedicle can be calculated by a first-order simultaneous equation that applies the straight line equation (D1:Z) of the first reference line and the straight line equation (BS:Z) of the outer side BS of the pedicle. The y-coordinate (Ez) and the z-coordinate (Ez) of the starting point can be calculated by a first-order simultaneous equation that applies the straight line equation (D1:Z) of the first reference line and the straight line equation (BS:Z) of the outer side BS of the pedicle.

[0159] D1:Z = Da (Y - C2.y) + C2.z

[0160] BS: Z = Ba(Y - B1.y) + B1.z

[0161] First, the two lines are the same at (Ey, Ez), and since Y and Z of the two straight lines are the same, the calculation is as follows.

[0162] D1a(Y - C2.y) + C2.z = Ba(Y - B1.y) + B1.z

[0163] The value of Ey can be obtained by grouping the terms related to Y in the above equation into one side and separating Y from the other.

[0164] Ey = Y = (B1.z - C2.z + Ba * B1.y - Da * C2.y) / (Da - Ba)

[0165] By substituting Ey obtained from the above equation into the straight line equation (D1:Z) of the reference line D1, the coordinate Ez in the z direction of the starting point can be obtained.

[0166] Ez = Z = Da (Ey - C2.y) + C2.z

[0167] The above calculation process is one of several calculation methods for obtaining the coordinates of the starting point (E), and can also be calculated by other methods, and it is obvious that a specific calculation method does not limit the technical scope of the present disclosure.

[0168] FIG. 14b illustrates another embodiment of the present disclosure, wherein a modified first baseline (D1a) is established based on the initial first baseline (D1). The modified first baseline (D1a) replaces the initial first baseline D1 obtained in FIG. 14a.

[0169] The above modified first reference line (D1a) is obtained by moving the initial first reference line (D1) parallel to the center of the vertebral body, and the distance of the parallel movement (G1) may correspond to the diameter or radius of the screw to be used in the surgery, and according to other embodiments, may be larger or smaller than the diameter of the screw.

[0170] The purpose of shifting the initial reference line parallel to the inside of the vertebral body in this way is to prevent the screw from being exposed to the outside of the vertebral body due to its size, and depending on the patient's condition, if the pedicle is not significantly offset from the vertebral body, the initial first reference line can be applied as the screw path reference line. However, if the pedicle is significantly offset from the vertebral body due to some reason or due to congenital causes, the modified first reference line can be used as the screw path reference line.

[0171] That is, the embodiment according to the present disclosure selectively applies one of the reference lines D1 and D1a as a reference line for setting the final first screw pass, and the first reference line referred to below is D1 or D1a.

[0172] Figures 15a and 15b show the setting of the entry points (E1, E2) and target points (T1, T2) of the surgical screw on the initial first reference line (D1) and the modified first reference line, respectively.

[0173] First, referring to Fig. 15a, a target point (T), which is an end point where the tip of the surgical screw is located on the initial first reference line (D1), is set on the yz plane.

[0174] The coordinates of the above target point (T) are determined by the total length (Ls) of the screw to be used. In one embodiment according to the present disclosure, the total length (Ls) of the screw is set so as not to exceed the length of the side (A1-A2) of the vertebral body region in the direction in which the surgical screw advances, and its diameter (Ds) is set so as not to be larger than the length of the outer side (B1-B4) where the screw entry point is located.

[0175]

[0176] Here, the coordinates of the target point T(y, z) are expressed by the equation below.

[0177] Ty = Ey + Ls * Cos(arctan(Da))

[0178] Tz = Ez + Ls * Sin(arctan(Da))

[0179] The length of the line connecting the coordinates of the starting point and the coordinates of the target point obtained through the above process corresponds to the length of the screw to be used in the surgery.

[0180] Meanwhile, referring to FIG. 15b, the entry point (E2) and target point (T2) of the surgical screw are set on the modified first screw pass (D1a).

[0181] The coordinates of the starting point (E2) and the target point (T2) established on the above-mentioned modified first reference line (D1a) are set by mathematical calculations as described above. As illustrated in Fig. 14b, the above-mentioned modified first reference line (D1a) passes through the vicinity of the center (B5) of the pedicle within the pedicle.

[0182] <S9-S10 단계>

[0183] This step obtains a second reference line, i.e., an axial slice path (2), which is paired with the first reference line in the yz plane.

[0184] Figure 16 is an image showing a state in which the second vertebral body region (a) and the second pedicle region (b, b') are extracted or separated using the first image segmentation model.

[0185] As illustrated in Fig. 16, the ROI (a) of the vertebral body and the left and right pedicles (b), (b') can be extracted through a segmentation model for the second image, i.e., the AP image.

[0186] A key point in establishing the second baseline in the AP image is that, when acquiring the third image described below, the AX image, the slice cross-section image that maximizes the left and right pedicles must be acquired. This requires reflecting the degree of vertebral rotation, thereby securing the largest pedicle area in the AX image.

[0187] Therefore, to reduce the slope of the ROI (a) of the vertebral body and the segmentation error of the two left and right pedicle regions b and b', the average slope value is obtained.

[0188] The above average slope increases or decreases depending on the arrangement of the left and right pedicles, and the temporary second reference line (d') representing this average slope is obtained as follows.

[0189] Figure 17 is an xz plane graph showing the elliptical second vertebral region (b, b') within the second vertebral region (a) having vertices a1, a2, a3, and a4 as a dotted line.

[0190] As shown in Fig. 17, first, a midpoint (d1) between the center (b5) of the left pedicle (b) and the corner (a1) of the vertebral body region closest thereto and a midpoint (d2) between the center (c5) of the right pedicle (b') and the corner (a2) of the vertebral body region closest thereto are obtained, and a temporary second reference line (d') having an average slope is obtained by connecting these two midpoints (d1, d2).

[0191] Figure 18 shows the first reference line and the second reference line set in the first image and the second image.

[0192] First, referring to FIG. 18, by applying the YZ position of the first reference line (D1 or D1a) in the first image obtained through the above process and the slope of the temporary reference line (d') in the X, Z plane in the second image to obtain a third image in the AX plane of the 3D image, a final second reference line (d) that passes through the center of both pedicles on average is obtained, as in the second image on the right side of FIG. 18. Therefore, through this process, it is possible to obtain a third image in the axial direction, i.e., an AX image, from the slice plane according to the first and second reference lines, as shown in (A) of FIG. 19.

[0193] Here, the temporary second reference line (d') illustrated in Fig. 17 is depicted as spanning the upper side of both pedicles (b, b'), and in Fig. 18 is depicted as passing through the central regions of both pedicles (b, b'), i.e., the vicinity of the center. The second reference line (d) in Fig. 18 represents the final second reference line when the slope of the second reference line in the XZ plane is applied to the YZ position of the first reference line. That is, the final second reference line is a line resulting from translating the temporary second reference line (d') in parallel to the average center of both pedicles (b, b').

[0194] <S11 단계>

[0195] Figure 19 (A) illustrates the third image described above, and (B) illustrates the vertebral body region, the pedicle region, and the neural canal region as ROIs extracted from the third image.

[0196] This step is a step of extracting AX parameters for the AX image of (A) of Fig. 19 obtained as described above.

[0197] The extraction of the above regions is performed by the object segmentation unit, and the segmentation of the object can be performed by applying the AX segmentation model trained by deep learning using labeled samples for the vertebra body, pedicle, and foramen in the AX image as shown in FIG. 18, similar to the LL segmentation model and AP segmentation model trained by deep learning as described above.

[0198] Figure 19 (B) shows the vertebral body region (a), left and right pedicle regions (b, b'), and the foramen region (f) between the left and right pedicles extracted by the segmentation for the AX image.

[0199] Figures 20a and 20b illustrate another embodiment of the process of setting a screw path baseline for the ROI of the third image.

[0200] First, the embodiment of Fig. 20a will be described.

[0201] (A) to (D) of FIG. 20a are processes for setting a screw path reference line on which a screw path is placed in an AX image, which is a surgical screw insertion or placement plane.

[0202] (A) A center point “fc” is set in the central region of the neural canal region, and a center point “ad” is also set in each central region of the spinal canal region, and then a third reference line (L) is set passing through these two points fc and ad.

[0203] (B) The intersection points a'c and ac between the third reference line and the inner and outer borders of the vertebral body region are set. This intersection point ac becomes the distal setting coordinate through which the screw path reference line passes. Meanwhile, the distance a'c - ac represents the width of the vertebral body in the direction of the third reference line, and this distance can be applied to determine the total length of the screw.

[0204] (C) Set the first screw pass reference line (LP) passing through the above intersection point ac and the center of the pedicle (b'c) on one side (left in the drawing), i.e., the proximal setting coordinate.

[0205] (D) A second screw pass reference line (RP) is set that passes through the above intersection point ac and the center of the other side (right side in the drawing) of the pedicle (bc), i.e., the proximal setting coordinate.

[0206] The first and second screw pass reference lines (LP, RP) set as above are the axis lines along which the surgical screw is installed.

[0207] The first and second screw pass reference lines (LP, RP) above are set in slightly different directions in the following manner.

[0208] The embodiment of Fig. 20b is described.

[0209] Figure 20b is, as in Figure 20a, a process of setting a screw path reference line on which a screw path is placed in the AX image, which is a surgical screw insertion or placement plane, in (A) to (D).

[0210] (A) A center point “fc” is set in the central region of the neural canal region, and a center point “ad” is also set in each central region of the spinal canal region, and then a third reference line (L) is set passing through these two points fc and ad.

[0211] (B) The intersection points a'c and ac1 between the third reference line and the inner and outer borders of the vertebral body region are set. This intersection point ac1 becomes the distal setting coordinate through which the screw path reference line passes. Here, a modified intersection point ac1', i.e., the distal setting coordinate, which is located distally from the intersection point ac1 on the screw path reference line (L) and away from the vertebral body, is set. This modified intersection point ac1' is located an arbitrary distance (G2) away from the intersection point ac1 above it.

[0212] This arbitrary distance (G2) is the average thickness or height (G2) of the pedicle, and according to other embodiments, may have another positive value smaller or larger than this.

[0213] Meanwhile, the distance a'c - ac1 represents the width of the vertebral body in the direction of the third reference line, and this distance can be applied to determine the total length of the screw.

[0214] (C) Establish the first screw pass reference line (LP) passing through the above modified intersection point ac2 and the center of the pedicle (b'c) on one side (left in the drawing).

[0215] (D) A second screw pass reference line (RP) is set that passes through the intersection point ac2 and the center of the pedicle (bc) on the other side (right side in the drawing).

[0216] The first and second screw pass reference lines (LP, RP) set as above are modified axial lines on which surgical screws are installed, and the angle between them is narrower than the angle between the first and second screw pass reference lines (LP, RP) obtained in the process of FIG. 20, and therefore, the tips of the two surgical screws placed on the two reference lines LP and RP do not meet within the vertebral body region.

[0217] Each of (A) of FIG. 22a and FIG. 22b shows the first and second screw pass reference lines set by the above process according to the embodiment of FIG. 20a and FIG. 20b, and each of (B) of these shows a state in which the reference lines are mapped (overlaid) on the AX image.

[0218] As illustrated in (B) of FIG. 22a following the embodiment of FIG. 20a, a starting point or entry point (LE, RE) of a surgical screw is formed at the intersection of the first and second screw pass reference lines and the outer borders of both pedicles. This entry point is set at the proximal set coordinates through which the screw pass reference lines pass.

[0219] And, as shown in (B) of FIG. 22b according to the embodiment of FIG. 20b,

[0220] The first and second screw pass reference lines pass through the center (bc) of each pedicle and the modified intersection point ac2, and the entry point of the screw is set on the outer surface of the pedicle.

[0221] <S12-S13 단계>

[0222] Steps S12-S13 are the steps to finalize the surgical plan by setting the screw target points (RT, LT) away from the starting point (LE, RE), which is the final entry point of the surgical screw.

[0223] The above entry point is located at the proximal set coordinate where the screw enters the pedicle.

[0224] In this step, the total screw length (Ls) obtained in the process of setting the first reference line mentioned above or the width of the vertebral body in the direction of the first reference line can be applied to determine the screw length, or the width of the vertebral body in the direction of the third reference line (the distance a'c-ac) can also be considered and determined together. In addition, it can be adjusted by the operating surgeon considering the above results as needed. Meanwhile, the diameter of the screw needs to be set to be smaller than the average diameter of the pedicle or the diameter of the smaller pedicle among the left and right pedicles, and this can also be changed by the operating surgeon.

[0225] According to the embodiment of FIGS. 20A and 20B, (A) of FIGS. 23A and 23B shows a screw pass with the screw total length (Ls) applied to the ROI area, and (B) shows a state in which the screw pass is determined in the third image.

[0226] The above screw path has 3D coordinates, which are recognized by the surgical system during surgery and the surgery is performed as is.

[0227] FIGS. 23a and 23b illustrate a state in which an entry point and a target point (or an end point) of a screw pass of a total length are set on the screw pass reference line of FIGS. 22a and 22b according to one or more embodiments.

[0228] Referring to FIG. 23a according to the embodiment of FIG. 20a, entry points (LE, RE) and target points (LT, RT) located on the surface of the pedicle are formed on the first and second screw pass reference lines (LP, RP), and the distance between the entry point (LE) and the target point (LT) on each reference line corresponds to the total length of the screw.

[0229] Referring to FIG. 23b according to the embodiment of FIG. 20b, there are modified entry points (LE', RE'). These modified entry points (LE', RE') are spaced apart from the entry points (LE, RE) on the surface of the pedicle by an arbitrary distance on the corresponding reference line. This spacing is to take into account the gap caused by the patient's skin and / or muscle tissue, etc., rather than directly contacting the rear end (head) of the screw held by the surgical drill when the screw is fixed to the pedicle during actual surgery. Accordingly, the target point within the vertebral body is also moved from the original position of LT, LP to the modified LT' and RT'. The positions of these entry points and target points can be flexibly modified and changed based on the screw pass criteria determined in the above process, according to the condition of the patient's surgical site, and according to the judgment of the surgeon.

[0230] The above processes are performed for one target, and if there are multiple target targets, that is, if there are multiple vertebrae to be operated on and it is not the last target yet in step S14, the process returns to step S6, repeats the process from step S6 to step S14, and then ends.

[0231] The embodiments of the present disclosure, as described above, provide an image-based automatic 3D surgical planning method and system. According to the present invention, safe and reliable surgical performance is possible without relying heavily on the surgeon's skill or experience.

[0232] Embodiments of the present disclosure may be implemented as a computer program medium storing software for performing the aforementioned method of automatically establishing a surgical plan on a computer. Furthermore, the present invention may be implemented through an image processing unit, an object separation unit, a baseline setting unit, a path setting unit, etc., utilizing a processor, memory, and a display, as components for performing the aforementioned method.

[0233] Additionally, the present invention can also be implemented by an image-based surgical robot system.

[0234] Referring to FIG. 24, a surgical robot system (1) according to one or more embodiments of the present disclosure applies an image acquisition device (100) as a main image acquisition device. In addition, the robot system (1) is configured to include a surgical robot (200), a position sensor (300), and a navigation system (400), and the surgical robot (200) is configured to include a main body (201), a robot arm (203) equipped with an end effector (203a), and a robot controller (205).

[0235] In an embodiment according to the present disclosure, except for the image acquisition device described above, the image processing unit, the object separation unit, the baseline generation unit, and the path determination unit may be functionally included in the devices mentioned above. In addition, the image processing unit, the object separation unit, the baseline setting unit, the path setting unit, etc. may be implemented by the surgical planning software, and in particular, the object separation unit performs extraction or separation of the vertebral body and the pedicle from the first image, the second image, and the third image by applying the LL segmentation model, the AP segmentation model, and the AX segmentation model trained by deep learning.

[0236] The robotic arm (203) is controlled according to the surgical plan and control software determined during the surgery, based on the initiation. The navigation system (400) can display surgical plan information for surgical tools or implants on images acquired during the surgery, or can assist the surgeon's surgical actions by displaying the real-time location of the surgical tools or implants on images or, in some cases, 3D images acquired before the surgery. To this end, a display that allows the surgeon to visually compare the real-time location of surgical tools, etc. with the surgical plan and surgical status during the surgery can be connected to the navigation system (400).

[0237] While various embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, modifications to future embodiments of the present invention will not depart from the scope of the present invention.

Claims

1. A step of one or more image acquisition devices acquiring one or more surgical site images including image information in multiple directions including different first directions, second directions, and third directions with respect to a spinal surgical site of a patient; A step of the image processing unit extracting a first image in a first direction and a second image in a second direction from an image of a surgical site; A step of the object separation unit extracting a first vertebral region and a first pedicle region from the first image, and extracting a second vertebral region and left and right second pedicle regions within the second vertebral region from the second image; A step of setting a reference line, wherein a first reference line passing through a first coordinate set in the central region of a first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through both a third coordinate and a fourth coordinate set in the central region of the left and right vertebral regions in the second image; A step of the image processing unit extracting a third image in the third direction from the surgical site image based on the first reference line and the second reference line; A step of extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and An image-based 3D surgical planning method, comprising: a step of setting a screw path passing through at least one central region of the left and right third pedicles and the third vertebral body region, wherein a proximal setting coordinate is set to the second pedicle, and a distal setting coordinate is set along which a reference line of the screw path passes through a third reference line passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

2. In paragraph 1, The above object extraction unit is an image-based 3D surgical planning method that extracts a vertebral body region and a pedicle region by applying a model generated by machine learning.

3. In paragraph 1, An image-based 3D surgical planning method, wherein the first reference line is set to be perpendicular or at an arbitrary angle to the midline passing through the area between the first pedicle region and the first vertebral body region, but passing through the central area of ​​the first pedicle region or being parallel thereto at an arbitrary set distance therefrom.

4. In paragraph 1, An image-based 3D surgical planning method, wherein the central region includes the center and / or vicinity of the corresponding vertebral body region and / or pedicle region.

5. In any one of paragraphs 1 to 4, An image-based 3D surgical planning method, wherein the third image is obtained from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

6. In any one of paragraphs 1 to 4, An image-based 3D surgical method, wherein the third image is acquired from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through the central region of each of the left and right pedicles in the second image.

7. In paragraph 6, The second reference line is set parallel to an arbitrary straight line connecting the center of one of the left and right spinal regions and the midpoint between the coordinates of one corner of the third image close thereto, and the midpoint between the center of the other of the left and right spinal regions and the coordinates of the other corner of the third image close thereto.

8. In any one of paragraphs 1 to 4, The above object separation unit extracts the neural canal area between the left and right third spinal cord areas from the third image, The above baseline setting section: In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and An image-based 3D surgical method, wherein the distal setting coordinates are set at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other, or at the point where the third reference line and the screw path reference line intersect outside the vertebral body.

9. In paragraph 8, An image-based 3D surgical planning method in which the above reference line setting unit sets the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

10. In any one of paragraphs 1 to 4, An image-based 3D surgical planning method, wherein a screw starting point for a pedicle and a target point within a vertebral body are set on the first reference line, and the distance between the screw starting point and the target point is set as the length of the screw used in the surgery.

11. One or more image acquisition devices for acquiring one or more surgical site images having image components in multiple directions with respect to the spinal surgical site; An image processing unit responsible for image processing related to surgery, including extracting a first image in a first direction, a second image in a second direction different from the first direction, and a third image in a third direction different from the first and second directions from one or more surgical site images; An object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebral body and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a third pedicle region from the third image; A reference line setting unit that sets a first reference line passing through a first coordinate set in the central region of the first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through the third coordinate and the fourth coordinate set in the central region of the left and right vertebral regions in the second image; and An image-based 3D surgical planning system, comprising: a screw path setting unit for determining a 3D screw path by setting a screw path passing through at least one central region of the left and right third pedicles and the third vertebral body region, setting proximal setting coordinates on the second pedicle, and setting distal setting coordinates through which a reference line of the screw path passes on a third reference line passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

12. In paragraph 11, The above reference line setting unit is an image-based 3D surgical planning system in which the first reference line crosses the midline passing through the area between the first pedicle region and the first vertebral body region, and the first reference line passes through the central region of the first pedicle region or is parallel to the central region at an arbitrarily set distance therefrom.

13. In paragraph 11, An image-based 3D surgical method, wherein the image processing unit obtains a third image from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

14. In paragraph 11, An image-based 3D surgical system, wherein the image processing unit obtains the third image from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through the central region of each of the left and right pedicles in the second image.

15. In paragraph 14, The second reference line is an image-based 3D surgical planning system, wherein the second reference line is set parallel to an arbitrary straight line connecting the midpoint between the center of one side of the spinal cord region and the coordinates of one corner of the third image close thereto, and the midpoint between the center of the other side of the spinal cord region and the coordinates of the other corner of the third image close thereto.

16. In paragraph 11, The above object separation unit extracts the neural canal area between the left and right third spinal cord areas from the third image, The above path setting section: In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and An image-based 3D surgical planning system, wherein the distal setting coordinates are set at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other, or at the point where the third reference line and the screw path reference line intersect outside the vertebral body.

17. In paragraph 16, An image-based 3D surgical planning system in which the path setting unit sets the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

18. In any one of paragraphs 11 to 17, An image-based 3D surgical planning system in which a screw starting point for a pedicle and a target point within a vertebral body are set on the first reference line, and the distance between the screw starting point and the target point is set as the length of the screw used in the surgery.

Images