Image-based 3D automatic surgical planning method and system

Abstract

An image-based 3D surgical planning method and system is disclosed, which includes the steps of acquiring first and second images from an image of a spinal surgical site, acquiring first and second reference lines from the first and second images, and acquiring a third image from the spinal surgical site based on the first and second reference lines, and determining a 3D screw path by setting a proximal setting coordinate for determining a screw path in a central region of at least one of the left and right third pedicles and a distal setting coordinate in or near the center of the vertebral body region.

Description

[Technical Field]

[0001] The present invention relates to an automatic surgical planning method and system in a surgical robot system, and more particularly to an automatic surgical planning method and system capable of generating a surgical planning path in a 3D image space. [Background technology]

[0002] A spinal surgical procedure, such as a pedicle screw fixation surgery using a surgical robot system, requires a 3D image capable of an axial view. Therefore, a 3D image capable of an axial view, such as a computed tomography (CT) image, rather than a 2D image, is used for the axial view image.

[0003] Typically, during intra-operation, a 2D image acquisition device, such as a C-arm, is used to verify in real time whether the surgery is proceeding according to the pre-operative plan. To navigate surgical tools according to the pre-planned plan using the acquired 2D images, the 2D images acquired during surgery must be registered with the 3D images acquired pre-operatively. However, the process of registering the 2D images with the 3D images can cause problems such as delays in the surgery and registration errors due to imperfect registration.

[0004] On the other hand, surgical planning based solely on 3D images acquired before surgery without the aid of 2D images during surgery cannot avoid problems that arise from radiation exposure that occurs while acquiring 3D images, such as CT images, and differences in the patient's condition before surgery and during surgery.

[0005] However, if a surgical plan is established based on 2D X-ray images acquired during surgery, there is no need to match the images acquired before surgery, and the surgical plan can be changed as needed during surgery. However, this process requires the operating surgeon to estimate the surgical location in 3D space while viewing the 2D images. As a result, such a surgical plan requires the surgeon to have a high level of surgical experience. Even if this were possible, the image quality of 2D images is inferior to that of 3D CT images, and the image information provided by 3D spatial information is lacking. Therefore, a successful surgery is unlikely unless performed by an experienced surgeon.

[0006] Considering these points, the development of an image-based surgical planning method and system based on an imaging system, which does not rely on subjective surgical planning that depends on the surgeon's experience, is desirable in many respects. Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention provides a method and system for automatically establishing an image-based 3D surgical plan based on multiple images taken in different directions and performing surgery based on the plan.

[0008] The present invention provides a method and system for automatically establishing a 3D surgical plan using three or more images from different directions acquired before and / or during surgery and performing surgery based on the plan. [Means for solving the problem]

[0009] The image-based 3D surgical planning method according to the present invention comprises the steps of: acquiring, by one or more image acquisition devices, one or more surgical site images including image information from a plurality of directions, including a first direction, a second direction, and a third direction, different from one another, relative to a spinal surgical site of a patient; extracting a first image in a first direction and a second image in a second direction from the surgical site image by an image processor; an object separating unit extracting a first vertebral body region and a first pedicle region from the first image, and extracting a second vertebral body region and left and right second pedicle regions within the second vertebral body region from the second image; a reference line setting unit setting a first reference line passing through a first coordinate set in a central region of the first pedicle region and a second coordinate set in the first vertebral body region, and a second reference line passing through both a third coordinate and a fourth coordinate set in central regions of the left and right pedicle regions in the second image; an image processor extracting a third image in the third direction from the surgical site image based on the first reference line and the second reference line; an object separating unit extracting a third vertebral body region and left and right third pedicle regions from the third image; The method includes a step in which a path setting unit sets a screw path that passes through the central region of at least one of the left and right third pedicles and the third vertebral body region, setting proximal setting coordinates to the second pedicle, and setting distal setting coordinates, through which the reference line of the screw path passes, on a third reference line that passes through the nerve canal region between the left and right third pedicle regions and the central region of the vertebral body.

[0010] According to one or more embodiments, The object extraction unit may extract the vertebral body region and the pedicle region by applying a model generated by machine learning.

[0011] According to one or more embodiments, The first reference line may be set to intersect the midline passing through the region between the first pedicle region and the first vertebral body region at a right angle or at any angle, and may be set to pass through the center region of the first pedicle region or to be parallel to the center region at an arbitrary distance from the center region.

[0012] According to one or more embodiments, The central region may include the center and / or near the center of the vertebral body region and / or pedicle region.

[0013] According to one or more embodiments, The third image may be acquired from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0014] According to one or more embodiments, The third image is acquired from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, where the second reference line may pass through the central regions of both the left and right pedicles in the second image.

[0015] According to one or more embodiments, The second reference line may be set parallel to an arbitrary straight line connecting the midpoint between the center of one pedicle region of the left or right pedicle region and the coordinate of one side corner of the third image closest thereto, and the midpoint between the center of the other pedicle region and the coordinate of the other side corner of the third image closest thereto.

[0016] According to one or more embodiments, the object separating unit extracts a neural foramen region between left and right third pedicle regions from the third image; The path setting unit In the third image, a third reference line is set that passes through the coordinates of the center or near the center of the third vertebral body region and the neural canal region, and The distal setting coordinate may be set at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the nerve canal region intersect with each other, or at the intersection where the third reference line and the reference line of the screw path intersect outside the vertebral body.

[0017] According to one or more embodiments, The path setting unit can set a screw path reference line on which the screw path is placed so that the screw path reference line passes through the distal setting coordinate.

[0018] According to one or more embodiments, A screw start point for the pedicle and a target point within the vertebral body are set on the first reference line, and the distance between the screw start point and the target point can be set as the screw length used in surgery.

[0019] According to one or more embodiments, The first pedicle region may have an inner edge facing the first vertebral body region and an opposite outer edge, the starting point may be located on or adjacent to the outer edge of the first pedicle region, and the target point may be located within the first vertebral body region.

[0020] According to one or more embodiments, The starting point may be located on one side edge of the second pedicle region toward the outside of the second vertebral body region or adjacent to the edge on the inside thereof, and the target point may be located on the other side edge opposite the one side or adjacent to the edge on the inside thereof.

[0021] According to one or more embodiments, The surgical site image including the image information from multiple directions may be acquired by a 3D image acquisition device.

[0022] The image-based 3D surgical planning system of the present invention is one or more image acquisition devices for acquiring one or more images of the affected area having image components in various directions relative to the spinal surgical site; an image processing unit that processes images related to surgery, including extracting a first image in a first direction, a second image in a second direction different from the first direction, and a third image in a third direction different from the first and second directions, from the one or more images of the affected area; an object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a pedicle region from the third image; a reference line setting unit that sets a first reference line that passes through a first coordinate that is set in a central region of the first pedicle region and a second coordinate that is set in the first vertebral body region, and a second reference line that passes through both a third coordinate and a fourth coordinate that are set in a central region of the left and right pedicle regions in the second image; The method includes a screw path setting unit that sets a screw path that passes through the central region of at least one of the left and right third pedicles and the third vertebral body region, setting a proximal setting coordinate on the second pedicle, and setting a distal setting coordinate where the reference line of the screw path passes on a third reference line that passes through the nerve canal region between the left and right third pedicle regions and the central region of the vertebral body, thereby determining a 3D screw path.

[0023] According to one or more embodiments, in the image-based 3D surgical planning system, the reference line setting unit can set the first reference line so that it intersects perpendicularly with a midline passing through the region between the first pedicle region and the first vertebral body region.

[0024] According to one or more embodiments, in the image-based 3D surgical planning system, the image processor may acquire a third image from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0025] According to one or more embodiments, in the image-based 3D surgical planning system, the image processor acquires the third image from the surgical site image based on the coordinates of the first reference line in the first image and the second reference line in the second image, wherein the second reference line may pass through the central regions of both the left and right pedicles in the second image.

[0026] According to one or more embodiments, in the image-based 3D surgical planning system, the second reference line may be set parallel to any straight line connecting the midpoint between the center of one pedicle region of the left or right pedicle region and the coordinate of one side corner of the third image closest to it, and the midpoint between the center of the other pedicle region of the right or left pedicle region and the coordinate of the other side corner of the third image closest to it.

[0027] According to one or more embodiments, in the image-based 3D surgical planning system, the object separator extracts a nerve canal region between left and right third pedicle regions from the third image; The path setting unit In the third image, a third reference line is set that passes through the coordinates of the center or near the center of the third vertebral body region and the nerve canal region, and The distal setting coordinate may be set at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the nerve canal region intersect with each other, or at the intersection where the third reference line and the reference line of the screw path intersect outside the vertebral body.

[0028] According to one or more embodiments, in the image-based 3D surgical planning system, the path setting unit can set a screw path reference line on which the screw path is placed so that the screw path reference line passes through the distal setting coordinate.

[0029] According to one or more embodiments, in the image-based 3D surgical planning system, a screw start point for the pedicle and a target point within the vertebral body are set on the first reference line, and the distance between the screw start point and the target point (or end point) can be set as the screw length to be used in the surgery. [Effects of the Invention]

[0030] The present invention proposes a method and system for automatically creating a 3D surgical plan using multiple image information. This method overcomes limitations inherent in existing surgical plans, such as the limited results depending on the surgeon's experience, and enables efficient creation and execution of a surgical plan. By automatically creating a surgical plan, the present invention can shorten the overall surgical time and maintain high-quality results. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a flowchart of the chronological processing steps of a method for image-based 3D surgical planning in accordance with one or more embodiments of the present invention. [Figure 2] 1 is a diagram that diagrammatically illustrates the definition of 3D coordinates for a virtual 3D surgical space of a first image and a second image, in accordance with one or more embodiments of the present invention. [Figure 3] 1 is a diagram showing an output image in which a vertebra body and a pedicle are extracted, segmented, or separated from an LL image by applying a first image segmentation model according to one or more embodiments of the present invention. [Figure 4] 10 is a diagram illustrating an output image in which a vertebra body and a pedicle are segmented from an AP image by applying a second image segmentation model according to one or more embodiments of the present invention. [Figure 5] 1 is a diagram illustrating a process for generating a deep learning model according to one or more embodiments of the present invention. [Figure 6] 1 is a diagram illustrating multiple training first images with vertebra bodies and pedicles labeled in accordance with one or more embodiments of the present invention; [Figure 7] 10 is a diagram illustrating multiple training second images with vertebra bodies and pedicles labeled in accordance with one or more embodiments of the present invention. [Figure 8] 10 is a diagram illustrating multiple training third images with vertebra bodies, pedicles, and nerve canals labeled in accordance with one or more embodiments of the present invention. [Figure 9] 10 is a diagram illustrating the results of labeling a target segmented into a first image and a second image, according to one or more embodiments of the present invention. [Figure 10] 1 is a diagram illustrating an example L1 target previously selected in accordance with one or more embodiments of the present invention. [Figure 11]1 is a diagram illustrating a vertebral body region (A) and a pedicle region (B) established in a first view according to one or more embodiments of the present invention. [Figure 12] 1 is a graph of the yz plane showing the vertex coordinates of the vertebral body region (A) and pedicle region (B) set in a first image and the midline between those regions, according to one or more embodiments of the present invention. [Figure 13] 12 is a graph in the yz plane showing an extraordinary line rotated 90 degrees from the midline of FIG. 11 in accordance with one or more embodiments of the present invention. [Figure 14A] 1 is a yz-plane graph showing an initial first baseline (D1) disposed parallel to a temporary straight line D' in accordance with one or more embodiments of the present invention. [Figure 14B] FIG. 14B is a yz-plane graph showing a modified first baseline (D1a) that replaces and is disposed parallel to the first baseline D1 of FIG. 14A, in accordance with one or more embodiments of the present invention. [Figure 15A] 14B is a yz-plane graph showing the set coordinates of the target point T or end point of the surgical screw on the initial first reference line of FIG. 14A in accordance with one or more embodiments of the present invention. [Figure 15B] 14C is a yz-plane graph illustrating the set coordinates of the target point T or end point of the surgical screw on the modified first reference line of FIG. 14B in accordance with one or more embodiments of the present invention. [Figure 16] 1 is an image showing a second vertebral body region (a) and a second pedicle region (b, b') extracted or separated using a first image segmentation model in accordance with one or more embodiments of the present invention. [Figure 17] 1 is an xz plane graph showing the elliptical second pedicle region (b, b') within the second vertebral body region (a) with vertices a1, a2, a3, a4 in a dotted line in one or more embodiments of the present invention. [Figure 18]1 is a diagram illustrating a state in which a first reference line and a second reference line are set on a first image and a second image acquired according to one or more embodiments. [Figure 19] (A) illustrates a third image acquired along an axial slice path according to one or more embodiments, and (B) is a drawing showing the vertebral body region, pedicle region, and nerve canal region as ROIs extracted from the third image. [Figure 20A] 10 is a diagram illustrating a process of setting a screw path reference line for an ROI of a third image according to one or more embodiments. [Figure 20B] 10 is a diagram illustrating another example process for setting an alternative screw path reference line for an ROI in a third image, according to one or more embodiments. [Figure 21A] 20B is a diagram showing an ROI and a third image in which a screw path reference line for the ROI of the third image is set according to the embodiment of FIG. 20A; [Figure 21B] 20C is a diagram showing an ROI and a third image in which a screw path reference line for the ROI of the third image is set according to the embodiment of FIG. 20B. [Figure 22A] 21B is a diagram showing a state in which the entry point and the destination point (or the end point) of the entire screw path are set on the screw path reference line of FIG. 21A according to the embodiment of FIG. 20. [Figure 22B] 21B is a diagram showing a state in which the entry point and the destination point (or the end point) of the entire screw path are set on the screw path reference line of FIG. 21B according to the embodiment of FIG. 20A. [Figure 23] 1 is a diagram illustrating a schematic configuration of a surgical robot system in accordance with one or more embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the following embodiments. It is preferable that the embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art. The same reference numerals refer to the same elements throughout. Furthermore, various elements and regions in the drawings are depicted in schematic form. Therefore, the present invention should not be limited by the relative sizes and spacings depicted in the accompanying drawings.

[0033] Terms such as "first" and "second" may be used to describe various components, but the components are not limited by these terms. These terms are used only to distinguish one component from another. For example, a first component may be designated a "second component" and conversely, the second component may be designated the "first component" without departing from the scope of the inventive concept.

[0034] The terms used in this application are used only to describe specific embodiments and are not intended to limit the concept of the present invention. The singular expressions include the plural expressions unless the context clearly dictates otherwise. In this application, the expressions "comprise" or "have" specify the presence of the features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preclude the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

[0035] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention pertains. Furthermore, commonly used and predefined terms should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an overly formal sense unless expressly defined herein.

[0036] If an embodiment can be implemented differently, the order of certain steps may be performed differently from that described. For example, two steps described in succession may be performed substantially simultaneously or in the reverse order from that described.

[0037] DETAILED DESCRIPTION OF THE INVENTION In accordance with one or more embodiments, a method and system for image-based 3D surgical planning is described in detail below.

[0038] FIG. 1 is a time-series flowchart of the image-based 3D surgical planning method according to the present invention.

[0039] The procedure of FIG. 1 includes steps S6 to S14 that are repeatedly performed for multiple target vertebrae. When there is only one vertebra, a surgical plan for one vertebra is established in one procedure.

[0040] Although one or more embodiments described below are described based on 3D images captured by a 3D image capturing device, it should be understood that the embodiments of the present invention are not limited to 3D images captured by a 3D image capturing device. The present invention also encompasses images captured by one or more image capturing devices capable of capturing image information in three different directions, particularly any device capable of creating combined or aligned three-directional images.

[0041] Figure 1 can be explained step by step as follows.

[0042] The method according to the present invention generally includes a preparatory work process, an LL point generation process, an AP point generation process, an axial (hereinafter often referred to as AX) point generation process, and an optimization and automation process.

[0043] <Step S1> Step S1 is a step of taking a plurality of images from various directions with respect to the affected part of the patient. This step uses 3D images by a 3D image acquisition device according to the present embodiment.

[0044] FIG. 2 schematically shows the definition of 3D coordinates related to the virtual 3D surgical space of the first image and the second image according to one or more embodiments of the present invention.

[0045] <Step S2> In step S2, it is a step of loading the 3D image into an image processing unit for processing the 3D image having axes in the X, Y, and Z directions. In this step, the 3D image and coordinate information are loaded.

[0046] <Step S3> In step S3, it is a step of processing the loaded 3D image in two directions to obtain the first image and the second image as DRR (Digitally Reconstructed Radiograph) images. In this step, for example, an LL image (Lateral-Lateral Image) is obtained as the first image, and for example, an AP image (Anterior-Posterior Image) is obtained as the second image. These images are obtained by orthogonal projection in the first direction and the second direction with respect to the 3D image.

[0047] In the LL image, i.e., the first image, the left - right direction (abdomen - back) is the Y - axis, and the up - down direction (head - foot) is also the Z - axis. And in the AP image, i.e., the second image, the up - down direction (head - foot) is the Z - axis, and the left - right direction (left arm - right arm) is the X - axis. Therefore, the first image, for example, the LL image, is arranged in the Y - Z plane, and the second image, for example, the AP image, is arranged in the X - Z plane.

[0048] <S4 step> In the S4 step, the extraction or segmentation or separation of the vertebral body and pedicle as the target object is carried out from the first and second images obtained in the above - mentioned process.

[0049] The segmentation of the object can be performed by applying the LL segmentation model and the AP segmentation model trained by deep learning.

[0050] FIG. 3 shows the output image in which the vertebral body and pedicle are segmented from the LL image by applying the first - image segmentation model.

[0051] FIG. 4 shows the output image in which the vertebral body and pedicle are segmented from the AP image by applying the second - image segmentation model.

[0052] As shown in FIG. 5, the above - mentioned segmentation model is based on the DeepLabv3+ model. According to other embodiments of the present invention, it is needless to say that other models can be applied.

[0053] The DeepLavb3+ model is one of the deep learning models used in the field of computer vision and is used to perform extraction or segmentation operations. This model is based on a deep convolutional neural network (CNN) architecture and particularly has an encoder structure and a decoder structure, making it suitable for image segmentation.

[0054] The training of the model includes the same process as general deep learning training. For the training of such a model, diverse spine-related video materials collected for training are required. The training videos include a first training video taken in a first direction and a second training video taken in a second direction. As shown in FIGS. 4 and 5, pixel-level labeling is performed on the first video and the second video, and then, by performing the learning process, the target first video segmentation model and second video segmentation model are obtained.

[0055] <着 FIGS. 6 and 7 show videos in which vertebra bodies and pedicles are labeled for a number of first learning videos and second learning videos, respectively.

[0056] FIG. 8 shows a video in which vertebra bodies, pedicles, and neural foramina are labeled for a number of third learning videos according to one or more embodiments of the present invention.

[0057] <S5 stage> This is the stage of performing labeling on the targets segmented in the previous stage. FIG. 9 shows the results of labeling the targets segmented in the AP video and the LL video.

[0058] As shown in FIG. 9, the divided targets were sequentially labeled as L1, L2, L3, L3, L4, and L5 from the top, and the same labels were attached to the same vertebral bodies.

[0059] <S6 stage> As shown in FIG. 10, at this stage, as a stage of selecting one target from a plurality of targets prior to the process of extracting the LL point, for example, the L1 target is first selected and undergoes subsequent processes.

[0060] <S7 - S8 stage> The extraction of the LL parameter for the selected labeled target is performed, and through this, the first reference line in the LL image plane is determined. This reference line corresponds to one axis of the image plane in the third direction to be obtained later, and is the first axial slice path (axial slice path(1)) for extracting the third image described above. When planning the first reference line, it must be set so as not to deviate from the pedicle and vertebral body regions. Here, if only the inclination of either one of the vertebral body region or the pedicle region is reflected, the path will deviate in other regions, and in order to prevent the path planning failure due to the detection error occurring during object segmentation, the first reference line is set to the average inclination of the inclinations of the two regions.

[0061] For this purpose, first, the ROI (region of interest) regions for the pedicle and vertebral body are set.

[0062] In FIG. 11, (A) is the vertebral body region, and (B) is the pedicle region. Such ROIs are obtained by the first image segmentation model described above.

[0063] As shown in FIG. 12, the vertebral body region (A) is a rectangular region formed by four vertices A1, A2, A3, and A4, and the pedicle region (B) is a rectangular region formed by four vertices B1, B2, B3, and B4, and the two regions are adjacent. An intermediate line C passes between the adjacent sides of the two adjacent regions.

[0064] The midline C passes through the center between the two adjacent sides (B2-B3, A1-A4) of both areas (A, B), and intersections C1 and C2 are located on the midline C. Intersection C1 is located midway between the line segment connecting vertices B2 and A1 of both areas, and C2 is located midway between the line segment connecting vertices B3 and A4. Therefore, the y coordinate of C1 (C1.y) is (A1.y + B2.y) / 2, and the z coordinate of C1 (C1.z) is (A1.z + B2.z) / 2. The y coordinate of C2 (C2.y) is (A4.y + B3.y) / 2, and the z coordinate of C2 (C2.z) is (A4.z + B3.z) / 2.

[0065] As a result, the intermediate line C has an average gradient (Ca) relative to the gradients of both regions in the LL image plane coordinates, i.e., the yz plane. Such an intermediate line C serves as a reference for setting a first reference line in the LL image plane, and the coordinates C1(y,z) and C2(y,z) of the intersection points C1 and C2 in the yz plane coordinate system and the gradient Ca of the intermediate line C are expressed by the following formula:

[0066]

number

[0067] FIG. 13 shows a temporary point C3 in the middle process of setting the first reference line from the midline C between the pedicle region (B) and the vertebral body region (A) in the yz plane.

[0068] The coordinates of the temporary point C3 on the intermediate line C coincide with the coordinates of the intersection point C1 rotated 90° around the intersection point C2. That is, if the coordinates of C1 are (y1, z1) in the yz plane coordinate system, the coordinates of C3 become (z1, -y1). As a result, the temporary straight line D'0 connecting the intersection point C2 and the temporary point C3 is a normal line perpendicular to the intermediate line C, and the equation of the normal line is D0:Z=Da(Y-C2.y)+C2.z.

[0069] 14A illustrates an initial first baseline (D1) that is arranged parallel to the temporary straight line D'. The initial first baseline D1 crosses the midline C at a predetermined angle or is perpendicular to it, and is aligned with the temporary straight line D'. The temporary straight line D' is translated a predetermined distance so that it passes through the central region of the pedicle region (B) having four vertices, such as B1, B2, B3, and B4, i.e., the center (B5) or its vicinity, to determine the initial first baseline D1 on the yz plane.

[0070] The y coordinate B5.y and z coordinate B5.z of B5 are expressed as follows: B5.y=(B1.y+B2.y+B3.y+B4.y) / 4 B5.z=(B1.z+B2.z+B3.z+B4.z) / 4 The linear equation of the first reference line D1 having a slope Da is expressed as follows: D1:Z=Da(Y-C2.y)+C2.z Da = (C3.z - C2.z) / (C3.y - C2.y)

[0071] The first reference line D1 expressed by the above linear equation (D1:z) passes through the outer edge of the pedicle (BS, B1-B4) and generates an intersection point E1. This intersection point E1 becomes the start point or entry point for determining the surgical screw length, which will be explained later.

[0072] The lateral side BS of the pedicle is expressed by the following formula: BS:Z=Ba(Y-B1.y)+B1.z Ba = (B4.z - B1.z) / (B4.y - B1.y)

[0073] Next, the coordinates of the start point located on the initial first reference line D1 and the target point, which is the terminating point where the end of the surgical screw reaches, are calculated.

[0074] The coordinates (Ey, Ez) of the starting point E located on the outer edge of the pedicle can be calculated by a linear simultaneous equation applying the linear equation of the first reference line (D1:Z) and the linear equation of the outer edge BS of the pedicle (BS:Z). D1:Z=Da(Y-C2.y)+C2.z BS:Z=Ba(Y-B1.y)+B1.z

[0075] First, the two lines have the same Y and Z in (Ey, Ez), so if we calculate it as follows, we get the following. D1a(Y-C2.y)+C2.z=Ba(Y-B1.y)+B1.z

[0076] The Ey value can be obtained by collecting the terms relating to Y in the above equation and separating Y. Ey=Y=(B1.z-C2.z+Ba*B1.y-Da*C2.y) / (Da-Ba)

[0077] By substituting Ey obtained by the above formula into the linear equation (D1:Z) of the reference line D1, the z-coordinate Ez of the starting point is obtained. Ez=Z=Da(Ey-C2.y)+C2.z

[0078] The above calculation process is one of various calculation methods for determining the coordinates of the starting point E, and it goes without saying that other methods can also be used for calculation, and a specific calculation method does not limit the technical scope of the present invention.

[0079] 14B shows a modified first baseline (D1a) obtained based on the initial first baseline D1 according to another embodiment of the present invention, where the modified first baseline D1a replaces the initial first baseline D1 obtained in FIG.

[0080] The modified first reference line D1a is obtained by translating the initial first reference line D1 toward the center of the vertebral body, and the translation distance G1 corresponds to the diameter or radius of the screw used in the surgery, and in other embodiments, it is larger or smaller than the diameter of the screw.

[0081] The reason for translating the initial reference line into the vertebral body in this way is to prevent the screw from being exposed outside the vertebral body due to its size, and if the pedicle does not significantly offset from the vertebral body due to the patient's condition, the initial first reference line can be used as the screw path reference line. However, if the pedicle is significantly offset from the vertebral body for some reason or congenital reasons, the modified first reference line can be used as the screw path reference line.

[0082] That is, in an embodiment according to the present invention, either one of the reference lines D1 and D1a is selectively applied as the reference line for setting the final first screw pass, and the first reference line referred to below is D1 or D1a.

[0083] 15A and 15B show the setting of surgical screw entry points E1, E2 and target points T1, T2 on the initial first reference line D1 and the corrected first reference line, respectively.

[0084] First, referring to FIG. 15A, a target point T is set on the yz plane, which is an end point where the end portion of the surgical screw is located on the initial first reference line D1.

[0085] The coordinates of the target point T are determined by the total length (Ls) of the screw used. In one embodiment of the present invention, the total length Ls of the screw is set so that it does not exceed the length of the side (A1-A2) of the vertebral body region in the direction in which the surgical screw advances, and its diameter Ds is set so that it is not larger than the length of the outer side B1-B4 where the screw entry point is located.

[0086]

number

[0087] Here, the coordinates T(y, z) of the target point are expressed by the following formula. T.y = E.y + Ls * Cos(arctan(Da)) T.z = E.z + Ls * Sin(arctan(Da))

[0088] The length of the line segment connecting the coordinates of the starting point and the target point obtained through the above process corresponds to the screw length used in the surgery.

[0089] On the other hand, referring to FIG. 15B, an entry point E2 and a target point T2 of the surgical screw are set on the corrected first screw path D1a. [[ID=1,9]]

[0090] On the corrected first reference line D1a, the coordinates of the set starting point E2 and target point T2 are set by the mathematical calculation of the above-described method. As shown in FIG. 14B, the corrected first reference line D1a passes through the vicinity outside the center B5 of the pedicle within the pedicle.

[0091] <S9 - S10 stage> In this stage, a second reference line paired with the first reference line in the y - z plane, that is, the axial slice path 2, is obtained.

[0092] FIG. 16 is an image showing a state in which the second vertebral body region (a) and the second pedicle regions (b, b') are extracted or separated using the first video segmentation model.

[0093] As shown in FIG. 16, the vertebral body ROI (a) and the left and right pedicles (b), (b') can be extracted through the segmentation model for the second video, that is, the AP video.

[0094] The key point in setting the second reference line in the AP image is that when acquiring the third image (i.e., the AX image) described below, a slice image in which the left / right pedicle is most visible must be acquired. To do this, the degree of vertebral rotation must be reflected, thereby ensuring the largest pedicle area in the AX image.

[0095] Therefore, the average tilt value is calculated to reduce the segmentation error of the vertebral body ROI (a) and the two left and right pedicle regions b and b'.

[0096] The average inclination increases or decreases depending on the arrangement of the right pedicle, and the temporary second reference line d' indicating such an average inclination is determined as follows.

[0097] FIG. 17 is an xz plane graph showing, by dotted lines, an elliptical second pedicle region (b, b') within a second vertebral body region (a) having vertices a1, a2, a3, and a4.

[0098] As shown in Figure 17, first, the midpoint (d1) between the center (b5) of the left pedicle (b) and the corner (a1) of the vertebral body region closest to it and the midpoint (d2) between the center (c5) of the right pedicle (b') and the corner (a2) of the vertebral body region closest to it are determined, and these two midpoints (d1, d2) are connected to determine a temporary second reference line (d') with an average slope.

[0099] FIG. 18 shows a state in which the first and second reference lines are set for the first and second images.

[0100] First, referring to FIG. 18, if the Y-Z position of the first reference line (D1 or D1a) in the first video obtained in the above process and the inclination of the provisional reference line (d') in the X, Z plane in the second video are applied to obtain a third video in the AX plane of the 3D video, a final second reference line (d) that passes through the centers of both pedicles on average, as shown in the second video on the right side of FIG. 18, can be obtained. Therefore, through this process, an axial third video, that is, an AX video (axial image), as shown in (A) of FIG. 19, can be obtained from the slice plane formed by the first reference line and the second reference line.

[0101] Here, the provisional second reference line (d') shown in FIG. 17 is shown to span the upper sides of both pedicles (b, b'), and in FIG. 18, it is shown to pass through both central regions of both pedicles (b, b'), that is, near the center. In FIG. 18, the second reference line (d) indicates the final second reference line when the inclination of the second reference line in the X-Z plane is applied to the Y-Z position of the first reference line. That is, the final second reference line is a line resulting from translating the provisional second reference line (d') parallel to the average center of both pedicles (b, b').

[0102] <Step S11> (A) of FIG. 19 illustrates the above-mentioned third video, and (B) illustrates the vertebral body region, pedicle region, and neural canal region as ROIs extracted from the third video.

[0103] This step is to extract the AX parameter (Axial parameter) related to the AX video of (A) in FIG. 19 obtained as described above.

[0104] The extraction of the above regions is performed by the object segmentation unit. The segmentation of the object can be performed by applying an AX segmentation model trained by deep learning using samples labeled for the vertebra body, pedicle, and foramen in the AX video, as shown in FIG. 18, similar to the LL segmentation model and the AP segmentation model trained by deep learning as described above.

[0105] FIG. 19(B) shows the vertebral body region (a), the left and right pedicle regions (b, b'), and the foramen region (f) between the left and right pedicles extracted by the segmentation of the AX image.

[0106] 20A and 20B illustrate another embodiment of the process of setting a screw path reference line for the ROI of the third image.

[0107] First, the embodiment of FIG. 20A will be described.

[0108] (A) to (D) of FIG. 20A show the process of setting a screw path reference line on which a screw path is placed in an AX image, which is a plane for inserting or placing a surgical screw.

[0109] (A) A center point "fc" is set in the central region of the neural tube region, and center points "ad" are also set in each central region of the vertebral canal region, and then a third reference line L is set that passes through these two points fc and ad.

[0110] (B) The intersection points a'c and ac1 between the third reference line and the inner and outer borders of the vertebral body region are set. This intersection point ac1 is the distal setting coordinate through which the screw path reference line passes. Meanwhile, the distance a'c-ac1 indicates the width of the vertebral body in the direction of the third reference line, and this distance can be applied to determine the overall length of the screw.

[0111] (C) The first screw path reference line (LP) is set through the intersection point ac1 and the center (b'c) of the pedicle on one side (left side in the drawing), i.e., the proximal setting coordinate.

[0112] (D) A second screw path reference line (RP) is set that passes through the intersection point ac and the center (bc) of the other (right side in the drawing) pedicle, i.e., the proximal setting coordinate.

[0113] The first and second screw path reference lines (LP, RP) set as described above are axes along which surgical screws are provided.

[0114] The first and second screw path reference lines (LP, RP) obtained in the process of FIG. 20B are set in slightly different directions in the following manner.

[0115] The embodiment of FIG. 20B will now be described.

[0116] FIG. 20B, like FIG. 20A, (A) to (D) show the process of setting a screw path reference line on which the screw path is placed in the AX image, which is the surgical screw insertion or placement plane.

[0117] (A) A center point "fc" is set in the central region of the neural tube region, and center points "ad" are also set in each central region of the vertebral canal region, and then a third reference line (L) is set that passes through these two points fc and ad.

[0118] (B) Intersection points a'c and ac1 between the third reference line and the inner and outer boundary lines of the vertebral body region are set. This intersection point ac1 is the distal setting coordinate through which the screw path reference line passes. Here, a corrected intersection point ac1', i.e., the distal setting coordinate, is set on the screw path reference line (L) distal to the vertebral body at the intersection point ac1. The corrected intersection point ac1' is separated from the intersection point ac1 above by an arbitrary distance (G2).

[0119] The arbitrary distance (G2) is the average thickness or height of the pedicle (G2) and may have other amounts less or greater according to other embodiments.

[0120] On the other hand, the distance a'c-ac1 indicates the width of the vertebral body in the direction of the third reference line, and this distance can be applied to determine the overall length of the screw.

[0121] (C) Set a first screw path reference line (LP) passing through the center (b’c) of the pedicle on one side (the left side in the drawing) of the corrected intersection point ac2.

[0122] (D) Set a second screw path reference line (RP) passing through the center (bc) of the pedicle on the other side (the right side in the drawing) of the intersection point ac2.

[0123] The first and second screw path reference lines (LP, RP) set as above are the corrected axes where surgical screws are provided, and the angle between them is narrowed compared to the angle between the first and second screw path reference lines (LP, RP) obtained in the process of FIG. 20. Therefore, the tip portions of the two surgical screws arranged on the two reference lines LP and RP do not meet within the vertebral body region.

[0124] Each of FIGS. 22A and 22B (A) is according to the embodiments of FIGS. 20A and 20B, and shows the first and second screw path reference lines set by the process as described above. Each of them (B) illustrates a state where the reference lines are mapped (overlaid) on the AX image.

[0125] As shown in (B) of FIG. 22A according to the embodiment of FIG. 20A, starting points or entry points (LE, RE) of surgical screws are formed at the portions where the first and second screw path reference lines intersect the outer boundary lines of the two pedicles. The entry points are set at the proximal set coordinates through which the screw path reference lines pass.

[0126] And as shown in (B) of FIG. 22B according to the embodiment of FIG. 20B, the first and second screw path reference lines pass through the center (bc) of each pedicle and the corrected intersection point ac2, and entry points of the screws are set on the outer surface of the pedicle.

[0127] <S12 - S13 stage> The S12 - S13 stage is a stage of determining the final surgical plan by setting screw target points (RT, LT) that are finally away from the starting points (LE, RE) which are the entry points of surgical screws.

[0128] The entry point is located at the proximal set coordinate where the screw enters the pedicle.

[0129] In this step, the screw length can be determined based on the overall screw length (Ls) or the vertebral body width in the direction of the first reference line obtained in the process of setting the first reference line, or it can be determined by taking into account the vertebral body width in the direction of the third reference line (the distance a'c-ac). Furthermore, the surgeon can adjust the screw length as needed, taking into account the results of the above. Meanwhile, the screw diameter must be set to be smaller than the average diameter of the pedicles or the diameter of the smaller of the left and right pedicles, and this can also be changed by the surgeon.

[0130] In the embodiment of Figures 20A and 20B, (A) of Figures 22A and 22B shows the screw path with the total length of the screw (Ls) applied to the ROI region, and (B) illustrates the state in which the screw path is determined in the third image.

[0131] The screw path has 3D coordinates, which are recognized by the surgical system during surgery and the surgery is performed as is.

[0132] 22A and 22B illustrate the state in which the entry and destination (or termination) points of the full length screw path are set on the screw path reference line of FIGS. 22A and 22B, according to one or more embodiments.

[0133] Referring to Figure 22A according to the embodiment of Figure 20A, entry points (LE, RE) and target points (LT, RT) located on the surface of the pedicle are formed on the first and second screw path reference lines (LP, RP), and the distance between the entry point (LE) and the target point (LT) on each reference line corresponds to the total length of the screw.

[0134] Referring to FIG. 22B according to the embodiment of FIG. 20B, modified entry points (LE', RE') are provided. These modified entry points (LE', RE') are spaced apart by a given distance from the entry points (LE, RE) on the surface of the pedicle on the reference line. This spacing takes into account the gaps created by the patient's skin and / or muscle tissue, etc., that may occur during actual surgery when the screw is fixed into the pedicle, rather than the posterior end (head) of the screw held in the surgical drill. Therefore, the target points within the vertebral body are also shifted to the modified LT' and RT' at the original positions of LT and LP. The positions of these entry points and target points can be flexibly modified and changed on the screw path reference line determined in the above process at the surgeon's discretion depending on the condition of the patient's surgical site.

[0135] The above process is performed for one target. If there are multiple targets, i.e., multiple vertebrae to be operated on, and step S14 is not the last target, the process returns to step S6, and steps S6 to S14 are repeatedly performed, and then the process ends.

[0136] The above-described embodiments of the present invention provide a method and system for image-based automated 3D surgical planning, which enables safe and reliable surgical procedures without relying heavily on the skill or experience of the surgeon.

[0137] An embodiment of the present invention may be embodied as a computer program medium storing software for executing the above-described method of automatically establishing a surgical plan on a computer. The present invention may also be embodied through an image processing unit, an object separation unit, a reference line setting unit, a path setting unit, etc., which perform the above-described method and which use a processor, a memory, a display, etc.

[0138] The present invention may also be embodied in a vision-based surgical robot system.

[0139] 23, a surgical robot system 1 according to one or more embodiments of the present invention employs an image capturing device 100 as a main image capturing device. The robot system 1 also includes a surgical robot 200, a position sensor 300, and a navigation system 400. The surgical robot 200 includes a main body 201, a robot arm 203 equipped with an end effector 203a, and a robot controller 205.

[0140] In an embodiment of the present invention, the image processing unit, object separation unit, reference line generation unit, and path determination unit may be functionally included in the above-described devices, except for the image acquisition device. The image processing unit, object separation unit, reference line setting unit, and path setting unit may be implemented by surgical planning software. In particular, the object separation unit extracts or separates vertebral bodies and pedicles from the first, second, and third images by applying an LL segmentation model, an AP segmentation model, and an AX segmentation model trained by deep learning.

[0141] The robot arm 203 is controlled according to the surgical plan and control software determined during surgery. The navigation system 400 can display surgical plan information related to surgical tools or implants on a display, either as an image acquired during surgery or as a real-time position of the surgical tools or implants on an image or, in some cases, as a 3D image acquired before surgery, to assist the surgeon in performing the surgical procedure. For this reason, a display that allows the surgeon to visually compare the real-time position of the surgical tools, etc. as the surgical plan and the current surgical status with the naked eye during surgery can be connected to the navigation system 400.

[0142] Although various embodiments of the present invention have been described in detail above, those skilled in the art will be able to implement the present invention in various modifications without departing from the spirit and scope of the present invention as defined in the appended claims. Therefore, future changes to the embodiments of the present invention will not be deemed to be outside the scope of the present invention.

Claims

1. acquiring, by one or more image acquisition devices, one or more surgical site images including image information from multiple directions, including a first direction, a second direction, and a third direction, relative to the spinal surgical site of the patient; extracting a first image in a first direction and a second image in a second direction from the surgical site image by an image processor; an object separating unit extracting a first vertebral body region and a first pedicle region from the first image, and extracting a second vertebral body region and left and right second pedicle regions within the second vertebral body region from the second image; a reference line setting unit setting a first reference line passing through a first coordinate set in a central region of the first pedicle region and a second coordinate set in the first vertebral body region, and a second reference line passing through both a third coordinate and a fourth coordinate set in central regions of the left and right pedicle regions in the second image; an image processor extracting a third image in the third direction from the surgical site image based on the first and second reference lines; an object separating unit extracting a third vertebral body region and left and right third pedicle regions from the third image; a path setting unit sets a screw path that passes through a central region of at least one of the left and right third pedicles and the third vertebral body region, setting a proximal setting coordinate on the second pedicle, and setting a distal setting coordinate through which a reference line of the screw path passes on a third reference line that passes through a nerve canal region between the left and right third pedicle regions and the central region of the vertebral body.

2. The image-based 3D surgical planning method of claim 1 , wherein the object extraction unit extracts the vertebral body region and the pedicle region by applying a model generated by machine learning.

3. 2. The image-based 3D surgical planning method of claim 1, wherein the first reference line intersects a midline passing through a region between a first pedicle region and a first vertebral body region at a right angle or at an arbitrary angle, and is set to pass through a central region of the first pedicle region or to be parallel to the central region at an arbitrary distance therefrom.

4. The image-guided 3D surgical planning method of claim 1 , wherein the central region includes a center and / or a vicinity of a center of the vertebral body region and / or the pedicle region.

5. The image-based 3D surgical planning method of claim 1 , wherein the third image is acquired from the surgical site image based on the coordinates of the first reference line and the inclination of the second reference line.

6. 5. The image-based 3D surgical method of claim 1, wherein the third image is acquired from the surgical site image based on coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through central regions of both the left and right pedicles in the second image.

7. 7. The image-based 3D surgical planning method according to claim 6, wherein the second reference line is set parallel to an arbitrary straight line connecting a midpoint between the center of one of the left and right pedicle regions and the coordinate of one side corner of the third image closest thereto, and a midpoint between the center of the other of the left and right pedicle regions and the coordinate of the other side corner of the third image closest thereto.

8. the object separating unit extracts a neural canal region between left and right third pedicle regions from the third image; The reference line setting unit a third reference line is set in the third image, the third reference line passing through the coordinates of the center or near the center of the third vertebral body region and the neural canal region; 5. The image-based 3D surgical method according to claim 1, wherein the distal setting coordinate is set at a position at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the nerve canal region intersect with each other, or at an intersection point where the third reference line and the reference line of the screw path intersect outside the vertebral body.

9. The image-based 3D surgical planning method according to claim 8 , wherein the reference line setting unit sets a screw path reference line on which the screw path is placed so as to pass through the distal setting coordinate.

10. 5. The image-based 3D surgical planning method according to claim 1, wherein a screw start point for a pedicle and a target point within a vertebral body are set on the first reference line, and the distance between the screw start point and the target point is set as the screw length to be used in surgery.

11. one or more image acquisition devices for acquiring one or more surgical site images having image components in various directions relative to the spinal surgical site; an image processing unit for processing images related to surgery, including extracting a first image in a first direction, a second image in a second direction different from the first direction, and a third image in a third direction different from the first direction and the second direction, from the one or more surgical site images; an object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebral body and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a third pedicle region from the third image; a reference line setting unit that sets a first reference line that passes through a first coordinate that is set in a central region of the first pedicle region and a second coordinate that is set in the first vertebral body region, and a second reference line that passes through both a third coordinate and a fourth coordinate that are set in a central region of the left and right pedicle regions in the second image; and a screw path setting unit that sets a screw path that passes through a central region of at least one of the left and right third pedicles and the third vertebral body region, by setting a proximal setting coordinate on the second pedicle and setting a distal setting coordinate through which a reference line of the screw path passes on a third reference line that passes through a nerve canal region between the left and right third pedicle regions and the central region of the vertebral body, thereby determining a 3D screw path.

12. 12. The image-based 3D surgical planning system of claim 11, wherein the reference line setting unit positions the first reference line so that the first reference line crosses a midline passing through a region between a first pedicle region and a first vertebral body region, but passes through a central region of the first pedicle region or is parallel to and spaced apart from this central region by an arbitrarily set distance.

13. The image-based 3D surgical method of claim 11 , wherein the image processor acquires a third image from the surgical site image based on the coordinates of the first reference line and the inclination of the second reference line.

14. 12. The image-based 3D surgical system of claim 11, wherein the image processor acquires the third image from the surgical site image based on coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through central regions of both the left and right pedicles in the second image.

15. 15. The image-based 3D surgical planning system of claim 14, wherein the second reference line is set parallel to an arbitrary straight line connecting the midpoint between the center of one pedicle region of the left and right pedicle regions and the coordinates of one side corner of the third image closest to the center of the pedicle region of the left and right pedicle regions, and the midpoint between the center of the other pedicle region of the left and right pedicle regions and the coordinates of the other side corner of the third image closest to the center of the other pedicle region.

16. the object separating unit extracts a neural canal region between left and right third pedicle regions from the third image; The path setting unit In the third image, a third reference line is set that passes through the coordinates of the center or near the center of the third vertebral body region and the neural canal region, and The image-based 3D surgical planning system of claim 11, wherein the distal setting coordinate is set at or near the point where the third reference line and the boundary line of the vertebral body region farthest from the nerve canal region intersect with each other, or at the intersection where the third reference line and the reference line of the screw path intersect outside the vertebral body.

17. The image-based 3D surgical planning system according to claim 16, wherein the path setting unit sets a screw path reference line on which the screw path is placed so as to pass through the distal setting coordinate.

18. 18. The image-based 3D surgical planning system of claim 11, wherein a screw start point for a pedicle and a target point within a vertebral body are set on the first reference line, and the distance between the screw start point and the target point is set as the screw length to be used in surgery.

Images