Image-based 3D automated surgical planning method and system

Abstract

An image-based 3D surgical planning method and system are disclosed. The planning method comprises the steps of: obtaining a first image and a second image from an image of a spinal surgical site; obtaining first and second reference lines from the first image and the second image; and obtaining a third image from the spinal surgical site on the basis of the first and second reference lines, and setting proximal setting coordinates for determining the screw path in at least one central region of the left or right third pedicle and distal setting coordinates set at or near the center of a vertebral body region, thereby determining a 3D screw path.

Description

Image-based 3D automatic surgical planning method and system

[0001] The present disclosure relates to an automatic surgical planning method and system in a surgical robot system, and more particularly, to an automatic surgical planning method and system capable of generating a surgical planning path in a 3D image space.

[0002] Spinal surgical procedures, such as pedicle screw fixation using a surgical robot, require 3D images capable of an axial view. Therefore, 3D images capable of an axial view, such as computed tomography (CT) images, rather than 2D images, are used as these axial view images.

[0003] Typically, intraoperatively, a 2D image acquisition device, such as a C-arm, is used to verify in real time whether the surgery is proceeding as planned. To navigate surgical tools as planned using these acquired 2D images, the intraoperative 2D images must be registered with the preoperative 3D images. However, the process of registering 2D images to 3D images can lead to delays in surgery and registration errors due to incomplete registration.

[0004] Meanwhile, surgical planning based only on 3D images acquired before surgery without the help of 2D images during surgery cannot avoid problems that arise due to radiation exposure caused by acquiring 3D images, such as CT images, and differences in the patient's condition before surgery and during surgery.

[0005] However, if a surgical plan is established based on 2D X-ray images acquired during surgery, there is no need for registration with images acquired before surgery, and the flexibility to change the surgical plan as needed can be expected even during surgery. However, this process requires the operating surgeon to estimate the location of the desired 3D space in 3D space while looking at the 2D image. Consequently, this surgical plan requires a high level of surgical experience from the operating surgeon. Even if this is realized, it is difficult to expect a successful surgery unless the surgery is performed by an experienced operating surgeon, because 2D images lack image information in terms of image quality and 3D spatial information compared to 3D CT images.

[0006] Considering these points, the development of an image-based surgical planning method and system based on an image system rather than a subjective surgical planning method that relies on the surgeon's experience is desirable in many respects.

[0007] The present disclosure proposes a method and system for automatically establishing an image-based 3D surgical plan based on multiple images taken in different directions and performing surgery based on the plan.

[0008] The present disclosure proposes a method and a system for applying the same, which can automatically establish a 3D surgical plan using three or more images from different directions acquired before and / or during surgery, and perform surgery based on the plan.

[0009] Image-based 3D surgical planning method according to the present disclosure:

[0010] A step of one or more image acquisition devices acquiring one or more surgical site images including image information in multiple directions including different first directions, second directions, and third directions with respect to a spinal surgical site of a patient;

[0011] A step of the image processing unit extracting a first image in a first direction and a second image in a second direction from the surgical site image;

[0012] A step of the object separation unit extracting a first vertebral body region and a first pedicle region corresponding to the vertebral body and pedicle from the first image, and extracting a second vertebral body region and left and right second pedicle regions within the second vertebral body region from the second image;

[0013] A step of setting a first reference line passing through a first coordinate set in a central region of a first vertebral region and a first reference line passing through a second coordinate set within a first vertebral body region, wherein the path generation unit;

[0014] A step of the path generation unit setting a second reference line passing through both the third coordinate and the fourth coordinate set in the central area of ​​the left and right spinal cord areas in the second image;

[0015] A step of the image processing unit extracting a third image in a third direction different from the first and second directions from an image of the surgical site acquired from the spinal surgical site of the patient based on the first and second reference lines;

[0016] A step of extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and

[0017] The path setting unit includes a step of setting proximal setting coordinates for determining a screw path, which are set in the central region of at least one of the left and right third pedicles in the third image, and distal setting coordinates, which are set within the vertebral body region.

[0018] According to one or more embodiments,

[0019] The first reference line may be perpendicular to a midline passing through an area between the first pedicle region and the first vertebral body region.

[0020] According to one or more embodiments,

[0021] The central region may include the center and / or vicinity of the corresponding vertebral body region and / or pedicle region.

[0022] According to one or more embodiments,

[0023] The third image can be obtained from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0024] According to one or more embodiments,

[0025] The third image is obtained from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line can pass through the central region of each of the left and right pedicles in the second image.

[0026] According to one or more embodiments,

[0027] The second reference line may be set parallel to any straight line connecting the midpoint between the center of one side of the spinal cord region and the coordinates of one corner of the third image close thereto, and the midpoint between the center of the other side of the spinal cord region and the coordinates of the other corner of the third image close thereto.

[0028] According to one or more embodiments,

[0029] The above object separation unit extracts the neural foramen region between the left and right third pedicle regions from the third image,

[0030] The above path setting section:

[0031] In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and

[0032] The point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other can be set as the distal setting coordinate.

[0033] According to one or more embodiments,

[0034] The above path setting unit can be set so that the screw path reference line on which the screw path is placed passes through the above distal setting coordinates.

[0035] According to one or more embodiments,

[0036] On the first reference line, the screw starting point and the end point within the vertebral body are set, and the distance between the screw starting point and the end point can be set as the length of the screw used in the surgery.

[0037] According to one or more embodiments,

[0038] The first vertebral body region has an inner edge facing the first vertebral body region and an outer edge opposite thereto, and the starting point may be located on or adjacent to the outer edge of the first vertebral body region, and the ending point may be located within the first vertebral body region.

[0039] According to one or more embodiments,

[0040] The starting point may be positioned on one side of the second vertebral body region facing outward from the second vertebral body region or adjacent to the edge from the inside thereof, and the ending point may be positioned on the other side opposite to the one side or adjacent to the edge from the inside thereof.

[0041] According to one or more embodiments,

[0042] The surgical site image including the above multi-directional image information can be acquired by a 3D image acquisition device.

[0043] Image-based 3D surgical planning system according to the present disclosure:

[0044] One or more image acquisition devices for acquiring one or more images of the affected area having image components in multiple directions with respect to the spinal surgical site;

[0045] An image processing unit responsible for image processing related to surgery, including extracting a first image in a first direction from one or more of the above-mentioned images, a second image in a second direction different from the first direction, and a third image in a third direction different from the first and second directions;

[0046] An object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebra and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a pedicle region from the third image;

[0047] A reference line setting unit that sets a first reference line passing through a first coordinate set in the central region of the first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through the third coordinate and the fourth coordinate set in the central region of the left and right vertebral regions in the second image; and

[0048] In the third image, a screw path setting unit is provided for setting a proximal setting coordinate for determining a screw path in the central region of at least one of the left and right third pedicles and a distal setting coordinate set at or near the center of the vertebral body region, thereby setting a 3D screw path.

[0049] According to one or more embodiments, in the image-based 3D surgical planning system, the reference line setting unit may set the first reference line to orthogonally cross a midline passing through an area between the first pedicle region and the first vertebral body region.

[0050] According to one or more embodiments, in the image-based 3D surgical planning system, the image processing unit can obtain a third image from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

[0051] According to one or more embodiments, in the image-based 3D surgical planning system, the image processing unit obtains the third image from the surgical site image based on coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line can pass through the central region of each of the left and right pedicles in the second image.

[0052] According to one or more embodiments, in the image-based 3D surgical planning system, the second reference line may be set parallel to an arbitrary straight line connecting a midpoint between the center of one of the spinal pedicle regions and a coordinate of a corner of the third image close thereto, and a midpoint between the center of the other spinal pedicle region and a coordinate of the corner of the other image close thereto.

[0053] According to one or more embodiments, in the image-based 3D surgical planning system, the object segmentation unit extracts a neural canal region between the left and right third pedicle regions from the third image,

[0054] The above path setting section:

[0055] In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and

[0056] The point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other can be set as the distal setting coordinate.

[0057] According to one or more embodiments, in the image-based 3D surgical planning system, the path setting unit can set the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

[0058] According to one or more embodiments, in the image-based 3D surgical planning system, a screw starting point for a pedicle and an end point within a vertebral body are set on the first reference line, and a distance between the screw starting point and the end point can be set as a length of a screw used in the surgery.

[0059] The present invention proposes a method and system capable of automatically establishing a 3D surgical plan using multiple image data. This invention addresses the limitations inherent in surgical methods, such as the limitations of existing surgical plans where the outcome is limited by the surgeon's experience, thereby enabling efficient surgical planning and execution. According to the present invention, by automatically generating a surgical plan, the overall surgical time can be shortened, while maintaining high-quality results.

[0060] FIG. 1 is a flowchart of sequential processing steps of an image-based 3D surgical planning method according to one or more embodiments of the present disclosure.

[0061] FIG. 2 schematically illustrates the definition of 3D coordinates for a virtual 3D surgical space of a first image and a second image, according to one or more embodiments of the present disclosure.

[0062] FIG. 3 shows an output image in which a vertebra body and a pedicle are segmented from an LL image by applying a first image segmentation model according to one or more embodiments of the present disclosure.

[0063] FIG. 4 shows an output image in which a vertebra body and a pedicle are segmented from an AP image by applying a second image segmentation model according to one or more embodiments of the present disclosure.

[0064] FIG. 5 schematically illustrates a process for generating a deep learning model applied to one or more embodiments according to the present disclosure.

[0065] FIG. 6 shows an image of a plurality of first learning images labeled with respect to a vertebra body and a pedicle according to one or more embodiments of the present disclosure.

[0066] FIG. 7 shows an image of a plurality of second learning images labeled with respect to the vertebra body and pedicle according to one or more embodiments of the present disclosure.

[0067] FIG. 8 shows an image of a plurality of third learning images labeled with respect to a vertebra body, a pedicle, and a foramen according to one or more embodiments of the present disclosure.

[0068] FIG. 9 shows the results of labeling targets segmented in a first image and a second image according to one or more embodiments of the present disclosure.

[0069] FIG. 10 illustrates a first selected L1 target according to one or more embodiments of the present disclosure.

[0070] FIG. 11 illustrates a vertebral body region (A) and a pedicle region (B) set in a first image, according to one or more embodiments of the present disclosure.

[0071] FIG. 12 is a graph of a yz plane showing the coordinates of vertices of a vertebral body region (A) and a pedicle region (B) set in a first image and the midline between these regions, according to one or more embodiments of the present disclosure.

[0072] FIG. 13 is a graph of the yz plane showing a temporary straight line obtained by rotating the midline of FIG. 11 by 90 degrees, according to one or more embodiments according to the present disclosure.

[0073] FIG. 14 is a yz-plane graph showing a first reference line D arranged parallel to a temporary straight line D' according to one or more embodiments of the present disclosure.

[0074] FIG. 15 is a yz plane graph showing the set coordinates of a target point (T) or an end point of a surgical screw on a screw pass reference line according to one or more embodiments of the present disclosure.

[0075] FIG. 16 is an image showing a state in which a second vertebral body region (a) and a second pedicle region (b, b') are extracted or separated through a first image segmentation model in one or more embodiments of the present disclosure.

[0076] FIG. 17 is an xz plane graph showing an elliptical second vertebral region (b, b') within a second vertebral region (a) having vertices a1, a2, a3, a4, in one or more embodiments of the present disclosure, represented by a dotted line.

[0077] FIG. 18 illustrates a state in which a first reference line and a second reference line are set in a first image and a second image acquired according to one or more embodiments.

[0078] FIG. 19 illustrates a third image acquired along an axial slice path according to one or more embodiments.

[0079] FIG. 20 illustrates a vertebral region, a pedicle region, and a neural canal region as ROIs extracted from a third image according to one or more embodiments.

[0080] FIG. 21 illustrates a process for setting a screw path baseline for an ROI of a third image, according to one or more embodiments.

[0081] FIG. 22 illustrates a third image and an ROI of a third image in a state of a screw path baseline for the ROI according to one or more embodiments.

[0082] FIG. 23 illustrates a state in which a screw path of total length is set on a screw path reference line for an ROI of a third image, according to one or more embodiments.

[0083] FIG. 24 shows a schematic configuration of a surgical robot system according to one or more embodiments of the present disclosure.

[0084] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the embodiments described below. It is preferable to interpret that the embodiments of the present invention are provided to more completely explain the present invention to a person having average knowledge in the art. Like reference numerals denote like elements throughout. Furthermore, various elements and areas in the drawings are schematically drawn. Therefore, the present invention is not limited by the relative sizes or intervals drawn in the accompanying drawings.

[0085] While terms like "first" and "second" may be used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a "second component," and vice versa, without departing from the scope of the present invention.

[0086] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the concept of the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the expressions "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

[0087] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, it is to be understood that commonly used terms, such as those defined in dictionaries, should be interpreted to have a meaning consistent with their meaning within the relevant technical context, and should not be interpreted in an overly formal sense unless explicitly defined herein.

[0088] In some embodiments, where the implementation is otherwise feasible, a particular process sequence may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0089] Hereinafter, an image-based 3D surgical planning method and system are described in detail according to one or more embodiments.

[0090] FIG. 1 is a sequential flowchart of an image-based 3D surgical planning method according to the present disclosure.

[0091] The process of Figure 1 includes steps S6 to S14 that are repeatedly performed for multiple target vertebral bodies, and in the case of one vertebral body, a surgical plan for one vertebra is established in one flow.

[0092] One or more embodiments described below are described based on a 3D image obtained by a 3D image acquisition device, but it should be noted that the embodiments of the present disclosure are not proposed by a 3D image obtained by a 3D image acquisition device. The present disclosure includes any device capable of producing an image obtained by one or more image acquisition devices capable of capturing image information from three different directions, particularly a combined or mutually aligned three-directional image.

[0093] A step-by-step explanation of Figure 1 is as follows.

[0094] The method according to the present disclosure generally includes a preparatory work process, an LL point generation process, an AP point generation process, an axial (hereinafter often referred to as AX) point generation process, and an optimization and automation process.

[0095] < Stage S1 >

[0096] Step S1 is a step of capturing multiple images of the patient's affected area from various directions. This step utilizes 3D images obtained by a 3D image acquisition device according to the present embodiment.

[0097] <S2 단계>

[0098] Step S2 is the step of loading a 3D image with axes in the X, Y, and Z directions into the image processing unit for processing. In this step, 3D images and coordinate information are loaded.

[0099] <S3 단계>

[0100] Step S3 involves processing the loaded 3D image from two directions to obtain the first and second images as DRR (Digitally Reconstructed Radiograph) images. In this step, the first image, for example, is a Lateral-Lateral Image (LL image), and the second image, for example, is an Anterior-Posterior Image (AP image). These images can be obtained by orthogonal projection in the first and second directions of the 3D image.

[0101] In the above LL image, i.e., the first image, the left-right direction (back to stomach) is the Y-axis, and the up-down direction (head to feet) is also the Z-axis. And in the above AP image, i.e., the second image, the up-down direction (head to feet) is the Z-axis, and the left-right direction (left arm to right arm) is the X-axis, so the first image, e.g., the LL image, is parallel to the YZ plane, and the second image, e.g., the AP plane, is parallel to the XZ plane.

[0102] < Stage S4 >

[0103] In step S4, the vertebral body and pedicle are extracted or segmented as target objects from the first and second images obtained in the aforementioned process.

[0104] Object segmentation can be performed by applying the LL segmentation model and the AP segmentation model trained by deep learning. Fig. 3 shows an output image in which the vertebra body and pedicle are segmented from the LL image by applying the first image segmentation model, and Fig. 4 shows an output image in which the vertebra body and pedicle are segmented from the AP image by applying the second image segmentation model.

[0105] The above segmentation models can be based on the DeepLabv3+ model as shown in FIG. 5, and it is obvious that other models can be applied according to other embodiments of the present disclosure.

[0106] The DeepLavb3+ model is a deep learning model used in computer vision, used for extraction and segmentation tasks. Based on a deep convolutional neural network (CNN) architecture, it features an encoder and decoder structure, making it particularly well-suited for image segmentation.

[0107] Training the above model involves the same process as general deep learning training. Training such a model requires various spine-related image data collected for training. The training images include a first training image captured from a first direction and a second training image captured from a second direction. As shown in FIGS. 4 and 5, pixel-level labeling is performed on the first and second images, and then a learning process is performed to obtain the desired first and second image segmentation models. FIGS. 6 and 7 described above show images in which the vertebrae body and pedicle are labeled in each of a plurality of first and second training images, respectively. FIG. 8 shows an image of a plurality of third learning images labeled with respect to a vertebra body, a pedicle, and a foramen according to one or more embodiments of the present disclosure.

[0108] < S5 stage >

[0109] This step involves labeling the targets segmented in the previous step. Figure 9 shows the results of labeling the targets segmented in the AP and LL images.

[0110] As shown in Figure 9, the segmented targets were sequentially labeled as L1, L2, L3, L3, L4, and L5 from the top, and the same label was given to the same vertebral body.

[0111] < S6 stage >

[0112] In this step, one of multiple targets is selected prior to the process of extracting LL points. For example, as shown in Fig. 10, the L1 target may be selected first and then subjected to subsequent processes.

[0113] < S7-S8 stage >

[0114] The extraction of LL parameters for the target of the selected label is performed, and the first reference line in the LL image plane is determined through this. This reference line is the first axial slice path (axial slice path (1)) for extracting the aforementioned third image, which corresponds to one axis of the third-direction image plane to be performed later. When planning the first reference line, it must be set so as not to go beyond the pedicle and vertebral body regions. Here, if the slope of only one region of the vertebral body region or the pedicle region is reflected, the path may go outside the other region, and in order to prevent a path planning failure due to a detection error that occurs during object segmentation, the first reference line is set to the average slope of the slopes of the two regions.

[0115] To this end, regions of interest (ROIs) are first established for the spinal column and vertebral body. In Fig. 11, (A) represents the vertebral body region, and (B) represents the spinal column region. These ROIs can be obtained using the first image segmentation model described above.

[0116] As illustrated in Figure 12, the vertebral body region (A) is a rectangular region defined by four vertices A1, A2, A3, and A4, and the pedicle region (B) is a rectangular region defined by four vertices B1, B2, B3, and B4, and these two regions are adjacent. A midline (C) passes between the adjacent edges of the two adjacent regions.

[0117] The above midline C passes through the exact center between adjacent sides (B2-B3, A1-A4) of both areas (A, B), and the intersections C1 and C2 are located on the midline C. The intersection C1 is located in the middle of the line segment connecting the vertices B2 and A1 of both areas, and C2 is located in the middle of the line segment connecting the vertices B3 and A4. Therefore, the y-coordinate of C1 (C1.y) is (A1.y+B2.y) / 2, and the z-coordinate of C1 (C1.z) is (A1.z+B2.z) / 2. In addition, the y-coordinate of C2 (C2.y) is (A4.y+B3.y) / 2, and the z-coordinate of C2 (C2.z) is (A4.z+B3.z) / 2.

[0118] Thus, the midline C has the average slope (Ca) for the slopes of both regions in the plane coordinates of the LL image, i.e., the yz plane. This midline C serves as a reference for setting the first reference line in the LL image plane, and the coordinates C1(y, z), C2(y, z) of the intersection points C1 and C2 in the yz plane coordinate system and the slope Ca of the midline C are expressed by the following formulas.

[0119]

[0120] Figure 13 is a graph showing a temporary point C3 in the intermediate process of establishing a first reference line from the midline C between the pedicle region (B) and the vertebral body region (A) in the yz plane.

[0121] The coordinates of the above temporary point C3 correspond to the coordinates of the intersection point C1 rotated 90 degrees around the intersection point C2 on the above midline C. That is, if the coordinates of C1 in the yz plane coordinate system are (y1, z1), the coordinates of C3 become (z1, -y1). Consequently, the temporary straight line D' connecting the intersection point C2 and the temporary point C3 is a normal line orthogonal to the above midline C, and the straight line equation of this normal line is D':Z = Da(Y - C2.y) + C2.z.

[0122] Fig. 14 illustrates a first reference line D arranged parallel to the temporary straight line D'. The first reference line D intersects or is perpendicular to the center line C at a predetermined angle, and is parallel to the temporary straight line D'. The temporary straight line D' is moved parallel to the center area (B) of the pedicle region having four vertices, such as B1, B2, B3, and B4, that is, the center (B5) or the vicinity of the center, to determine the first reference line D in the y-z plane.

[0123] The y-coordinate B5.y and z-coordinate B5.z of the above B5 are expressed as follows.

[0124] B5.y = (B1.y + B2.y + B3.y + B4.y) / 4

[0125] B5.z = (B1.z + B2.z + B3.z + B4.z) / 4

[0126]

[0127] The equation of the straight line of the first reference line (D) with slope Da is expressed as follows.

[0128] D:Z = Da (Y - C2.y) + C2.z

[0129] Da = (C3.z - C2.z) / (C3.y - C2.y)

[0130]

[0131] The first reference line D, expressed by the straight line equation (D:z) as above, passes through the outer side of the pedicle (BS, B1-B4) and creates an intersection point E, which becomes the starting point for determining the length of the surgical screw to be described later.

[0132] The outer side (BS) of the above pedicle is expressed as follows.

[0133] BS: Z = Ba(Y - B1.y) + B1.z

[0134] Ba = (B4.z + B1.z) / (B4.y + B1.y)

[0135]

[0136] Next, the coordinates of the start point located on the first reference line D and the terminating point reached by the end of the surgical screw are calculated.

[0137] The coordinates (Ey, Ez) of the starting point E located on the outer side of the pedicle can be calculated by a first-order simultaneous equation that applies the straight line equation (D:Z) of the first reference line and the straight line equation (BS:Z) of the outer side BS of the pedicle. The y-coordinate (Ez) and the z-coordinate (Ez) of the starting point can be calculated by a first-order simultaneous equation that applies the straight line equation (D:Z) of the first reference line and the straight line equation (BS:Z) of the outer side BS of the pedicle.

[0138] D:Z = Da (Y - C2.y) + C2.z

[0139] BS: Z = Ba(Y - B1.y) + B1.z

[0140] First, the two lines are the same at (Ey, Ez), and since Y and Z of the two straight lines are the same, the calculation is as follows.

[0141] Da (Y - C2.y) + C2.z = Ba(Y - B1.y) + B1.z

[0142] The Ey value can be obtained by grouping the terms related to Y in the above equation into one side and separating Y from the other.

[0143] Ey = Y = (B1.z - C2.z + Ba * B1.y - Da * C2.y) / (Da - Ba)

[0144] By substituting Ey obtained from the above equation into the straight line equation (D:Z) of the screw pass, the z-direction coordinate Ez of the starting point can be obtained.

[0145] Ez = Z = Da (Ey - C2.y) + C2.z

[0146] The above calculation process is one of several calculation methods for obtaining the coordinates of the starting point (E), and can also be calculated by other methods, and it is obvious that a specific calculation method does not limit the technical scope of the present disclosure.

[0147] Fig. 15 illustrates the set coordinates of the end point (T) of the surgical screw on the screw path on the yz plane. The coordinates of the end point are determined by the total length (Ls) of the screw to be used. In one embodiment according to the present disclosure, the total length (Ls) of the screw is set so as not to exceed the length of the side (A1-A2) of the vertebral body region in the direction in which the surgical screw advances, and the diameter (Ds) thereof is set so as not to be larger than the length of the outer side (B1-B4) where the screw entry point is located.

[0148]

[0149] Here, the coordinates of the endpoint T(y, z) are expressed by the equation below.

[0150] Ty = Ey + Ls * Cos(arctan(Da))

[0151] Tz = Ez + Ls * Sin(arctan(Da))

[0152] The length of the line segment connecting the coordinates of the starting point and the coordinates of the ending point obtained through the above process corresponds to the length of the screw to be used in the surgery.

[0153] < S9-S10 steps >

[0154] This step obtains a second reference line, i.e., an axial slice path (2), which is paired with the first reference line in the yz plane.

[0155] As illustrated in Fig. 16, the ROI (a) of the vertebral body and the left and right pedicles (b), (b') can be extracted through a segmentation model for the second image, i.e., the AP image.

[0156] A key point in establishing the second baseline in the AP image is that, when acquiring the third image described below, the AX image, it is necessary to acquire an image of the slice cross-section that maximizes the left and right pedicles. This requires reflecting the degree of vertebral rotation, thereby securing the largest pedicle area in the AX image.

[0157] Therefore, the average slope value is calculated to reflect the slope of the ROI (a) of the vertebral body and the segmentation error of the two left and right pedicle regions b and b'.

[0158] The average slope increases or decreases depending on the arrangement of the left and right pedicles, and the temporary second reference line (d') representing this average slope is obtained as follows.

[0159] As shown in Fig. 17, first, in one embodiment, a midpoint (d1) between the center (b5) of the left pedicle (b) and the corner (a1) of the vertebral body region closest thereto and a midpoint (d2) between the center (c5) of the right pedicle (b') and the corner (a2) of the vertebral body region closest thereto are obtained, and a temporary second reference line (d') having an average slope is obtained by connecting these two midpoints (d1, d2).

[0160] As shown in FIG. 18, by applying the YZ position of the first reference line (D) in the first image obtained through the above process and the slope of the temporary reference line (d') in the X, Z plane in the second image to obtain a third image in the AX plane of the 3D image, a final second reference line (d) that passes through the centers of both pedicles on average is obtained, as in the second image on the right side of FIG. 18, and through this, an axial third image, i.e., an AX image, can be obtained from the slice plane according to the first and second reference lines as shown in FIG. 19.

[0161] Here, the second reference line (d') illustrated in Fig. 17 is depicted as spanning the upper side of both pedicles (b, b'), and in Fig. 18 is depicted as passing through the central regions of both pedicles (b, b'), i.e., the vicinity of the center. The second reference line (d) in Fig. 18 represents the final second reference line when the slope of the second reference line in the XZ plane is applied to the YZ position of the first reference line. That is, the final second reference line is a line resulting from translating the temporary second reference line (d') in parallel to the average center of both pedicles (b, b').

[0162] < Step S11 >

[0163] This is a step of extracting AX parameters for the AX image obtained as described above.

[0164] Figure 20 shows the vertebral body region (a), the left and right pedicle regions (b, b'), and the neural canal region (f) between the left and right pedicles extracted by segmentation for the above AX image.

[0165] The extraction of the above regions is performed by the object segmentation unit, and the segmentation of the object can be performed by applying the AX segmentation model trained by deep learning using labeled samples for the vertebra body, pedicle, and foramen in the AX image as shown in FIG. 18, similar to the LL segmentation model and AP segmentation model trained by deep learning as described above.

[0166] Figures 21(a) to (b) show the process of setting a screw path reference line on which a screw path is placed in an AX image, which is a surgical screw insertion or placement plane, and each process is explained separately.

[0167] (a) Set a center point “fc” in the central region of the neural canal region, and also set a center point “ad” in each central region of the spinal canal region, and then set a third reference line (L) passing through these two points fc and ad.

[0168] (I) Set the intersection points a'c and ac between the third reference line and the inner and outer borders of the vertebral body region. This intersection point ac becomes the distal setting coordinate through which the screw path reference line passes. Meanwhile, the distance a'c - ac represents the width of the vertebral body in the direction of the third reference line, and this distance can be applied to determine the total length of the screw.

[0169] (c) Establish the first screw pass reference line (LP) passing through the above intersection point ac and the center of the pedicle (b'c) on one side (left side in the drawing).

[0170] (a) Establish a second screw pass reference line (LP) passing through the intersection point ac and the center of the pedicle (bc) on the other side (right side in the drawing).

[0171] The first and second screw pass reference lines set as above are the axis lines along which the surgical screw is installed.

[0172] Figures 22(a) and (b) show the first and second screw pass reference lines set through the same process as above.

[0173] As illustrated in Fig. 22, the starting point or entry point (LE, RE) of the surgical screw is formed at the intersection of the first and second screw pass reference lines and the outer borders of both pedicles. This entry point is set at the proximal set coordinates through which the screw pass reference lines pass.

[0174] <S12-S13 단계>

[0175] Steps S12-S13 are the steps to finalize the surgical plan by setting the screw end point (RT, LT) away from the starting point (LE, RE), which is the final entry point of the surgical screw.

[0176] The above entry point is located at the proximal set coordinate where the screw enters the pedicle.

[0177] In this step, the total screw length (Ls) obtained in the process of setting the first reference line mentioned above or the width of the vertebral body in the direction of the first reference line can be applied to determine the screw length, or the width of the vertebral body in the direction of the third reference line (the distance a'c-ac) can also be considered and determined together. In addition, it can be adjusted by the operating surgeon considering the above results as needed. Meanwhile, the diameter of the screw needs to be set to be smaller than the average diameter of the pedicle or the diameter of the smaller pedicle among the left and right pedicles, and this can also be changed by the operating surgeon.

[0178] The outer picture of Fig. 23 shows the screw pass with the screw total length (Ls) applied to the ROI area, and the right picture shows the state in which the screw pass is determined in the third image.

[0179] The above screw pass has 3D coordinates, which are recognized by the surgical system during surgery and the surgery is performed as is.

[0180] The above processes were performed for one target, but if there are multiple target targets, that is, if there are multiple target spines and it is not the last target yet in step S14, the process returns to step S6, repeats the process from step S6 to step S14, and then ends.

[0181] The embodiments of the present disclosure, as described above, provide an image-based automatic 3D surgical planning method and system. According to the present invention, safe and reliable surgical performance is possible without relying heavily on the surgeon's skill or experience.

[0182] Embodiments of the present disclosure may be implemented as a computer program medium storing software for performing the aforementioned method of automatically establishing a surgical plan on a computer. Furthermore, the present invention may be implemented through an image processing unit, an object separation unit, a baseline setting unit, a path setting unit, etc., utilizing a processor, memory, and a display, as components for performing the aforementioned method.

[0183] Additionally, the present invention can also be implemented by an image-based surgical robot system.

[0184] Referring to FIG. 24, a surgical robot system (1) according to one or more embodiments of the present disclosure applies an image acquisition device (100) as a main image acquisition device. In addition, the robot system (1) is configured to include a surgical robot (200), a position sensor (300), and a navigation system (400), and the surgical robot (200) is configured to include a main body (201), a robot arm (203) equipped with an end effector (203a), and a robot controller (205).

[0185] In an embodiment according to the present disclosure, except for the image acquisition device described above, the image processing unit, the object separation unit, the baseline generation unit, and the path determination unit may be functionally included in the devices mentioned above. In addition, the image processing unit, the object separation unit, the baseline setting unit, the path setting unit, etc. may be implemented by the surgical planning software, and in particular, the object separation unit performs extraction or separation of the vertebral body and the pedicle from the first image, the second image, and the third image by applying the LL segmentation model, the AP segmentation model, and the AX segmentation model trained by deep learning.

[0186] The robotic arm (203) is controlled according to the surgical plan and control software determined during the surgery, based on the initiation. The navigation system (400) can display surgical plan information for surgical tools or implants on images acquired during the surgery, or can assist the surgeon's surgical actions by displaying the real-time location of the surgical tools or implants on images or, in some cases, 3D images acquired before the surgery. To this end, a display that allows the surgeon to visually compare the real-time location of surgical tools, etc. with the surgical plan and surgical status during the surgery can be connected to the navigation system (400).

[0187] While various embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the spirit and scope of the invention as defined in the appended claims. Therefore, modifications to future embodiments of the present invention will not depart from the scope of the present invention.

Claims

1. A step of one or more image acquisition devices acquiring one or more surgical site images including image information in multiple directions including different first directions, second directions, and third directions with respect to a spinal surgical site of a patient; A step of the image processing unit extracting a first image in a first direction and a second image in a second direction from an image of a surgical site; A step of the object separation unit extracting a first vertebral region and a first pedicle region from the first image, and extracting a second vertebral region and left and right second pedicle regions within the second vertebral region from the second image; A step of setting a first reference line, wherein the first coordinate is set in the central region of the first vertebral region and the second coordinate is set within the first vertebral body region; A step of setting a second reference line that passes through both the third coordinate and the fourth coordinate set in the central area of ​​the left and right spinal cord areas in the second image; A step of the image processing unit extracting a third image in the third direction from the surgical site image based on the first reference line and the second reference line; A step of extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and An image-based 3D surgical planning method comprising: a step of setting a proximal setting coordinate for determining a screw path in the central region of at least one of the left and right third pedicles in the third image, and a distal setting coordinate set at or near the center of the vertebral body region.

2. In paragraph 1, The above object extraction unit is an image-based 3D surgical planning method that extracts a vertebral body region and a pedicle region by applying a model generated by machine learning.

3. In paragraph 1, An image-based 3D surgical planning method, wherein the first reference line orthogonally crosses the midline passing through the area between the first pedicle region and the first vertebral body region.

4. In paragraph 1, An image-based 3D surgical planning method, wherein the central region includes the center and / or vicinity of the corresponding vertebral body region and / or pedicle region.

5. In any one of paragraphs 1 to 4, An image-based 3D surgical planning method, wherein the third image is obtained from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

6. In any one of paragraphs 1 to 4, An image-based 3D surgical method, wherein the third image is acquired from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through the central region of each of the left and right pedicles in the second image.

7. In paragraph 6, The second reference line is set parallel to an arbitrary straight line connecting the midpoint between the center of one spinal region and the coordinates of one corner of the third image close to it among the left and right spinal regions, and the midpoint between the center of the other spinal region and the coordinates of the other corner of the third image close to it.

8. In any one of paragraphs 1 to 4, The above object separation unit extracts the neural canal area between the left and right third spinal cord areas from the third image, The above baseline setting section: In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and An image-based 3D surgical method, wherein the point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region are mutually intersected is set as the distal setting coordinate.

9. In paragraph 8, An image-based 3D surgical planning method in which the above reference line setting unit sets the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

10. In any one of paragraphs 1 to 4, An image-based 3D surgical planning method, wherein a screw starting point and an end point within a vertebral body are set on the first reference line, and the distance between the screw starting point and the end point is set as the length of the screw used in the surgery.

11. One or more image acquisition devices for acquiring one or more surgical site images having image components in multiple directions with respect to the spinal surgical site; An image processing unit responsible for image processing related to surgery, including extracting a first image in a first direction, a second image in a second direction different from the first direction, and a third image in a third direction different from the first and second directions from one or more surgical site images; An object separation unit that extracts a first vertebral body region and a first pedicle region corresponding to a vertebral body and a pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a third pedicle region from the third image; A reference line setting unit that sets a first reference line passing through a first coordinate set in the central region of the first vertebral region and a second coordinate set within the first vertebral body region, and a second reference line passing through the third coordinate and the fourth coordinate set in the central region of the left and right vertebral regions in the second image; and An image-based 3D surgical planning system, comprising: a screw path setting unit for determining a 3D screw path by setting a proximal setting coordinate for determining a screw path in the central region of at least one of the left and right third pedicles in the third image and a distal setting coordinate set at or near the center of the vertebral body region.

12. In paragraph 11, An image-based 3D surgical planning system, wherein the reference line setting unit sets the first reference line to orthogonally cross the midline passing through the area between the first pedicle region and the first vertebral body region.

13. In paragraph 11, An image-based 3D surgical method, wherein the image processing unit obtains a third image from the surgical site image based on the coordinates of the first reference line and the slope of the second reference line.

14. In paragraph 11, An image-based 3D surgical system, wherein the image processing unit obtains the third image from the surgical site image based on the coordinates of the first reference line of the first image and the second reference line of the second image, wherein the second reference line passes through the central region of each of the left and right pedicles in the second image.

15. In paragraph 14, The second reference line is an image-based 3D surgical planning system, wherein the second reference line is set parallel to an arbitrary straight line connecting the midpoint between the center of one side of the spinal cord region and the coordinates of one corner of the third image close thereto, and the midpoint between the center of the other side of the spinal cord region and the coordinates of the other corner of the third image close thereto.

16. In paragraph 11, The above object separation unit extracts the neural canal area between the left and right third spinal cord areas from the third image, The above path setting section: In the third image, a third reference line is set that passes through the coordinates of the center or near the center of each of the third vertebral body region and the neural canal region, and An image-based 3D surgical planning system that sets the point where the third reference line and the boundary line of the vertebral body region farthest from the neural canal region intersect each other as the distal setting coordinate.

17. In paragraph 16, An image-based 3D surgical planning system in which the path setting unit sets the screw path reference line on which the screw path is placed to pass through the distal setting coordinates.

18. In any one of paragraphs 11 to 17, An image-based 3D surgical planning system in which a screw starting point and an end point within the vertebral body are set on the first reference line, and the distance between the screw starting point and the end point is set as the length of the screw used in the surgery.

Images