Image-based three-dimensional automatic operation planning method and system

Abstract

An image-based three-dimensional (3D) surgical planning method and system are disclosed. The planning method comprises: a step of obtaining a first image and a second image from a spinal site image; a step of obtaining a first baseline and a second baseline from the first image and the second image; and determining a three-dimensional screw path by acquiring a third image from the spinal surgical site on the basis of the first baseline and the second baseline and setting proximal set coordinates for determining the screw path and distal set coordinates for determining the screw path in a central region of at least one of the left and right third pedicles, the distal setting coordinates are set as the center of the spinal body region or near the center.

Description

Technical Field

[0001] The present disclosure relates to an automatic surgery planning method and system in a surgical robot system, and more particularly, to an automatic surgery planning method and system, which can be used to generate a surgery planning path in a three-dimensional image space. Background Art

[0002] During spinal surgeries such as pedicle screw fixation surgery using a surgical robot system, a three-dimensional image that can achieve axial view is required. Therefore, instead of a two-dimensional image, a three-dimensional image that can achieve axial view, such as a computed tomography (CT) image, is used as the axial view image. Typically, a two-dimensional (2D) image acquisition device such as a so-called C-arm is used during intra-operation to verify in real time whether the operation is being performed according to the plan established before the operation. In order to use the 2D image thus obtained to navigate the surgical tool according to the pre-established plan, it is necessary to register the three-dimensional image obtained before the operation with the two-dimensional image obtained during the operation. However, in the process of registering the 3D image with the 2D image, problems such as delayed operation time and incomplete registration may occur, resulting in registration errors.

[0003] For surgical planning based solely on 3D images obtained before surgery without the aid of 2D images during surgery, exposure to radiation when obtaining 3D images such as CT images and problems arising from the difference between the patient's status before surgery and that during surgery are inevitable.

[0004] However, if a surgical plan is established based on the two-dimensional X-ray image obtained during surgery, it is not necessary to align with the image obtained before the surgery, and the flexibility of changing the surgical plan as needed can be expected even during surgery. However, in this process, the operating surgeon should estimate the position of the 3D space of the operation in the 3D space while viewing the 2D image. As a result, this surgical plan requires the operating surgeon to have experience in difficult surgeries. Even if this surgical plan is implemented, the two-dimensional image lacks clarity and image information based on three-dimensional space information relative to the three-dimensional CT image, so unless the operation is performed by an experienced operating surgeon, it is difficult to expect a successful operation.

[0005] In view of these, the development of image-based surgical planning methods and systems, rather than subjective surgical planning that relies on the experience of the surgeon, is preferred in various aspects.

[0006] <Prior Art Literature>

[0007] US8,335,553

[0008] CN107157579

[0009] US 7,787,932B2

[0010] KR101264198B1

[0011] US0659599B2

[0012] US6226548B1

[0013] US20070270866A1

[0014] US20050192575A1 Summary of the invention

[0015] Technical issues

[0016] The present disclosure provides methods and systems for automatically establishing an image-based three-dimensional (3D) surgical plan based on a plurality of images in different directions, and performing surgery based on the established plan.

[0017] The present disclosure provides a method and system for automatically establishing a 3D surgical plan using three or more images in different directions acquired before and / or during surgery, and for performing surgery based on the established plan.

[0018] Technical Solution

[0019] An image-based three-dimensional (3D) surgical planning method according to the present disclosure includes:

[0020] The step of obtaining one or more surgical site images including image information in multiple directions of the patient's spinal surgical site by one or more image obtaining devices, wherein the multiple directions include a first direction, a second direction, and a third direction that are different from each other;

[0021] A step of extracting, by an image processing unit, a first image in a first direction and a second image in a second direction from the surgical site image;

[0022] The object separation unit extracts a first vertebral body region and a first pedicle region from the first image, and extracts a second vertebral body region and left and right second pedicle regions within the second vertebral body region from the second image;

[0023] A first baseline and a second baseline are set by a baseline setting unit, wherein the first baseline is set by a first coordinate in a central region of a first pedicle region and a second coordinate in a first vertebral body region, and the second baseline is set by a third coordinate and a fourth coordinate in central regions of left and right pedicle regions in the second image;

[0024] extracting, by an image processing unit, a third image in the third direction from the surgical site image based on the first baseline and the second baseline;

[0025] extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and

[0026] A step of setting a screw path through a central region of at least one of the left and right third pedicles and the third vertebral body region by a path setting unit, setting proximal setting coordinates on the second pedicle, and setting distal setting coordinates through which the baseline of the screw path passes on a third baseline passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

[0027] According to one or more embodiments,

[0028] The object extraction unit may apply a model generated by machine learning to extract the vertebral body region and the pedicle region.

[0029] According to one or more embodiments,

[0030] The first baseline may be set to cross a middle line passing through a region between the first pedicle region and the first vertebral body region at an orthogonal or arbitrary angle, and pass through a central region of the first pedicle region, or be parallel to the central region of the first pedicle region at an arbitrarily set distance.

[0031] According to one or more embodiments,

[0032] The central region may include the center and / or vicinity of the center of the corresponding vertebral body region and / or pedicle region.

[0033] According to one or more embodiments,

[0034] The third image is obtainable from the surgical site image based on the coordinates of the first baseline and the tilt of the second baseline.

[0035] According to one or more embodiments,

[0036] The third image is obtained from the surgical site image based on coordinates of the first baseline in the first image and the second baseline in the second image, where the second baseline may pass through respective central areas of left and right pedicles in the second image.

[0037] According to one or more embodiments,

[0038] The second baseline can be set to be parallel to an arbitrary straight line, which connects the center of one pedicle area among the left and right pedicle areas, the midpoint between the coordinates of the corner on one side of the third image close to the center, the center of the other pedicle area, and the midpoint between the coordinates of the corner on the other side of the third image close to the center.

[0039] According to one or more embodiments,

[0040] The object separation unit extracts a neural foramen region between left and right third pedicle regions from the third image, and

[0041] The path setting unit

[0042] A third baseline is set in the third image, the third baseline passing through coordinates of the center or the vicinity of the center of each of the third vertebral body region and the neural tube region, and,

[0043] The distal end setting coordinate may be set at a point where the third baseline intersects with a boundary line of a vertebral body region farthest from the neural tube region or a position near the point or at an intersection where the third baseline intersects with the screw path baseline outside the vertebral body.

[0044] According to one or more embodiments,

[0045] The path setting unit may set a screw path baseline for placing the screw path to pass through the distal end setting coordinates.

[0046] According to one or more embodiments,

[0047] The screw start point for the pedicle and the target point in the vertebral body are set on the first baseline, and the distance between the screw start point and the target point can be set as the length of the screw used for surgery.

[0048] According to one or more embodiments,

[0049] The first pedicle region has an inner edge opposite to the first vertebral body region and an outer edge opposite to the inner edge, the starting point is located on or near the outer edge of the first pedicle region, and the target point may be located in the first vertebral body region.

[0050] According to one or more embodiments,

[0051] The starting point is arranged to be located on one side edge of the second pedicle region facing the outside of the second vertebral body region or close to the edge in the second pedicle region, and the target point can be arranged to be located on the other side edge opposite to the one side edge or close to the edge on the inner side of the other side.

[0052] According to one or more embodiments,

[0053] The surgical site image including image information in multiple directions can be obtained by a 3D image obtaining device.

[0054] An image-based 3D surgical planning system according to the present disclosure includes:

[0055] One or more image acquisition devices, which acquire one or more affected part images for obtaining a spinal surgery site, wherein the one or more affected part images have image components in various directions;

[0056] An image processing unit, which is responsible for image processing related to surgery, wherein the image processing includes extracting a first image in a first direction, a second image in a second direction, and a third image in a third direction from the one or more images of the surgical affected part, wherein the second direction is different from the first direction, and the third direction is different from the first direction and the second direction;

[0057] an object separation unit, which extracts a first vertebral body region and a first pedicle image corresponding to the vertebra and the pedicle from the first image, extracts a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extracts a third vertebral body region and a pedicle region from the third image; a baseline setting unit, which sets a first baseline and a second baseline, wherein the first baseline is set by a first coordinate set in a central region of the first pedicle region and a second coordinate set in the first vertebral body region, and the second baseline is set by a third coordinate and a fourth coordinate set in the central region of the left and right pedicle regions in the second image; and

[0058] A screw path setting unit sets a screw path through at least one central area of ​​the left and right third pedicles and the third vertebral body area, sets a proximal setting coordinate at the second pedicle, and sets a distal setting coordinate through which the baseline of the screw path passes on a third baseline passing through a neural canal area between the left and right third pedicle areas and the central area of ​​the vertebral body to determine a 3D screw path.

[0059] According to one or more embodiments, in the image-based 3D surgical planning system, the baseline setting unit may orthogonally cross the first baseline across a middle line passing through a region between the first pedicle region and the first vertebral body region.

[0060] According to one or more embodiments, in the image-based 3D surgical planning system, the image processing unit may obtain a third image from the surgical site image based on the coordinates of the first baseline and the tilt of the second baseline.

[0061] According to one or more embodiments, in an image-based 3D surgical planning system, the image processing unit obtains a third image from the surgical site image based on the coordinates of the first baseline of the first image and the second baseline of the second image, where the second baseline may pass through the central area of ​​each of the left and right pedicles in the second image.

[0062] According to one or more embodiments, in an image-based 3D surgical planning system, the second baseline can be set to be parallel to an arbitrary straight line, which connects the center of one pedicle region, the midpoint between the coordinates of the corner on one side of the third image close to the center, the center of the other pedicle region, and the midpoint between the coordinates of the corner on the other side of the third image close to the center in the left and right pedicle regions.

[0063] According to one or more embodiments, in the image-based 3D surgical planning system, the object separation unit extracts the neural canal region between the left and right third pedicle regions from the third image, and the path setting unit

[0064] A third baseline is set in the third image, the third baseline passing through coordinates of the center or the vicinity of the center of each of the third vertebral body region and the neural tube region, and,

[0065] The distal end setting coordinate may be set at a point where the third baseline intersects with a boundary line of a vertebral body region farthest from the neural tube region or at a position near the point, or at an intersection where the third baseline intersects with the screw path baseline outside the vertebral body.

[0066] According to one or more embodiments, in the image-based three-dimensional surgical planning system, the path setting unit may set a screw path baseline for placing the screw path to pass through the distal setting coordinate.

[0067] According to one or more embodiments, in an image-based 3D surgical planning system, the screw starting point for the pedicle and the target point within the vertebral body can be set on the first baseline, and the distance between the screw starting point and the target point (or terminal point) can be set as the length of the screw used for the surgery.

[0068] Beneficial Effects

[0069] The present invention provides a method and system that can be used to automatically establish a three-dimensional (3D) surgical plan using multiple image information. According to the present invention, the problem that the surgical results in existing surgical methods are limited by the limitations of the surgical method, such as the experience of the surgeon, can be solved, so that the surgical plan can be effectively established and executed. According to the present invention, the overall surgical time can be shortened by automatically generating a surgical plan, and a high level of surgical results can be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The present invention is a flowchart of sequential processing steps of an image-based three-dimensional (3D) surgical planning method according to one or more embodiments of the present disclosure.

[0071] Figure 2 According to one or more embodiments of the present disclosure, the definition of 3D coordinates of a virtual 3D surgical space in a first image and a second image is schematically shown.

[0072] Figure 3 An output image is shown obtained by extracting, segmenting or separating the vertebra and pedicle from the lateral-lateral (LL) image using a first image segmentation model according to one or more embodiments of the present disclosure. Figure 4 An output image obtained by applying a second image segmentation model according to one or more embodiments of the present disclosure to segment a vertebral body and a pedicle from an anterior-posterior (AP) image is shown.

[0073] Figure 5 A schematic diagram of a process for generating a deep learning model according to one or more embodiments of the present disclosure.

[0074] Figure 6 An image showing a state in which labels are attached to a vertebra body and a pedicle in a plurality of first learning images according to one or more embodiments of the present disclosure.

[0075] Figure 7 An image showing a state in which labels are attached to a vertebra body and a pedicle in a plurality of second learning images according to one or more embodiments of the present disclosure.

[0076] Figure 8An image showing a state in which labels are attached to a vertebral body, a pedicle, and a foramen in a plurality of third learning images according to one or more embodiments of the present disclosure.

[0077] Fig. 9 Results of attaching labels to objects segmented in a first image and a second image according to one or more embodiments of the present disclosure are shown.

[0078] Fig.10 An example of an L1 target being first selected according to one or more embodiments of the present disclosure is shown.

[0079] Fig.11 A vertebral body region A and a pedicle region B set in a first image according to one or more embodiments of the present disclosure are shown.

[0080] Fig.12 is a graph of the yz plane, which shows the vertex coordinates of the vertebral body region A and the pedicle region B and the middle line between the vertebral body region A and the pedicle region B set in the first image according to one or more embodiments of the present disclosure.

[0081] Fig.13 is a diagram of the yz plane, which illustrates the use of one or more embodiments of the present disclosure. Fig.11 The middle line shown is rotated 90° to form a temporary straight line.

[0082] FIG. 14 a is a yz plane diagram illustrating an initial first baseline D1 arranged parallel to a temporary straight line D′ according to one or more embodiments of the present disclosure.

[0083] FIG. 14 b is a yz plane diagram illustrating a modified first baseline D1 a according to one or more embodiments of the present disclosure, wherein the modified first baseline D1 a replaces the first baseline D1 shown in FIG. 14 a and is arranged in parallel with the first baseline D1 .

[0084] FIG. 15 a is a yz plane diagram showing set coordinates of a target point T or a terminal point of a surgical screw on the initial first baseline shown in FIG. 14 a according to one or more embodiments of the present disclosure.

[0085] 15 b is a yz plane graph showing set coordinates of a target point T or a terminal point of a surgical screw on the modified first baseline shown in FIG. 14 b according to one or more embodiments of the present disclosure.

[0086] Fig.16is an image, which shows a state in which the second vertebral body region a and the second pedicle regions b and b′ are extracted or separated by means of the first image segmentation model in one or more embodiments of the present disclosure.

[0087] Fig.17 is an xz plane diagram according to one or more embodiments of the present invention, wherein dotted lines are used to indicate elliptical second pedicle regions b and b' within a second vertebral body region a formed by vertices a1, a2, a3, and a4.

[0088] Fig.18 A state in which a first baseline and a second baseline are set in an acquired first image and a second image according to one or more embodiments is shown.

[0089] Fig.19 (A) illustrates an example of a third image obtained along an axial slice path according to one or more embodiments, and (B) shows a vertebral body region, a pedicle region, and a neural canal region as regions of interest (ROIs) extracted from the third image.

[0090] FIG. 20 a illustrates a process of setting a screw path baseline for a ROI in a third image, according to one or more embodiments.

[0091] 20b shows another illustration of an alternative screw path baselining process for a ROI in a third image in accordance with one or more embodiments.

[0092] FIG. 21 a shows the ROI and the third image in which the screw path baseline has been set for the third image ROI according to the embodiment shown in FIG. 20 a .

[0093] FIG. 21 b shows the ROI and the third image in which the screw path baseline has been set for the third image ROI according to the embodiment shown in FIG. 20 b .

[0094] FIG. 22 a shows an example of a state in which the entry point and the target point (or terminal point) of the total length screw path have been set on the screw path baseline of FIG. 21 a according to the embodiment of FIG. 20 .

[0095] FIG. 22 b shows an example of a state in which the entry point and the target point (or terminal point) of the total length screw path have been set on the screw path baseline of FIG. 21 b according to the embodiment of FIG. 20 a .

[0096] Fig.23 The schematic configuration of a surgical robot system according to one or more embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0097] Hereinafter, preferred embodiments of the inventive concept will be described in detail with reference to the accompanying drawings. However, the embodiments of the inventive concept may be changed into various other forms, and the scope of the inventive concept should not be interpreted as being limited to the embodiments described in detail below. The embodiments of the inventive concept are preferably interpreted as providing a more complete description of the inventive concept to those of ordinary skill in the art. The same symbols always represent the same elements. Furthermore, the various elements and regions in the accompanying drawings are schematically illustrated. Therefore, the inventive concept is not limited to the relative sizes or intervals illustrated in the accompanying drawings.

[0098] The terms "first", "second" and the like can be used to describe various constituent elements, but the constituent elements are not limited by the terms. The terms are only used to distinguish one constituent element from other constituent elements. For example, without departing from the scope of the invention, the first constituent element can be named as the second constituent element, and conversely, the second constituent element can be named as the first constituent element.

[0099] The terms used in this application are only used to illustrate specific embodiments and are not intended to limit the concept of the present invention. Unless otherwise apparent in the context, a singular form of expression includes a plural form of expression. In this application, expressions such as "including" or "having" should be understood to specify the presence of features, quantities, steps, operations, constituent elements, parts or combinations thereof recorded in the specification, without excluding in advance the presence or additional possibility of the quantity, operation, constituent element, part or combination of one or more other features.

[0100] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meanings as commonly understood by those of ordinary skill in the art to which the inventive concept belongs. Also, it should be understood that commonly used and predefined terms should be interpreted as having a meaning consistent with the meaning of these terms in the context of the relevant art, and, unless explicitly defined herein, should not be interpreted as excessive formal meanings.

[0101] When some embodiments are implemented differently, a specific process sequence may be performed in a different order than the order described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order to the order described.

[0102] Hereinafter, according to one or more embodiments, an image-based three-dimensional surgical planning method and system will be described in detail.

[0103] Figure 1 A time sequence flow chart of an image-based three-dimensional (3D) surgical planning method according to the present disclosure. Figure 1The process shown includes steps S6 to S14 repeatedly performed on a plurality of vertebral bodies as the target. When the number of vertebral bodies is one, a surgical plan for one vertebra is established in one process.

[0104] In one or more embodiments described below, the description is based on the three-dimensional images generated by the image acquisition device. However, it is obvious that the embodiments of the present disclosure are not limited to the three-dimensional images of the three-dimensional image acquisition device. The present disclosure includes any form of device that can produce images obtained by one or more image acquisition devices capable of photographing image information in three different directions, especially three-way images combined or registered with each other.

[0105] According to the step description Figure 1 As follows.

[0106] The method according to the present disclosure generally includes a preparatory work process, a lateral-lateral (LL) point generation process, an anterior-posterior (AP) point generation process, an axial (Axial, hereinafter often referred to as AX) point generation, and an optimization and automation process.

[0107] <Step S1>

[0108] Step S1 is a step of photographing a plurality of images of the affected part of the patient in various directions. In this step, 3D images obtained by means of a 3D image acquisition device are used according to this embodiment.

[0109] Figure 2 Schematically shows the definition of the 3D coordinates of the virtual 3D surgical space in the first image and the second image according to one or more embodiments of the present disclosure.

[0110] <Step S2>

[0111] In step S2, a 3D image having axes in the X, Y, and Z directions is loaded into the image processing unit for processing. In this step, the 3D image and coordinate information are loaded.

[0112] <Step S3>

[0113] In step S3, a first image and a second image are obtained as digitally reconstructed radiographs (DDRs) by processing the loaded 3D image in two directions. In this step, for example, a lateral-lateral image (LL image) is obtained as the first image, and, for example, an anterior-posterior image (AP image) is obtained as the second image. These images can be obtained by performing orthogonal projection on the 3D image in a first direction and a second direction. In the LL image, which is the first image, the left-right direction (abdomen-back) is along the Y-axis, and the up-down direction (head-feet) is along the Z-axis. Also, in the AP image, which is the second image, the up-down direction (head-feet) is along the Z-axis, and the left-right direction (left arm-right arm) is along the X-axis. Thus, the first image, for example, the LL image, is juxtaposed in the Y-Z plane, and the second image, for example, the AP image, is juxtaposed in the X-Z plane.

[0114] <Step S4>

[0115] In step S4, the vertebral bodies and pedicles, which are the objects of interest in the first and second images obtained in the above process, are extracted or segmented or separated.

[0116] The segmentation of the object can be performed by applying an LL segmentation model and an AP segmentation model trained by deep learning.

[0117] Figure 3 An output image obtained by segmenting the vertebral body and pedicle from the LL image using the first image segmentation model is shown.

[0118] Figure 4 An output image obtained by segmenting the vertebral body and pedicle from the AP image using the second image segmentation model is shown.

[0119] As Figure 5 shown, the segmentation models described above can be based on the DeepLavb3+ model, and it is obvious that other models can be applied according to other embodiments of the present disclosure.

[0120] The DeepLavb3+ model is one of the deep learning models used in the field of computer vision and is used to perform extraction or segmentation work. This model is based on the structure of a deep convolutional neural network (CNN) and particularly has an encoder and a decoder structure, and is thus suitable for image segmentation.

[0121] Training of the model includes processes such as in ordinary deep learning training. To train such a model, various spine-related image materials collected for training are required. The training images include a first training image photographed in a first direction and a second training image photographed in a second direction. Among them, as Figure 4 and Figure 5 shown, labels are attached to the first image and the second image at the pixel level, and then a learning process is performed, so that the required first image segmentation model and second image segmentation model can be obtained. Figure 6 and Figure 7 respectively show images of the states in which labels are attached to the vertebra body and the pedicle of each of a plurality of first learning images and second learning images.

[0122] Figure 8 Shows an image of the state in which labels are attached to the vertebra body, the pedicle, and the foramen in a plurality of third learning images of one or more embodiments of the present disclosure.

[0123] <Step S5>

[0124] This step is to perform labeling on the target segmented in the previous step. Fig. 9 Shows the result of attaching labels to the target segmented in the AP image and the LL image.

[0125] As Fig. 9 shown, labels L1, L2, L3, L3, L4, and L5 are attached to the segmented target from the previous time, and the same label is given to the same vertebra body.

[0126] <Step S6>

[0127] As Fig.10 shown, in this step, one of a plurality of targets is selected before the process of extracting the LL point. For example, the L1 target can be selected first and go through the following process.

[0128] <Steps S7 to S8>

[0129] The LL parameters of the selected label are extracted to determine the first baseline in the LL image plane. The baseline is the first axial slice path (axial slice path (1)), which is equivalent to an axis in the front of the image in the third direction obtained later and is used to extract the third image. When planning the first baseline, it should be set so as not to deviate from the pedicle and vertebral body regions. Here, if only the inclination of either the vertebral body region or the pedicle region is reflected, the path may deviate from the other region, and in order to prevent the path planning failure caused by the detection error occurring when the object is segmented, the first baseline is set based on the average inclination of the inclinations of the two regions.

[0130] To this end, the region of interest (ROI) for the pedicle and vertebral body is first set.

[0131] exist Fig.11 In FIG. 1 , A is the vertebral body region, and B is the pedicle region. Such a ROI can be obtained by the first image segmentation model described above.

[0132] like Fig.12 As shown, the vertebral body region A is a quadrilateral region consisting of four vertices A1, A2, A3 and A4, and the pedicle region B is a quadrilateral region consisting of four vertices B1, B2, B3 and B4, and the two regions are adjacent. The middle line C passes between the adjacent sides of the two adjacent regions.

[0133] The middle line C passes through the center between the two adjacent sides B2-B3 and A1-A4 of the two areas A and B, and the intersection points C1 and C2 are located on the middle line C. The intersection point C1 is located in the middle of the line connecting the vertices B2 and A1 of the two areas, and C2 is located in the middle of the line connecting the vertices B3 and A4. Therefore, the y coordinate (C1.y) of C1 is (A1.y+B2.y) / 2, and the z coordinate (C1.z) of C1 is (A1.z+B2.z) / 2. In addition, the y coordinate (C2.y) of C2 is (A4.y+B3.y) / 2, and the z coordinate (C2.z) of C2 is (A4.z+B3.z)2.

[0134] Thus, the middle line C has the plane coordinates of the LL image, that is, the average tilt Ca of the tilts of the two regions in the yz plane. This middle line C serves as a reference for setting the first baseline in the LL image plane. In the yz plane coordinate system, the coordinates C1 (y, z), C2 (y, z) of the intersection points C1 and C2 and the tilt Ca of the middle line C are expressed as follows.

[0135]

[0136] Fig.13A temporary point C3 midway through the process of setting the first baseline from the midline C between the pedicle region B and the vertebral body region A in the yz plane is shown.

[0137] The coordinates of the temporary point C3 are consistent with the coordinates obtained by rotating the intersection point C1 by 90° with the intersection point C2 as the center in the middle line C. That is, if the coordinates of C1 in the yz plane coordinate system are (y1, z1), the coordinates of C3 are (z1, -y1). As a result, the temporary straight line D'0 connecting the intersection point C2 and the temporary point C3 is a normal line orthogonal to the middle line C, and the straight line equation of the normal line is D0: Z = Da (Y-C2.y) + C2.z.

[0138] FIG. 14a shows an initial first baseline D1 arranged to be parallel to the temporary straight line D'. The initial first baseline D1 crosses the middle line C or is orthogonal to the middle line C at a preset angle and is parallel to the temporary straight line D'. The temporary straight line D' is moved in parallel by a preset distance to the central area of ​​the pedicle region B having four vertices, namely, the center B5 or the vicinity of the center, thereby determining the initial first baseline D1 on the yz plane. The y coordinate B5.y and the z coordinate B5.z of B5 are expressed as follows.

[0139] B5.y-(B1.y+B2.y+B3.y+B4.y) / 4

[0140] B5.z=(B1.z+B2.z+B3.z+B4.z) / 4

[0141] The straight line equation for the first base line D1 having the inclination Da is expressed as follows.

[0142] D1:Z=Da(Y-C2.y)+C2.zDa=(C3.z-C2.z) / (C3.y-C2.y)

[0143] When the first base line D1 represented by the above straight line equation (D1:z) passes through the outer sides BS, B1 to B4 of the pedicles, an intersection point E is generated, which serves as a starting point or entry point for determining the length of a surgical screw described later.

[0144] The outer side BS of the pedicle is expressed as follows.

[0145] BS:Z=Ba(Y-B1.y)+B1.zBa=(B4.z-B1.z) / (B4.y-B1.y)

[0146] Next, the coordinates of the start point and the target point located on the initial first baseline D1 are calculated, wherein the target point is the terminating point reached by the end of the surgical screw.

[0147] For the coordinates (Ey, Ez) of the starting point E located on the outer side of the pedicle, the y coordinate (Ez) and z coordinate (Ez) of the starting point can be calculated by using a linear equation of the first baseline (D1:Z) and a linear equation of the outer side BS of the pedicle (BS:Z).

[0148] D1:Z=Da(Y-C2.y)+C2.z

[0149] BS:Z=Ba(Y-B1.y)+B1.z

[0150] First, at (Ey, Ez), the Y and Z of the two lines are the same, so they can be calculated as follows.

[0151] D1a(Y-C2.y)+C2.z=Ba(Y-B1.y)+B1.z

[0152] In the above formula, the terms related to Y can be folded to one side and Y can be separated from each to obtain the Ey value.

[0153] Ey=Y=(B1.z-C2.z+Ba*B1.y-Da*C2.y) / (Da-Ba)

[0154] The coordinate Ez of the start point in the z direction is obtained by substituting Ey obtained from the above formula into the straight line equation of the base line D1 (D1:Z).

[0155] Ez=Z=Da(Ey-C2.y)+C2.z

[0156] The above calculation process is one of many calculation methods for calculating the coordinates of the starting point E, and the coordinates of the starting point E can also be calculated by other methods. It is obvious that the technical scope of the present disclosure is not limited to a specific calculation method.

[0157] Fig. 14b shows the setting of a modified first baseline D1a obtained based on the initial first baseline D1 according to another embodiment of the present disclosure. The modified first baseline D1a replaces the initial first baseline D1 obtained in Fig. 14a.

[0158] The modified first baseline D1a is obtained by parallel moving the initial first baseline D1 along the central direction of the vertebral body, and the parallel moving distance G1 may be equivalent to the diameter or radius of the screw to be used in the operation.

[0159] Furthermore, according to another embodiment, it may also be larger or smaller than the diameter of the screw.

[0160] In this way, the initial baseline is moved parallel to the inside of the vertebral body to prevent the screw from being exposed outside the vertebral body due to its size. According to the patient's condition, if the pedicle has no height offset relative to the vertebral body, the initial first baseline can be used as the baseline of the screw path. However, if the pedicle has a height offset relative to the vertebral body due to some reason or congenital reasons, the modified first baseline can be used as the baseline of the screw path.

[0161] That is, in the embodiment according to the present disclosure, any one of the baselines D1 and D1a is selectively used as the baseline for setting the final first screw path, and the first baseline referred to below is D1 or D1a. Figures 15a and 15b respectively show the entry points E1, E2 and target points T1, T2 of the surgical screws set on the initial first baseline D1 and the modified first baseline.

[0162] First, referring to FIG. 15 a , a target point T, ie, a terminal point where the distal end portion of the surgical screw is located, is set on the initial first baseline D1 on the yz plane.

[0163] The coordinates of the target point T depend on the total length Ls of the screw to be used. In one embodiment of the present disclosure, the total length Ls of the screw is set to not exceed the length of the side A1-A2 of the vertebral body region in the direction of the surgical screw, and the diameter Ds is set to not be greater than the length of the outer side B1-B4 where the screw entry point is located.

[0164]

[0165] Here, the coordinates T (y, z) of the target point are expressed using the following equation.

[0166] Ty=E.y+Ls*Cos(arctan(Da))

[0167] Tz=E.z+Ls*Sin(arctan(Da))

[0168] The length of the line segment connecting the start point coordinates and the target point coordinates obtained through the above process is equivalent to the length of the screw to be used for surgery.

[0169] Referring to FIG. 15b, an entry point E2 and a target point T2 of the surgical screw are set on the modified first screw path D1a.

[0170] The coordinates of the starting point E2 and the target point T2 set on the modified first baseline D1a are set by mathematical calculations in the above manner. As shown in FIG. 14b, the modified first baseline D1a is offset from the center B5 of the pedicle within the pedicle and passes near it.

[0171] <Steps S9 to S10>

[0172] In this step, a second baseline paired with the first baseline in the y-z plane is obtained, that is, the axial slice path (2).

[0173] Fig.16 An image showing the state of extracting or separating the second vertebral body region a and the second pedicle regions b, b' using the first image segmentation model.

[0174] As Fig.16 shown, the ROI a of the vertebral body and the left and right pedicles b, b' can be extracted by the segmentation model of the second image, that is, the AP image.

[0175] The key when setting the second baseline in the AP image is that when obtaining the third image, that is, the AX image described later, an image in which the left / right pedicles are represented as the largest slice profile is required. For this purpose, the degree of vertebral body rotation needs to be reflected so that a pedicle region with the largest size can be ensured in the AX image.

[0176] Therefore, the average tilt value is calculated to reduce the tilt of the ROI a of the vertebral body and the segmentation error of the left and right two pedicle regions b, b'.

[0177] The average tilt increases or decreases according to the configuration form of the left and right pedicles. The temporary second baseline d' used to represent this average tilt is calculated in the following manner.

[0178] Fig.17 It is a x-z plane chart, which uses a dotted line to represent the second pedicle regions b, b' within the elliptical second vertebral body region a having vertices a1, a2, a3, a4.

[0179] As Fig.17 shown, first, the midpoint d1 between the center b5 of the left pedicle b and the corner a1 of the pedicle region closest to the center b5 and the midpoint d2 between the center c5 of the right pedicle b' and the corner a2 of the vertebral body region closest to the center c5 are calculated, and the two midpoints d1, d2 are connected to calculate the temporary second baseline d' with an average tilt.

[0180] Fig.18 The first baseline and the second baseline are set in the first image and the second image.

[0181] First, referring to Fig.18 , when obtaining a third image from the AX plane of the 3D image by applying the Y-Z position of the first baseline D1 or D1a in the first image obtained through the above process and the inclination of the temporary baseline d' on the X, Z plane in the second image, a final second baseline d is obtained, as Fig.18 The second image on the right passes through the centers of the two pedicles on average. Therefore, through such a process, an axial third image, that is, an axial image (AX image), can be obtained from the slice plane based on the first baseline and the second baseline as shown in Fig.19 (A).

[0182] Here, Fig.17 The illustrated temporary second baseline d' is shown as resting on the upper sides of the two pedicles b, b', and is illustrated in Fig.18 as passing through the two central regions of the two pedicles b, b', that is, near the center. Fig.18 The illustrated second baseline d shows the final second baseline when applying the inclination of the second baseline on the X-Z plane to the Y-Z position of the first baseline. That is, the final second baseline is a line obtained by parallelly moving the temporary second baseline d' to the average center of the two pedicles b, b'.

[0183] <Step S11>

[0184] Fig.19 (A) of

[0185] shows an example of the above-mentioned third image, and (B) shows the vertebral body region, the pedicle region, and the neural canal region as ROIs extracted from the third image. Fig.19 This step is to extract the Axial parameter from the AX image in (A) of

[0186] The extraction of the region is performed by an object separation unit, and the segmentation of the object can be performed by applying an AX segmentation model, which is trained by deep learning on samples with labels attached to the vertebra body, pedicle, and foramen in the AX image as shown in Fig.18 . Similar to the LL segmentation model and the AP segmentation model trained by deep learning as above.

[0187] Fig.19(B) shows a vertebral body region a, left and right pedicle regions b, b', and a neural tube (Foramen) region f between the left and right pedicles (Pedicles) extracted by performing the segmentation on the AX image.

[0188] 20a and 20b illustrate another embodiment of a process of baseline setting of a screw path for an ROI of a third image.

[0189] First, the embodiment of FIG. 20a will be described.

[0190] (A) to (D) of FIG. 20a illustrate a process of setting a screw path baseline on which a screw path is to be placed in an AX image, which is a plane for inserting or deploying surgical screws.

[0191] (A) After setting a center point "fc" in the central region of the neural tube region and a center point "ad" in each central region of the spinal canal region, a third baseline L is set, which passes through the two points fc and ad.

[0192] (B) Set the intersection points a'c and ac1 between the third baseline and the medial and lateral boundary lines of the vertebral body region. The intersection point ac is the distal coordinate set for the screw path baseline. The distance from a'c to ac1 represents the width of the vertebral body along the third baseline direction, and this distance can be used to determine the total length of the screw.

[0193] (C) The first screw path baseline LP is set, which passes through the intersection ac1 and the center b'c of the pedicle on one side (the left side of the figure), that is, the proximal setting coordinate.

[0194] (D) The second screw path baseline RP is set, which passes through the intersection ac1 and the center bc of the pedicle on the other side (right side of the drawing), that is, the proximal setting coordinate.

[0195] The first screw path base line LP and the second screw path base line RP set as described above are axes on which the surgical screws are installed.

[0196] The first screw path base line LP and the second screw path base line RP are set in slightly different directions as follows.

[0197] The embodiment of FIG. 20b is described.

[0198] As shown in FIG. 20a, (A) to (D) in FIG. 20b show the process of setting a screw path baseline where a screw path is to be placed in the AX image of the surgical screw insertion or deployment plane.

[0199] (A) After setting a center point "fc" in the central region of the neural tube region and a center point "ad" in each central region of the spinal canal region, a third baseline L is set, which passes through the two points fc and ad.

[0200] (B) Set the intersection points a'c and ac1 between the third baseline and the medial and lateral economic lines of the vertebral body area. The intersection point ac1 is the distal setting coordinate through which the screw path baseline passes. Here, a modified intersection point ac1', that is, the distal setting coordinate, is set on the screw path baseline L, which is isolated from the intersection point ac1 at the distal end of the vertebral body. The modified intersection point ac1' is separated from the intersection point ac1 from above by an arbitrary distance G2. The distance G2 is the average thickness or height G2 of the pedicle, and according to another embodiment, it can have other smaller or larger positive values.

[0201] The distance from a'c to ac1 represents the width of the vertebral body along the third baseline direction, and this distance can be used to determine the total length of the screw.

[0202] (C) The first screw path baseline LP is set, which passes through the modified intersection ac2 and the pedicle center b'c on one side (left side in the figure).

[0203] (D) The second screw path baseline RP is set, which passes through the intersection ac2 and the pedicle center bc on the other side (right side of the figure).

[0204] The first screw path baseline LP and the second screw path baseline RP set as above are modified axes on which surgical screws are provided. The angle between the first screw path baseline LP and the second screw path baseline RP is narrower than the angle between the first screw path baseline LP and the second screw path baseline RP obtained in the process of Figure 20b. Therefore, the tips of the two surgical screws arranged on the two baselines LP and RP will not meet each other in the vertebral body area.

[0205] (A) of Figures 22a and 22b respectively show the first screw path baseline and the second screw path baseline set by the above process according to the embodiments of Figures 20a and 20b, wherein (B) respectively show the state where the baselines are drawn (overlaid) on the AX image.

[0206] As shown in FIG. 22a (B) according to the embodiment of FIG. 20a, the first screw path baseline, the second screw path baseline and the outer boundary lines of the two pedicles intersect to form the starting point or entry point LE, RE of the surgical screw. The entry point is set at the proximal set coordinates through which the screw path baseline passes.

[0207] And, as shown in (B) of FIG. 22b according to the embodiment of FIG. 20b,

[0208] The first screw path baseline and the second screw path baseline pass through the centers b'c, bc of the respective pedicles and the modified intersection point ac2, and the entry points of the screws are set on the outer surface of the pedicles.

[0209] <Steps S12 to S13>

[0210] S12 to S13 are steps for determining the final surgical plan by finally setting the screw target points RT, LT away from the entry points LE, RE of the surgical screws, which are the starting points.

[0211] The entry points are located at the proximal set coordinates where the screws enter the pedicles.

[0212] In this step, the total screw length Ls obtained during the process of setting the above-mentioned first baseline or the vertebral body width in the direction of the first baseline can be applied to determine the screw length, or the vertebral body width in the direction of the third baseline (the distance from a'c to ac) can also be considered simultaneously to determine the screw length. And, the main surgeon can adjust according to the above results as needed. The screw diameter needs to be set to be smaller than the average diameter of the pedicle or the diameter of the smaller of the left and right pedicles, and this setting can also be changed by the main surgeon.

[0213] In FIGS. 22a and 22b based on the embodiments of FIGS. 20a and 20b, (A) shows the screw path applying the total screw length Ls in the ROI region, and (B) shows the state of determining the screw path in the third image. The screw path has 3D coordinates, which are recognized by the surgical system during the operation, and the operation is performed as it is.

[0214] FIGS. 22a and 22b show the state where the entry points and the target points (or end points) of the total length screw paths are set on the screw path baselines of FIGS. 22a and 22b according to one or more embodiments.

[0215] Referring to FIG. 22a according to the embodiment of FIG. 20a, entry points LE, RE and target points LT, RT are formed on the first screw path baseline LP and the second screw path baseline RP. The entry points LE, RE and the target points LT, RT are respectively located on the pedicle surface, and the distance between the entry point LE and the target point LT on each baseline corresponds to the total screw length.

[0216] Referring to FIG. 22b according to the embodiment of FIG. 20b, there are modified entry points LE', RE'. These modified entry points LE', RE' are separated from the entry points LE, RE on the surface of the pedicle at an arbitrary distance on the corresponding baseline. This separation is formed by considering that the screw will not directly contact the rear end (head) of the screw grasped by the surgical drill when it is fixed to the pedicle during actual surgery, and due to the gap in the patient's skin and / or muscle tissue. Therefore, the target point in the vertebral body is also moved from the position of the original target point LT, LP to the modified target point LT' and RT'. The position of this entry point and target point can be flexibly modified and changed on the screw path baseline determined in the process according to the state of the patient's surgical site and the judgment of the surgeon. The above process is all performed on one target. However, in the case of multiple object targets, that is, when there are multiple surgical target vertebrae and the last target has not been reached in step S14, return to step S6 and repeatedly execute the process from step S6 to step S14, and then end.

[0217] The embodiments of the present disclosure as described above provide an image-based automatic three-dimensional surgical intensification method and system. According to the present invention, safe and correct surgery can be performed without relying too much on the proficiency or experience of the surgeon. The embodiments of the present disclosure can be implemented by a computer program medium, and the computer program medium is configured to store software on a computer that executes the method of automatically establishing a surgical plan as described above. In addition, the present invention is used to execute the above method and can be implemented by an image processing unit, an object separation unit, a baseline setting unit, a path setting unit, etc. using a processor, a memory, a display, etc.

[0218] Furthermore, the present invention can also be implemented by an image-based surgical robot system.

[0219] Reference Fig.23 According to one or more embodiments of the present disclosure, the image acquisition device 100 is applied as the main image acquisition device in the surgical robot system 1. In addition, the present robot system 1 is configured to include a surgical robot 200, a position sensor 300 and a navigation system 400, and the surgical robot 200 is configured to include a body 201, a robot arm 203 including an end effector 203a, and a robot controller 205. In the embodiment according to the present disclosure, in addition to the above-mentioned image acquisition device, an image processing unit, an object separation unit, a baseline generation unit, and a path determination unit can be functionally included in the above-mentioned devices. In addition, the image processing unit, the object separation unit, the baseline setting unit, the path setting unit, etc. can be embodied by the surgical planning software, and in particular, the object separation unit uses the LL segmentation model, the AP segmentation model and the AX segmentation model trained by deep learning to extract or separate the vertebral body and the pedicle from the first image, the second image and the third image.

[0220] According to the disclosure, the robot arm 203 is controlled based on the determined surgical plan and control software during the operation. The navigation system 400 can assist the doctor's surgical behavior by displaying surgical plan information about surgical tools or implants or images obtained during the operation on a display, and by displaying the real-time positions of surgical tools or implants in images, or in some cases, in 3D images obtained before the operation. To this end, a display that enables the doctor to compare and view the surgical plan and the real-time positions of surgical tools, etc. as the actual operation with the naked eye during the operation can be connected to the navigation system 400.

[0221] Although various embodiments of the present invention are described in detail above, as long as one is of ordinary skill in the art of the present invention, various modifications should be made to the present invention without departing from the idea and scope of the present invention defined by the appended claims. Therefore, modifications to the embodiments of the present invention implemented later will not depart from the technical scope of the present invention.

Claims

1. An image-based three-dimensional (3D) surgical planning method, comprising: The step of obtaining one or more surgical site images including image information in multiple directions of the patient's spinal surgical site by one or more image obtaining devices, wherein the multiple directions include a first direction, a second direction, and a third direction that are different from each other; A step of extracting, by an image processing unit, a first image in a first direction and a second image in a second direction from the surgical site image; The object separation unit extracts a first vertebral body region and a first pedicle region from the first image, and extracts a second vertebral body region and left and right second pedicle regions within the second vertebral body region from the second image; The first baseline and the second baseline are set by a baseline setting unit, wherein the first baseline is set by a first coordinate in the central region of the first pedicle region and a second coordinate in the first vertebral body region, and the second baseline is set by a third coordinate and a fourth coordinate in the central region of the left and right pedicle regions in the second image; A step of extracting, by an image processing unit, a third image in the third direction from the surgical site image based on the first baseline and the second baseline; A step of extracting a third vertebral body region and left and right third pedicle regions from the third image by an object separation unit; and A step of setting a screw path through a central region of at least one of the left and right third pedicles and the third vertebral body region by a path setting unit, setting a proximal setting coordinate in the second pedicle, and setting a distal setting coordinate through which a baseline of the screw path passes on a third baseline passing through a neural canal region between the left and right third pedicle regions and the central region of the vertebral body.

2. The image-based 3D surgical planning method according to claim 1, wherein: The object extraction unit applies a model generated by machine learning to extract a vertebral body region and a pedicle region.

3. The image-based 3D surgical planning method according to claim 1, wherein: The first baseline is set to cross the middle line of the area between the first pedicle area and the first vertebral body area at an orthogonal or arbitrary angle, and passes through the central area of ​​the first pedicle area or is parallel to the central area of ​​the first pedicle area at an arbitrarily set distance.

4. The image-based 3D surgical planning method according to claim 1, wherein: The central region includes the center and / or vicinity of the center of the corresponding vertebral body region and / or pedicle region.

5. The image-based 3D surgical planning method according to any one of claims 1 to 4, wherein: The third image is obtained from the surgical site image based on the coordinates of the first baseline and the tilt of the second baseline.

6. The image-based 3D surgical planning method according to any one of claims 1 to 4, wherein: The third image is obtained from the surgical site image based on coordinates of the first baseline of the first image and the second baseline of the second image, where the second baseline passes through a central region of each of left and right pedicles in the second image.

7. The image-based 3D surgical planning method according to claim 6, wherein: The second baseline is set to be parallel to an arbitrary straight line, which connects the center of one pedicle area in the left and right pedicle areas, the midpoint between the corner coordinates on one side in the third image close to the center, the center of the other pedicle area in the left and right pedicle areas, and the midpoint between the corner coordinates on the other side in the third image close to the center.

8. The image-based three-dimensional surgical planning method according to any one of claims 1 to 4, wherein: The object separation unit extracts the neural tube region between the left and right third pedicle regions from the third image, and, The baseline setting unit A third baseline is set in the third image, the third baseline passing through coordinates of the center or the vicinity of the center of each of the third vertebral body region and the neural tube region, and, The distal end setting coordinate is set at a point where the third baseline intersects with a boundary line of a vertebral body region farthest from the neural tube region or a position near the point or at an intersection where the third baseline intersects with the screw path baseline outside the vertebral body.

9. The image-based 3D surgical planning method according to claim 8, wherein: The baseline setting unit sets a screw path baseline for placing the screw path to pass through the distal end setting coordinates.

10. The image-based 3D surgical planning method according to any one of claims 1 to 4, wherein: A screw start point for the pedicle and a target point in the vertebral body are set on the first baseline, and the distance between the screw start point and the target point is set as the length of the screw used for surgery.

11. An image-based three-dimensional (3D) surgical planning system comprising: one or more image acquisition devices configured to acquire one or more surgical site images having multiple directional image components of the spinal surgical site; an image processing unit configured to be responsible for image processing related to surgery, the image processing comprising extracting a first image in a first direction, a second image in a second direction, and a third image in a third direction from the one or more surgical site images, wherein the second direction is different from the first direction, and the third direction is different from the first direction and the second direction; an object separation unit configured to extract a first vertebral body region and a first pedicle region corresponding to the vertebral body and the pedicle from the first image, extract a second vertebral body region and a second pedicle region within the second vertebral body region from the second image, and extract a third vertebral body region and a third pedicle region from the third image; a baseline setting unit configured to set a first baseline and a second baseline, wherein the first baseline is set by a first coordinate in a central region of the first pedicle region and a second coordinate in the first vertebral body region, and the second baseline is set by a third coordinate and a fourth coordinate in central regions of left and right pedicle regions in the second image; and A screw path setting unit is configured to determine a three-dimensional screw path by setting a screw path through at least one central area of ​​the left and right third pedicles and the third vertebral body area, setting a proximal setting coordinate on the second pedicle, and setting a distal setting coordinate through which a baseline of the screw path passes on a third baseline passing through a neural canal area between the left and right third pedicle areas and through a central area of ​​the vertebral body.

12. The image-based 3D surgical planning system according to claim 11, wherein: The baseline setting unit arranges the first baseline to cross a middle line passing through a region between the first pedicle region and the first vertebral body region and pass through a central region of the first pedicle region or to be parallel to the central region at an arbitrarily set distance.

13. The image-based 3D surgical planning system according to claim 11, wherein: The image processing unit obtains the third image from the surgical site image based on the coordinates of the first baseline and the tilt of the second baseline.

14. The image-based 3D surgical system according to claim 11, wherein: The image processing unit obtains the third image from the surgical site image based on coordinates of the first baseline of the first image and the second baseline of the second image, where the second baseline passes through the central areas of the left and right pedicles in the second image.

15. The image-based 3D surgical planning system of claim 14, wherein: The second baseline is set to be parallel to an arbitrary straight line, which connects the center of one pedicle area of ​​the left and right pedicle areas, the midpoint between the coordinates of the corner on one side of the third image close to the center, the center of the other pedicle area, and the midpoint between the coordinates of the corner on the other side of the third image close to the center.

16. The image-based 3D surgical planning system of claim 11, wherein: The object separation unit extracts the neural tube region between the left and right third pedicle regions from the third image, and The path setting unit A third baseline is set in the third image, the third baseline passing through the coordinates of the center or the vicinity of the center of each of the third vertebral body region and the neural tube region, and, The distal end setting coordinate is set at a point where the third baseline intersects with a boundary line of a vertebral body region farthest from the neural tube region or a position near the point or at an intersection where the third baseline intersects with the screw path baseline outside the vertebral body.

17. The image-based 3D surgical planning system of claim 16, wherein: The path setting unit sets a screw path baseline for placing the screw path through the distal setting coordinates.

18. The image-based 3D surgical planning system according to any one of claims 11 to 17, wherein: A screw start point for the pedicle and a target point in the vertebral body are set on the first baseline, and the distance between the screw start point and the target point is set as the length of the screw used for surgery.

Images