3D navigation system and method for surgical and procedural instrument position tracking

Abstract

A 3D navigation system for surgical and procedural instrument position tracking, according to one embodiment of the present invention, comprises: a camera, which photographs an instrument for a surgery and a procedure of a patient so as to obtain a camera image; a skin surface extraction unit for generating, from an MRI image of the patient, a 3D point cloud having a first surface point corresponding to the skin surface of the patient; a landmark matching unit for calculating the position correspondence relationship between the MRI image and the camera image on the basis of landmark matching, and then matching same so as to generate a first matching image; a surface point acquisition unit for extracting a second skin surface point on the basis of a physical marker of the patient in the camera image; a surface matching unit for calculating, on the basis of surface matching, the position correspondence relationship between the 3D point cloud having the surface point and the camera image from which the second skin surface point has been extracted, and then matching same so as to generate a second matching image; and a coordinate conversion unit for finely adjusting the first matching image by matching, through a conversion matrix calculation method, the first matching image and the second matching image, which are different from each other.

Description

3D navigation system and method for tracking the position of surgical and surgical tools

[0001] The present invention relates to a 3D navigation system and method for tracking the position of surgical and surgical tools, and more particularly, to a 3D navigation system and method for tracking the position of surgical and surgical tools in the real world by obtaining a registered image in which the real space coordinates of the surgical and surgical tools are converted into image space coordinates using landmark registration and surface registration.

[0002] Ultrasound is a sound wave with a frequency higher than the audible frequency (16 Hz to 20 kHz), and is widely used in existing clinical settings, such as in ultrasonic medical imaging devices, high-intensity focused ultrasound surgical devices, and ultrasonic tumor treatment devices.

[0003] Recent neuroscience research on major depressive disorder has revealed the involvement of specific brain regions (deep regions), including the dorsolateral prefrontal cortex (DLPFC), cingulate cortex (CC), and nucleus accumbens (NAcc). Furthermore, selective modulation of these neurological circuits has been shown to be effective in improving major depressive disorder.

[0004] Meanwhile, conventional pharmacological approaches aim to modulate the uptake of neurotransmitters. However, they have the disadvantage of failing to target specific brain regions, and some patients may experience symptoms that do not respond to medication. Non-invasive brain stimulation techniques are being applied as non-pharmacological interventions, such as transcranial magnetic stimulation (TMS), which noninvasively stimulates the prefrontal cortex (PFC).

[0005] This non-invasive ultrasound examination method effectively stimulates the affected area non-invasively only when the ultrasound is irradiated toward the desired affected area (e.g., deep brain area) by the user, who is the operator (or medical professional). However, there was a problem in that it was difficult to accurately track the direction of the ultrasound irradiation because the user could not confirm whether the ultrasound was accurately irradiated to the affected area.

[0006] Accordingly, the present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a 3D navigation system and method for tracking the location of surgical and surgical tools in the real world by obtaining a registered image in which the real space coordinates of surgical and surgical tools are converted into image space coordinates using landmark registration and surface registration.

[0007] However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0008] In order to achieve the above object, a 3D navigation system for tracking the position of surgical and treatment tools according to an embodiment of the present invention may include: a camera for capturing a camera image by photographing surgical and treatment tools for a patient's surgery and treatment; a skin surface extraction unit for generating a 3D point cloud composed of first skin surface points corresponding to the patient's skin surface from an MRI image of the patient; a landmark matching unit for generating a first registration image by calculating a positional correspondence between the MRI image and the camera image based on landmark matching and then matching the calculated positional correspondence; a surface point acquisition unit for acquiring a second skin surface point existing on the patient's body surface from the camera image; a surface matching unit for generating a second registration image by calculating a positional correspondence between a 3D point cloud composed of the first skin surface points and a camera image from which the second skin surface points are extracted based on surface matching; and a coordinate transformation unit for transforming real coordinates into MRI / CT image coordinates using a transform matrix generated through surface matching of the first and second registration images.

[0009] In addition, the skin surface extraction unit performs a process of applying a convex hull algorithm to the MRI image, and then removing a portion that is not the patient's body surface from the result of the convex hull, a process of generating a 3D point cloud in which a plurality of points from which the non-body surface portions are removed from a 3D point cloud generated from the MRI image are partially collected, and a process of finding a cell and a cell adjacent to the cell based on an edge connecting the plurality of points, thereby expanding a point area on the 3D point cloud, through a surface extraction filter. The 3D point cloud generated from the MRI image can be passed through the surface extraction filter to extract the first skin surface point, thereby generating a 3D point cloud composed of the first skin surface point.

[0010] Additionally, the first aligned image can be finely aligned with a 3D point cloud composed of the first skin surface points by being finely adjusted through surface alignment.

[0011] Additionally, the camera may include a 3D camera for photographing surgical and surgical tools used outside the patient's body; and a thermal imaging camera for photographing surgical and surgical tools inserted outside the patient's body.

[0012] According to one embodiment of the present invention, since the location of surgical and surgical tools can be tracked in the real world, misuse of surgical and surgical tools can be prevented, thereby providing patients with safe surgery and procedures.

[0013] According to one embodiment of the present invention, since the location of surgical and surgical tools can be tracked in the real world, even when surgical and surgical tools are inserted into a patient's body, a user can check the location of the surgical and surgical tools inserted into the patient's body, thereby providing a safe surgery and procedure to the patient.

[0014] However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0015] FIG. 1 is a conceptual diagram of a 3D navigation system for tracking the position of surgical and surgical tools according to one embodiment of the present invention.

[0016] FIG. 2 is a block diagram illustrating components of a tracking device according to one embodiment of the present invention.

[0017] FIG. 3 is a flowchart illustrating a process of a 3D navigation method using landmark alignment and surface alignment according to one embodiment of the present invention.

[0018] Figure 4 is a flowchart illustrating the detailed process of the skin surface extraction step illustrated in Figure 3.

[0019] FIG. 5 is a drawing for explaining the convex hull algorithm shown in FIG. 4 and the step of removing points that do not exist on the skin surface from the result of the convex hull.

[0020] FIG. 6 is a drawing showing an example of a 3D point cloud generated through a process of removing non-surface points from the convex hull and the result of the convex hull shown in FIG. 4.

[0021] FIG. 7 is a drawing for explaining the adjacent cell search step illustrated in FIG. 4.

[0022] FIG. 8 is a diagram illustrating an example of a 3D point cloud according to the number of repetitions of the adjacent cell search step illustrated in FIG. 4.

[0023] FIG. 9 is a drawing illustrating an example of a 3D point cloud composed of first skin surface points generated through the 3D point cloud generation step illustrated in FIG. 4.

[0024] Figure 10 is a drawing for explaining the landmark alignment step.

[0025] Figure 11 is a drawing for explaining the steps of obtaining real body surface points.

[0026] Figure 12 is a drawing for explaining the surface alignment step of a real body surface point and a surface point extracted from MRI.

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the description of the present invention is merely an embodiment for structural and functional explanation, and therefore the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be modified in various ways and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purposes or effects presented in the present invention do not mean that a specific embodiment must include all of them or only such effects, and therefore the scope of the present invention should not be construed as being limited thereby.

[0028] The meanings of terms described in the present invention should be understood as follows.

[0029] Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, a first component could be referred to as a second component, and similarly, a second component could also be referred to as a first component. When a component is referred to as being "connected" to another component, it should be understood that it may be directly connected to that other component, but there may also be other components in between. Conversely, when a component is referred to as being "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationship between components, such as "between" and "immediately between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0030] Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "has" should be understood to specify the presence of stated features, numbers, steps, operations, components, parts, or combinations thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with their meaning within the context of the relevant technology, and should not be interpreted as having ideal or overly formal meanings unless explicitly defined herein.

[0032]

[0033] 3D navigation system

[0034] A 3D navigation system (100) according to one embodiment of the present invention is a system for tracking where a surgical and treatment tool (110) is currently located on a body of a patient (A) in the real world, and a device for implementing this is composed of a surgical and treatment device for performing surgery and treatment on a patient (A) and a tracking device for tracking the current location of the surgical and treatment tool (110).

[0035] FIG. 1 is a conceptual diagram of a 3D navigation system for tracking the position of surgical and surgical tools according to one embodiment of the present invention.

[0036] Referring to FIG. 1, the surgical and treatment device of the 3D navigation system (100) may include a surgical and treatment tool (110), a camera (120), and a terminal (130).

[0037] In one embodiment, the surgical and treatment tool (110) is a tool used by a medical professional user (B) to perform a surgery and treatment on a patient (A), and may vary depending on the type of surgery and treatment provided to the patient (A).

[0038] As a specific example, the surgical and treatment tool (110) may be an ultrasonic transducer capable of irradiating ultrasound to the brain of a patient (A) for a procedure of irradiating ultrasound to the brain of the patient (A).

[0039] In one embodiment, the camera (120) is a device that photographs surgical and surgical tools (110) in the real world where surgery and procedures are performed on a patient (A).

[0040] In addition, the camera (120) may be a 3D camera capable of taking three-dimensional (3D) pictures of surgical and treatment tools (110) outside the body of a patient (A), and a thermal imaging (infrared light, IR) camera capable of taking pictures of surgical and treatment tools (110) inserted into the body of a patient (A) with infrared light.

[0041] In one embodiment, the camera (120) may be implemented as a device in which a 3D camera and a thermal imaging camera are combined (or fused) to be able to capture images of surgical and surgical tools (110) regardless of the type of surgery and procedure.

[0042] And it is desirable for the camera (120) to capture not only the surgical and treatment tool (110), but also a body part (e.g., brain, spine, etc.) of the patient (A) surrounding the surgical and treatment tool (110), and accordingly, the camera image (121) captured by the camera (120) may include the surgical and treatment tool (110) and a body part of the patient (A).

[0043] In one embodiment, the terminal (130) is connected to a camera (120) and can receive an image including a surgical and treatment tool (110) and a body part of a patient (A) from the camera (120) and then output it on a display.

[0044] In addition, when the terminal (130) receives an MRI (Magnetic Resonance Imaging) image (10) of a patient (A) from a magnetic resonance imaging device not shown in the drawing, it can output the MRI image (10) of the patient (A) on the display.

[0045] And when the terminal (130) receives a CT (Computed Tomography) image of a patient (A) from a computed tomography device not shown in the drawing, it can output the CT image of the patient (A) on the display.

[0046] These terminals (130) can display images received from a camera (120), MRI images (10), CT images, etc., in a split display screen.

[0047] In one embodiment, when the terminal (130) receives a matching image (130a) generated from a tracking device to be described later, the terminal (130) can split and display the matching image (130a) on the screen.

[0048] Although the data displayed on the display may vary depending on the environment settings of the user (B), it is preferable that the terminal (130) be set to display the aligned image (130a) on the screen in a split manner so that the user (B) can check the position of the surgical and surgical tool (110) before, after, or during the surgical and surgical procedure.

[0049] In addition, when a plurality of surgical and treatment tools (110) are placed to perform surgery and treatment on various body parts of a patient (A), the terminal (130) can display a plurality of aligned images (130a) for each body part where the surgical and treatment tools (110) are placed on a screen in a divided manner as illustrated in FIG. 1.

[0050] Hereinafter, the surgery and procedure of the patient (A) is a surgery and procedure in which multiple ultrasound transducers are attached to the head of the patient (A) to irradiate ultrasound to the brain of the patient (A), and accordingly, the camera image (121) is an image taken of the head of the patient (A) to which multiple ultrasound transducers are attached, and the matching image (130a) is generated based on the MRI image (10) and the camera image (121), and the tracking device of the 3D navigation system (100) is described in detail based on tracking ultrasound irradiated from multiple ultrasound transducers through the matching image (130a).

[0051] FIG. 2 is a block diagram illustrating components of a tracking device according to one embodiment of the present invention.

[0052] Referring to FIG. 2, the tracking device of the 3D navigation system (100) may include a data receiving unit (140), a skin surface extraction unit (150), a landmark matching unit (160), a surface point acquisition unit (170), a surface matching unit (180), and a coordinate conversion unit (190).

[0053] In one embodiment, the data receiving unit (140) can receive an MRI Dicom file including an MRI image (10) of the brain and head of a patient (A) and data related to the patient (A) from a database that stores and manages MRI images (10) and data related to the patient (A).

[0054] In addition, when the data receiving unit (140) receives an MRI Dicom file, it can transmit the MRI Dicom file to the skin surface extraction unit (150) and the landmark matching unit (160), respectively.

[0055] In one embodiment, the skin surface extraction unit (150) extracts an MRI image (10) of the brain and head of a patient (A) from an MRI Dicom file received from a data receiving unit (140), and applies a convex hull algorithm to the MRI image (10) to remove a portion that is not the body surface (151) of the patient (A).

[0056] Here, the body surface (151) of the patient (A) refers to the nose, lips, etc. exposed on the skin surface based on the head of the patient (A).

[0057] Also, when there are multiple points, the convex hull means creating an area that contains all points and has only convex parts without concave parts, and can be used to extract multiple points that form the sides and points of the polygon.

[0058] And the part that is not the body surface (151) of the patient (A) refers to the part that is separated from the head of the patient (A) among the sides (or lines) forming the convex polygon generated by applying the convex hull algorithm.

[0059] In one embodiment, the skin surface extraction unit (150) can generate a 3D point cloud (152) corresponding to the body surface (151) of the patient (A) by applying a convex hull algorithm and then removing a portion that is not the body surface (151) from the convex hull.

[0060] Here, a point cloud means a set cloud of multiple points spread over a three-dimensional space, and in the present invention, a 3D point cloud (152) can be generated from a skin surface extraction unit (150) in a state in which a point area (152a) corresponding to a body surface (151) is displayed in a specific color (e.g., yellow).

[0061] In one embodiment, the skin surface extraction unit (150) can find a cell from a plurality of points (152b) forming a point area (152a) displayed in a 3D point cloud (152).

[0062] As a specific example, the skin surface extraction unit (150) can determine an edge connecting a plurality of points (152b) and determine that a cell is a cell in which the connection shape of the edge forms a polygonal shape.

[0063] In one embodiment, the point area (152a) means an area in which a plurality of points (152b) corresponding to the body surface (151) are subset on the 3D point cloud (152).

[0064] In one embodiment, the skin surface extraction unit (150) can find a cell ('yellow cell' in (b) of FIG. 7) that shares two or more points (152b) among cells, and can find a cell adjacent to the cell ('red cell' in (b) of FIG. 7) based on the same method.

[0065] In one embodiment, the skin surface extraction unit (150) can expand the point area (152a) on the 3D point cloud (152) through a process of finding adjacent cells.

[0066] In addition, the extent (size) of the point area (152a) on the 3D point cloud (152) may increase as the number of times the skin surface extraction unit (150) repeats the process of finding adjacent cells increases, and based on this characteristic, the skin surface extraction unit (150) may repeat the process of finding adjacent cells until the proportion of the point area (152a) in the 3D point cloud (152) becomes a certain level or greater.

[0067] In one embodiment, the skin surface extraction unit (150) extracts the first skin surface point (153) through the surface extraction filter illustrated in FIG. 4, thereby generating a 3D point cloud (152) composed of the first skin surface point (153), as illustrated in FIG. 9.

[0068] Here, the first skin surface point (153) refers to a portion of the patient's (A) head that corresponds to the skin surface.

[0069] Additionally, the skin surface extraction unit (150) can transmit a 3D point cloud (152) composed of first skin surface points (153) to the surface matching unit (180).

[0070] In one embodiment, the 3D point cloud (152) is composed of first skin surface points (153), which may contribute to improving the surface matching speed and result accuracy of the surface matching unit (180).

[0071] In one embodiment, the landmark matching unit (160) can calculate the positional correspondence between an MRI image (10) received from a data receiving unit (140) and a camera image (121) received from a camera (120) based on landmark registration, which is one of rigid registration, and then match them to generate a first matching image.

[0072] Here, landmark matching refers to a method of matching by designating a limited number of markers to prominent feature areas of two images and then calculating the distance difference between the markers. In the present invention, the markers may be markers artificially attached to the body or any part of the body.

[0073] In one embodiment, the first aligned image can be aligned in the form of an MRI image (10) in which multiple ultrasound transducers are displayed, as illustrated in FIG. 10, by converting the real space coordinates of multiple ultrasound transducers, which are surgical and treatment tools (110), into image space coordinates through landmark alignment.

[0074] Additionally, the landmark alignment unit (160) may generate the first alignment image as a first transformation process (Transform 1) for converting the real space coordinates of the ultrasonic transducer into image space coordinates.

[0075] In one embodiment, the first transformation process is not a process of completely transforming the real space coordinates of the ultrasound transducer into image space coordinates through the first registration image in the form of an MRI image (10), but may be a preprocessing process of transforming the real space coordinates of the ultrasound transducer into image space coordinates.

[0076] And the landmark alignment unit (160) can transmit the generated first alignment image to the coordinate transformation unit (190).

[0077] In one embodiment, the surface point acquisition unit (170) receives a camera image (121) from the camera (120) and can extract a second skin surface point (171) from the head of the patient (A) as illustrated in FIG. 11.

[0078] As a specific example, the surface point acquisition unit (170) can first extract the face area of ​​the patient (A) from the camera image (121), and then extract the second skin surface point (171) based on the body marker among the face area of ​​the patient (A).

[0079] Here, the body markers may include physical features of the patient's (A) face, such as eyes, nose, and cheekbones.

[0080] In one embodiment, the surface point acquisition unit (170) can transmit the camera image (121) from which the second skin surface point (171) is extracted to the surface matching unit (180).

[0081] In one embodiment, the surface matching unit (180) receives a 3D point cloud (152) composed of first skin surface points (153) from the skin surface extraction unit (150), and receives a camera image (121) in which second skin surface points (171) are extracted from the surface point acquisition unit (170).

[0082] This surface matching unit (180) can generate a second matching image by calculating the positional correspondence between the 3D point cloud (152) and the camera image (121) from which the second skin surface point (171) is extracted based on surface registration, which is one of the rigid registrations.

[0083] Here, surface matching refers to a method of matching based on dividing the surface area in each of the two images and performing a similarity evaluation using the boundary thereof.

[0084] In one embodiment, the second aligned image can be aligned in the form of a 3D point cloud (152) in which the first skin surface points (153) are clustered on the skin surface portion corresponding to the body markers by reflecting the head shape and facial physical features of the patient (A) through surface alignment.

[0085] In addition, the surface alignment unit (180) may generate a second alignment image as a second transformation process (Transform 2) for transforming the real space coordinates of multiple ultrasound transducers, which are surgical and treatment tools (110), into image space coordinates.

[0086] In one embodiment, the second transformation process may be a process of generating a second matching image so that the first matching image is finely adjusted in the coordinate transformation unit (190).

[0087] And the surface matching unit (180) can transmit the generated second matching image to the coordinate transformation unit (190).

[0088] In one embodiment, the coordinate transformation unit (190) can receive the result Transform of the second alignment from the surface alignment unit (180) using the result Transform of the first alignment from the landmark alignment unit (160) as an initial value.

[0089] In addition, the coordinate transformation unit (190) can perform a navigation function that ultimately transforms real world coordinates into image coordinates by using the second registration result transform generated by performing surface registration on the surface 3D points extracted from the first registration image and the MRI image.

[0090] As a specific example, the coordinate transformation unit (190) can fine-tune the first matching image by performing surface matching in a manner that reflects the head shape and facial physical features of the patient (A) of the second matching image to the first matching image.

[0091] In one embodiment, the first alignment image can be finely aligned to a 3D point cloud (152) composed of first skin surface points (153) through fine adjustment by a coordinate transformation unit (190).

[0092] In one embodiment, the aligned image (130a) generated from the coordinate transformation unit (190) is image data that converts the real space coordinates of the ultrasound transducers placed on the head of the patient (A) into image space coordinates, and when transmitted to the terminal (130) with the positions of a plurality of ultrasound transducers displayed, it can be split and displayed on the screen of the display of the terminal (130).

[0093] The 3D navigation system (100) of the present invention may include a simulation unit that simulates the ultrasonic range of a plurality of ultrasonic transducers so that the user (B) can track the direction of the ultrasonic irradiation, if the surgery or treatment is a method of non-invasively irradiating the affected area with ultrasonic waves, although not shown in the drawing.

[0094] In one embodiment, the simulation unit can simulate the ultrasonic range of multiple ultrasonic transducers and display it in a matching image (130a).

[0095] In one embodiment, the user (B) performs surgery and treatment while visually checking a matching image (130a) showing the positions and ultrasound ranges of multiple ultrasound transducers through the terminal (130), and can track the direction of ultrasound irradiation through the matching image (130a).

[0096]

[0097] 3D navigation method

[0098] Hereinafter, as described above, the surgery and procedure of the patient (A) is a surgery and procedure that uses an ultrasound transducer to irradiate ultrasound to the brain of the patient (A), and the camera image (121) according to this is an image taken of the head of the patient (A) to which a plurality of ultrasound transducers are attached, and a matching image (130a) is generated based on the MRI image (10) and the camera image (121), and based on tracking ultrasound irradiated from the plurality of ultrasound transducers through the matching image (130a), the process of the 3D navigation method (S100) for tracking the location of a surgical and surgical tool according to an embodiment of the present invention performed by the 3D navigation system (100) will be described in detail.

[0099] FIG. 3 is a flowchart illustrating a process of a 3D navigation method using landmark alignment and surface alignment according to one embodiment of the present invention.

[0100] First, the magnetic resonance imaging device performs MRI imaging of the brain and head of the patient (A) in the MRI imaging space to create a matching image (130a), thereby creating an MRI image (10) (S110).

[0101] After MRI imaging, the user (B) can attach (or use) multiple ultrasound transducers to the head of the patient (A) to use multiple ultrasound transducers for surgery and treatment when the patient (A) arrives at the surgery and treatment space (S120).

[0102] After that, the camera (120) is installed in the surgical and treatment space, and a plurality of ultrasound transducers can be photographed through movement control by the user (B) to obtain a camera image (121) (S130).

[0103] At this time, the camera image (121) may include the head of a patient (A) to which multiple ultrasound transducers are attached.

[0104] After that, the data receiving unit (140) can receive an MRI Dicom file containing an MRI image (10) of the brain and head of the patient (A) and data related to the patient (A) from a database that stores and manages MRI images (10) and data related to the patient (A) (S140).

[0105] After that, when the skin surface extraction unit (150) receives an MRI Dicom file from the data receiving unit (140), it extracts an MRI image (10) of the brain and head of the patient (A) from the MRI Dicom file, and then extracts the skin surface of the patient (A) (S150).

[0106] The detailed process of extracting the skin surface of this patient (A) is as shown in Figures 4 to 9.

[0107] FIG. 4 is a flowchart illustrating a detailed process of the skin surface extraction step illustrated in FIG. 3, FIG. 5 is a diagram for explaining a step of removing points that do not exist on the skin surface from the convex hull algorithm and the result of the convex hull illustrated in FIG. 4, FIG. 6 is a diagram illustrating an example of a 3D point cloud generated through a process of removing non-surface points from the convex hull and the result of the convex hull illustrated in FIG. 4, FIG. 7 is a diagram for explaining an adjacent cell search step illustrated in FIG. 4, FIG. 8 is a diagram illustrating an example of a 3D point cloud according to the number of repetitions of the adjacent cell search step illustrated in FIG. 4, and FIG. 9 is a diagram illustrating an example of a 3D point cloud composed of first skin surface points generated through the 3D point cloud generation step illustrated in FIG. 4.

[0108] Referring to FIG. 4, the skin extraction step (S150) may include an MRI image extraction step (S151), a 3D point cloud generation step (S152), a convex hull algorithm application step (S153), a point removal step that is not on the body surface (S154), an adjacent cell search repetition step (S155), and a 3D point cloud generation step (S156) composed of first skin surface points.

[0109] Referring to FIG. 5, the skin surface extraction unit (150) applies an MRI image extracted from an MRI image (10) of the brain and head of a patient (A) to a convex hull algorithm (S153), removes points that are not body surfaces from the convex hull result (S154), and then repeats the process of finding neighbor cells (S155), so that only the part corresponding to the body surface (151) can be found in the MRI image.

[0110] Referring to FIG. 6, the skin surface extraction unit (150) can generate a 3D point cloud (152) composed of first skin surface points (153) by removing non-body surface parts from the convex hull algorithm and the result of the convex hull in the 3D point cloud generation step (S156).

[0111] Referring to FIG. 7, the skin surface extraction unit (150) can find a cell from a plurality of points (152b) forming a point area (152a) displayed in a 3D point cloud (152) in the adjacent cell search repetition step (S155).

[0112] Referring to FIG. 8, the skin surface extraction unit (150) can repeat the process of finding adjacent cells until the proportion of the point area (152a) occupying the 3D point cloud (152) reaches a certain level or more.

[0113] Thereafter, the skin surface extraction unit (150) can generate a first skin surface point (153) based on finding a 3D point cloud (152) that has repeated the adjacent cell finding repetition step (S155).

[0114] Referring to FIG. 9, the skin surface extraction unit (150) can generate a 3D point cloud (152) composed of first skin surface points (153) in a 3D point cloud generation step (S156) composed of first skin surface points.

[0115] Referring again to FIG. 3, when the landmark matching unit (160) receives an MRI Dicom file from the data receiving unit (140), it calculates the positional correspondence between the MRI image (10) and the camera image (121) based on landmark registration, which is one of the rigid registrations, and then matches them to generate a first matching image as shown in FIG. 10 (S160).

[0116] Figure 10 is a drawing for explaining the landmark alignment step.

[0117] Referring to Fig. 10, the first registration image is a result of registration using multiple body landmarks in reality and body landmarks in the MRI image.

[0118] Referring again to FIG. 3, when the surface point acquisition unit (170) receives a camera image (121) from the camera (120), it can extract a second skin surface point (171) from the head of the patient (A) as shown in FIG. 11 (S170).

[0119] Figure 11 is a drawing for explaining a step of obtaining a real body surface point, and Figure 12 is a drawing for explaining a step of matching a real body surface point with a surface point extracted from MRI.

[0120] Referring to FIGS. 11 and 12, the surface point acquisition unit (170) can first extract the face area of ​​the patient (A) from the camera image (121), and then extract the second skin surface point (171) based on the body marker among the face areas of the patient (A).

[0121] Referring again to FIG. 3, when the skin surface extraction step (S150) and the surface point acquisition step (S170) are completed, the surface matching unit (180) can receive a 3D point cloud (152) composed of first skin surface points (153) from the skin surface extraction unit (150) and can receive a camera image (121) in which second skin surface points (171) are extracted from the surface point acquisition unit (170).

[0122] After that, the surface matching unit (180) can calculate the positional correspondence between the 3D point cloud (152) and the camera image (121) from which the second skin surface point (171) is extracted based on surface registration, which is one of the rigid registrations, and then match them to generate a second matching image (S180).

[0123] At this time, the second aligned image can be aligned in the form of a 3D point cloud (152) in which the first skin surface points (153) are clustered on the skin surface portion corresponding to the body marker by reflecting the head shape and facial physical features of the patient (A) through surface alignment.

[0124] When the surface matching step (S180) is completed, the coordinate transformation unit (190) can receive a first matching image from the landmark matching unit (160) and a second matching image from the surface matching unit (180).

[0125] After that, the coordinate transformation unit (190) can perform a navigation function by transforming the real coordinates into coordinates in the MRI / CT image using the Transform generated by creating the second aligned image.

[0126] At this time, the coordinate transformation unit (190) can transform the real coordinates into MRI / CT image coordinates using Transform, which is a method of fine-tuning the first aligned image by surface matching, which reflects the shape of the patient's (A) head and physical features of the face in the second aligned image.

[0127] In this way, the first aligned image finely adjusted from the coordinate transformation unit (190) may be the aligned image (130a) illustrated in FIG. 1, and this aligned image (130a) is image data that converts the real space coordinates of the ultrasonic transducers placed on the head of the patient (A) into image space coordinates, and when transmitted to the terminal (130) to provide the user with the positions of a plurality of ultrasonic transducers displayed, it may be displayed in a split manner on the screen of the display of the terminal (130).

[0128] At this time, if the surgery and treatment are methods of non-invasively irradiating the affected area with ultrasound, the alignment image (130a) can display the positions of multiple ultrasound transducers and simultaneously display the ultrasound range by the simulation unit.

[0129] In addition, the user (B) can perform surgery and treatment while visually checking the alignment image (130a) showing the positions of multiple ultrasound transducers and the ultrasound range through the alignment image (130a), and can track the direction of ultrasound irradiation through the alignment image (130a).

[0130]

[0131] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can utilize the individual components described in the above-described embodiments in combination with each other. Accordingly, the present invention is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0132] The present invention may be embodied in other specific forms without departing from the technical spirit and essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in all respects but should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes coming within the equivalent scope of the present invention are intended to be included therein. The present invention is not intended to be limited to the embodiments set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Furthermore, claims that are not explicitly cited in the claims may be combined to form an embodiment or incorporated into a new claim by post-application amendment.

[0133] Description of the symbol

[0134] 10: MRI images,

[0135] 100: 3D navigation system,

[0136] 110: Surgical and surgical instruments,

[0137] 120: Camera,

[0138] 121: Camera footage,

[0139] 130: Terminal,

[0140] 130a: Matched image,

[0141] 140: Data receiving unit,

[0142] 150: Skin surface extraction area,

[0143] 151: Body surface,

[0144] 152: 3D point cloud,

[0145] 152a: Point area,

[0146] 152b: Point,

[0147] 153: First skin surface point,

[0148] 160: Landmark alignment section,

[0149] 170: Surface Point Acquisition Section,

[0150] 171: Second skin surface point,

[0151] 180: Surface alignment,

[0152] 190: Coordinate transformation section.

Claims

1. A camera that obtains images by photographing surgical and surgical tools for the patient's surgery and procedure; A skin surface extraction unit that generates a 3D point cloud composed of first skin surface points corresponding to the skin surface of the patient from the MRI image of the patient; A landmark matching unit that calculates the positional correspondence between the MRI image and the camera image based on landmark matching and then matches them to generate a first matching image; A surface point acquisition unit for acquiring a second skin surface point existing on the body surface of the patient from the camera image; A surface matching unit that calculates a positional correspondence between a 3D point cloud composed of the first skin surface points and a camera image from which the second skin surface points are extracted based on surface matching and then matches them to generate a second matching image; and A 3D navigation system for tracking the position of surgical and surgical tools, characterized by including a coordinate transformation unit that transforms real coordinates into MRI / CT image coordinates using a transform matrix generated through surface matching of the first and second aligned images.

2. In paragraph 1, The above skin surface extraction part, After applying a convex hull algorithm to the MRI image, a process of removing a portion that is not the patient's body surface from the result of the convex hull, a process of generating a 3D point cloud in which a plurality of points from which the portion that is not the body surface is removed from a 3D point cloud generated from the MRI image are subsets, and a process of expanding a point area on the 3D point cloud by repeating the process of finding a cell and a cell adjacent to the cell based on an edge connecting the plurality of points. A 3D navigation system for tracking the position of surgical and surgical tools, characterized in that it generates a 3D point cloud composed of the first skin surface points by passing the 3D point cloud generated from the MRI image through the surface extraction filter to extract the first skin surface points.

3. In paragraph 1, The above first aligned image is, A 3D navigation system for tracking the position of surgical and surgical tools, characterized in that the system is finely aligned with a 3D point cloud composed of the first skin surface points through surface alignment.

4. In paragraph 1, The above camera, A 3D camera for photographing surgical and procedural tools used outside the patient's body; and A 3D navigation system for tracking the location of surgical and surgical tools, comprising: a thermal imaging camera for photographing surgical and surgical tools inserted into the body of the patient;

Images