Computer program product, device and system for verifying surgical target matching

Abstract

The present disclosure relates to a method for verifying surgical target matching, an apparatus for the method, and a system including the apparatus. The method for verifying surgical target matching according to the present disclosure includes the following steps: preparing a 3D model including shape information of a surgical target; obtaining information on the position and posture of the surgical target and information on the position and posture of a target marker attached to the surgical target through a tracker; performing matching with the 3D model by deriving the correlation of the position and posture between the surgical target and the target marker; tracking changes in the position and posture of a probe tip moving along the surface of the surgical target; and generating matching verification information and displaying the matching verification information as an image, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model corresponding to at least one of the position and posture of the probe tip. Accordingly, the matching situation is verified intuitively and visually. In addition, the matching degree of a certain continuous region rather than a single point can be checked intuitively.

Description

Technical Field

[0001] The present disclosure relates to a method for verifying a surgical target match, an apparatus for the method, and a system including the apparatus. More specifically, the present disclosure relates to a method for verifying a surgical target match to check a spatial match degree between a surgical target and a surgical image, an apparatus for the method, and a system including the apparatus. Background Art

[0002] With the development of medical technology in recent years, navigated surgery using robots and computer systems has been actively introduced, and such surgery has also been applied to the field of artificial joint surgery.

[0003] In the case of the knee joint, when external injuries, various infections, and diseases cause knee pain and physical damage, the entire or part of the knee joint is treated through orthopedic surgery based on replacement. Approximately 10% to 30% of the patients who will undergo partial replacement have worn the interior of the knee joint.

[0004] Among the robots used for such joint orthopedic surgery, there is a robot that can automatically complete the entire surgical process. Such a surgical robot automatically cuts bones along a pre-planned path without manual intervention.

[0005] For knee surgery using a surgical robot, it is necessary to first install optical markers on the surgical target, and then use an optical sensor to track the position and posture of the optical markers, so as to perform spatial matching processing on the surgical target and its shape model in advance. In addition, it is also necessary to check whether the matching result is appropriate.

[0006] Conventional methods for verifying surgical target matching generally adopt methods for checking the quantitative value of the matching result, or methods for checking the matching degree of multiple pre-registered points.

[0007] However, since the matching result is simply represented as a quantitative value, it is difficult for the user to determine whether the matching result is appropriate in the former method, and there is a problem in the latter method that only the matching degree of the pre-registered points is checked, so the matching degree of other regions cannot be checked.

[0008] (Related Art Patent Document 1) Korean Patent Application with Publication No. 102015-0102936 (September 9, 2015). Summary of the Invention

[0009] To solve the above problems, one aspect of the present disclosure is to provide a method for visually and intuitively verifying the matching degree. Further, one aspect of the present disclosure is to provide a method, apparatus, and system for verifying the matching of surgical targets, in which the matching degree of a certain continuous region rather than a single point can be visually inspected. In addition, one aspect of the present disclosure is to provide a method, apparatus, and system for verifying the matching of surgical targets, in which when the matching result is inappropriate, the region that needs to be rematched is rematched again.

[0010] Aspects of the present disclosure can be achieved by providing a method for verifying the matching of surgical targets, the method comprising: preparing a three-dimensional (3D) model including the shape information of the surgical target; obtaining information about the position and pose of the surgical target, and the position and pose of the target marker attached to the surgical target, through a tracker; performing matching with the 3D model by deriving the correlation between the position and pose of the surgical target and the target marker; tracking the change in the position and pose of the probe tip moving along the surface of the surgical target; and generating matching verification information and displaying the matching verification information as an image, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model corresponding to at least one change in the position and pose of the probe tip.

[0011] Meanwhile, aspects of the present disclosure can also be achieved by providing a method for verifying the matching of surgical targets, the method comprising: receiving the information about the position and pose of the surgical target obtained through a tracker, and the position and pose of the target marker attached to the surgical target; performing the matching of the surgical target with the 3D model by deriving the correlation between the position and pose of the surgical target and the target marker; receiving the information about the position and pose of the probe tip moving along the surface of the surgical target obtained through the tracker; and generating matching verification information and displaying the matching verification information as an image, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model corresponding to the change in the position and pose of the probe tip.

[0012] In addition, generating the matching verification information and displaying the image may include generating 3D graphic icons and displaying the 3D graphic icons on the 3D model, the 3D graphic icons representing a plurality of spheres with different radii centered at the position of the probe tip.

[0013] In addition, generating the matching verification information and displaying the image may include changing the shape of the 3D graphic icon to represent the relative position relationship between the position of the probe tip and the surface of the 3D model.

[0014] In addition, generating matching verification information and displaying an image may include displaying images with different perspectives according to changes in the position or posture of the probe tip.

[0015] In addition, the image displaying the matching verification information may include a first region, in the first region, a first image showing a perspective change of the 3D model according to changes in the position or posture of the probe tip, and the first region in the first image is displayed from at least one fixed perspective of the 3D model.

[0016] Multiple spheres may be represented as visually distinguishable from each other.

[0017] In addition, the method may further include: quantitatively calculating and displaying a matching result, where quantitatively calculating and displaying the matching result may include at least one of the following: calculating and displaying the distance deviation between multiple points on the surgical target obtained for matching and multiple points on the corresponding 3D model; calculating and displaying the distance between the point where the probe tip is located corresponding to the probe movement and a landmark point on the 3D model; and calculating and displaying the distance between the point where the probe tip is located corresponding to the probe movement and a corresponding point on the 3D model.

[0018] Meanwhile, aspects of the present disclosure can be implemented by providing a device for verifying surgical target matching, the device including: a signal receiver configured to receive information on the position and posture of the surgical target obtained by a tracker and information on the position and posture of a target marker attached to the surgical target; a target matcher configured to perform matching between the surgical target and a three-dimensional (3D) model by deriving the correlation of the position and posture between the surgical target and the target marker; a matching validator configured to generate matching verification information based on changes in the information on the position and posture of the probe tip moving along the surface of the surgical target received through a data input unit, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the image model.

[0019] In addition, the matching validator may generate 3D graphic icons representing multiple spheres with different radii centered on the position of the probe tip, and display the 3D graphic icons on the 3D model, and the multiple spheres may be represented as visually distinguishable from each other.

[0020] In addition, the matching validator may generate matching verification information such that a first image with at least one fixed perspective of the 3D model and a second image with a perspective change according to changes in the position and posture of the probe tip can be respectively displayed in different regions.

[0021] Meanwhile, aspects of the present disclosure can be realized by providing a system for verifying surgical target matching, the system including: a tracker configured to track the position and pose of a target marker attached to a surgical target and the position and pose of a probe; and a memory configured to store a three-dimensional (3D) model having shape information of the surgical target acquired before surgery; a target matcher configured to perform matching with the 3D model by deriving the correlation of the position and pose between the surgical target and the target marker attached to the surgical target; a matching validator configured to generate matching verification information based on the change in the information of the position and pose of the probe tip moving along the surface of the surgical target acquired by the tracker, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the image model; and a display unit configured to display an image based on the matching verification information.

[0022] Here, the matching validator can generate 3D graphic icons representing a plurality of spheres with different radii centered on the position of the probe tip and display the 3D graphic icons on the 3D model, and the plurality of spheres can be represented as visually distinguishable from each other.

[0023] As described above, the present disclosure provides a method for verifying surgical target matching, an apparatus for the method, and a system including the apparatus, in which the matching can be verified intuitively and visually. In addition, the matching degree of a certain continuous region rather than a single point can be checked intuitively. In addition, when the matching result is inappropriate, only the region that needs to be rematched is rematched, thereby shortening the time required for matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematically shows a system for verifying surgical target matching according to an embodiment of the present disclosure.

[0025] Figure 2 Is a control block diagram of an apparatus for verifying surgical target matching according to an embodiment of the present disclosure.

[0026] Figure 3 Is an operation schematic diagram of a controller according to an embodiment of the present disclosure.

[0027] Figure 4 Shows an example of matching verification information generated by a matching validator according to an embodiment of the present disclosure and displayed as an image.

[0028] Figure 5 Shows an example of a 3D graphic icon representing the position of a probe tip according to an embodiment of the present disclosure.

[0029] Figure 6Shows the matching verification information in a state where, while the position of the probe tip moves upward from the position of Figure 3 the pose of the probe tip turns slightly to the right.

[0030] Figure 7 is a schematic diagram for explaining a method for a matching validator to calculate a matching fitness according to an embodiment of the present disclosure.

[0031] Figure 8 Shows the case where the probe tip moves to a landmark point according to an embodiment of the present disclosure;

[0032] Figure 9 Shows according to an embodiment of the present disclosure Figure 8 the corresponding matching verification information for the case where the probe tip shown moves to a landmark point; and

[0033] Figure 10 is a flowchart showing a method for verifying a surgical target match by a matching verification device according to an embodiment of the present disclosure. Detailed Description of the Invention

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0035] Figure 1 Schematically shows a system for verifying a surgical target match according to an embodiment of the present disclosure. Refer to Figure 1 , a system for verifying an image match of a surgical target according to an embodiment of the present disclosure includes a target marker 20 attached to a surgical target 10, a tracker 30, and a matching verification device 40.

[0036] The surgical target 10 refers to the target of a surgery. In an embodiment of the present disclosure, the knee joint between the femur and the tibia is taken as an example to illustrate the surgical target 10, so the target markers 20 are respectively attached to the femur and the tibia. In addition, according to the present disclosure, the term "surgical target" in the present disclosure can be used in a broad sense, including surgical objects such as the femur and the tibia, or can be used in a narrow sense to represent a specific position or surgical site of the surgical target, such as an implant origin or a joint center. During the matching process of the surgical target 10, the user uses a probe 50 to register positions corresponding to multiple points of the surgical target 10.

[0037] Optical markers can be used for the target marker 20 and the probe 50, including three or four bars branching in different directions relative to a center point, and high-reflection ball markers are respectively formed at their ends.

[0038] The tracker 30 is configured to track the position and pose of the probe 50 and the position and pose of the target marker 20 attached to the surgical target 10. The tracker 30 detects the position and pose of the marker in three-dimensional (3D) space coordinates and sends the detected position and pose to a matching verification device 40 (described later). According to an embodiment of the present disclosure, an optical tracking system (OTS) can be used as an example of the tracker 30.

[0039] The matching verification device 40 for the surgical target 10 is configured to verify the spatial matching and matching results of the surgical target 10 and may include a computer (or a processor) and a display unit. The matching verification device 40 for the surgical target 10 can be provided as a stand-alone device. As needed, the computer (or processor) and the display unit can be provided separately in such a way that the computer (or processor) is set in the surgical robot, while the display unit is connected and integrated into the tracker 30. In this case, the computer (or processor) provided in the surgical robot is connected to the tracker 30, receives the position / pose information of the marker, processes the matching and matching results, and sends the matching results to the display unit.

[0040] Figure 2 is a control block diagram of the device 40 for verifying the matching of a surgical target according to an embodiment of the present disclosure. Refer to Figure 2 According to an embodiment of the present disclosure, the matching verification device 40 for a surgical target includes a signal receiver 41, a user input unit 42, a display unit 43, a memory 44, and a controller 45.

[0041] The signal receiver 41 is configured to receive signals from the outside and may include, for example, a high-definition multimedia interface (HDMI) connector and a D-sub connector for connecting to external devices, or a communication module for connecting to a wired / wireless network such as the Internet. The signal receiver 41 may include a wired / wireless communication module for communicating with the tracker 30.

[0042] The user input unit 42 is configured to receive commands from the user and send the commands to the controller 45 (described later) and may include various input units such as a keyboard, a mouse, or a button.

[0043] The display unit 43 is configured to display images on the screen and may be implemented as a liquid crystal display (LCD) panel, a light-emitting diode (LED) panel, an organic light-emitting diode (OLED) panel, etc.

[0044] Here, the user input unit 42 and the display unit 43 can be physically separated from other components. For example, the signal receiver 41, the controller 45, and the memory 44 can be integrated into the main body of the surgical robot, and the user input unit 42 and the display unit 43 can be configured as separate devices communicatively connected to the tracker 30 and the surgical robot.

[0045] The memory 44 can be configured to store various operating systems (OSs), middleware, platforms, and various applications of the matching verification device 40 for the surgical target, and can be configured to store program codes, video and audio signals, and signal-processed data. The memory 44 is configured to store a 3D model 10a having the shape information of the surgical target 10, which has been obtained before the surgery. Here, the 3D model 10a of the surgical target 10 can be generated, for example, from a set of image data, such as computed tomography (CT) or magnetic resonance imaging (MRI) images of a patient.

[0046] The controller 45 is configured to perform overall control of the matching verification device 40 of the surgical target based on the user input received through the user input unit 42 or an internal program. The controller 45 can include program codes for signal processing and control, and a processor for executing the programs. The controller 45 performs position tracking and matching based on the position information / pose information of the target marker 20 received from the tracker 30 through the signal receiver 41, and the position information / pose information of the probe 50, and generates matching verification information about the matching result, which represents the degree of matching.

[0047] Reference Figure 2 , the controller 45 includes a target matcher 45a and a matching validator 45b.

[0048] The target matcher 45a performs matching between the surgical target 10 and the 3D model 10a having the shape information of the surgical target 10 based on the position information / pose information of the target marker 20 obtained from the tracker 30, and the position information / pose information of the probe 50. The target matcher 45a derives the position / pose correction between the surgical target 10 and the target marker (or bone marker) 20 attached to the surgical target from the position information / pose information of the target marker 20, and the position information / pose information of the probe 50.

[0049] For example, after the target marker 20 is installed on the surgical target 10, the probe 50 simultaneously detects the positions / postures of multiple specific points of the preset surgical target 10 and the position / posture of the target marker 20. The target matcher 45a receives the position / posture of the target marker 20 obtained by the tracker 30 and the information of the specific points indicated by the probe 50, and derives the position / posture correlation between the target marker 20 and the surgical target 10 from the received information, thereby performing matching with the 3D model 10a of the surgical target 10 previously stored in the memory 44. The target matcher 45a may include program code and / or software for calculating positions and performing matching.

[0050] The matching validator 45b is configured to generate matching verification information regarding the matching result of the target matcher 45a, and based on the change in the position and posture information of the tip of the probe 50 moving along the surface of the surgical target 10, generate matching verification information including a 3D graphic icon representing the relative position relationship between the tip of the probe 50 and the image model. In addition, the matching validator 45b can quantitatively calculate the matching result. Meanwhile, the quantitative value of the matching degree and the matching verification information generated by the matching validator 45b are provided to the display unit 43 and displayed as an image. In this way, the user can visually and intuitively identify the matching degree. The matching validator 45b may include program code and / or software for calculating the matching result and generating the matching verification information and the graphic.

[0051] Figure 3 is a schematic diagram of the operation of the controller 45 according to an embodiment of the present disclosure. For example, referring to Figure 3 , the femur 11 is shown as the surgical target 10, and the target marker 21 is attached to the femur.

[0052] Referring to Figure 3 , the user uses the probe 50 to register the positions corresponding to the designated points of the preset surgical target 11. The tracker 30 tracks the position / posture of the target marker 21 indicating the designated points on the surface of the surgical target 11 and the position / posture of the probe 50, and sends the tracked position / posture to the matching verification device 40. The matching verification device 40 performs matching with the 3D model 10a corresponding to the surgical target 11 pre-stored based on the position / posture of the probe 50 and the position / posture information of the target marker 21 received from the tracker 30. Generally, four views such as the far view, rear view, medial view, and front view of the 3D model 10a corresponding to the surgical target 10 or 11 are sequentially switched, and the probe 50 indicates the points corresponding to the positions of the points preset in the switched views, so as to register the corresponding positions by receiving the position information / posture information of the probe 50 and the position information / posture information of the target marker 21. Here, about 20 to 40 points are registered. In Figure 3In [the text], "LP" is an abbreviation of landmark point, which refers to the reference point of the posture of the surgical target 10 or 11. Therefore, when the user sequentially designates corresponding points with the probe 50 and the matching verification device 40 registers the designated points, the matching is performed.

[0053] After Figure 3 the matching, in the spatial relationship, the position / orientation correlation between the probe 50 and the 3D model 10a corresponding to the surgical target 10 or 11 is represented as follows.

[0054]

Equation 1

[0055]

[0056] (where PTB is the position / orientation correlation between the probe 50 and the surgical target 10 or 11, OTSTP is the transformation matrix of the probe 50 relative to the tracker 30, OTSTBM is the transformation matrix of the target marker 21 relative to the tracker 30, and BMTB is the transformation matrix of the surgical targets 10 and 11 relative to the target marker 21)

[0057] When the registration of multiple preset points is completed, the matching validator 45b generates matching verification information of the matching result. The matching validator 45b generates a 3D graphic icon A, which represents the position of the tip of the probe 50 and displays it on the 3D model 10a so that the relative position relationship with the 3D model 10a can be represented based on the change in the position and orientation information of the tip of the probe 50.

[0058] Figure 4 Shows an example of the matching verification information generated by the matching validator 45b and displayed as an image according to an embodiment of the present disclosure. Refer to Figure 4, the image displaying the matching verification information according to the present disclosure has a first region. In the first region, a first image that changes in the viewing angle of the image according to the change in the position or posture of the tip of the probe 50 is displayed. In the image displaying the matching verification information, the first image displayed on the region R1 represents the 3D model 10a in the viewing angle observed by the tip of the probe 50 based on the position / pose information of the probe 50. The matching validator 45b uses the tip of the probe 50 as the center, uses the extension line connected to the tip and oriented in a specific direction as the pose axis, and gives the shape of the position of the tip of the probe 50 on a partial region of the 3D model 10a observed from the angle of the corresponding axis, thereby generating the matching verification information. Therefore, the viewing angle and region of the image for displaying the matching verification information change according to the change in the position or posture of the tip of the probe 50. That is, when the tip of the probe 50 slides on the surface of the surgical target 10 or 11, the position and posture of the tip of the probe 50 change, and the first image of the matching verification information displayed in the region R1 relatively changes the position and posture of the 3D model 10a according to the change in the position / pose of the tip of the probe 50. In this way, the user can visually check the relative relationship with the 3D model 10a in real time when swiping the surface of the surgical target 10 or 11 with the tip of the probe 50.

[0059] Back to Figure 4 , "A" represents the 3D graphic icon A of the position of the tip of the probe 50, so that the user can know the current position of the tip of the probe 50 on the 3D model 10a corresponding to the surgical target 10 or 11 through the 3D graphic icon A. Here, the 3D graphic icon A giving the shape of the position of the tip of the probe 50 does not merely represent the position of the tip of the probe 50, but itself represents the relative position relationship between the tip of the probe 50 and the 3D model 10a.

[0060] For this purpose, the 3D graphic icon A giving the shape of the position of the tip of the probe 50 can represent the relative position relationship with the 3D model 10a through a plurality of spheres, these spheres using the position of the tip of the probe 50 as the center and having different radii.

[0061] Figure 5 Shows an example of the 3D graphic icon A representing the position of the tip of the probe 50 according to an embodiment of the present disclosure. Refer to Figure 5 , the 3D graphic icon A representing the tip of the probe 50 may include two spheres A1 and A2 having the same center and different radii. In Figure 5Among them, A1 represents a circle with a radius of 1 mm, and A2 represents a circle with a radius of 2 mm. The center of the 3D graphic icon A, A1, or A2 corresponds to the tip of the probe 50, that is, the point where the tip of the probe 50 is located. When the tip of the probe 50 contacts the surface of the surgical target 10 or 11, a partial area of the sphere is located within the 3D model 10a, and the center of this partial area indicates the position of the tip of the probe 50. Therefore, the partial area of the sphere that enters the 3D model 10a is not displayed in the first image, so that the shape of the sphere changes according to the viewing angle. Therefore, the 3D graphic icon A representing the position of the tip of the probe 50 can be displayed so that the whole or part of the circle can be seen according to the positional and postural relationship with the surgical target 10 or 11. In this way, according to the present disclosure, the shape of the 3D graphic icon A changes according to the relative positional relationship between the tip of the probe 50 and the surface of the 3D model 10a. Therefore, the user can intuitively know the relative positional relationship. For example, by changing the position and posture of the probe 50 while watching the screen, the shape of the 3D model 10a and the distance between the point where the tip of the probe 50 is currently located on the surgical target 10 or 11, etc.

[0062] In addition, according to the present disclosure, the 3D graphic icon A is represented by overlapping spheres A1 and A2 with different radii, and these spheres are displayed in different colors or different contrasts so as to be visually distinguishable from each other. Therefore, the user can intuitively know the relative positional relationship based on the shape of the 3D graphic icon A. For example, the distance or space between the 3D model 10a and the tip of the probe 40. In addition, the user can intuitively recognize the relative positional relationship (for example, the distance and space between the 3D model 10a and the surface of the surgical target 10 or 11) while swiping the tip of the probe 50 across the surface of the surgical target 10 or 11, that is, the tip of the probe 40 is not at a specific point on the surgical target 10 or 11, but within certain continuous areas.

[0063] In Figure 5 (a) shows the case where, from the perspective of the tip of the probe 50, two 100% spheres A1 and A2 of the 3D graphic icon A are seen according to the relative positional relationship with the 3D model 10a. In this case, the 3D model 10a may be located behind the tip of the probe 50, or may be located at a lateral position 2 mm away from the tip of the probe 50. In any case, this means that the tip of the probe 50 and the 3D model 10a are spaced 2 mm or more apart from each other. Therefore, it can be intuitively understood that the difference in matching is 2 mm or more.

[0064] In Figure 5In (b), it shows the situation where, from the perspective of the tip of the probe 50, two 50% spheres A1 and A2 of the 3D graphic icon A are seen according to the relative positional relationship with the 3D model 10a. In this case, it means that 50% of the two spheres of the 3D graphic icon A are located inside the 3D model 10a. That is to say, it can be intuitively understood that the tip of the probe 50 is directly on the surface of the 3D model 10a. On the other hand, Figure 5 In (c) of it, it shows the situation where two spheres of about 20% are seen, which means that about 80% of the two spheres A1 and A2 of the 3D graphic icon A are located inside the 3D model 10a. In other words, it can be intuitively understood that the tip of the probe 50 is within a distance of 1 mm from the inner surface of the 3D model 10a. Figure 5 It shows that the 3D graphic icon A includes two spheres with different radii from each other, but the 3D graphic icon A can include three to five spheres.

[0065] Reference Figure 4 , the image showing the matching verification information according to the present invention includes a second region, in which a second image of the 3D model with at least one fixed perspective is shown, regardless of the change in the position or posture of the tip of the probe 50. In the second image showing the positional relationship between the 3D model 10a and the tip of the probe 50 at at least one fixed perspective, the 3D graphic icon A is shown on the 3D model 10a with a specific perspective. In this embodiment, as an example, the second image shows two side views of the 3D model 10a.

[0066] Referring to Figure 4 , the first image shown in the first region changes according to the position / pose of the probe 50 in terms of the display area and the perspective of the 3D model 10a, but the second image shown in the second region shows the 3D model 10a with a specific perspective regardless of the position / pose of the probe 50. Therefore, when the user compares the two images according to the shape of the 3D graphic icon A representing the position of the tip of the probe 50, the matching degree can be intuitively recognized.

[0067] Figure 4 It shows the matching verification information when the position / pose of the probe 50 is as Figure 3 shown, where the tip of the probe 50 is almost at a right angle to the surgical target 10 or 11. Therefore, although Figure 4The first image shows that, from the perspective of the tip of the probe 50, 100% of the 3D graphic icon A is seen as exposed on the 3D model 10a. However, the second image shows that, with the perspective of the 3D model 10a fixed to the side, the tip of the probe 50 is on the lower side, so only about 50% of the sphere corresponding to the 3D graphic icon A, that is, a hemisphere, is seen. Although it is impossible to accurately know the matching degree in the first image, the result shows that the matching between the surgical target 10 or 11 and the 3D model 10a at the corresponding position is almost accurate. However, the second image is based on a fixed perspective. Therefore, if the point where the tip of the probe 50 is currently located is the point with the lowest curvature in the curve of the surgical target 10 or 11, it may be difficult to see the sphere from the side. Therefore, it is difficult to know the accurate matching degree in the second image. By controlling the posture of the probe 50 in the first image, the perspective in which the 3D graphic icon A is seen as a hemisphere is relatively accurate.

[0068] Figure 6 Shows the matching verification information according to an embodiment of the present disclosure when the position of the probe tip moves upward from Figure 3 the position, with the posture of the tip of the probe 50 slightly turned to the right. Similar to Figure 4 the Figure 5 first image shows that 100% of the sphere, that is, the 3D graphic icon A is seen as exposed on the 3D model 10a from the perspective of the tip of the probe 50. This is because the tip of the probe 50 still has a posture towards the surgical target 10 or 11. However, in the second image, the sphere is not shown in its entirety but only partially, that is, almost 80% or more of the sphere is exposed because the perspective of the 3D model 10a is fixed to the side and the tip of the probe 50 is on the lower side. Therefore, it can be intuitively understood that there is a deviation of at least 1.5 mm between the surgical target 10 or 11 and the 3D model 10a even if the tip of the probe 50 is currently located at the point with the highest curvature in the curve of the surgical target 10 or 11.

[0069] As described above, the matching verification information according to the present disclosure shows the position of the tip of the probe 50 represented by a plurality of spheres with different radii. Therefore, while changing the posture of the tip of the probe 50, the user can intuitively know how far the point of the surgical target 10 or 11 indicated by the tip of the probe 50 is from the 3D model 10a according to the shape of the 3D graphic icon A represented on the 3D model 10a. In addition, the corresponding perspective and position of the 3D model 10a are shown according to the change of the position / posture of the probe 50, so that when the surface of the surgical target 10 or 11 is swiped with the tip of the probe 50, the matching degree of the area to be inspected can be easily checked. Therefore, it is possible to intuitively know the matching deviation degree, that is, which part of the surgical target 10 or 11 does not match properly.

[0070] The matching validator 45b can calculate the matching result as a quantitative value and display this value as included in the matching verification information. For example, the matching validator 45b can calculate the root mean square (d) error (RMSE) value, which is related to the distance deviation between multiple points on the surgical target 10 or 10 obtained for matching and multiple points on the corresponding 3D model 10a, as the fitness evaluation value of the matching result. Refer to Figure 4 , the RMSE value is displayed in the third area. Through this value, the user can numerically check the fitness of the matching.

[0071] Figure 7 is a schematic diagram for explaining the method of calculating the matching fitness by the matching validator 45b according to an embodiment of the present disclosure. In Figure 7 , the light gray points represent the points obtained for matching in the surgical target 10 or 11, and the black points represent the points on the 3D model 10a that are closest to the matching points.

[0072] The matching validator 45b calculates the deviation between the matching points obtained for matching and the points on the 3D model 10a that are closest to the matching points through the following formula.

[0073]

Equation 2

[0074]

[0075] (where is the point obtained from the surgical target 10 or 11 for matching, is the point on the 3D model 10a that is closest to the matching point, and N is the number of points)

[0076] Figure 4 shows that the RMSE value is 0.171.

[0077] Meanwhile, the matching validator 45b can calculate the distance between the landmark point LP and the current position of the tip of the probe 50 as the matching result and generate matching verification information based on the calculated distance. Refer to Figure 4 , the landmark point LP is displayed in gray in the second image, and the distance value 28.80 mm between the landmark point LP and the current position of the tip of the probe 50 is displayed in the fourth area. This value changes as the tip position of the probe 50 changes.

[0078] Figure 8 represents the case where the tip of the probe 50 moves to the landmark point LP, Figure 9The corresponding matching verification information is displayed. In the first image and the second image, the 3D graphic icon A overlaps with the landmark point LP, and the calculated distance of 0.51 mm between the landmark point LP and the tip of the probe 50 is displayed in the fourth region. In the first image, according to the posture of the probe 50, the 3D graphic icon A is presented as a semi-circular shape, from which it can be intuitively understood that the matching fitness at the corresponding point is high. Instead of the distance from the landmark point LP, the distance between the point where the tip of the probe 50 is located corresponding to the movement of the probe 50 and the point on the 3D model 10a closest to this point can be calculated and displayed. In this case, this value can easily check the matching fitness of the corresponding point.

[0079] As described above, according to the present disclosure, in addition to the quantitative value, the matching fitness is visually expressed so that the user can intuitively know the matching degree.

[0080] Figure 10 It is a flowchart showing a method for verifying the matching of the surgical target 10 or 11 by the matching verification device 40 according to an embodiment of the present disclosure. Repeated descriptions of the foregoing embodiments will be avoided if necessary.

[0081] Reference Figure 10 , first, a 3D model 10a of the surgical target 10 or 11 is generated based on an image obtained by photographing the surgical target 10 or 11, etc. (S10). The 3D model 10a is stored in the memory 44. The matching verification device 40 receives the information on the position and posture of the surgical target 10 or 11 obtained by the tracker 30, and the information on the position and posture of the target marker 21 attached to the surgical target 10 or 11 (S11). The position / posture information of the surgical target 10 or 11 can be obtained by various methods. According to an embodiment of the present disclosure, the probe 50 is used to obtain the position / posture information of the surgical target 10 or 11.

[0082] The target matcher 45a derives the correlation between the position and posture of the surgical target 10 or 11 and the target marker 20 or 21 from the information on the position and posture of the target marker 21 attached to the surgical target 10 or 11 received by the tracker 30 and the information on the position and posture of the probe 50, and thus performs the matching with the 3D model 10a (S12). In this case, a plurality of preset points are indicated by the probe 50, thereby performing the matching process.

[0083] When the matching is completed, the matching validator 45b generates matching verification information regarding the matching result, and the display unit 43 displays the matching verification information. At the same time, the tracker 30 tracks the position / pose of the tip of the probe 50 moving along the surface of the surgical target 10 or 11, and sends the tracked position / position to the matching verification device 40 (S13). The matching validator 45b generates matching verification information, and the matching verification information includes a 3D graphic icon A, and the 3D graphic icon A includes the relative position relationship between the tracked position of the tip of the probe 50 and the 3D model 10a (S14). In addition, the matching validator 45b calculates the matching result as a quantitative value and generates matching verification information including the quantitative value (S14).

[0084] The matching verification information generated by the matching validator 45b is displayed on the display unit 43 (S15). Thus, the user can intuitively recognize the relative position relationship between the 3D model 10a and the tip of the probe 50 based on the shape of the 3D graphic icon A represented by a plurality of overlapping spheres with different radii.

[0085] In addition, while changing the position and pose of the probe 50, a series of regions of the surgical target 10 or 11 can be traced with the probe 50 instead of a single point, so as to verify the matching fitness while checking the shape of the 3D graphic icon a.

[0086] When the matching verification process is completed (S16), it is identified whether rematching is required. When the quantitatively calculated value does not satisfy a certain reference value, the image matching device can forcibly re-execute the matching for the region that requires rematching, or when it is identified that the matching degree of the points checked by tracing with the probe 50 is low, the user can request rematching (S17). In this case, the matching can be performed again from the beginning, or only the view corresponding to the region that requires rematching can be matched again.

[0087] Although all the elements constituting the embodiments of the present disclosure have been described above as being combined into one unit or being coupled to operate as one unit, the present disclosure is not necessarily limited to such embodiments. That is, at least two elements can be selectively combined and operated without departing from the scope of the present disclosure. In addition, each element can be implemented as independent hardware, but some or all of the elements can be selectively combined with each other, so they can be implemented as a computer program having program modules that execute part or all of the functions, combined in one or more hardware. The code and code segments constituting the computer program can be easily conceived by those of ordinary skill in the technical field of the present disclosure. Such a computer program can implement the embodiments of the present invention, that is, stored in a computer-readable medium and read and executed by a computer. The medium for the computer program can include magnetic recording media and optical recording media.

[0088] In addition, unless otherwise explicitly described, terms such as "comprising", "including", or "having" mean the presence of the corresponding elements and should be construed as including one or more other elements without excluding their presence. Unless otherwise defined, all terms, including technical or scientific terms, have the same meaning as understood by a person of ordinary skill in the art to which this disclosure pertains. General terms defined in a dictionary should be construed as having a meaning consistent with the context of the relevant art and not be construed as having an idealistic or overly formalistic meaning, unless otherwise explicitly defined in this disclosure.

[0089] Although embodiments of the present disclosure have been described for illustrative purposes, those of ordinary skill in the art will understand that various modifications and changes can be made without departing from the basic features of the present disclosure. Therefore, the embodiments disclosed by the present invention are only used to illustrate the technical spirit of the present invention, rather than to limit the technical spirit of the present invention, and the technical spirit of the present invention is not limited by these embodiments. In addition, the scope of the present invention should be interpreted by the appended claims, and all technical ideas within the scope equivalent to the appended claims should also be construed as being included within the scope of the present invention.

Claims

1. An apparatus for verifying surgical target matching, the apparatus comprising: a signal receiver configured to receive information on the position and pose of the surgical target obtained by a tracker, and information on the position and pose of a target marker attached to the surgical target; a target matcher configured to perform matching with a three-dimensional (3D) model of the surgical target by deriving the correlation of the position and pose between the surgical target and the target marker; and a matching validator configured to generate matching verification information based on changes in the position and pose information of a probe tip moving along the surface of the surgical target received by the signal receiver, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model; wherein the matching validator generates the 3D graphic icon representing a plurality of spheres with different radii centered on the position of the probe tip and displays the 3D graphic icon on the 3D model.

2. The apparatus according to claim 1, wherein the plurality of spheres are represented as being visually distinguishable from each other.

3. The apparatus according to claim 1 or 2, wherein the matching validator generates the matching verification information such that a first image with at least one fixed viewing angle of the 3D model and a second image with a viewing angle changed according to changes in the position and pose of the probe tip can be respectively displayed in different regions.

4. A computer program product comprising instructions for causing the apparatus according to any one of claims 1-3 to perform the following steps in a method for verifying surgical target matching: preparing, using the apparatus, a three-dimensional (3D) model including the shape information of the surgical target; acquiring, using the apparatus, information on the position and pose of the surgical target and information on the position and pose of a target marker attached to the surgical target through a tracker; performing, using the apparatus, matching with the 3D model by deriving the correlation of the position and pose between the surgical target and the target marker; tracking, using the apparatus, changes in the position and pose of a probe tip moving along the surface of the surgical target; and generating, using the apparatus, matching verification information and displaying the matching verification information as an image, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model corresponding to at least one change in the position and pose of the probe tip; wherein generating the matching verification information and displaying the image includes: generating the 3D graphic icon and displaying the 3D graphic icon on the 3D model, the 3D graphic icon representing a plurality of spheres with different radii centered on the position of the probe tip.

5. A computer program product comprising instructions for causing the apparatus according to any one of claims 1-3 to perform the following steps in a method for verifying surgical target matching: The device is used to receive information on the position and pose of the surgical target obtained by the tracker, as well as information on the position and pose of the target marker attached to the surgical target; The device is used to perform the matching between the surgical target and the 3D model of the surgical target by deriving the correlation of the position and pose between the surgical target and the target marker; The device is used to receive information on the position and pose of the probe tip moving along the surface of the surgical target obtained by the tracker; and The device is used to generate matching verification information and display the matching verification information as an image, where the matching verification information includes a 3D graphic icon, and the 3D graphic icon represents the relative position relationship between the probe tip and the 3D model, corresponding to the change in the position and pose of the probe tip; Among them, the generating of the matching verification information and the displaying of the image include: generating the 3D graphic icon and displaying the 3D graphic icon on the 3D model, and the 3D graphic icon represents a plurality of spheres with different radii centered on the position of the probe tip.

6. The computer program product according to claim 4 or 5, wherein, the generating of the matching verification information and the displaying of the image include: using the device to change the shape of the 3D graphic icon to represent the relative position relationship between the position of the probe tip and the surface of the 3D model.

7. The computer program product according to claim 4 or 5, wherein, the generating of the matching verification information and the displaying of the image include: using the device to display the image with different viewing angles according to the change in the position or pose of the probe tip.

8. The computer program product according to claim 6, wherein, the image for displaying the matching verification information includes a first area, in the first area, a first image of the viewing angle change of the 3D model is displayed according to the change in the position or pose of the probe tip, and a second area for displaying a second image at at least one fixed viewing angle of the 3D model.

9. The computer program product according to claim 4 or 5, wherein, the plurality of spheres are represented as visually distinguishable from each other.

10. The computer program product according to claim 4 or 5 further includes using the device to quantitatively calculate and display the matching result, wherein, the using of the device to quantitatively calculate and display the matching result includes at least one of the following: using the device to calculate and display the distance deviation between a plurality of points on the surgical target obtained for the matching and a plurality of corresponding points on the 3D model; using the device to calculate and display the distance between the point where the probe tip is located corresponding to the probe movement and a landmark point on the 3D model; and using the device to calculate and display the distance between the point where the probe tip is located corresponding to the probe movement and a corresponding point on the 3D model.

11. A system, the system includes the device according to any one of claims 1 to 3, and the system further includes: The tracker is configured to track the position and pose of the target marker attached to the surgical target and the position and pose of the probe; A memory configured to store the three-dimensional (3D) model having shape information of the surgical target obtained before surgery; A target matcher configured to perform matching with the 3D model by deriving the correlation of the position and pose between the surgical target and the target marker attached to the surgical target; A matching validator configured to generate matching verification information based on the change in the information of the position and pose of the probe tip moving along the surface of the surgical target obtained by the tracker, the matching verification information including a 3D graphic icon representing the relative position relationship between the probe tip and the 3D model; A display unit configured to display an image based on the matching verification information; And wherein the matching validator generates the 3D graphic icon representing a plurality of spheres having different radii and centered on the position of the probe tip, and displays the 3D graphic icon on the 3D model.

12. The system according to claim 11, wherein, the plurality of spheres are represented as being visually distinguishable from each other.

13. The system according to claim 11 or 12, wherein, the matching validator generates the matching verification information such that a first image having at least one fixed viewing angle for the 3D model and a second image having a viewing angle changed according to the change in the position and pose of the probe tip can be respectively displayed in different regions.

Images