System and method for automatic surgical position marking in robot-assisted surgery

Abstract

The present teachings relate to surgical location marking. The 3D model of the organ includes cutting points forming a surgical trajectory. Each cutting point has 3D coordinates and a surface normal in the model space. When the cutting points are projected into the workspace, mapped cutting points are created using the mapped 3D coordinates and the mapped surface normals in the workspace. Some mapped cutting points are marked with a surgical instrument having a tip and having a force sensor attached thereto along a direction determined based on a reaction force sensed by the force sensor when the tip contacts the cutting point.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Patent Application No. 18 / 163,715 (Attorney Docket No. 140551.569668), entitled "System and Method for Automatic Surgical Position Marking in Robotic-Assisted Surgery," which is incorporated herein by reference in its entirety. Background Art 1. Technical Field

[0002] The present teachings relate generally to computers and more particularly to signal processing. 2. Technical Background

[0003] Robotics is a field that has seen significant progress over the past few decades. Robots are not only used in industrial settings (such as on assembly lines) to produce products around the clock, but are also being deployed in other types of environments to transport goods in warehouses or assist surgeons in certain surgical procedures. For example, robotic surgery has become widely accepted for liver resections due to the unparalleled precision, reach, and dexterity of robots in various tasks. An added benefit of surgical robots is that their performance does not degrade over time, unlike humans who get tired, need to eat and sleep, and may be distracted.

[0004] Various surgical robotic solutions have been developed for human-guided minimally invasive surgery. In these solutions, the surgical robot is instructed on what to do and how to do it. This type of human-guided robotic surgery may not fully exploit the robot's advantages. For example, humans can be imprecise even when not fatigued. This imprecision can affect the quality of their work, especially in procedures that require precise movements, accurate observations, and precise adjustments based on these observations. Robots may be more adept at handling high-precision procedures. Furthermore, robots can be better positioned and provide more accurate observations and measurements, leading to improved precision.

[0005] Therefore, there is a need to develop solutions that address the shortcomings of the current state of the art. Summary of the Invention

[0006] The teachings disclosed herein relate to methods, systems, and programming for information management. More specifically, the teachings relate to methods, systems, and programming related to hash tables and storage management using hash tables.

[0007] In one example, a method is implemented on a machine having at least one processor, a storage device, and a communication platform capable of connecting to a network for surgical position marking. A three-dimensional (3D) model of an organ includes a surgical trajectory on the surface of the organ. The surgical trajectory can be formed by a set of discrete cutting points, or can be represented as a continuous contour on the surface of the organ. Each cutting point is represented by a 3D coordinate and a surface normal relative to a first coordinate system. The cutting point is then projected onto the organ present in a second coordinate system to create a mapped cutting point. Each mapped cutting point is represented by a mapped 3D coordinate and a mapped surface normal in the second coordinate system. A surgical instrument having a tip with a force sensor attached thereto is controlled to mark at least some of the mapped cutting points on the organ along a marking direction determined based on a reaction force sensed by the force sensor when the tip contacts the mapped cutting point.

[0008] In a different example, a system for surgical location marking is disclosed. The system includes a registration unit, a 3D resection trajectory mapper, and a 3D cutting point marking mechanism. The registration unit is configured to align a 3D model of an organ with a workspace, wherein the 3D model includes cutting points that form a surgical trajectory on a surface of the organ. Each cutting point is represented by a 3D coordinate and a surface normal relative to a first coordinate system. The 3D resection trajectory mapper projects the cutting points onto the organ existing in a second coordinate system to create mapped cutting points. Each mapped cutting point is represented by a mapped 3D coordinate and a mapped surface normal in the second coordinate system. The 3D cutting point marking mechanism marks at least some of the mapped cutting points on the organ by controlling the movement of a surgical instrument having a tip having a force sensor attached thereto, along a marking direction determined based on a reaction force sensed by the force sensor when the tip contacts the mapped cutting point.

[0009] Other concepts relate to software for implementing the present teachings. A software product according to this concept includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium can be executable program code data, parameters associated with the executable program code, and / or information related to a user, a request, content, or other additional information.

[0010] Another example is a machine-readable, non-transient and tangible medium on which is recorded information for surgical position marking. When the information is read by a machine, the machine performs the following steps. A three-dimensional (3D) model of an organ is received, and the three-dimensional (3D) model of the organ has a list of cutting points that form a surgical trajectory on the surface of the organ. Each cutting point is represented by a 3D coordinate and a surface normal relative to a first coordinate system. The cutting point is then projected onto the organ in a second coordinate system to create a mapped cutting point. Each mapped cutting point is represented by a mapped 3D coordinate and a mapped surface normal in the second coordinate system. A surgical instrument having a tip with a force sensor attached thereto is controlled to mark at least some of the mapped cutting points on the organ along a marking direction determined based on a reaction force sensed by the force sensor when the tip contacts the mapped cutting point.

[0011] Additional advantages and novel features will be set forth in part in the following description and in part will be apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The advantages of the present teachings may be realized and obtained by practice or use of various aspects of the methods, tools, and combinations set forth in the detailed examples discussed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The methods, systems, and / or programs described herein will be further described by way of exemplary embodiments. These exemplary embodiments will be described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals denote similar structures throughout the several views of the drawings, and in which:

[0013] Figure 1A-1B A human liver is shown having a growth therein and a planned resection trajectory to remove the growth;

[0014] Figure 1C shows an exemplary portion of a planned resection trajectory with discrete cutting points, each of which is represented by a coordinate and a surface normal;

[0015] Figure 2A An exemplary medical device having a tip and a rigid body with a tracking device attached thereto is shown;

[0016] Figure 2B The figure shows a surgical scene in which the tip of a medical instrument is continuously tracked and visual information around the tip is continuously acquired in real time;

[0017] Figure 3A depicts an exemplary high-level system diagram of a surgical location marking system for automatically marking cutting points along a resection trajectory according to an embodiment of the present teachings;

[0018] Figure 3Bis a flow chart of an exemplary process of a surgical location marking system for automatically marking cutting points along a surgical trajectory according to an embodiment of the present teachings;

[0019] Figure 4A depicts an exemplary high-level system diagram of a registration unit according to an embodiment of the present teachings;

[0020] Figure 4B is a flow chart of an exemplary process of a registration unit according to an embodiment of the present teachings;

[0021] Figure 5A depicts an exemplary high-level system diagram of a 3D cutting point marker direction determiner according to an embodiment of the present teachings;

[0022] Figure 5B is a flow chart of an exemplary process of a 3D cutting point marking direction determiner according to an embodiment of the present teachings;

[0023] Figure 6A Figure illustrates an intermediate point of a surface normal for a first cutting point along a resection trajectory and a sensed force direction for correcting the surface normal for subsequent cutting points according to an embodiment of the present teachings;

[0024] Figure 6B illustrates how the sensed force direction at a previous cutting point is used to correct the surface normal direction at the current cutting point, according to an embodiment of the present teachings;

[0025] Figure 7 is a schematic diagram of an exemplary mobile device architecture that can be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments; and

[0026] Figure 8 is a schematic diagram of an exemplary computing device architecture that can be used to implement a special-purpose system for implementing the present teachings in accordance with various embodiments. DETAILED DESCRIPTION

[0027] In the following detailed description, numerous specific details are set forth by way of example in order to facilitate a thorough understanding of the relevant teachings. However, it should be apparent to one skilled in the art that the present teachings may be practiced without these details. In other instances, well-known methods, processes, components, and / or systems have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0028] The present teachings disclose exemplary methods, systems, and implementations for automated robotically guided surgical position marking on an organ surface along a pre-planned resection trajectory. Figure 1A A human organ 100 is shown having a growth (110) therein. Figure 1BA planned resection trajectory 130 is shown on an organ for resecting a portion of the organ 120 to remove a growth. Figure 1C An exemplary portion of a planned resection trajectory 130 is shown having discrete cutting points on the trajectory, each of which can be represented by three-dimensional (3D) coordinates (X, Y, Z) relative to a known coordinate system and a surface normal N. Assuming that the pre-planned resection trajectory includes m cutting points, the representation of the resection trajectory can be expressed as {(x1, y1, z1, N1), (x2, y2, z2, N2), ..., (Xi, Yi, Zi, Ni), ..., (Xm, Ym, Zm, Nm)}, {(Xi, Yi, Zi, Ni)}, 1 <= i <= m.

[0029] Traditionally, to remove a portion of an organ, a surgeon manually manipulates a surgical instrument inserted into the patient's body and gradually moves it toward a cutting point along a pre-planned surgical path. The surgeon manipulates the instrument by relying on visual information (e.g., laparoscopic images) continuously acquired by a camera inside the patient's body from the area near the instrument. Figure 2A An exemplary surgical instrument is shown having a tip 220 attached to a rigid instrument body 200, with a tracking device 210 attached to the opposite end of the rigid body 200. Such a surgical instrument can be inserted into a patient. The surgical instrument can be tracked via the tracking device attached to the surgical instrument. Because the surgical instrument has a rigid body, by tracking the tracking device 210 on one end of the medical instrument, the 3D position of the instrument tip 220 can also be tracked.

[0030] This is illustrated by an exemplary surgical scenario. Figure 2B As shown in the figure. Figure 2B As shown, a patient 240 is lying on an operating table 230. In this exemplary scenario, surgery will be performed on the patient 240 on the operating table 230, for example, by removing a portion of the patient's organ. The surgical procedure will follow the following steps: Figure 1B A pre-planned surgical trajectory on the surface of the organ is shown. The surgical instrument 200 can be inserted into the patient's body so that the tip 220 of the surgical instrument is located inside the body, while the end of the instrument with the tracking device 210 attached thereto remains outside the patient's body. In order to track the position of the tip 220 of the surgical instrument, the tracking device 270 (e.g., a camera) is configured so that the tracking device 270 can dynamically track the position of the tracking device 210. When the tracking camera 270 is calibrated, each time it detects the tracking device 240, the 3D position of the tip 220 of the instrument (e.g., relative to the coordinate system of the tracking device 270) can also be determined.

[0031] During minimally invasive surgery, a laparoscopic instrument 250 may be inserted into a patient's body along with a laparoscopic camera 260 at its end. The laparoscopic instrument 250 is inserted into the body in such a way that the laparoscopic camera 250 can capture images of the target organ (the organ to be operated on) and the tip 220 of the surgical instrument. These captured images provide the surgeon with visual information regarding the target organ, the position of the surgical instrument, and the spatial relationship between the two. This visual information provides guidance to the surgeon during the procedure, for example, by controlling the movement of the instrument toward the target organ based on visual observations.

[0032] The target organ can be modeled based on images previously acquired from the patient prior to surgery. For example, a 3D model can be obtained based on CT scan images of the target organ. Based on these scanned images, the target organ can be modeled in 3D space, with detected growths detected and labeled. Various measurements of the growths and their spatial orientation can also be obtained by analyzing the image data. Based on this extracted information, the resection area can be determined, and accordingly, the surgical trajectory for removing the area containing the growth can be determined prior to surgery. Figure 1B Such a pre-planned surgical trajectory can be incorporated into the 3D model of the patient's target organ in the form of multiple discrete cutting points from the trajectory, as shown in FIG. Figure 1C As shown, each of the cutting points is represented as a tuple with 3D coordinates and a surface normal. The coordinates of each cutting point provide the target position to be reached by the tip of the surgical instrument, and the surface normal provides guidance on the angle at which the tip of the surgical instrument approaches the organ surface.

[0033] During surgery, a pre-planned surgical trajectory can be projected onto a laparoscopic image of the target organ acquired during surgery. The pre-planned surgical trajectory is generated pre-operatively based on pre-operative data. To project the cutting points into the correct locations in the laparoscopic image, registration is required. The image of the target organ acquired by the laparoscopic camera 260 can be registered with a 3D model of the target organ, so that the cutting points along the pre-planned surgical trajectory can then be projected onto the laparoscopic image of the target organ based on the registration results. Ideally, during surgery, the cutting points projected onto the laparoscopic image based on the registration results provide the surgeon with an on-the-fly visual roadmap of where to cut the target organ. However, in practice, there are a number of reasons why the cutting points may not be projected into the correct locations in the laparoscopic image. First, the target organ may appear different during surgery than it did before surgery. For example, the patient may breathe during surgery, causing the target organ to deform. Second, the surface of the organ also deforms accordingly, causing the surface normals at different cutting points to differ from those captured at a different time (pre-surgery). There may be other reasons for the mismatch.

[0034] The present teaching provides a robotic guidance solution for automatically marking cutting points of a surgical trajectory, which controls the surgical instrument to approach each cutting point in a corrected surface normal direction by instantly determining the correction required for each cutting point, wherein the corrected surface normal direction is determined based on the maximum force direction sensed by the force sensor when contacting the target surface. The process is automated, i.e., contacting the cutting point, sensing the force direction at the cutting point, adjusting the contact direction of the next cutting point based on the sensed maximum force direction, marking the next cutting point in the adjusted direction, sending the force direction again, and so on, repeating these until all cutting points in the surgical trajectory are marked. In this process, the precise characteristics of the robot are utilized and used to guide the operation. Reference below Figures 3A-6B The following disclosure is presented using an example of a resection trajectory. This is for illustrative purposes only and does not limit the scope of the present teaching.

[0035] Figure 3A Depicted is an exemplary high-level system diagram of a surgical position marking system 300 for automatically marking cutting points of a resection trajectory according to an embodiment of the present teachings. In the illustrated embodiment, the surgical position marking system 300 includes a 3D needle tracking mechanism 330, a registration unit 310, a 3D resection trajectory mapper 340, a 3D cutting point marking mechanism 390 (which includes a 3D cutting point marking direction determiner 360, a surgical robot 370, and a force sensor reading collector 380). The 3D needle tracking mechanism 330 can be provided as a subsystem for tracking the tip (needle) of a surgical instrument. Such a tracking mechanism can be configured as follows Figure 2B 2. As shown, the surgical instrument 200 includes a tracking device 270 and a tracking apparatus attached to the surgical instrument 200. In this example, the tracking device 270 corresponds to a calibrated camera with a field of view covering the entire space around the operating table so that the position of the tracking tip (needle) 220 is continuously tracked.

[0036] The tracked needle position can be sent to the registration unit 310 to facilitate registration. The (one or more) 3D models 320 of the target organ can provide 3D modeling of the organ and a pre-planned resection trajectory represented by, for example, a list of discrete cutting points, each discrete cutting point can be represented by a tuple with 3D coordinates and surface normals calculated based on the 3D model of the target organ. A user (e.g., a surgeon) can use the needle of the instrument to select some feature points on the target organ based on what the user sees from the displayed 2D laparoscopic image capturing the anatomical structure near the target organ. When such feature points are selected using the tip of the surgical instrument, the 3D coordinates of such feature points can be determined based on the tip coordinates tracked by the tracking mechanism 330. Feature points on the 3D model corresponding to the feature points selected using the tip of the instrument can be identified. Based on these two sets of corresponding feature points, the registration unit 310 obtains a transformation matrix 350, which can be used to transform or map any point of the 3D model 320 to a point on the target organ, or vice versa. Reference Figure 4A-4B Details of the registration unit 310 are provided.

[0037] A 3D resection trajectory mapper 340 may be provided to map or transform the cutting points along the pre-planned resection trajectory represented in the 3D model space to the target organ in the workspace, i.e., projecting the cutting points (represented as 3D coordinates and their corresponding surface normals in the model coordinate system) onto corresponding points on the target organ in the coordinate system of the workspace. This may be achieved by transforming the 3D coordinates and surface normals of each cutting point in the model space into 3D coordinates in the tracking 3D coordinate system of the workspace using a transformation matrix 350. The mapped cutting points may then be visualized by overlaying them onto the 2D laparoscopic image so that the surgeon can visualize these cutting points on the target organ from a 2D display device in the operating room.

[0038] Based on the transformed cutting points, the 3D cutting point marking direction determiner 360 determines, for each cutting point, the direction in which the needle approaches the cutting point in order to mark the cutting point. Ideally, this direction is the surface normal at the cutting point. As discussed herein, the shape of the target organ may be different than before surgery due to various reasons. Therefore, the surface normal at the cutting point calculated before surgery may not reflect the actual surface normal of the cutting point during surgery. Furthermore, because the registration may not be completely accurate, the position of the cutting point mapped on the target organ may not be the exact same point on the target surface. Therefore, the surface normal of the cutting point from the 3D model may be different from the surface normal of the actual corresponding point on the target surface. For these reasons, the surface normal associated with each mapped cutting point on the target surface is not perpendicular to the actual target surface. Therefore, a 3D cutting point marking direction determiner 360 is provided for determining a corrected surface normal direction for the surgical instrument to mark the cutting point relative to each cutting point on the resection trajectory. Reference Figures 5A-6B Details of the 3D cutting point marker direction determiner 360 are provided.

[0039] The corrected surface normal direction of each cutting point can then be used to control the surgical robot 370 to mark the cutting point in the calculated direction. The surgical robot 370 can be aligned in the workspace where the surgery is to be performed. In some embodiments, marking can be achieved by controlling the surgical instrument to first reach an intermediate point along the corrected surface normal direction, and starting from the intermediate point, the needle or tip of the instrument moves along this direction to mark the cutting point. The needle is configured to have a force sensor attached thereto so that when the needle contacts (applies force to) the surface of the target organ, the force sensor senses a reaction force from the surface of the target organ. Such reaction force has not only the magnitude of the force, but also the direction of the reaction force. Such sensed information is obtained by the force sensor reading collector 380 and sent to the 3D cutting point marking direction determiner 360, where the sensed force direction can then be used to determine the corrected surface normal of the next cutting point according to the present teachings. For details on how to use the sensed force direction, refer to Figure 6A and Figure 6B supply.

[0040] Figure 3Bis a flow chart of an exemplary process of a surgical position marking system 300 for automatically marking cutting points along a surgical trajectory according to an embodiment of the present teachings. In order to align a 3D model of a target organ (constructed before surgery) with the target organ during surgery, a registration unit 310 receives the 3D model including the resection trajectory at 305. The 3D needle tracking mechanism 330 receives a 2D video image acquired by, for example, a laparoscopic camera, in which the needle (or tip) of the surgical instrument is visible, and displays such 2D video image on a 2D display device at 315 to facilitate interaction with the user. When the user selects a registration feature point using the tip of the surgical instrument, the 3D needle tracking mechanism 330 receives the selected feature point at 325 and obtains its corresponding 3D coordinates at 335 by tracking the position of the tip of the instrument by the tracking device 270, as described herein with respect to Figure 2B Based on the selected feature points, the registration unit 310 may then identify corresponding feature points on the 3D model at 345 .

[0041] Based on the selected 3D feature points in the tracking space (expressed as 3D coordinates relative to the tracking coordinate system) and the identified corresponding 3D feature points in the 3D model space (expressed as 3D coordinates relative to the modeling space), the registration unit 310 calculates a transformation matrix 350 at 355. The transformation matrix 350 is then used by the 3D resection trajectory mapper 340 to map the cutting points on the pre-planned resection trajectory to the workspace at 365. That is, based on the transformation matrix 350, the 3D coordinates and surface normals of the cutting points in the model space are transformed into 3D coordinates and surface normals relative to the workspace coordinate system. Then, at 375, a 3D cutting point marking mechanism including a 3D cutting point marking direction determiner 360, a robot 370 (which can be registered with the workspace as discussed herein), and a force sensor reading collector 380 marks each cutting point on the 3D resection trajectory based on a corrected marking direction determined based on the direction of the reaction force sensed by the force sensor of the needle attached to the instrument.

[0042] Figure 4A An exemplary high-level system diagram of a registration unit 310 according to an embodiment of the present teachings is depicted. As discussed herein, the function of the registration unit 310 is to determine a transformation matrix 350 based on two sets of 3D coordinates, the two sets of 3D coordinates representing a first set of feature points selected from a target organ during surgery and a second set of corresponding feature points identified from a 3D model of the target organ derived preoperatively. To this end, in the illustrated embodiment, the registration unit 310 includes a needle tip 3D coordinate obtainer 400, a 2D laparoscopic image renderer 420, a user-interactive feature point selector 410, a 3D model feature point identifier 430, and a transformation matrix generator 440.

[0043] Figure 4B is a flow chart of an exemplary process of the registration unit 310 according to an embodiment of the present teachings. In operation, the 2D laparoscopic image renderer 420 of the registration unit 310 receives at 405 a laparoscopic image generated by, for example, Figure 2B The illustrated laparoscopic camera 260 captures a 2D laparoscopic image. The 2D laparoscopic image captures the target organ and the tip of the surgical instrument 220 in real time. Then, at 415, a 2D laparoscopic image renderer 420 can render the 2D laparoscopic image on a display device to provide a visual aid to the user (e.g., surgeon). Utilizing this visual information, whenever the user moves the surgical instrument, the displayed visual information reflects the movement of the instrument's tip in real time.

[0044] Then, at 425, the user interactive feature point selector 410 facilitates the user to select a plurality of feature points on the target organ for registration. The selection can be made by the user moving the needle tip of the instrument to the location of each feature point on the target organ based on the visual information on the display device. In some embodiments, through a user interface (e.g., provided by the user interactive feature point selector 410), the user can determine the location on the target organ in the 2D laparoscopic image as a feature point. The user can then manipulate the surgical instrument so that the needle tip of the instrument reaches the desired 2D feature point location in the laparoscopic image. During the movement, the needle tip 3D coordinate obtainer 400 obtains the desired 2D feature point location from the scene (see Figure 2B ) tracks the 3D coordinates of the needle tip 220. When the needle tip observed in the 2D laparoscopic image reaches the 2D feature point position desired by the user, the 3D coordinates of the needle tip tracked at that moment correspond to the 3D coordinates of the selected feature point on the 3D target organ, and the needle tip 3D coordinate obtainer 400 obtains the tracked 3D coordinates of the needle tip at 435.

[0045] Therefore, selecting feature points for registration can be an interactive process in which a user selects feature points in a 2D image (laparoscopic image). During this process, such interaction can produce selected feature points. Using such information, the robot can be guided to control the movement of the surgical instrument to a 3D position on the organ that corresponds to the selected feature point in the 2D image when viewed from the 2D laparoscopic image. Such an interactive process produces a set of 3D feature points, which are represented as a set of 3D coordinates in a coordinate system for the surgical scene. Using the first set of 3D coordinates in the surgical coordinate system, the 3D model feature point identifier 430 identifies corresponding feature points from the 3D model of the target organ at 445. This produces a second set of 3D coordinates that represent corresponding feature points on the 3D model in the model coordinate space. The transformation matrix generator 440 then performs the registration at 455 and generates the transformation matrix 350.

[0046] As about Figures 3A to 3BAs discussed, the transformation matrix 350 obtained as disclosed herein is then used by the 3D resection trajectory mapper 340 to transform the 3D coordinates and surface normals of the cutting points included in the pre-planned resection trajectory from the 3D model 320 into mapped 3D coordinates and mapped surface normals of the cutting points expressed relative to the coordinate system of the workspace. By the transformation, the pre-planned resection trajectory is mapped onto the target organ during surgery. That is, the cutting points on the pre-planned resection trajectory are now projected onto the target organ during surgery. As described herein with reference to Figures 3A-3B As discussed, these cut points are then marked one at a time in sequence by a combination of a 3D cut point marking direction determiner 360, a surgical robot 370, and a force sensor reading collector 380. As disclosed, for each cut point, the surface normal from the 3D model is adjusted based on the direction of the maximum force determined from the force sensor readings. The adjusted surface normal can then be used as the direction for the surgical instrument to mark the cut point. The force sensor readings can include the magnitude of the sensed force, and importantly, the direction of the force. The force sensor readings associated with the cut point can be used to determine the adjustment of the surface normal for the next cut point. Details of how the marking direction for each cut point is determined are provided below.

[0047] Figure 5A An exemplary high-level system diagram of a 3D cutting point marking direction determiner 360 according to an embodiment of the present teachings is depicted. In the illustrated embodiment, the 3D cutting point marking direction determiner 360 includes a trajectory marking controller 500, a needle contact direction determiner 520, an intermediate point determiner 530, a robot point-to-point (PP) marking trajectory generator 540, a robot configuration generator 550, and a force sensor reading processor 560. Figure 1C As described above, the resection trajectory can be generated as a list of cutting points, {(Xi, Yi, Zi, Ni)}, 1 <= i <= m, where (Xi, Yi, Zi) represents the 3D coordinates of cutting point i, and Ni represents the surface normal of cutting point i. A list of mapped cutting points can be similarly represented, where each mapped cutting point has an associated surface normal from the 3D model of the underlying target organ.

[0048] When the trajectory marking controller 500 receives the mapped list of cutting points, it controls the process of marking one cutting point at a time in a series of marking steps. For the first cutting point, the trajectory marking controller 500 may activate the needle contact direction determiner 520 to determine the direction the needle tip should follow to mark the first cutting point. In some embodiments, the contact direction of the first cutting point may be the surface normal direction. Based on the surface normal direction, the midpoint along the direction of the surface normal may be determined by the midpoint determiner 530 based on the parameters configured in 510. The combination of the needle contact direction and the midpoint along this direction may be sent to the robot PP marking trajectory generator 540 to generate a marking trajectory that the robot arm will follow to mark the midpoint and, from the midpoint, to reach the first cutting point along the surface normal at the first cutting point. To enable the robot to follow the marking trajectory to mark the first point, the robot configuration generator 550 configures the parameters of the robot arm to enable the robot arm to move to manipulate the surgical instrument. These configured parameters are then sent to the robot to perform marking at the first cutting point.

[0049] When the robotic arm is controlled accordingly to mark the first cutting point, the force sensor attached to the needle of the instrument can sense the reaction force from the target organ and generate a force sensor reading. When the force sensor reading processor 560 receives such a reading, the force sensor reading processor 560 can process the received reading and identify the maximum force direction 570. The maximum force direction 570 can then be used to adjust the marking direction of the next cutting point.

[0050] exist Figure 6A The middle figure shows the determination of a first intermediate point for marking a first cutting point. In this illustration, a surgical instrument is inserted into a patient's body from the skin 600 via a trocar point T610. In the body, a target organ 620 has an organ surface 630. The first cutting point M1 is on the organ surface 630 and has coordinates and a surface normal V1n. As discussed herein, the surgical instrument will mark the first cutting point M1 from the direction of the surface normal of M1 or V1n. Because the trocar point T already exists and it connects the first cutting point M1 in the direction V1t, the direction V1t may not be aligned with V1n. To ensure that M1 is marked from the direction V1n, an intermediate point 640 is determined along V1n (at Figure 6A ), so that the surgical instrument can be controlled so that its needle first reaches the middle point (to avoid any collision), and from the middle point, a first cutting point M1 is marked by traveling along the direction V1n to the organ surface 630. The middle point can be determined based on different considerations. For example, it can be determined based on a fixed distance from the cutting point or a function of different parameters. Thus, the first cutting point according to the present teachings can be marked by the surgical instrument along the direction of the surface normal of the 3D model of the target organ.

[0051] When the surgical instrument marks the first cutting point, the force sensor attached to the needle of the instrument can sense the reaction force from the organ surface. In the case where the actual surface normal at the first cutting point during surgery is roughly the same as the 3D model surface normal Vln, the direction of the sensed reaction force may be very close to Vln. However, due to different reasons, the organ surface during surgery may be shaped into a different shape, so that the surface normal at the cutting point may also be different. In this case, the direction of the sensed reaction force may be different from the model surface normal V1n. Figure 6A As shown, when the surgical instrument tip contacts a first cutting point M1 on the organ surface, the force sensor can sense a reaction force having a direction V1f, defined as a vector of the reaction force. When the force direction sensed at the first cutting point deviates from the surface normal V1n from the 3D model, this indicates that the organ surface 630 at cutting point M1 may be shaped differently than the organ surface 630 from the 3D model of the target organ. In this case, the force direction sensed at the first cutting point can be utilized when determining the direction for marking the next cutting point.

[0052] Figure 6B The figure illustrates the concept of using the reaction force and direction sensed when marking a cutting point to determine the direction of marking the next cutting point according to an embodiment of the present teaching. Figure 6A As shown, at the first cutting point, there is a sensed force with direction V1f. After completing marking M1, the next cutting point to be marked is M2, as shown in FIG. Figure 6B To mark the cutting point M2, the surface normal V2n is obtained from the 3D model of the target organ. Assume that V2t is the unit vector from the cutting point M2 to the trocar point T610. The corrected surface normal direction is V'2n, which is Figure 6B The combination of V2n, V1f and V2t shown. The middle point P2 of M2 is in the direction of V'2n and its position can be determined based on different criteria. In some embodiments, it can be calculated based on, for example, the weighted sum: P2=w1*V 2f +w2*V 2n +w3*V 2t Wherein, w1, w2, and w3 are weight factors with values ​​in the range [0, 1]. As discussed herein, these weights can be provided via configuration at step 515 based on weights archived in and retrieved from storage device 510. In some embodiments, these weight factors can be different for different markers. The weight factors can be learned, for example, via machine learning via an iterative optimization scheme.

[0053] The robot PP trajectory generator 540 can then use such determined direction for marking the second cutting point (from the needle contact direction determiner 520) and the intermediate point associated with the second cutting point (from the intermediate point determiner 530) to determine the robot arm trajectory, and the robot configuration generator 550 can then configure the parameters to control the robot to mark the second cutting point. This process iteratively marks one cutting point at a time by utilizing the force direction sensed when the instrument contacts the previous cutting point to adjust the surface normal. In general, the intermediate point p of the i-th (i≠1) cutting point can be generated using the following general formula i : P i =w1*V if +w2*V in +w3*V it Among them 1 <i<K。

[0054] Figure 5B is a flow chart of an exemplary process of a 3D resection trajectory marking unit according to an embodiment of the present teachings. Upon receiving a mapped (transformed) resection trajectory with a cutting point at 505, the trajectory marking controller 500 may configure parameters for generating a corrected surface normal or calculating an intermediate point along the correct surface normal at 515. Using these configured parameters, the trajectory marking controller 500 invokes the needle contact direction determiner 520 to determine the surface normal associated with the first cutting point and, at 535, calculates the intermediate point to be used to mark the first cutting point. Based on the surface normal and the intermediate point through which the first cutting point is marked, the robot PP trajectory generator 540 calculates a robot trajectory from the first intermediate point to the first cutting point at 545. The robot configuration generator 550 then configures kinematic parameters for the robot to implement the marking motion. Using these configured parameters, the robot is controlled at 555 to move the surgical instrument so that the needle tip of the surgical instrument reaches the first cutting point via the first intermediate point. Because the intermediate point is in the direction of the surface normal, the tip of the surgical instrument contacts the cutting point in the direction of the surface normal. When the instrument tip contacts the first cutting point, the reaction force sensed by the force sensor at the needle tip has a maximum force direction.

[0055] When it is determined at 565 that there are more cutting points to mark, processing proceeds to determine the corrected surface normal of the next cutting point at 575 based on the force direction sensed at the first cutting point, the surface normal from the model, and the vector formed by connecting the cutting point and the cannula needle point. The intermediate point along the corrected surface normal direction can then be determined and obtained by the intermediate point determiner 530 at 585. As discussed herein, to mark the cutting point, the robot is controlled to move the surgical instrument to first reach the intermediate point, and then the surgical instrument is moved from the intermediate point so that the tip contacts the cutting point along the corrected surface normal. This process is repeated for each cutting point on the surgical trajectory until all cutting points have been marked and determined at 565. When this occurs, the process ends at 595.

[0056] Figure 7 is a schematic diagram of an exemplary mobile device architecture that may be used to implement a dedicated system for implementing the present teachings in accordance with various embodiments. In this example, the user device on which the present teachings may be implemented corresponds to a mobile device 700, including but not limited to a smartphone, a tablet computer, a music player, a handheld game console, a global positioning system (GPS) receiver, and a wearable computing device, or in any other form factor. The mobile device 700 may include one or more central processing units ("CPUs") 740, one or more graphics processing units ("GPUs") 730, a display 720, a memory 760, a communication platform 710 (such as a wireless communication module), a storage device 790, and one or more input / output (I / O) devices 750. Any other suitable components, including but not limited to a system bus or controller (not shown), may also be included in the mobile device 700. As Figure 7 As shown, a mobile operating system 770 (e.g., iOS, Android, Windows Phone, etc.) and one or more applications 780 can be loaded from storage 790 into memory 760 for execution by CPU 740. Application 780 can include, at least in part, a user interface for information analysis and management according to the present teachings or any other suitable mobile application on mobile device 700. User interaction, if any, can be implemented via I / O device 750 and provided to various components connected via network(s).

[0057] In order to realize each module, unit and function thereof described in the present disclosure, computer hardware platform can be used as (one or more) hardware platform of one or more elements described herein.The hardware components, operating system and programming language of this type of computer are conventional in nature, and it is assumed that those skilled in the art are fully familiar with to adapt these technologies to the appropriate settings described herein.The computer with user interface element can be used to realize personal computer (PC) or other types of workstations or terminal equipment, although if suitably programmed, computer can also serve as server.It is believed that those skilled in the art are familiar with the structure, programming and general operation of this type of computer equipment, so accompanying drawing should be self-explanatory.

[0058] Figure 8 800 is a schematic diagram of an exemplary computing device architecture that can be used to implement a dedicated system for implementing the present teachings according to various embodiments. Such a dedicated system in conjunction with the present teachings has a functional block diagram of a hardware platform including user interface elements. The computer can be a general-purpose computer or a special-purpose computer. Both can be used to implement a dedicated system for the present teachings. The computer 800 can be used to implement any component or aspect of the framework disclosed herein. For example, the information analysis and management methods and systems disclosed herein can be implemented on a computer such as computer 800 via the computer's hardware, software programs, firmware, or a combination thereof. Although only one such computer is shown for convenience, the computer functions described herein in connection with the present teachings can be implemented in a distributed manner on several similar platforms to distribute the processing load.

[0059] The computer 800 includes, for example, a COM port 850 that is connected to and from a network connected to the COM port 850 to facilitate data communications. The computer 800 also includes a central processing unit (CPU) 820 in the form of one or more processors for executing program instructions. The exemplary computer platform includes an internal communication bus 810, various forms of program storage and data storage (e.g., disk 870, read-only memory (ROM) 830, or random access memory (RAM) 840) for various data files to be processed and / or transferred by the computer 800 and possibly program instructions to be executed by the CPU 820. The computer 800 also includes an I / O component 860 that supports input / output flows between the computer and other components in the computer (such as user interface elements 880). The computer 800 can also receive programming and data via network communications.

[0060] Thus, as described above, aspects of the methods and / or other processes of information analysis and management may be embodied in programming. The programmatic aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of executable code and / or associated data executed on or implemented in some type of machine-readable medium. Tangible, non-transitory "storage" type media include any or all of memory or other storage devices for a computer, processor, or the like, or its associated modules (such as various semiconductor memories, tape drives, disk drives, etc.) that can provide storage for software programming at any time.

[0061] All or part of the software may sometimes be delivered over a network, such as the Internet or various other telecommunication networks. Such communications, for example, may enable software to be loaded from one computer or processor to another, for example, in connection with information analysis and management. Thus, another type of medium that may carry software elements includes optical, electrical, and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical ground networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, or the like, may also be considered to be the medium that carries the software. As used herein, unless limited to tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0062] Thus, a machine-readable medium can take many forms, including but not limited to tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks that can be used to implement the system shown in the accompanying drawings or any of the components of the system, such as any of the storage devices or the like in any (one or more) computers. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and optical fiber, including the wires that form a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, punched card stock tape, any other physical storage medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cassette, a carrier wave that transports data or instructions, a cable or link that transports such a carrier wave, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0063] Those skilled in the art will recognize that the teachings herein are amenable to various modifications and / or enhancements. For example, while the implementation of the various components described above may be embodied in a hardware device, it may also be implemented as a software-only solution, for example, installed on an existing server. Additionally, the technology disclosed herein may be implemented as firmware, a firmware / software combination, a firmware / hardware combination, or a hardware / firmware / software combination.

[0064] Although the foregoing has described what is considered to constitute the present teachings and / or other examples, it should be understood that various modifications may be made thereto, and the subject matter disclosed herein may be implemented in various forms and examples, and the teachings may be applied to many applications, only some of which have been described herein. It is intended that the appended claims claim any and all applications, modifications, and variations that fall within the true scope of the present teachings.

Claims

1. A method implemented on at least one processor, memory, and communication platform, the method comprising: a three-dimensional 3D model of a receiving organ, wherein the 3D model comprises a list of cutting points forming a surgical trajectory on a surface of the organ, wherein each of the cutting points is represented by a 3D coordinate and a surface normal relative to a first coordinate system; projecting the list of cut points onto the organ existing in a second coordinate system to create a mapped list of cut points, each of the mapped cut points being represented by a mapped 3D coordinate in the second coordinate system and a mapped surface normal; as well as At least some of the mapped cutting points are marked on the organ by controlling movement of a surgical instrument having a tip having a force sensor attached thereto, along marking directions determined based on reaction forces sensed by the force sensor when the tip contacts the mapped cutting points.

2. The method according to claim 1, wherein The marking steps include: marking a first of the mapped cut points on the surface of the organ in the direction of a mapped surface normal associated with the first mapped cut point; and Each of the remaining mapped cut points is marked in the direction of a corrected surface normal of the mapped cut point, wherein the corrected surface normal is determined based on a reaction force sensed by the force sensor when the tip marked the previously mapped cut point.

3. The method according to claim 1, wherein The projecting step includes: selecting a plurality of feature points on the organ using the tip of the surgical instrument to obtain a first set of 3D coordinates in the second coordinate system, wherein the first set of 3D coordinates respectively represent the plurality of feature points; identifying a plurality of corresponding feature points from the 3D model, each of the plurality of corresponding feature points corresponding to one of the selected plurality of feature points; Obtaining a second set of 3D coordinates of the plurality of corresponding feature points in the first coordinate system; determining a transformation matrix based on the first set of 3D coordinates and the second set of 3D coordinates; and The 3D coordinates and the surface normal of each in the list of cutting points relative to the first coordinate system are transformed into mapped 3D coordinates and a mapped surface normal of a mapped cutting point in the second coordinate system.

4. The method according to claim 3, wherein The steps of selecting include: displaying a two-dimensional (2D) image capturing the organ and the tip of the surgical instrument; continuously tracking the 3D coordinates of the tip in the second coordinate system; receiving information indicating a selection of each of the plurality of feature points on the organ; and When the information is received, the 3D coordinates of the tip are recorded as the 3D coordinates of each of the selected plurality of feature points.

5. The method according to claim 2, wherein The step of marking the first mapped cutting point comprises: contacting the organ at the mapped 3D coordinates of the first mapped cutting point via the tip of the surgical instrument along the direction of the mapped surface normal of the first mapped cutting point; and When the tip contacts the organ, a reaction force including a direction and a magnitude of the force is sensed by the force sensor, wherein the sensed reaction force is used to mark a next mapped cutting point among the mapped cutting points.

6. The method according to claim 5, wherein The contacting step comprises: determining a midpoint in the direction of the mapped surface normal of the first mapped cutting point; Controlling the surgical instrument to move to the intermediate point; and Movement of the surgical instrument from the intermediate point to the mapped 3D coordinates of the first mapped cutting point is controlled so that the tip contacts the organ along the direction of the mapped surface normal of the first mapped cutting point.

7. The method according to claim 2, wherein The step of marking each of the remaining mapped cutting points comprises: for each of the remaining mapped cutting points, determining a corrected surface normal for a previously mapped cutting point based on the reaction force sensed when the tip contacts the previously mapped cutting point, contacting the organ via the tip at the mapped cutting point in the direction of the corrected surface normal; and When the mapped cutting point is contacted, a reaction force having a force direction and magnitude is sensed by the force sensor to be used to mark the next mapped cutting point.

8. The method according to claim 7, wherein The corrected surface normal is determined by a combination of: the mapped surface normal of the mapped cutting point; the force direction of the reaction force sensed by the force sensor while marking a previously mapped cutting point; as well as A trocar direction is determined by connecting an entry point of the surgical instrument and the mapped cutting point.

9. The method according to claim 7, wherein Contacting the organ at the mapped cutting point in the direction of the corrected surface normal comprises: determining a midpoint in the direction of the corrected surface normal of the mapped cutting point; controlling the surgical instrument to move to the intermediate point; and Movement of the surgical instrument from the intermediate point to the mapped cutting point is controlled so that the tip contacts the organ along the direction of the corrected surface normal.

10. A machine-readable and non-transitory medium having information recorded thereon, characterized in that: The information, when read by the machine, causes the machine to perform the following steps: a three-dimensional 3D model of a receiving organ, wherein the 3D model comprises a list of cutting points forming a surgical trajectory on a surface of the organ, wherein each of the cutting points is represented by a 3D coordinate and a surface normal relative to a first coordinate system; projecting the list of cut points onto the organ existing in a second coordinate system to create a mapped list of cut points, each of the mapped cut points being represented by a mapped 3D coordinate in the second coordinate system and a mapped surface normal; as well as At least some of the mapped cutting points are marked on the organ by controlling movement of a surgical instrument having a tip having a force sensor attached thereto, along marking directions determined based on reaction forces sensed by the force sensor when the tip contacts the mapped cutting points.

11. The medium according to claim 10, wherein The marking steps include: marking a first of the mapped cut points on the surface of the organ in the direction of a mapped surface normal associated with the first mapped cut point; and Each of the remaining mapped cut points is marked in the direction of a corrected surface normal of the mapped cut point, wherein the corrected surface normal is determined based on a reaction force sensed by the force sensor when the tip marked the previously mapped cut point.

12. The medium according to claim 10, wherein The projecting step includes: selecting a plurality of feature points on the organ using the tip of the surgical instrument to obtain a first set of 3D coordinates in the second coordinate system, wherein the first set of 3D coordinates respectively represent the plurality of feature points; identifying a plurality of corresponding feature points from the 3D model, each of the plurality of corresponding feature points corresponding to one of the selected plurality of feature points; Obtaining a second set of 3D coordinates of the plurality of corresponding feature points in the first coordinate system; determining a transformation matrix based on the first set of 3D coordinates and the second set of 3D coordinates; and The 3D coordinates and the surface normal of each in the list of cutting points relative to the first coordinate system are transformed into mapped 3D coordinates and a mapped surface normal of a mapped cutting point in the second coordinate system.

13. The medium according to claim 12, wherein The steps of selecting include: displaying a two-dimensional (2D) image capturing the organ and the tip of the surgical instrument; continuously tracking the 3D coordinates of the tip in the second coordinate system; receiving information indicating a selection of each of the plurality of feature points on the organ; and When the information is received, the 3D coordinates of the tip are recorded as the 3D coordinates of each of the selected plurality of feature points.

14. The medium according to claim 11, wherein The step of marking the first mapped cutting point comprises: contacting the organ at the mapped 3D coordinates of the first mapped cutting point via the tip of the surgical instrument along the direction of the mapped surface normal of the first mapped cutting point; and When the tip contacts the organ, a reaction force including a direction and a magnitude of the force is sensed by the force sensor, wherein the sensed reaction force is used to mark a next mapped cutting point among the mapped cutting points.

15. The medium according to claim 14, wherein The contacting step comprises: determining a midpoint in the direction of the mapped surface normal of the first mapped cutting point; controlling the surgical instrument to move to the intermediate point; and Movement of the surgical instrument from the intermediate point to the mapped 3D coordinates of the first mapped cutting point is controlled so that the tip contacts the organ along the direction of the mapped surface normal of the first mapped cutting point.

16. The medium according to claim 11, wherein The step of marking each of the remaining mapped cutting points comprises: for each of the remaining mapped cutting points, determining a corrected surface normal for a previously mapped cutting point based on the reaction force sensed when the tip contacts the previously mapped cutting point, contacting the organ via the tip at the mapped cutting point in the direction of the corrected surface normal; and When the mapped cutting point is contacted, a reaction force having a force direction and magnitude is sensed by the force sensor to be used to mark the next mapped cutting point.

17. The medium according to claim 16, wherein The corrected surface normal is determined by a combination of: the mapped surface normal of the mapped cutting point; the force direction of the reaction force sensed by the force sensor while marking a previously mapped cutting point; as well as A trocar direction is determined by connecting an entry point of the surgical instrument and the mapped cutting point.

18. The medium according to claim 16, wherein Contacting the organ at the mapped cutting point in the direction of the corrected surface normal comprises: determining a midpoint in the direction of the corrected surface normal of the mapped cutting point; Controlling the surgical instrument to move to the intermediate point; and Movement of the surgical instrument from the intermediate point to the mapped cutting point is controlled so that the tip contacts the organ along the direction of the corrected surface normal.

19. A system comprising: a registration unit implemented by a processor and configured to receive a three-dimensional 3D model of an organ, wherein the 3D model comprises a list of cutting points forming a surgical trajectory on a surface of the organ, wherein each of the cutting points is represented by a 3D coordinate and a surface normal relative to a first coordinate system; a 3D resection trajectory mapper implemented by a processor and configured to project the list of cut points onto the organ existing in a second coordinate system to create a mapped list of cut points, each of the mapped cut points being represented by a mapped 3D coordinate in the second coordinate system and a mapped surface normal; as well as A 3D cutting point marking mechanism is implemented by a processor and is configured to mark at least some of the mapped cutting points on the organ along marking directions determined based on reaction forces sensed by the force sensor when the tip contacts the mapped cutting points by controlling movement of a surgical instrument having a tip having a force sensor attached thereto.

20. The system of claim 19, wherein: The 3D cutting point marking mechanism is further configured to: marking, on the surface of the organ, a first of the mapped cut points in a direction of a mapped surface normal associated with the first mapped cut point; and Each of the remaining mapped cut points is marked in the direction of a corrected surface normal of the mapped cut point, wherein the corrected surface normal is determined based on a reaction force sensed by the force sensor when the tip marked the previously mapped cut point.

21. The system of claim 19, wherein: The registration unit comprises: a user interface feature point selector, implemented by a processor and configured to select a plurality of feature points on the organ via the tip of the surgical instrument to obtain a first set of 3D coordinates in the second coordinate system respectively representing the plurality of feature points; A 3D model feature point identifier, implemented by a processor and configured to: identifying a plurality of corresponding feature points from the 3D model, each of the plurality of corresponding feature points corresponding to one of the selected plurality of feature points; and Obtaining a second set of 3D coordinates of the plurality of corresponding feature points in the first coordinate system; and a transformation matrix generator implemented by a processor and configured to determine a transformation matrix based on the first set of 3D coordinates and the second set of 3D coordinates, wherein the transformation matrix is ​​used to transform the 3D coordinates and the surface normal of each of the cutting points in the list relative to the first coordinate system into the mapped 3D coordinates and the mapped surface normal of the mapped cutting point in the second coordinate system.

22. The system of claim 20, wherein: The user interface feature point selector is configured to select a feature point by: displaying a two-dimensional (2D) image capturing the organ and the tip of the surgical instrument; continuously tracking the 3D coordinates of the tip in the second coordinate system; receiving information indicating a selection of each of the plurality of feature points on the organ; as well as When the information is received, the 3D coordinates of the tip are recorded as the 3D coordinates of each of the selected plurality of feature points.

23. The system of claim 20, wherein: The 3D cutting point marking mechanism is configured to mark the first mapped cutting point by: contacting the organ via the tip of the surgical instrument at the mapped 3D coordinates of the first mapped cutting point along the direction of the mapped surface normal of the first mapped cutting point; as well as When the tip contacts the organ, a reaction force including a direction and a magnitude of the force is sensed by the force sensor, wherein the sensed reaction force is used to mark a next mapped cutting point among the mapped cutting points.

24. The system of claim 23, wherein: The step of contacting the first mapped cut point is performed by: determining a midpoint in the direction of the mapped surface normal of the first mapped cutting point; controlling the surgical instrument to move to the intermediate point; as well as Movement of the surgical instrument from the intermediate point to the mapped 3D coordinates of the first mapped cutting point is controlled so that the tip contacts the organ along the direction of the mapped surface normal of the first mapped cutting point.

25. The system of claim 20, wherein: The 3D cutting point marking mechanism marks each of the remaining mapped cutting points by performing the following operations for each of the remaining mapped cutting points, determining a corrected surface normal for a previously mapped cutting point based on the reaction force sensed when the tip contacts the previously mapped cutting point, contacting the organ via the tip at the mapped cutting point in the direction of the corrected surface normal; as well as When the mapped cutting point is contacted, a reaction force having a force direction and magnitude is sensed by the force sensor to be used to mark the next mapped cutting point.

26. The system of claim 25, wherein: The corrected surface normal is determined by a combination of: the mapped surface normal of the mapped cutting point; the force direction of the reaction force sensed by the force sensor while marking a previously mapped cutting point; as well as A trocar direction is determined by connecting an entry point of the surgical instrument and the mapped cutting point.

27. The system of claim 25, wherein: The step of contacting the organ at the mapped cutting point in the direction of the corrected surface normal comprises: determining a midpoint in the direction of the corrected surface normal of the mapped cutting point; Controlling the surgical instrument to move to the intermediate point; and Movement of the surgical instrument from the intermediate point to the mapped cutting point is controlled so that the tip contacts the organ along the direction of the corrected surface normal.

Images