System and method for determining force applied to anatomical object in subject based on deformable three-dimensional model

Abstract

An illustrative force determination system may be configured to detect an amount of deformation of a deformable 3D model of a scene occurring when an anatomical object located in the scene deforms, and determine a force value representative of a force causing deformation of the anatomical object based on the amount of deformation of the deformable 3D model. The force determination system may also be configured to provide haptic feedback to the user based on the force value.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 409,981, filed on September 26, 2022, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] In some cases, a computer-assisted medical system may be used to perform a medical procedure. For example, one or more instruments may be at least partially inserted into a subject's body so that a surgeon may use the computer-assisted medical system to manipulate the one or more instruments within the subject's body.

[0004] Unfortunately, the surgeon may not receive feedback (e.g., tactile feedback) from one or more instruments during a medical procedure, which may make it difficult for the surgeon to determine the amount of force applied by the one or more instruments to the subject's internal anatomical structures. The lack of such feedback from the one or more instruments may cause the one or more instruments to apply an undesirable amount of force within the subject. Furthermore, due to the lack of feedback from the one or more instruments, the surgeon may not be aware of instrument errors (e.g., stapler misfires) and / or conditions of the internal anatomical structures within the subject. Summary of the invention

[0005] The following description presents a simplified overview of one or more aspects of the systems and methods described herein. This overview is not an extensive review of all contemplated aspects, is not intended to identify key or important elements of all aspects, nor is it intended to outline the scope of any or all aspects. Its purpose is to present one or more aspects of the systems and methods described herein as a prelude to the specific embodiments presented below.

[0006] An illustrative system includes a memory storing instructions and one or more processors communicatively coupled to the memory. The one or more processors may be configured to execute the instructions to perform a process including: detecting an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene is deformed, and determining a force value representing a force causing the anatomical object to deform based on the amount of deformation of the deformable 3D model.

[0007] An illustrative method includes detecting, by at least one computing device, an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene is deformed, and determining a force value representing a force causing the deformation of the anatomical object based on the amount of deformation of the deformable 3D model.

[0008] An illustrative non-temporary computer-readable medium may store instructions that, when executed, direct a processor of a computing device to perform a process comprising: detecting an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene is deformed, and determining a force value representing a force that causes the deformation of the anatomical object based on the amount of deformation of the deformable 3D model. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of the present disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0010] Figure 1 An illustrative implementation including a force determination system is shown.

[0011] Figure 2 Another illustrative implementation including a force determination system is shown.

[0012] Figure 3 An illustrative method of operating a force determination system is shown.

[0013] Figure 4 Another illustrative method of operating a force determination system is shown.

[0014] Figure 5A and Figure 5B An illustrative implementation of generating a deformable 3D model using a force determination system is shown.

[0015] Fig. 6A and Figure 6B An illustrative implementation of determining a force value using a force determination system is shown.

[0016] Figure 7 An illustrative embodiment of a display that may be generated using a force determination system is shown.

[0017] Figure 8 An exemplary computer-assisted medical system that may include a force determination system is shown.

[0018] Fig. 9 An illustrative computing system in accordance with the principles described herein is shown. DETAILED DESCRIPTION

[0019] An illustrative force determination system may be configured to determine an amount of force that causes deformation of an anatomical object within the scene (e.g., a region within a subject of a medical procedure) based on a deformable 3D model of the scene. For example, the force determination system may be configured to detect an amount of deformation of the deformable 3D model that occurs when an anatomical object located in the scene is deformed, and determine a force value representing the force that causes deformation of the anatomical object based on the amount of deformation of the deformable 3D model.

[0020] In some embodiments, a deformable 3D model can be generated in real time during a medical procedure based on an image of a scene (e.g., captured by an imaging device). As the anatomical object deforms (e.g., due to forces applied by an instrument during the medical procedure), the deformable 3D model can deform to depict the movement of the anatomical object within the scene. This can allow a force determination system to intuitively determine a force value based on the amount of deformation of the deformable 3D model. In some embodiments, tactile feedback representing the force can be provided to a user (e.g., a surgeon applying the force using an instrument).

[0021] Compared to conventional techniques that are not based on deformable 3D models, the principles described herein can produce improved force determination and provide other benefits as described herein. For example, determining force values ​​based on a deformable 3D model can allow for more accurate and / or more efficient determination of force values. To illustrate, determining force values ​​based on a deformable 3D model can be used to provide real-time and realistic tactile feedback to a surgeon so that a desired amount of force can be applied by one or more instruments in a subject's body. In addition, determining force values ​​based on a deformable 3D model can allow a surgeon to be aware of instrument errors (e.g., stapler misfires) and / or conditions of internal anatomical structures in a subject's body.

[0022] Figure 1 An exemplary embodiment 100 is shown, which is configured to determine a force value representing a force causing deformation of an anatomical object within the scene based on a deformable 3D model of a scene. As shown, embodiment 100 includes a force determination system 102, which is configured to generate a deformable 3D model of the scene based on an image of the scene, detect the amount of deformation of the deformable 3D model that occurs when an anatomical object located in the scene is deformed, and determine a force value representing the force causing deformation of the anatomical object based on the amount of deformation of the deformable 3D model. Embodiment 100 may include additional or alternative components that may serve a particular embodiment. In some examples, embodiment 100 or certain components of embodiment 100 may be implemented by a computer-assisted medical system.

[0023] The force determination system 102 may be implemented by one or more computing devices and / or computer resources (e.g., processors, memory devices, storage devices, etc.) that may serve a particular implementation. As shown, the force determination system 102 may include, but is not limited to, a memory 104 and a processor 106 selectively and communicatively coupled to each other. The memory 104 and the processor 106 may each include or be implemented by computer hardware configured to store and / or process computer software. Figure 1 Various other components of computer hardware and / or software not explicitly shown in FIG. 1 may also be included in force determination system 102. In some examples, memory 104 and / or processor 106 may be distributed among multiple devices and / or multiple locations as may serve particular implementations.

[0024] The memory 104 may store and / or otherwise maintain executable data used by the processor 106 to perform any of the functions described herein. For example, the memory 104 may store instructions 108 that may be executed by the processor 106. The memory 104 may be implemented by one or more memories or storage devices (including any memory or storage device described herein that is configured to store data in a temporary or non-temporary manner). The instructions 108 may be executed by the processor 106 to cause the force determination system 102 to perform any of the functions described herein. The instructions 108 may be implemented by any suitable application, software, code, and / or other executable data instance. In addition, the memory 104 may also maintain any other data that is accessed, managed, used, and / or transmitted by the processor 106 in a particular embodiment.

[0025] Processor 106 may be implemented by one or more computer processing devices including a general purpose processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, etc.), a special purpose processor (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc.), an image signal processor, or the like. Using processor 106 (e.g., when processor 106 is directed to perform operations represented by instructions 108 stored in memory 104), force determination system 102 may perform various operations described herein.

[0026] Figure 2Another illustrative embodiment 200 is shown, which is configured to determine a force value representing a force causing deformation of an anatomical object in the scene based on a deformable 3D model of a scene. As shown, the embodiment 200 includes a force determination system 202 coupled to an imaging device 204 and a user interface 206 (e.g., wired and / or wirelessly). The embodiment 200 may include additional or alternative components that can serve a specific embodiment. In some examples, the embodiment 200 or certain components of the embodiment 200 may be implemented by a computer-assisted medical system.

[0027] The imaging device 204 may be implemented by an endoscope or other suitable device configured to capture and output images (e.g., images, videos, image frame sequences, etc.) of the scene 208. In some embodiments, the imaging device 204 may include, but is not limited to, one or more of the following: a video imaging device, an infrared imaging device, a visible light imaging device, an invisible light imaging device, an intensity imaging device (e.g., a color, grayscale, black and white imaging device), a depth imaging device (e.g., a stereo imaging device, a time-of-flight imaging device, an infrared imaging device, a red, green, and blue (RGB) imaging device, a red, green, and blue and depth (RGB-D) imaging device, a light detection and ranging (LIDAR) imaging device, etc.).

[0028] In some implementations, the imagery may include image data (e.g., color, grayscale, saturation, intensity, brightness, depth, etc.) captured by imaging device 204. In some cases, the image data may be associated with data points represented in a common coordinate system, such as 3D voxels or two-dimensional (2D) pixels of an image captured by imaging device 204. In some implementations, imaging device 204 may be moved relative to scene 208 to capture imagery of scene 208 at different viewpoints.

[0029] Scene 208 may include an environment (e.g., a region within a subject of a medical procedure) and / or one or more objects within the environment. For example, scene 208 may include anatomical object 210. Anatomical object 210 may include an object associated with a subject (e.g., a body of a living animal, a human or animal cadaver, a portion of a human or animal anatomical structure, tissue removed from a human or animal anatomical structure, a non-tissue artifact, a training model, etc.). In some embodiments, anatomical object 210 may include tissue of a subject (e.g., an organ, soft tissue, connective tissue, etc.).

[0030] The imaging device 204 can capture deformation of an anatomical object 210 within the scene 208. To illustrate, a force 212 can be applied to the anatomical object 210, which can cause the anatomical object 210 to deform. The force 212 can be applied to the anatomical object 210 by one or more physical tools 214 (e.g., instruments, scalpels, scissors, forceps, clamps, etc.) and / or other objects. Although the illustrated embodiment shows the physical tools 214 positioned outside of the scene 208, the physical tools 214 can additionally or alternatively be included within the scene 208. Other non-anatomical objects used in medical procedures (e.g., staplers, meshes, sponges, etc.) can also be included within the scene 208.

[0031] The force determination system 202 can implement or be similar to the force determination system 102, and can be configured to receive images of the scene 208 from the imaging device 204. In some embodiments, the force determination system 202 can be configured to fuse images of the scene 208 captured by the imaging device 204 at different viewpoints of the scene 208. In some examples, the fusion can include merging aligned (or overlapping) voxels or pixels, such as by blending the intensity and / or depth values ​​of the aligned voxels or pixels. The blending can include weighted blending, in which the blended data points are weighted based on one or more factors, such as which camera of the stereoscopic device has the best view of the data point (e.g., by weighting the data captured by the camera with the best viewing angle more heavily). The fusion can additionally or alternatively include stitching together non-overlapping voxels or pixels, such as by stitching the images together along non-overlapping boundaries of the images. Thus, the fusion of images at different viewpoints can allow the image of the scene 208 to include a larger area than the single field of view of the imaging device 204.

[0032] As shown, the force determination system 202 includes a deformable 3D model generator 216 that is configured to generate a deformable 3D model 218 based on an image of the scene 208. For example, the deformable 3D model generator 216 can be configured to generate a point cloud having a plurality of nodes that represent surface points on one or more objects (e.g., anatomical object 210) within the scene 208 as depicted in the image captured by the imaging device 204. The deformable 3D model generator 216 can also be configured to generate vertices associated with 3D positions corresponding to the 3D positions of the plurality of nodes from the image. In the case where the plurality of nodes are based on 2D imagery, the deformable 3D model generator 216 can be configured to determine depths associated with the plurality of nodes, such as by processing a stereo image captured by the imaging device 204. Additionally or alternatively, a depth sensor can be used to generate a depth map of the scene 208.

[0033] The deformable 3D model generator 216 may also be configured to deform the deformable 3D model 218 over time as one or more objects within the scene 208 move. For example, the anatomical object 210 within the scene 208 may deform during a medical procedure (e.g., due to the force 212), such that the deformable 3D model 218 may deform to correspond to the deformation of the anatomical object 210. To illustrate, the 3D positions of the vertices of the deformable 3D model 218 may track the 3D positions of the plurality of nodes associated with the vertices as the 3D positions of the plurality of nodes are updated in the imagery captured by the imaging device 204 as one or more objects within the scene 208 move. This may allow the deformable 3D model 218 to deform over time as one or more objects within the scene 208 move. In some implementations, the deformable 3D model generator 216 may be configured to detect deformation of the deformable 3D model 218, such as by comparing the 3D positions of the vertices of the deformable 3D model 218 at two or more different points in time. Additionally or alternatively, a deformable or non-deformable first 3D model may be generated at a first point in time, and a deformable or non-deformable second 3D model may be generated at a second point in time different from the first point in time, such that the first 3D model and the second 3D model may be compared to each other to detect deformation.

[0034] In some implementations, the deformable 3D model generator 216 may use a simultaneous localization and mapping (SLAM) heuristic to build and / or update a map of the scene 208 while keeping track of the locations of objects within the scene 208. For example, the SLAM heuristic may be configured to generate a point cloud having a plurality of nodes representing surface points on one or more objects within the scene 208, and the SLAM heuristic may be configured to derive and / or associate vertices of the deformable 3D model 218 with 3D locations corresponding to the 3D locations of the plurality of nodes as the imaging device 204 views the scene 208 in real time. The SLAM heuristic may also be configured to derive and / or associate additional vertices of the deformable 3D model 218 with 3D locations corresponding to the 3D locations of the additional nodes as the imaging device 204 moves relative to the scene 208 to capture additional areas of the scene 208, while also keeping track of the 3D locations of previous vertices of the deformable 3D model 218 and the 3D locations of the associated previous nodes as one or more objects within the scene 208 move and / or deform. In some implementations, SLAM heuristics can be configured to track the pose of the imaging device 204 as it moves relative to the scene 208 (eg, using vision software).

[0035] Other suitable configurations may also be used to generate and / or update the deformable 3D model 218 as one or more objects within the scene 208 move. For example, the deformable 3D model generator 216 may be configured to generate the deformable 3D model 218 based on preoperative images of the scene 208. Additionally or alternatively, the movement of the one or more objects within the scene 208 may be determined based on kinematic data representing the movement of the one or more objects over time. For example, the kinematic data may be generated by or associated with a computer-assisted medical system that is communicatively coupled to the one or more objects (e.g., a physical tool).

[0036] The force determination system 202 also includes a force value module 220 configured to determine a force value representing a force (e.g., force 212) that causes deformation of the anatomical object 210 in the scene 208 based on the amount of deformation of the deformable 3D model 218. The force value can be represented by any suitable value, such as a discrete value (e.g., an integer, a range, a level, a percentage, etc.) representing the force applied to the anatomical object 210.

[0037] To illustrate, the force value module 220 can be configured to detect an amount of deformation of the deformable 3D model 218 that occurs when an anatomical object 210 located in the scene 208 is deformed, such as by a force 212 applied to the anatomical object 210 by a physical tool 214. For example, the force value module 220 can be configured to track the movement of a plurality of nodes representing surface points on the anatomical object 210 over time when the force 212 is applied to the anatomical object 210 based on an image of the scene 208 captured by the imaging device 204. The force value module 220 can also be configured to update 3D positions of vertices of the deformable 3D model 218 associated with the 3D positions of the plurality of nodes, and compare the updated 3D positions of the vertices with the previous 3D positions of the vertices to determine an amount of deformation of the deformable 3D model 218. Based on the amount of deformation of the deformable 3D model 218, the force value module 220 can determine a force value representing the force 212 that caused the deformation of the anatomical object 210. In some implementations, the force value can represent an amount of physical force applied to the anatomical object 210. Additionally or alternatively, the force value may represent a relative force based on a change in deformation of the anatomical object 210. As one illustrative example, the force value may increase as the amount of deformation increases, and / or the force value may decrease as the amount of deformation decreases.

[0038] The user interface 206 can be configured to receive force values ​​from the force determination system 202. The user interface 206 of the illustrated embodiment includes a display device 222 and a user input device 224. The display device 222 can be implemented by a monitor or other suitable device configured to display information to a user. For example, the display device 222 can be configured to display the force values ​​received from the force determination system 202. In some embodiments, the display device 222 can be further configured to display an image of the scene 208 captured by the imaging device 204 and / or the deformable 3D model 218 generated by the force determination system 202. The user input device 224 can include any suitable device (e.g., a button, a joystick, a touch screen, a keyboard, a handle, etc.) that is configured to receive user input, such as for manipulating the physical tool 214.

[0039] In some embodiments, the force determination system 202 can be configured to determine multiple force values ​​based on the deformable 3D model 218. As an illustrative example, the force determination system 202 can be configured to determine the force values ​​based on multiple deformation areas of one or more anatomical objects 210 within the scene 208 that may be caused by one or more physical tools 214. In addition, the force determination system 202 can be configured to mark, track and / or present multiple force values. For example, the force determination system 202 can be configured to mark (e.g., highlight) objects in the scene 208 that are involved in determining force values, such as anatomical objects 210 and / or physical tools 214. The force determination system 202 can also be configured to track and update multiple force values ​​when one or more physical tools 214 move relative to one or more anatomical objects 210. The force determination system 202 can also be configured to present multiple force values ​​to a user, such as on a display device 222. In some embodiments, the multiple force values ​​can be marked on the display device 222 for reference.

[0040] Figure 3 An illustrative method 300 is shown that may be performed by force determination system 202. Figure 3 Exemplary operations according to one embodiment are illustrated, but other embodiments may omit, add to, reorder, and / or modify Figure 3 In addition, Figure 3 Each operation depicted in may be performed in any manner described herein.

[0041] As shown, the force determination system 202 can detect the amount of deformation of the deformable 3D model 218 of the scene 208 that occurs when the anatomical object 210 located in the scene 208 is deformed at operation 302. In some embodiments, the force determination system 202 can generate a point cloud having a plurality of nodes representing surface points on the anatomical object 210 within the scene 208 based on the imagery of the scene 208 captured by the imaging device 204 (e.g., using SLAM heuristics), and derive vertices associated with 3D positions corresponding to the 3D positions of the plurality of nodes. When the 3D positions of the plurality of nodes move with the movement of the anatomical object 210 within the scene 208, the 3D positions of the vertices of the deformable 3D model 218 can be updated with the corresponding 3D positions of the plurality of nodes. This can allow the deformable 3D model 218 to deform over time with the deformation of the anatomical object 210 within the scene 208.

[0042] The force determination system 202 can be configured to track the movement of the 3D positions of the plurality of nodes over time (e.g., when the force 212 is applied to the anatomical object 210) based on the image of the scene 208 captured by the imaging device 204. The force determination system 202 can also be configured to update the 3D positions of the vertices of the deformable 3D model 218 associated with the 3D positions of the plurality of nodes, and compare the updated 3D positions of the vertices with the previous 3D positions of the vertices to determine an amount of deformation of the deformable 3D model 218. In some embodiments, determining the amount of deformation can further include determining a displacement value of the vertex (e.g., a change in position of the vertex in terms of direction, angle, velocity, acceleration, etc.) caused by the force 212 applied to the anatomical object 210 by the physical tool 214.

[0043] The force determination system 202 may also determine a force value representing the force 212 causing the deformation of the anatomical object 210 based on the amount of deformation of the deformable 3D model 218 at operation 304. For example, the amount of deformation of the deformable 3D model 218 may indicate the amount of force applied to the anatomical object 210. As an illustrative example, the force value may increase as the amount of deformation increases, and / or the force value may decrease as the amount of deformation decreases. Other suitable methods may also be used to determine the force value.

[0044] To illustrate, determining the force value may also be based on one or more material property values ​​of the deformable 3D model 218, which may represent one or more material properties of the anatomical object 210. For example, the material properties of the anatomical object 210 (e.g., mass, stiffness, stress, strain, strength, hardness, elasticity, Young's modulus, Poisson's ratio, etc.) may affect how the anatomical object 210 deforms in response to the force 212 (e.g., based on Hooke's law, F=kx, where F is the force, k is a constant based on the material properties, and x is the amount of deformation). In some embodiments, the deformable 3D model 218 may include baseline material property values ​​that may be used for any anatomical object 210, predetermined material property values ​​that may be used for anatomical objects 210 with known material properties, and / or relative material property values ​​that may vary between different regions of the anatomical object and / or between different anatomical objects 210. For example, the material property values ​​may vary between a more rigid anatomical object 210 (e.g., that may exhibit a lower amount of deformation) and a more flexible anatomical object 210 (e.g., that may exhibit a higher amount of deformation). In some implementations, material property values ​​of the deformable 3D model 218 may be adjusted (eg, from initial baseline material property values ​​to predetermined material property values ​​or relative material property values).

[0045] Material property values ​​may be associated with vertices of the deformable 3D model 218. For example, material property values ​​at vertices of the deformable 3D model 218 may remain constant throughout the vertices and / or material property values ​​may vary between vertices of the deformable 3D model 218. For example, some anatomical objects 210 may have non-uniform material properties such that the material properties may vary at various vertices throughout the anatomical object 210. In instances where the anatomical object 210 includes non-uniform material properties, one or more material property values ​​may be associated with a region of the anatomical object 210. To illustrate, a first material property value may represent a first material property associated with a first region of the anatomical object 210, while a second material property value may represent a second material property associated with a second region of the anatomical object 210, and / or a first material property value may represent a material property associated with a first region of the anatomical object 210, while a second material property value may represent the same material property associated with a second region of the anatomical object 210. Additionally, force determination system 202 may be configured to determine which portion of anatomical object 210 is deformed and / or in contact with physical tool 214 such that force determination system 202 may determine force values ​​using one or more material property values ​​associated with that portion of anatomical object 210 .

[0046] In some embodiments, determining the force value can further include implementing and applying an artificial intelligence algorithm (such as a machine learning algorithm) to identify the type of the anatomical object 210 (e.g., an organ (e.g., kidney, intestine, etc.), tissue (e.g., connective tissue, muscle, neural tissue, etc.), etc.). In some embodiments, the type of the identified anatomical object 210 can be used to associate material property values ​​with the deformable 3D model 218 (e.g., based on known material properties of the type of the identified anatomical object 210). Any suitable form of artificial intelligence and / or machine learning can be used, including, for example, deep learning, neural networks, etc. For example, a machine learning algorithm can be generated by a machine learning program and applied to a recognition operation. The machine learning algorithm can operate as a recognition function applied to individual and / or fused images to classify the anatomical objects 210 in the images.

[0047] In addition to or in lieu of a machine learning algorithm, other suitable methods may be used to identify the type of anatomical object 210. For example, the force determination system 202 may be configured to identify the type of anatomical object 210 within the scene 208 by implementing and applying an object recognition algorithm. For example, the object recognition algorithm may be used to identify a predetermined type of object (e.g., anatomical object 210) within image data received from the imaging device 204, such as by comparing the image data received from the imaging device 304 with model object data of the predetermined type of object. Such model object data may be stored in a model database that may be communicatively coupled to the force determination system 202.

[0048] In some embodiments, determining the force value can also be based on the position of the physical tool 214 that applies the force 212 to cause the deformation of the anatomical object 210. For example, detecting the position of the physical tool 214 can be based on kinematic data representing the movement of the physical tool 214 over time. Such kinematic data can be generated by or associated with a computer-assisted medical system that is communicatively coupled to the physical tool 214. Therefore, the known position of the physical tool 214 relative to the anatomical object 210 can allow the force determination system 202 to determine when the physical tool 214 is pressing against the anatomical object 210.

[0049] Additionally, determining the force value can be based on detecting the amount of deformation caused by the position of the physical tool 214. For example, the known position of the physical tool 214 can be used to determine the amount of deformation that occurs at or near the known position of the physical tool 214, which can allow the force determination system 202 to determine the amount of deformation caused by the physical tool 214 and estimate the associated force 212. In some implementations, the force determination system 202 can only calculate the amount of deformation caused by the physical tool 214 at the known position of the physical tool 214 (e.g., to reduce processing burden). Additionally or alternatively, determining the force value can also be based on one or more physical properties of the physical tool 214 (e.g., type, material, size, etc.).

[0050] In some embodiments, the force value can be based on detecting an amount of deformation caused by the movement of the physical tool 214. For example, the force determination system 202 can be configured to determine the movement of the physical tool 214 that applies the force to cause the deformation of the anatomical object 210, such that determining the force value can be further based on the movement of the physical tool 214. In addition, detecting the amount of deformation can include determining the amount of tool-induced deformation (e.g., deformation of the anatomical object 210 caused by the physical tool 214) based on the amount of deformation of the deformable 3D model 218 and the movement of the physical tool 214, such that determining the force value can be based on the amount of tool-induced deformation. Determining the movement of the physical tool 214 can be based on kinematic data representing the movement of the physical tool 214 over time. The kinematic data can be generated by a computer-assisted medical system communicatively coupled to the physical tool 214.

[0051] Figure 4 Another illustrative method 400 that may be performed by force determination system 202 is shown. Figure 4 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Figure 4 In addition, Figure 4 Each operation depicted in may be performed in any manner described herein.

[0052] As shown, the force determination system 202 may be configured at operation 402 to generate a point cloud having a plurality of nodes based on an image of the scene 208 (e.g., captured by the imaging device 204), the plurality of nodes representing surface points on the anatomical object 210 within the scene 208. At operation 404, the force determination system 202 may also be configured to generate a deformable 3D model 218 having vertices having 3D positions associated with the 3D positions of the plurality of nodes.

[0053] The force determination system 202 may also track the movement of the plurality of nodes over time as the force 212 is applied to the anatomical object 210 based on the image of the scene 208 at operation 406. The force determination system 202 may also update the 3D positions of the vertices of the deformable 3D model associated with the 3D positions of the plurality of nodes based on the movement of the plurality of nodes at operation 408. The force determination system 202 may also compare the updated 3D positions of the vertices to the previous 3D positions of the vertices at operation 410 (e.g., to determine an amount of deformation of the deformable 3D model 218).

[0054] The force determination system 202 may also determine a force value representing the force 212 applied to the anatomical object 210 based on the comparison of the 3D positions at operation 412. In some embodiments, the force determination system 202 may dynamically update the force value as the physical tool 214 and / or the anatomical object 210 moves. For example, the 3D positions of the vertices of the deformable 3D model 218 associated with multiple nodes may move as the anatomical object 210 deforms. These changes in the 3D positions of the vertices may affect the amount of deformation of the deformable 3D model 218, so that the force value can be dynamically updated. In some cases, the force determination system 202 may determine and / or update the force value only when a change in the amount of deformation of the deformable 3D model 218 has been detected (e.g., in order to reduce processing burden).

[0055] In some implementations, the force value may be dynamically updated based on the 3D position of the vertex corresponding to each consecutive image frame of the imagery captured by the imaging device 204. Additionally or alternatively, the force value may be dynamically updated based on the 3D position of the vertex corresponding to multiple image frames over time. For example, the force value may represent a combination of force measurements over multiple image frames (e.g., an average, mean, median, etc.).

[0056] In some embodiments, method 400 may also include performing, by force determination system 202, an operation based on the force value. For example, force determination system 202 may be configured to provide haptic feedback to a user based on the force value. Such haptic feedback may include auditory feedback, visual feedback, and / or tactile feedback. Auditory feedback may include an audio output (e.g., a noise, a buzzer, etc.) emitted based on the force value.

[0057] The visual feedback may include displaying the force value on one or more display devices (e.g., display device 222). For example, the force value may be displayed as a readout on one or more display devices. In some embodiments, the force determination system 202 may also be configured to instruct the one or more display devices (e.g., display device 222) to display an image depicting the scene 208 and / or the deformable 3D model 218 so that the force value may be marked on the image depicting the scene 208 and / or the deformable 3D model 218. Additionally or alternatively, the visual feedback may include shading and / or color coding (e.g., red, yellow, green, etc.) on the image depicting the scene 208 and / or the deformable 3D model 218 representing the force value. For example, an area of ​​the deformable 3D model 218 having a larger force value may be darker than an area of ​​the deformable 3D model 218 having a smaller force value.

[0058] The tactile feedback may include adjusting the resistance and / or degrees of freedom of the user input device 224 that may be used to manipulate the physical tool 214. As an illustrative example, the degrees of freedom of the user input device 224 may be constrained in the direction of the anatomical object 504 based on a force value.

[0059] In some implementations, the degree of tactile feedback can be based on the force value, such that the degree of tactile feedback increases or decreases as the force value increases or decreases, respectively. As an illustrative example, the constraints of the degrees of freedom of user input device 224 can increase or decrease as the force value increases or decreases, respectively.

[0060] Additionally or alternatively, the force value may be used to identify errors with the physical tool 214. To illustrate, monitoring the amount of deformation of the deformable 3D model 218 may indicate when a stapler misfire and / or misalignment has occurred on the anatomical object 210 (e.g., if the amount of deformation of the anatomical object 210 depicted by the deformable 3D model 218 is different from the typical amount of deformation caused by the insertion of staplers). If a misfire and / or misalignment has occurred, the force determination system 202 may provide a notification or alert to the user and / or inhibit the operation of the physical tool 214 from firing another stapler.

[0061] As another example, the force value can be used to determine the health of the anatomical object 210. For example, as the health decreases, the tissue of the anatomical object 210 may calcify or harden. Therefore, in some cases, an increase in the force value may indicate a decrease in the health of the anatomical object 210. Additionally or alternatively, a higher force value may indicate a characteristic of the anatomical object. To illustrate, a higher force value may indicate the location of a conduit within the anatomical object 210, such that it may be undesirable for the user to cut the conduit. In some embodiments, the force determination system 202 may provide a notification when the force value exceeds a threshold.

[0062] In some embodiments, the operation may include providing a notification when the force value exceeds or drops below a threshold value. For example, the threshold value may be associated with a successful stapler launch, such that the notification may indicate a stapler misfire. Additionally or alternatively, the threshold value may be associated with a healthy anatomical object, such that the notification may indicate an unhealthy anatomical object 210.

[0063] Figures 5A-6B 2 shows an illustrative example of determining a force value that may be performed by force determination system 202. For example, Figure 5A An embodiment 500 of an image 502 of a scene (e.g., scene 208) that may be captured by an imaging device 204 is shown. As shown, the image 502 includes an anatomical object 504. In the illustrated embodiment 500, the force determination system 202 has generated a point cloud that includes a plurality of nodes 506 (e.g., nodes 506-1 through 506-n). The nodes 506 may represent surface points on the anatomical object 504 within the scene captured by the image 502. Although the illustrated embodiment shows a single anatomical object 504 located in the scene, other embodiments may include additional anatomical objects 504 located in the scene.

[0064] Figure 5B An illustrative implementation 508 of a deformable 3D model 510 that can be generated by the force determination system 202 based on the imagery 502 is shown. The deformable 3D model 510 can implement or be similar to the deformable 3D model 218. As shown, the force determination system 202 can generate the deformable 3D model 510 by deriving vertices 512 (e.g., vertices 512-1 through 512-n) having 3D positions associated with the 3D positions of the plurality of nodes 506. Thus, the deformable 3D model 510 can depict the anatomical object 504 within the scene captured by the imagery 502. In some implementations, the deformable 3D model 510 can be generated using a SLAM algorithm that can derive the vertices 512 and track the positions of the vertices 516 as the imaging device 204 captures the imagery 502.

[0065] In some embodiments, deformation of anatomical object 504 within the scene depicted by image 502 may cause deformation of deformable 3D model 510. To illustrate, Fig. 6A Another embodiment 600 of an image 502 of an anatomical object 504 that may be captured by an imaging device 204 is shown. As shown, the anatomical object 504 has moved to a deformed state. The deformation of the anatomical object 504 may be caused by a force applied to the anatomical object 504 by a physical tool 602. The physical tool 602 may implement or be similar to the physical tool 214. The deformation of the anatomical object 504 may cause the 3D position of one or more of the nodes 506 to move as the anatomical object 504 deforms.

[0066] Figure 6B Another embodiment 604 of the deformable 3D model 510 is shown in a deformed state, which may correspond to the deformed state of the anatomical object 504. As shown, the 3D positions of the vertices 512 of the deformable 3D model 510 have been updated using the 3D positions of the nodes 506 associated with the vertices 512. The updated 3D positions of the vertices 512 may allow the deformable 3D model 510 to deform as the anatomical object 504 deforms.

[0067] Force determination system 202 can determine the force value based on the amount of deformation of deformable 3D model 510. For example, force determination system 202 can update the 3D position of vertex 512 (e.g., Figure 6B ) and the previous 3D position of vertex 512 (e.g., Figure 5B In some embodiments, the force value may be based on factors other than the amount of deformation of the deformable 3D model 510 (e.g., the orientation of the physical tool 602 relative to the anatomical object 504, material property values ​​representing material properties of the anatomical object 504, the type of anatomical object 504, the amount of deformation at or near the physical tool 602, etc.).

[0068] In some embodiments, force values ​​can be calculated at each vertex 512 and / or a group of vertices 512 to provide force values ​​at different regions of the deformable 3D model 510. Alternatively, the force values ​​can be calculated as a general force (e.g., maximum force, average force, median force, mean force, minimum force, etc.) applied to the anatomical object 504. In some embodiments, the force values ​​can be recalculated based on changes in the amount of deformation of the deformable 3D model 510 and / or movement of the physical tool 602 relative to the anatomical object 504.

[0069] In the illustrated embodiment, the physical tool 602 is not included in the deformable 3D model 510. For example, the physical tool 602 may be removed when the deformable 3D model 510 is generated because the position of the physical tool 602 may be known, such that any nodes 506 associated with the physical tool 602 may be removed and / or the vertices 512 may not be associated with those nodes 506. Alternatively, the physical tool 602 may be included in the deformable 3D model 510.

[0070] In some cases, the deformable 3D model 510 may be incomplete (e.g., in areas not captured by the imaging device 204), such that there may be missing vertices (e.g., vertex 512). In these cases, the force determination system 202 can be configured to perform dynamic interpolation to estimate the 3D positions of the missing vertices. For example, the force determination system 202 can interpolate the 3D positions of the missing vertices based on the 3D positions of nearby vertices. In addition, the force determination system 202 can update the 3D positions of the missing vertices based on the movement of nearby vertices as the anatomical objects within the scene move. The force determination system 202 can be configured to perform dynamic interpolation (e.g., to reduce processing burden) when deformation of the anatomical objects occurs in the incomplete areas of the deformable 3D model 510.

[0071] Figure 7 An illustrative implementation 700 of a display 702 is shown that may be displayed on display device 222. As shown, display 702 includes a display of a deformable 3D model 704 that may be generated by force determination system 202. Deformable 3D model 704 may implement or be similar to deformable 3D model 218 and / or deformable 3D model 510.

[0072] The display 702 may also depict levels of shading 706 (e.g., shading 706-1 to 706-3), which may represent the degree of force causing the deformation of the deformable 3D model 704. For example, the first level shading 706-1 may depict a darker shading, which may represent a higher degree of force than the second level shading 706-2 and / or the third level shading 706-3. The display 702 may also include a reference label 708 that may display the force value. In some embodiments, the display 702 may be updated in real time, so that the shading 706 and / or the label 708 depicting the force value may be updated as the deformation of the deformable 3D model 704 changes.

[0073] As described above, in some examples, force determination system 202, imaging device 204, user interface 206, and / or physical tool 214 may be associated with a computer-assisted medical system for performing a medical procedure on a body. Figure 8 An exemplary computer-assisted medical system 800 is shown that may be used to perform various types of medical procedures, including surgical and / or non-surgical procedures.

[0074] As shown, the computer-assisted medical system 800 may include a manipulator component 802 ( Figure 8800 ), a user control device 804, and an auxiliary device 806, all of which are communicatively coupled to each other. The medical team can use the computer-assisted medical system 800 to perform computer-assisted medical procedures or other similar operations on the body of a patient 808 or on any other subject that can serve a specific embodiment. As shown, the medical team can include a first user 810-1 (such as a surgeon for a surgical procedure), a second user 810-2 (such as a patient-side assistant), a third user 810-3 (such as another assistant, a nurse, an intern, etc.), and a fourth user 810-4 (such as an anesthesiologist for a surgical procedure), all of which can be collectively referred to as users 810, and each user can control the computer-assisted medical system 800, interact with the computer-assisted medical system 800, or otherwise become a user of the computer-assisted medical system 800. During a medical procedure, there may be more, fewer, or alternative users that can serve a specific embodiment. For example, the team composition for different medical procedures or non-medical procedures may be different and include users with different roles.

[0075] Although Figure 8 A minimally invasive medical procedure (such as a minimally invasive surgical procedure) is shown being performed, but it should be understood that the computer-assisted medical system 800 can be similarly used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations can also be performed.

[0076] like Figure 8 As shown, manipulator assembly 802 can include one or more manipulator arms 812 (e.g., manipulator arms 812-1 through 812-4) to which one or more instruments can be coupled. The instruments can be used in a computer-assisted medical procedure on patient 808 (e.g., in a surgical example, by being at least partially inserted into patient 808 and manipulated within patient 808). Although manipulator assembly 802 is depicted and described herein as including four manipulator arms 812, it should be appreciated that manipulator assembly 802 can include a single manipulator arm 812 or any other number of manipulator arms that can serve a particular embodiment. Although Figure 8 The example of the manipulator arm 812 is shown as a robotic manipulator arm, but it should be understood that in some examples, one or more instruments can be partially or fully manually controlled, such as by being held and manually controlled by a person. For example, these partially or fully manually controlled instruments can be coupled to Figure 8 The computer-assisted instrumentation of the manipulator arm 812 is shown for use in conjunction with, or as an alternative to, it.

[0077] During a medical procedure, the user control device 804 can be configured to facilitate remote operation control of the manipulator arm 812 and instruments attached to the manipulator arm 812 by the user 810-1. To this end, the user control device 804 can provide the user 810-1 with an image of the operating area associated with the patient 808 captured by the imaging device. To facilitate the control of the instruments, the user control device 804 can include a set of master controls. These master controls can be manipulated by the user 810-1 to control the movement of the manipulator arm 812 or any instrument coupled to the manipulator arm 812.

[0078] The auxiliary device 806 may include one or more computing devices configured to perform auxiliary functions to support the medical procedure, such as providing insufflation, electrocautery energy, lighting or other energy for imaging equipment, image processing, or coordinating components of the computer-assisted medical system 800. In some examples, the auxiliary device 806 may be configured with a display monitor 814 that is configured to display one or more user interfaces, or graphical or textual information that supports the medical procedure. In some cases, the display monitor 814 may be implemented by a touch screen display and provide user input capabilities. The enhanced content provided by the region-based augmentation system may be similar to or different from the content associated with the display monitor 814 or one or more display devices (not shown) in the operating area.

[0079] Manipulator assembly 802, user control device 804, and auxiliary device 806 may be communicatively coupled to each other in any suitable manner. Figure 8 As shown, the manipulator assembly 802, the user control device 804, and the auxiliary device 806 can be communicatively coupled via a control line 816, which can represent any wired or wireless communication link that can serve a particular embodiment. To this end, the manipulator assembly 802, the user control device 804, and the auxiliary device 806 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and the like.

[0080] In some embodiments, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. In general, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes, including one or more of the processes described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0081] Computer-readable media (also referred to as processor-readable media) include any non-transitory media that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take a variety of forms, including, but not limited to, non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memory. Volatile media can include, for example, dynamic random access memory ("DRAM"), which typically constitutes main memory. Common forms of computer-readable media include, for example, magnetic disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital video disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM (flash EEPROM), any other memory chip or cartridge memory, or any other tangible medium from which a computer can read.

[0082] Fig. 9 An exemplary computing device 900 is shown, which may be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein may be implemented by the computing device 900 .

[0083] like Fig. 9 As shown, computing device 900 may include a communication interface 902, a processor 904, a storage device 906, and an input / output ("I / O") module 908 communicatively coupled to one another via a communication infrastructure 910. Fig. 9 An illustrative computing device 900 is shown in Fig. 9 The components shown in are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig. 9 Components of computing device 900 are shown in .

[0084] The communication interface 902 may be configured to communicate with one or more computing devices. Examples of the communication interface 902 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0085] The processor 904 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, processes, and / or operations described herein. The processor 904 may perform operations by executing computer-executable instructions 912 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 906.

[0086] The storage device 906 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, the storage device 906 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including data described herein) may be temporarily and / or permanently stored in the storage device 906. For example, data representing computer executable instructions 912 configured to direct the processor 904 to perform any of the operations described herein may be stored within the storage device 906. In some examples, the data may be arranged in one or more databases residing within the storage device 906.

[0087] The I / O module 908 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 908 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 908 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0088] The I / O module 908 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 908 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0089] In the foregoing description, various exemplary embodiments have been described with reference to the attached drawings. However, it is apparent that various modifications and changes may be made to the various exemplary embodiments, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the following claims. For example, certain features of one embodiment described herein may be combined or replaced with features of another embodiment described herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory for storing instructions; as well as one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: detecting an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene is deformed; and A force value representing a force causing the deformation of the anatomical object is determined based on an amount of the deformation of the deformable 3D model.

2. The system of claim 1 , further comprising generating the deformable 3D model based on an image of the scene.

3. The system of claim 2, wherein generating the deformable 3D model comprises: The deformable 3D model is generated using a simultaneous localization and mapping heuristic method when an imaging device captures an image of the scene.

4. The system of claim 1 , wherein detecting the amount of deformation comprises: tracking, based on an image of the scene, movement over time of a plurality of nodes representing surface points on the anatomical object as the force is applied to the anatomical object; updating 3D positions of vertices of the deformable 3D model associated with positions of the plurality of nodes; as well as The updated 3D position of the vertex is compared to the previous 3D position of the vertex.

5. The system of claim 4, wherein the imagery comprises a stereoscopic image of the scene captured by an imaging device, and wherein tracking the movement of the plurality of nodes comprises determining the depth of the plurality of nodes by processing the stereoscopic image.

6. The system of claim 1, wherein the force value represents an amount of physical force applied to the anatomical object.

7. The system of claim 1, wherein the force value represents a relative force associated with a change in the deformation of the anatomical object.

8. The system of claim 1, wherein determining the force value comprises using a machine learning algorithm to identify a type of the anatomical object.

9. The system of claim 1, wherein determining the force value is also based on one or more material property values ​​of the deformable 3D model, the one or more material property values ​​representing one or more material properties of the anatomical object.

10. The system of claim 9, wherein the one or more material property values ​​include: a first material property value representing a first material property associated with a first portion of the anatomical object; as well as A second material property value representing a second material property associated with a second portion of the anatomical object.

11. The system of claim 10, wherein detecting the amount of deformation comprises: detecting a first deformation amount of the deformable 3D model associated with the first part; as well as A second deformation amount of the deformable 3D model associated with the second part is detected.

12. The system of claim 9, wherein the one or more material property values ​​include: a first material property value representing a material property associated with a first portion of the anatomical object; as well as A second material property value representing the same material property associated with a second portion of the anatomical object.

13. The system of claim 1, wherein determining the force value is further based on an orientation of a physical tool applying the force to cause the deformation of the anatomical object.

14. The system of claim 13, further comprising detecting the position of the physical tool based on kinematic data representing movement of the physical tool over time, the kinematic data generated by a computer-assisted medical system communicatively coupled to the physical tool.

15. The system of claim 13, wherein determining the force value is further based on detecting an amount of the deformation due to the orientation of the physical tool.

16. The system of claim 1, wherein the process further comprises determining movement of a physical tool that applies the force to cause the deformation of the anatomical object, wherein determining the force value is also based on the movement of the physical tool.

17. The system of claim 16, wherein the process further comprises determining an amount of tool-induced deformation based on the amount of the deformation of the deformable 3D model and the movement of the physical tool, wherein determining the force value is based on the amount of the tool-induced deformation.

18. The system of claim 16, wherein determining the movement of the physical tool is based on kinematic data representing the movement of the physical tool over time, the kinematic data being generated by a computer-assisted medical system communicatively coupled to the physical tool.

19. The system of any one of claims 1 to 18, wherein the process further comprises performing an operation based on the force value.

20. The system of claim 19, wherein the operation comprises providing tactile feedback to a user.

21. The system of claim 20, wherein the haptic feedback comprises one or more of tactile feedback, visual feedback, or auditory feedback.

22. The system of claim 20, wherein the degree of tactile feedback is based on the force value such that the degree of tactile feedback increases or decreases as the force value increases or decreases, respectively.

23. The system of claim 19, wherein the operation includes providing a notification when the force value exceeds or drops below a threshold value.

24. The system of claim 23, wherein the threshold is associated with a successful stapler firing, wherein the notification indicates a stapler misfire.

25. The system of claim 23, wherein the threshold is associated with a healthy anatomical object, wherein the notification indicates an unhealthy anatomical object.

26. A method comprising: detecting, by at least one computing device, an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene deforms; as well as A force value representing a force causing the deformation of the anatomical object is determined, by the at least one computing device and based on the amount of the deformation of the deformable 3D model.

27. The method of claim 26, further comprising generating the deformable 3D model based on an image of the scene.

28. The method of claim 27, wherein generating the deformable 3D model comprises using a simultaneous localization and mapping heuristic to generate the deformable 3D model when an imaging device captures an image of the scene.

29. The method of claim 26, wherein detecting the amount of deformation comprises: tracking, based on an image of the scene, movement over time of a plurality of nodes representing surface points on the anatomical object as the force is applied to the anatomical object; updating the 3D positions of the vertices of the deformable 3D model associated with the positions of the plurality of nodes and The updated 3D position of the vertex is compared to the previous 3D position of the vertex.

30. The method of claim 29, wherein the imagery comprises a stereoscopic image of the scene captured by an imaging device, and wherein tracking the movement of the plurality of nodes comprises determining the depth of the plurality of nodes by processing the stereoscopic image.

31. The method of claim 26, wherein the force value represents an amount of physical force applied to the anatomical object.

32. The method of claim 26, wherein the force value represents a relative force associated with a change in the deformation of the anatomical object.

33. The method of claim 26, wherein determining the force value comprises using a machine learning algorithm to identify a type of the anatomical object.

34. The method of claim 26, wherein determining the force value is further based on one or more material property values ​​of the deformable 3D model, the one or more material property values ​​representing one or more material properties of the anatomical object.

35. The method of claim 34, wherein the one or more material property values ​​include: a first material property value representing a first material property associated with a first portion of the anatomical object; as well as A second material property value representing a second material property associated with a second portion of the anatomical object.

36. The method of claim 35, wherein detecting the amount of deformation comprises: detecting a first deformation amount of the deformable 3D model associated with the first part; as well as A second deformation amount of the deformable 3D model associated with the second part is detected.

37. The method of claim 34, wherein the one or more material property values ​​include: a first material property value representing a material property associated with a first portion of the anatomical object; as well as A second material property value representing the material property associated with a second location of the anatomical object.

38. The method of claim 26, wherein determining the force value is further based on an orientation of a physical tool applying the force to cause the deformation of the anatomical object.

39. The method of claim 38, further comprising detecting the position of the physical tool based on kinematic data representing movement of the physical tool over time, the kinematic data generated by a computer-assisted medical system communicatively coupled to the physical tool.

40. The method of claim 38, wherein determining the force value is further based on detecting an amount of the deformation due to the orientation of the physical tool.

41. The method of claim 26, further comprising determining movement of a physical tool that applies the force to cause the deformation of the anatomical object, wherein determining the force value is also based on the movement of the physical tool.

42. The method of claim 41 further comprising determining an amount of tool-induced deformation based on the amount of the deformation of the deformable 3D model and the movement of the physical tool, wherein determining the force value is based on the amount of the tool-induced deformation.

43. The method of claim 41, wherein determining the movement of the physical tool is based on kinematic data representing the movement of the physical tool over time, the kinematic data being generated by a computer-assisted medical system communicatively coupled to the physical tool.

44. The method of any one of claims 26 to 43, further comprising performing an operation based on the force value.

45. The method of claim 44, wherein the operating comprises providing tactile feedback to a user.

46. ​​The method of claim 45, wherein the tactile feedback comprises one or more of haptic feedback, visual feedback, or auditory feedback.

47. The method of claim 45, wherein the degree of tactile feedback is based on the force value such that the degree of tactile feedback increases or decreases as the force value increases or decreases, respectively.

48. The method of claim 44, wherein the operation includes providing a notification when the force value exceeds or drops below a threshold value.

49. The method of claim 48, wherein the threshold is associated with a successful stapler firing, wherein the notification indicates a stapler misfire.

50. The method of claim 48, wherein the threshold is associated with a healthy anatomical object, wherein the notification indicates an unhealthy anatomical object.

51. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform a process comprising: detecting an amount of deformation of a deformable three-dimensional (3D) model of a scene that occurs when an anatomical object located in the scene is deformed; as well as A force value representing a force causing the deformation of the anatomical object is determined based on an amount of the deformation of the deformable 3D model.

Images