Physical medical element sizing system and method

Abstract

An example system is configured to instruct a display device to render a virtual medical element representing a physical medical element over a target region within an image of an internal space of a patient, the target region depicting an anatomical surface to be covered by the physical medical element; receive user input while the virtual medical element is rendered over the target region, the user input setting at least one of a pose of the virtual medical element within the image or a size of the virtual medical element; and determine a physical size of the physical medical element based on the user input and depth data of a depth map representing the internal space.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 911,851, filed October 7, 2019, entitled “PHYSICAL MEDICAL ELEMENTSIZING SYSTEMS AND METHODS”, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] Computer-assisted surgical systems are commonly used to perform hernia repair procedures within a patient's body. As part of the hernia repair procedure, a mesh patch can be placed over the hernia and attached (e.g., sutured) to the tissue surrounding it. As the tissue heals, the mesh patch provides support to the damaged tissue.

[0004] During a hernia repair procedure, the surgeon interacting with a computer-assisted surgical system must determine the appropriate size of the mesh patch. Once the mesh patch has been sized (e.g., by cutting the mesh patch from the mesh material), the surgeon must place it in the appropriate position within the patient's body. These and other types of operations involving determining the appropriate physical size of the medical element to be incorporated into the patent can be time-consuming and lengthy. Summary of the Invention

[0005] The following description presents a brief overview of one or more aspects of the systems and methods described herein. This overview is not an extensive description of all anticipated aspects, and is neither intended to identify key or essential elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the specific implementations presented below.

[0006] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to instruct a display device to render a virtual medical element representing a physical medical element over a target location within an image of a patient's internal space, the target location depicting an anatomical surface to be covered by the physical medical element; receiving user input while the virtual medical element is rendered over the target location, the user input setting at least one of the pose of the virtual medical element within the image and the size of the virtual medical element; and determining the physical size of the physical medical element based on the user input and depth data of a depth map representing the internal space.

[0007] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to access image data representing an image acquired by an imaging device and depicting the patient's internal space; obtaining depth data representing a depth map of the internal space depicted in the image acquired by the imaging device; identifying a target location within the image based on the image data and the depth data, the target location depicting an anatomical surface to be covered by a physical medical element; and instructing a display device to render a virtual medical element representing the physical medical element above the identified target location within the image.

[0008] An exemplary method includes instructing a display device via a medical component management system to render a virtual medical component representing a physical medical component over a target location within an image of a patient's internal space, the target location depicting an anatomical surface to be covered by the physical medical component; receiving user input via the medical component management system while the virtual medical component is rendered over the target location, the user input setting at least one of the pose and size of the virtual medical component within the image; and determining the physical size of the physical medical component by the medical component management system based on the user input and depth data from a depth map representing the internal space.

[0009] An exemplary method includes accessing image data via a medical component management system, the image data representing an image acquired by an imaging device and depicting the patient's internal space; obtaining depth data via the medical component management system, the depth data representing a depth map of the internal space depicted in the image acquired by the imaging device; identifying a target location within the image based on the image data and the depth data, the target location depicting an anatomical surface to be covered by a physical medical component; and instructing a display device by the medical component management system to render a virtual medical component representing the physical medical component above the identified target location within the image.

[0010] An exemplary non-transitory computer-readable medium storage instruction, which, when executed, instructs a processor of a computing device to instruct a display device to render a virtual medical element representing a physical medical element over a target location within an image of a patient's internal space, the target location depicting an anatomical surface to be covered by the physical medical element; receives user input while the virtual medical element is rendered over the target location, the user input setting at least one of the pose and size of the virtual medical element within the image; and determines the physical size of the physical medical element based on the user input and depth data of a depth map representing the internal space.

[0011] An exemplary non-transitory computer-readable medium storage instruction, which, when executed, instructs a processor of a computing device to access image data representing an image acquired by an imaging device and depicting the internal space of a patient; obtain depth data representing a depth map of the internal space depicted in the image acquired by the imaging device; identify a target location within the image based on the image data and the depth data, the target location depicting an anatomical surface to be covered by a physical medical element; and instruct a display device to render a virtual medical element representing the physical medical element above the identified target location within the image. Attached Figure Description

[0012] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements.

[0013] Figure 1 An exemplary medical component management system based on the principles described herein is illustrated.

[0014] Figure 2 An exemplary configuration based on the principles described herein is illustrated, wherein the system is configured to identify a target location within an image acquired by an imaging device, and the target location delineates an anatomical surface to be covered by a physical medical element.

[0015] Figure 3 An exemplary configuration based on the principles described herein is illustrated, wherein the imaging device includes a visible light camera and a depth sensor.

[0016] Figure 4 An exemplary configuration based on the principles described herein is illustrated, wherein the depth sensor is implemented by a visible light camera.

[0017] Figure 5 An exemplary configuration based on the principles described herein is shown, in which the system obtains depth data from a preoperative image source.

[0018] Figure 6 Exemplary images are shown that can be captured by an imaging device based on the principles described herein.

[0019] Figures 7-9 An exemplary virtual medical element is shown rendered above a target region identified within an image, based on the principles described herein.

[0020] Figures 10-11 An exemplary configuration based on the principles described herein is illustrated, in which the system generates physical size data.

[0021] Figure 12 An exemplary physical material is shown for cutting physical medical components according to the principles described herein.

[0022] Figures 13-14 An exemplary configuration based on the principles described herein is illustrated, in which the system determines where to place the boot parameter group.

[0023] Figures 15-16 An exemplary scenario is shown in which a physical medical element is placed on an anatomical surface comprising a portion of tissue with and without defects, in accordance with the principles described herein.

[0024] Figures 17-20 An exemplary method based on the principles described herein is illustrated.

[0025] Figure 21 An exemplary computer-assisted surgical system based on the principles described herein is illustrated.

[0026] Figure 22 An exemplary computing device based on the principles described herein is illustrated. Detailed Implementation

[0027] This document describes a system and method for determining the size of a physical medical component. As described herein, an exemplary medical component management system can instruct a display device to render a virtual medical component (e.g., a virtual mesh patch) representing a physical medical component (e.g., a mesh patch for a hernia) over a target location within an image of a patient's internal space, wherein the target location depicts the anatomical surface to be covered by the physical medical component. While the virtual medical component is rendered over the target location, the medical component management system can receive user input that sets at least one of the pose and size of the virtual medical component within the image. Based on the user input and depth data from a depth map representing the internal space, the medical component management system can determine the physical size of the physical medical component.

[0028] The systems and methods described herein advantageously facilitate the determination of optimal dimensions for physical medical components that will be introduced into the patient's body during a medical procedure and placed above anatomical surfaces within the body. Furthermore, by determining the physical size of the physical medical component based on depth data from a depth map representing the internal space, the systems and methods described herein advantageously take into account the peaks, valleys, contours, and / or other depth variations of the anatomical surface when determining the optimal size of the physical medical component. Such depth variations may not be visually apparent in images of the internal space and can affect the size of the physical medical component. For example, large variations in depth between multiple portions of the anatomical surface to be covered by the physical medical component may necessitate a physical medical component size larger than that visually identifiable in images of the internal space.

[0029] As an illustration, if a user manually adjusts the size of a virtual medical element displayed within an image depicting an anatomical surface so that the virtual medical element appears to cover the anatomical surface, the size of the virtual medical element may not actually correspond to the actual size required for the physical medical element, unless the depth data represents the three-dimensional contour of the anatomical surface. Therefore, the systems and methods described herein base the size of the anatomical surface on depth data that indicates such a contour.

[0030] The systems and methods described herein can also advantageously determine where virtual medical elements will be initially rendered within an image of the patient's interior space. This can facilitate the relatively rapid determination of the optimal dimensions of the physical medical elements represented by the virtual medical elements, as will be described in more detail herein.

[0031] These and other advantages and benefits of the systems and methods described in this paper will become apparent in this paper.

[0032] As used herein, a physical therapy element refers to any element that is foreign to the patient's body and is configured to be placed on and cover an anatomical surface within the patient's body. For example, a physical therapy element may be implemented by a patch (e.g., a mesh or wire patch) configured to cover tissue defects (e.g., hernias, incisions, or other types of injuries) within the patient's body. Other examples of physical therapy elements that may be used in combination with the systems and methods described herein include, but are not limited to, gauze, bandages, plates, prostheses (e.g., artificial intervertebral discs, joint implants (e.g., knee implants), etc.), sample collection bags, bone grafts, fasteners (e.g., clips, clamps, nails, etc.). Physical therapy elements may be placed on an anatomical surface in any suitable manner. For example, a physical therapy element may be sutured, anchored, or otherwise secured to an anatomical surface.

[0033] Figure 1An exemplary medical component management system 100 (“System 100”) configured to perform the various operations described herein is illustrated. As shown, System 100 may include, but is not limited to, storage facility 102 and processing facility 104 selectively and communicatively coupled to each other. Facilities 102 and 104 may each include or be implemented with hardware and / or software components (e.g., processor, memory, communication interface, instructions stored in memory for execution by the processor, etc.). For example, facilities 102 and / or 104 may be implemented by any component of a computer-assisted surgical system configured to perform medical procedures in which physical medical components are introduced into and placed on anatomical surfaces within the patient's body. As another example, facilities 102 and / or 104 may be implemented by a computing device separate from and communicatively coupled to the computer-assisted surgical system. In some examples, facilities 102 and 104 may be distributed among multiple devices and / or multiple locations that may serve a particular implementation.

[0034] Storage facility 102 may maintain (e.g., store) executable data used by processing facility 104 to perform one or more of the operations described herein. For example, storage facility 102 may store instructions 106 that can be executed by processing facility 104 to implement one or more of the operations described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 102 may also maintain any data received, generated, managed, used, and / or transmitted by processing facility 104.

[0035] Processing facility 104 can be configured to perform (e.g., implement instructions 106 stored in storage facility 102 to perform) the various operations described herein.

[0036] For example, processing facility 104 can be configured to instruct a display device to render a virtual medical element representing a physical medical element over a target location within an image of the patient's internal space, the target location depicting an anatomical surface to be covered by the physical medical element. While the virtual medical element is being rendered over the target location, user input is received, which sets at least one of the pose and size of the virtual medical element within the image, and the physical size of the physical medical element is determined based on the user input and depth data from a depth map representing the internal space. As described herein, the physical size can define the surface area of ​​the physical medical element.

[0037] As another example, processing facility 104 may be configured to access image data representing an image acquired by an imaging device and depicting the patient's internal space; obtain depth data representing a depth map of the internal space depicted in the image acquired by the imaging device; identify a target location within the image based on the image data and the depth data, the target location depicting an anatomical surface to be covered by a physical medical element; and instruct a display device to render a virtual medical element representing the physical medical element above the identified target location within the image.

[0038] As another example, processing facility 104 may be configured to acquire anatomical characteristic data and determine a set of placement guidance parameters based on the anatomical characteristic data, which represents characteristics associated with the anatomical surface to be covered by the physical medical element, and the anatomical surface within the patient's internal space. As described herein, the set of placement guidance parameters may include one or more parameters configured to guide the placement of the physical medical element on the anatomical surface using one or more surgical instruments controlled by a computer-assisted surgical system.

[0039] As another example, the processing facility 104 may be configured to acquire anatomical characteristic data representing characteristics associated with the anatomical surface to be covered by the physical medical element, the anatomical surface within the patient's internal space; determine a suture plan (or any other type of fixation plan) based on the anatomical characteristic data for suturing the physical medical element to the anatomical surface while one or more surgical instruments controlled by a computer-assisted surgical system hold the physical medical element in place on the anatomical surface; and graphically indicate the suture plan within an image of the internal space.

[0040] This document describes these and other operations that can be performed by system 100 (e.g., processing facility 104).

[0041] Figure 2 An exemplary configuration 200 is illustrated, in which system 100 is configured to identify a target location within an image acquired by an imaging device, the target location depicting an anatomical surface to be covered by a physical medical element. As shown, system 100 can access image data 202, which represents an image acquired by the imaging device and depicts the patient's internal space. System 100 can also obtain depth data 204, which represents a depth map of the internal space depicted in the image acquired by the imaging device. Based on image data 202 and depth data 204, system 100 can identify the target location within the image (which depicts an anatomical surface to be covered by a physical medical element) and output target location data 206 representing the identified target location.

[0042] The target location data 202 can be in any suitable format. For example, the target location data 202 can include two-dimensional or three-dimensional pixel coordinates representing pixels that depict the anatomical surface to be covered by physical medical components.

[0043] Exemplary methods for generating image data 202 and depth data 204 will now be described.

[0044] Figure 3 An exemplary configuration 300 is illustrated, wherein the imaging device 302 includes a visible light camera 304 configured to generate and output image data 202 and a depth sensor 306 configured to generate and output depth data 204.

[0045] Imaging device 302 may be implemented by an endoscope or other camera device configured to capture images of the scene. In some examples, imaging device 302 may be configured to be attached to and controlled by a computer-assisted surgical system. In alternative examples, imaging device 302 may be handheld and manually operated by an operator (e.g., a surgeon).

[0046] In some examples, the scene captured by imaging device 302 may include a surgical area associated with the patient. In some examples, the surgical area may be entirely located within the patient and may include the area within or near the location of a surgical procedure planned, being performed, or already performed within the patient. For example, for a minimally invasive surgical procedure performed on tissue within the patient, the surgical area may include the tissue, the anatomical structures beneath the tissue, and the space surrounding the tissue (e.g., the location of surgical instruments used to perform the surgical procedure). In some example embodiments, a surgical area entirely located within the patient may be referred to as “internal space.” As described herein, any internal anatomical structures of the patient (e.g., blood vessels, organs, and / or tissues) and / or surgical instruments located within the internal space may be referred to as objects and / or structures.

[0047] A visible light camera 304 (“camera 304”) is configured to generate image data 202 representing a two-dimensional visible light image of a scene. Camera 304 can be implemented by any suitable image sensor, such as a charge-coupled device (“CCD”) image sensor, a complementary metal-oxide-semiconductor (“CMOS”) image sensor, a hyperspectral camera, a multispectral camera, etc.

[0048] Depth sensor 306 can be implemented by any suitable sensor configured to generate depth data 204. For example, depth sensor 306 can be implemented by a time-of-flight sensor, structured light sensor, interferometer, hyperspectral camera, multispectral camera, and / or any other suitable sensor configured to acquire depth data, which may serve a particular implementation. When depth sensor 306 is implemented by a time-of-flight sensor, the time-of-flight sensor can be implemented by one or more photodetectors (e.g., one or more single-photon avalanche diode (“SPAD”) detectors), CCD sensor, CMOS sensor, and / or any other suitable configuration. Figure 3 In the example, the depth sensor 306 is separate from the camera 304 (i.e., physically different).

[0049] In configuration 300, system 100 can obtain image data 202 by instructing camera 304 to acquire image data 202 and receiving image data 202 from camera 304. Similarly, system 100 can obtain depth data 204 by instructing depth sensor 306 to acquire depth data 204 and receiving depth data 204 from depth sensor 306.

[0050] Therefore, in configuration 300, system 100 is communicatively coupled to imaging device 302 via bidirectional communication link 308 and to illumination system 310 via communication link 312. Communication links 308 and 312 can each be implemented using any suitable wired and / or wireless communication medium that can serve a particular implementation. As described herein, system 100 can use communication links 308 and 312 to direct camera 304 and depth sensor 306 to acquire and receive image data 202 and depth data 204.

[0051] The lighting system 310 can be configured to emit light 314 (e.g., in the direction of system 100) to illuminate a scene to be imaged by the imaging device 302. The light 314 emitted by the lighting system 310 may include visible and / or invisible light (e.g., infrared light). As shown, the light 314 can travel to the scene via the imaging device 302 (e.g., through an illumination channel within the imaging device 302 that may be implemented by one or more optical fibers, light guides, lenses, etc.).

[0052] As shown in the figure, the light 314 emitted by the illumination system 310 can be reflected from the surface 316 within the scene imaged by the imaging device 302. When the imaging device 302 is aimed at the patient's internal space, the surface 316 represents a surface within the internal space (e.g., an anatomical surface).

[0053] Camera 304 and depth sensor 306 can each detect reflected light 314. Camera 304 can be configured to generate image data 202 based on the detected light, which represents a two-dimensional visible light image of the scene including surface 316. Depth sensor 306 can be configured to generate depth data 204 based on the detected light. Image data 202 and depth data 204 can each have any suitable format.

[0054] To generate a stereoscopic image of the scene, system 100 can instruct lighting system 310 to emit light 314. System 100 can also activate (e.g., turn on) visible light camera 304 and depth sensor 306. Light 314 travels into the scene and is reflected from surface 316 (and, in some examples, one or more other surfaces in the scene). Both camera 304 and depth sensor 306 detect the reflected light 314.

[0055] Camera 304 (and / or other circuitry included in imaging device 302) can generate image data 202 representing a two-dimensional visible light image of the scene based on detected light 314. This can be performed in any suitable manner. Visible light camera 304 (and / or other circuitry included in imaging device 302) can transmit image data 202 to system 100. This can also be performed in any suitable manner.

[0056] Depth sensor 306 can generate depth data 204 representing a depth map of the scene (e.g., a depth map of surface 316) based on detected light 314. This can be performed in any suitable manner. For example, depth sensor 306 can measure the amount of time it takes for photons of light 314 to travel from illumination system 310 to depth sensor 306. Based on this amount of time, depth sensor 306 can determine the depth of surface 316 relative to the orientation of depth sensor 306. The data representing this depth can be represented in depth data 204 in any suitable manner. For example, the depth map represented by depth data 204 can include an array of depth values ​​(e.g., depth buffer (Z-buffer) values) corresponding to each pixel in the image.

[0057] Depth sensor 306 (and / or other circuitry included in imaging device 302) can transmit depth data 204 to system 100. This can be done in any suitable manner.

[0058] System 100 can receive image data 202 and depth data 204 and perform one or more processing operations on the image data 202 and depth data 204. For example, as will be described in more detail below, system 100 can generate target location data 206 based on image data 202 and depth data 204.

[0059] As another example, system 100 can generate a right-side perspective image of the scene and a left-side perspective image representing the scene based on image data 202 and depth data 204. This can be performed in any suitable manner. System 100 can then instruct a display device to simultaneously display the right-side and left-side perspective images in a manner that forms a stereoscopic image of the scene. In some examples, the display device is included in and / or communicatively coupled to the computer-assisted surgical system 204.

[0060] Figure 4 An exemplary configuration 400 is illustrated, wherein the depth sensor 402 is implemented by visible light cameras 304-1 and 304-2 included in the imaging device 302. In configuration 400, system 100 can obtain depth data 204 by instructing camera 304-1 to acquire a first image (e.g., a first two-dimensional image) of the patient's internal space, instructing camera 304-2 to acquire a second image (e.g., a second two-dimensional image) of the patient's internal space, and generating a depth map represented by depth data 204 based on the first and second images.

[0061] In Figure 4 In the figure, the first image acquired by camera 304-1 is represented by image data 202-1, and the second image acquired by camera 304-2 is represented by image data 202-2. As shown, image data 202-1 and 202-2 are transmitted to a depth data generator 402 implemented by system 100. The depth data generator 402 can use any visible image-based technique to determine depth data 204 based on image data 202-1 and 202-2.

[0062] Other configurations of the imaging device 302 are possible based on the systems and methods described herein. For example, the imaging device 302 may include multiple cameras 304 and / or multiple depth sensors 306. For illustration, the imaging device 302 may include two cameras 304 combined with a single depth sensor 306. In these embodiments, depth data may be generated based on images acquired by the two cameras 304. The depth data generated by the depth sensors 304 can be used to fine-tune or otherwise enhance the depth data generated based on the images acquired by the two cameras 304.

[0063] In some examples, system 100 can obtain depth data 204 by accessing a preoperative image registered to an image from a source different from imaging device 204. For example, Figure 5An exemplary configuration 500 is shown, in which system 100 obtains depth data 204 from a preoperative image source 502. Source 502 can be implemented by a computer-aided computed tomography (CT) scanner, magnetic resonance imaging (MRI) device, ultrasound device, three-dimensional scanning (LIDAR) device, and / or any other suitable alternative imaging device configured to generate preoperative images of the patient. The preoperative images can be registered with the images represented by image data 202, thereby providing depth data 204.

[0064] System 100 can identify target locations based on image data 202 and depth data 204 in any suitable manner. For example, based on image data 202 and depth data 204, system 100 can identify a location within an image represented by image data 202 that depicts tissue that needs to be covered by a physical medical element, and then designate the identified location as a target location.

[0065] System 100 can identify regions of an image that depict tissues that need to be covered by physical medical components in any suitable manner. For example, based on image data 202 and depth data 204, system 100 can segment the image (e.g., by classifying different parts of the image to correspond to different types of tissues) and identify regions based on the segmentation.

[0066] Additionally or alternatively, system 100 can identify the location of tissue that needs to be covered by physical medical components by inputting image data 202 and depth data 204 into a machine learning model configured to identify tissue abnormalities. The machine learning model can be trained and / or used in any suitable manner.

[0067] In some examples, system 100 can determine a stage within a surgical procedure being performed on a patient, and further based on the identification of locations depicting tissue that need to be covered by physical medical components at the determined stage. For example, system 100 can receive input from a surgeon attempting to locate tissue defects within a patient by scanning the patient's internal space with an imaging device. Based on this input, system 100 can initiate a location identification heuristic that uses image data 202 and depth data 204 to automatically identify locations within an image acquired by the imaging device that depict tissue that needs to be covered by physical medical components.

[0068] Once the target location of the anatomical surface to be covered by the physical medical element is identified, the system 100 can instruct the display device to render a virtual medical element representing the physical medical element above the identified target location within the image. This can be done in any suitable manner.

[0069] For example, Figure 6An exemplary image 602 is shown, which can be captured by an imaging device targeting the internal space of a patient and can be represented by image data 202. As shown, image 602 depicts a tissue defect 604 surrounded by non-defective tissue 606. Tissue defect 604 can be a hernia, incision, or other type of injury. Non-defective tissue 606 can include, for example, healthy tissue unaffected by tissue defect 604.

[0070] System 100 can identify target locations within image 602 based on image data 202 and / or depth data 204, which delineate an anatomical surface to be covered by a physical medical element. This can be performed in any of the manner described herein. In some examples, the anatomical surface to be covered by the physical medical element includes at least a portion of tissue defect 604. The anatomical surface to be covered by the physical medical element may also include at least a portion of non-defective tissue 606 in some cases. For example, the anatomical surface to be covered by the physical medical element may include the entire tissue defect 604 and an overlapping location consisting of non-defective tissue 606 surrounding the tissue defect 604. The overlapping location may have any suitable width (e.g., between one and five centimeters) and can be used to attach the physical medical element to the anatomical surface as described herein.

[0071] Tissue defect 604 and non-defective tissue 606 can have different relative depths. For example, tissue defect 604 and the non-defective tissue surrounding tissue defect 604 can have various ridges, peaks, valleys, contours, and / or other uneven surfaces. However, such depth variations may not be visually discernible within image 602, especially if image 602 is two-dimensional. Therefore, as described herein, system 100 can take this depth variation into account when determining the physical size of a physical medical element that will cover the overlapping area consisting of tissue defect 604 and the non-defective tissue 606 surrounding tissue defect 604.

[0072] System 100 can instruct a display device to render a virtual medical element representing a physical medical element above a target area identified within image 602. For example, Figure 7 An exemplary virtual medical element 702 is shown rendered above a target region identified within image 602. Figure 7 In the example, the virtual medical element 702 includes a dashed line and a location enclosed by the dashed line. Furthermore, in... Figure 7 In the example, the identified target location directly corresponds to the virtual medical element 702 (i.e., is completely covered by the virtual medical element 702).

[0073] Although the virtual medical element 702 is illustrated as a dashed rectangle positioned above a portion of the tissue defect 604 and the non-defective tissue 606, it will be appreciated that the virtual medical element 702 may alternatively be rendered in any other suitable manner. For example, when the virtual medical element 702 is rendered above the target site, the virtual medical element 702 may be at least partially transparent to allow the user to visualize the target site and / or the tissue defect 604.

[0074] When the virtual medical element 702 is rendered over the target area, the user can provide user input to set the orientation and / or size of the virtual medical element 702. For example, if the user determines that the initially determined orientation and size of the virtual medical element 702 are satisfactory (e.g., if the virtual medical element 702 adequately covers the depiction of the tissue defect 604), the user can provide user input confirming that the orientation and size of the virtual medical element 702 are correct. Such user input can be provided in any suitable manner. For example, such user input can be provided by the user selecting an option displayed in image 602, by the user selecting a user input button on a component of the computer-assisted surgical system, by the user providing a verbal command, and / or in any other manner.

[0075] In some cases, the user may determine that the initially determined pose and / or size of the virtual medical element 702 needs further refinement. For example, the user may wish to zoom in or out of the virtual medical element 702, reposition one or more edges of the virtual medical element 702, and / or otherwise adjust the pose and / or size of the virtual medical element 702. In these cases, user input for setting the pose and / or size of the virtual medical element 702 may include one or more user input commands for adjusting the pose and / or size of the virtual medical element 702. Such user input may be provided in any suitable manner. For example, user input may be provided through user interaction with a virtual handle displayed as part of the virtual medical element 702, through a graphical user interface including image 602, through one or more keyboard or other input device commands, and / or in any other suitable manner. In response to receiving user input adjusting the pose and / or size of the virtual medical element 702, system 100 may dynamically adjust the rendering of the virtual medical element 702 substantially in real time to depict at least one of the pose and size of the virtual medical element adjusted by the user input.

[0076] To illustrate, Figure 8 The image shows the virtual medical element 702 after the user has adjusted its shape (and therefore its size). As shown in the figure... Figure 8 The shape and size of the virtual medical element 702 shown are different from the virtual medical element 702 before the user adjusted it. Figure 7 It more closely matches the actual shape and size of tissue defect 604.

[0077] In some examples, the initial shape of the virtual medical element 702 is a standard size used for a specific type of physical medical element. For example, the initial shape of the virtual medical element 702 could be rectangular, such as... Figure 7 As shown, this is used for physical medical components that are typically rectangular in shape.

[0078] Alternatively, the initial shape of the virtual medical element 702 can more closely conform to the actual shape of the anatomical surface that will be covered by the physical medical element. For example, Figure 9 An exemplary embodiment is shown, wherein the virtual medical element 702 has a contour shape that follows the outer edge of the tissue defect 604.

[0079] The initial shape of the virtual medical element 702 can be automatically determined based on one or more attributes of the physical medical element that will cover the anatomical surface. Alternatively or additionally, the initial shape of the virtual medical element 702 can be specified by the user by providing one or more user input commands.

[0080] Based on user input setting the posture and / or size of the virtual medical element 702 and depth data 204, system 100 can determine the physical size of the physical medical element that will cover the anatomical surface. For example, Figure 10 An exemplary configuration 1000 is illustrated, in which system 100 accesses virtual data 1002 and depth data 204 and generates physical size data 1004 based on the virtual data 1002 and depth data 204. Virtual data 1002 may represent the pose, size, and / or positioning of a virtual medical element 702 set by the user. As explained herein, depth data 204 represents a depth map of the interior space depicted in image 602. Physical size data 1004 represents the physical size of the physical medical element determined by system 100. As described herein, physical size may represent the surface area of ​​the physical medical element.

[0081] System 100 can generate physical size data 1004 based on virtual data 1002 and depth data 204 in any suitable manner. For example, system 100 can identify multiple pixels within image 602 covered by virtual medical element 702 based on the pose and size of the virtual medical element represented by virtual data 1002. System 100 can determine two-dimensional pixel regions of the multiple pixels. System 100 can determine the depth value of each of the multiple pixels covered by virtual medical element 702 based on depth data 204. Based on the two-dimensional pixel regions and depth values, system 100 can determine the surface area and / or other size of the anatomical surface to be covered by the physical medical element. System 100 can determine the physical size of the physical medical element based on the surface area of ​​the anatomical surface. Each of these operations can be performed in any suitable manner.

[0082] As described above, the systems and methods described herein can be used to determine the physical size of a mesh patch used to cover a hernia. As another example, a virtual medical element 702 may represent a fastener (e.g., a clip or nail) and be placed on an image of two tissue slices that will be fastened together. Based on the size determination and / or orientation changes of the virtual medical element 702, system 100 can determine the physical size of the actual fastener to be used to fasten the two tissue slices. As another example, the virtual medical element 702 may represent bone slurry or another type of material to be used to fill voids in bone. Based on the size determination and / or orientation changes of the virtual medical element 702, system 100 can determine the physical size (e.g., volume) of the actual bone slurry to be used to fill the voids. These examples illustrate just the many different ways system 100 can determine the physical size of a medical element to be introduced into a patient.

[0083] Due to the potential depth variations on an anatomical surface, the surface area of ​​a physical medical element, defined by the physical size of the virtual medical element 702, may in some cases be larger than the physical area represented by a two-dimensional pixel region. For example, the physical medical element may be made of a material configured to "shrink and wrap" or otherwise adhere to all surface areas of the anatomical surface. In this example, if the anatomical surface has protruding ridges that need to be covered by the physical medical element, this depth variation could result in the surface area defined by the physical size of the physical medical element, as determined by system 100, being larger than the physical area represented by a two-dimensional pixel region of the virtual medical element 702.

[0084] Figure 11An exemplary configuration 1100 is illustrated, wherein system 100 further determines the generation of physical size data 1004 based on deformation data 1102 in addition to virtual data 1002 and depth data 204. Deformation data 1102 represents a deformation model of a physical medical element. The deformation model may indicate one or more physical properties of the physical medical element that affect its ability to deform (e.g., stretch, compress, etc.) in response to forces applied to it. For example, the deformation model may indicate the type of material used to manufacture the physical medical element, the tensile strength of the material, and / or any other measure representing deformation that may serve a particular implementation.

[0085] System 100 can determine the physical size of the physical medical element based on deformation data 1102 in any suitable manner. For example, if deformation data 1102 indicates that the physical medical element is relatively tensile, system 100 can specify that the physical size will be relatively smaller than that of different types of physical medical elements exhibiting high tensile strength.

[0086] System 100 can acquire the deformation data 1102 in any suitable manner. For example, system 100 can maintain or access a database of deformation data 1102 that includes various types of physical medical components. System 100 can automatically determine which physical medical component will be used and / or which specific type of surgical procedure is being performed based on user input indicating which physical medical component will be used, tracking trends of a specific user, and / or in any other suitable manner, to confirm which specific physical medical component will be used to cover the anatomical surface.

[0087] System 100 can output data representing physical size in any suitable manner. For example, system 100 can output data representing physical size by displaying the physical size within image 602 or within any other suitable user interface. Additionally or alternatively, system 100 can output data representing physical size by outputting physical measurements of the physical medical element (e.g., width and length in centimeters). Additionally or alternatively, system 100 can output data representing physical size by outputting data representing a pattern that will be used to cut the physical medical element from the material.

[0088] In some examples, system 100 can project a virtual medical element 702 onto a physical material from which the physical medical element will be cut. In this way, the virtual medical element 702 can guide the user to cut the physical medical element from the physical material.

[0089] To illustrate, Figure 12 An exemplary physical material 1202 from which a physical medical component will be cut is shown. Figure 12In the example, physical material 1202 is a mesh material from which patches for hernias and / or other types of tissue defects can be cut. As shown, system 100 has projected virtual medical element 702 onto physical material 1202. Virtual medical element 702 can be projected onto physical material 1202 in any suitable manner.

[0090] Once ready to introduce a physical medical device into a patient, system 100 can be configured to provide placement guidance, which is configured to guide and / or assist the user in placing the physical medical device on an anatomical surface. This placement guidance can be provided in a variety of different ways.

[0091] For example, Figure 13 An exemplary configuration 1300 is illustrated, wherein system 100 obtains anatomical characteristic data 1302 representing characteristics associated with an anatomical surface to be covered by a physical medical element, and determines a placement guidance parameter set 1304 based on the anatomical characteristic data 1302. The placement guidance parameter set 1304 includes one or more parameters configured to guide the placement of the physical medical element on the anatomical surface using one or more surgical instruments controlled by a computer-assisted surgical system.

[0092] Anatomical characteristic data 1302 may represent one or more characteristics associated with the anatomical surface to be covered by the physical medical element. For example, anatomical characteristic data 1302 may represent the tissue type constituting the anatomical surface, the size of the anatomical surface, and / or the location of the anatomical surface within the patient's body. These characteristics may affect how the physical medical element is placed on the anatomical surface. For example, if the anatomical surface is near an organ, system 100 may generate one or more parameters to be included in placement guidance parameter set 1304, which indicate that the anatomical surface is near an organ and can be used to identify a suture plan for attaching the physical medical element to the anatomical surface in a manner that does not damage or otherwise affect the organ. As another example, if the tissue constituting the anatomical surface is relatively weak, system 100 may generate one or more parameters to be included in placement guidance parameter set 1304, which may be used to increase the relative number of sutures used to attach the physical medical element to the anatomical surface.

[0093] System 100 can acquire anatomical characteristic data 1302 in any suitable manner. For example, system 100 can acquire anatomical characteristic data 1302 by accessing image data 202, acquiring depth data 204, and determining anatomical characteristic data 1302 based on image data 202 and depth data 204. For illustration, image data 202 and depth data 204 can be used to determine the location, size, and / or any other characteristics of the anatomical surface.

[0094] In some examples, system 100 can use image data 202 and depth data 204 to segment an image represented by image data 202. Segmentation may include classifying different parts of the image into corresponding items (e.g., tissue type). Based on the segmentation, system 100 may determine one or more characteristics of an anatomical surface, such as tissue type.

[0095] Figure 14 An exemplary configuration 1400 is illustrated, in which system 100 obtains component characteristic data 1402 in addition to anatomical characteristic data 1302. As shown, in configuration 1400, system 100 generates a placement guide parameter set 1304 based on both anatomical characteristic data 1302 and component characteristic data 1402. In an alternative example, system 100 generates the placement guide parameter set 1304 based solely on component characteristic data 1402 (and not on anatomical characteristic data 1302).

[0096] Component characteristic data 1402 represents one or more characteristics associated with a physical medical component that will cover an anatomical surface. For example, component characteristic data 1402 may represent the physical size of the physical medical component, the type of material used for the physical medical component, the tensile strength of the physical medical component, and / or the deformation characteristics of the physical medical component.

[0097] System 100 can access component characteristic data 1402 in any suitable manner. For example, system 100 can maintain or access a database that includes component characteristic data 1402. As another example, system 100 can access component characteristic data 1402 by receiving user input indicating access to component characteristic data 1402.

[0098] In some examples, the placement guidance parameter group 1304 includes one or more parameters configured to specify the optimal orientation of the physical medical element when it is placed on an anatomical surface, wherein one or more surgical instruments are controlled by a computer-assisted surgical system. In these examples, system 100 may instruct the user on the optimal orientation of the physical medical element in any suitable manner.

[0099] For example, system 100 can be configured to indicate the optimal orientation of a physical medical component to a user by graphically indicating the optimal orientation within an image of the interior space (e.g., image 602). For illustration, Figure 15 An exemplary scenario is shown in which a physical medical element 1502 is placed on an anatomical surface comprising a portion of tissue defect 604 and non-defective tissue 606.

[0100] As shown in the figure, a physical medical element 1502 is placed on an anatomical surface by surgical instruments 1504-1 and 1504-2 (“surgical instruments 1504”). Surgical instruments 1504 may include any suitable grasping tool configured to hold and guide the physical medical element 1502 into place. In some examples, surgical instruments 1504 are controlled by a computer-aided surgical system (e.g., in response to user input commands provided by a surgeon or other user).

[0101] As shown in the figure, system 100 can render a virtual medical element 702 within image 602 while a user is using surgical instruments 1504 to place a physical medical element 1502 on an anatomical surface. In this configuration, the virtual medical element 702 graphically indicates the optimal orientation of the physical medical element. Furthermore, graphical arrows 1506-1 and 1506-2 can be rendered by system 100 within image 602 to indicate the direction in which the user should rotate the physical medical element 1502 to achieve the optimal orientation. System 100 may additionally or alternatively present any other suitable virtual, auditory, and / or graphical aids that may serve a particular implementation and indicate the optimal orientation of the physical medical element 1502.

[0102] In some examples, the placement guide parameter group 1304 includes one or more parameters configured to define an optimal insertion path within the internal space, which the surgical instrument 1504 will follow when the physical medical element 1502 is brought from outside the patient to contact the anatomical surface. In these examples, the system 100 may indicate the optimal insertion path of the physical medical element 1502 to the user in any suitable manner. For example, the system 100 may be configured to indicate the optimal insertion path of the physical medical element graphically within an image of the internal space (e.g., image 602). The system 100 may additionally or alternatively present any other suitable virtual, auditory, and / or graphical aids that may serve a particular implementation to indicate the optimal insertion path.

[0103] The optimal insertion path, defined by one or more parameters in the placement guide parameter group 1304, can be configured such that the surgical instrument 1504 and / or physical medical element 1502 avoid collisions with tissues and / or other objects (e.g., other surgical instruments) in the internal space when the physical medical element 1502 is brought from outside the patient to contact the anatomical surface.

[0104] In some examples, the placement guide parameter group 1304 includes one or more parameters configured to specify a suture plan for suturing the physical medical element 1502 to the anatomical surface when the surgical instrument 1504 holds the physical medical element 1502 in place on the anatomical surface. The suture plan may include information specifying the suture pattern to be used to suture the physical medical element 1502 to the anatomical surface, the spacing between the sutures used to suture the physical medical element 1502 to the anatomical surface, the type of thread to be used to suture the physical medical element 1502 to the anatomical surface, the length of thread required to suture the physical medical element 1502 to the anatomical surface, and / or any other aspect of the suture plan that may serve a particular implementation. While exemplary suture plans have been described herein, it should be appreciated that the placement guide parameter group 1304 may alternatively include one or more parameters configured to specify any other type of fixation plan that may be used to secure the physical medical element 1502 to the anatomical surface.

[0105] For example, an exemplary suture plan that can be defined by one or more parameters in the placement guide parameter set 1304 may indicate a recommended suture pattern, suture spacing, thread type, and / or thread length for a specific type of tissue as indicated in anatomical characteristic data 1302, a specific type and / or size of tissue defect 604 as indicated in anatomical characteristic data 1302, the proximity of other objects (e.g., organs) to tissue defect 604 as indicated in anatomical characteristic data 1302, and / or any other characteristic of the anatomical surface as indicated in anatomical characteristic data 1302. Additionally or alternatively, an exemplary suture plan that can be defined by one or more parameters in the placement guide parameter set 1304 may indicate a recommended suture pattern, suture spacing, thread type, and / or thread length for one or more of the characteristics of the physical medical component 1502 as indicated in component characteristic data 1402.

[0106] In some examples, system 100 can be configured to graphically indicate the suture plan within image 602. For example, Figure 16A physical medical element 1502 is shown before being sutured to an anatomical surface. System 100 can graphically indicate the suture plan within image 602 by presenting virtual markers (e.g., virtual markers 1602-1 to 1602-5) that indicate the locations where sutures will be placed to attach the physical medical element 1502 to the anatomical surface. The orientation and spacing of each virtual marker 1602 can be determined based on anatomical characteristic data 1302 and / or element characteristic data 1402. For example, most of the virtual markers 1602 are positioned at approximately the same distance from the outer edge of the physical medical element 1502. However, in this example, virtual marker 1602-2 is offset relative to its adjacent virtual markers 1602-1 and 1602-3 (i.e., virtual marker 1602-2 is farther from the edge of the physical medical element 1502 than virtual markers 1602-1 and 1602-3). This may be because the anatomical characteristic data 1302 may indicate that the tissue near the location of the physical medical element 1502 corresponding to the virtual mark 1602-2 is relatively weak, thus requiring a larger gap between the suture location and the edge of the physical medical element 1502.

[0107] In some examples, a user may provide user input configured to modify a suture plan graphically indicated within image 602. For example, a user may provide input to adjust the orientation of one or more virtual markers 1602, remove a specific virtual marker 1602, add a new virtual marker, and / or otherwise modify the suture plan. In response to this user input, system 100 may dynamically adjust the suture plan based on the user's repositioning of a specific virtual marker 1602. For example, based on the user's repositioning of a specific virtual marker 1602, system 100 may update the amount of suture required to perform the suture, adjust the positioning of the suture device and / or one or more other surgical instruments, and / or otherwise adjust the suture plan.

[0108] Figure 16 Also shown is a suturing device 1604 positioned on the physical medical element 1502 at locations corresponding to specific virtual markers 1602-4. The suturing device 1604 can be configured to suture the physical medical element 1502 to an anatomical surface in any suitable manner. For example, the suturing device 1604 can apply continuous sutures (or any other type of suture) around the periphery of the physical medical element 1502 at various locations corresponding to the virtual markers 1602.

[0109] In some examples, the suture device 1604 is controlled by a computer-assisted surgical system (e.g., via a manipulator arm connected to the computer-assisted surgical system). In these examples, the suture device 1604 may be referred to as some type of surgical instrument coupled to and controlled by the computer-assisted surgical system. In alternative examples, the suture device 1604 is not controlled by a computer-assisted surgical system. In these alternative examples, the suture device 1604 may be manually held and / or otherwise controlled by a user.

[0110] When the suturing device 1604 is controlled by a computer-assisted surgical system, the positioning and / or operation of the suturing device 1604 can be set in response to user input (e.g., the user can provide input commands to move and / or operate the suturing device 1604). For example, the user can provide input commands to instruct the computer-assisted surgical system to move the suturing device 1604 from a suturing position to a suturing position guided by a suturing plan, such as indicated by virtual markers 1602. For example, after the suturing device 1604 has been used to suture the physical medical element 1502 to the anatomical surface at suturing positions corresponding to virtual markers 1602-4, the user can provide input commands to cause the suturing device 1604 to move to a suturing position corresponding to virtual markers 1602-5 (e.g., by manipulating the master controls as part of the computer-assisted surgical system). Once at that position, the suturing device 1604 can be used to suture the physical medical element 1502 to the anatomical surface at the suturing positions corresponding to virtual markers 1602-5. This repositioning of the suture device 1604 can be performed automatically by a computer-aided surgical system without requiring specific user input to control the positioning and / or operation of the suture device 1604.

[0111] In some examples, system 100 can instruct a computer-assisted surgical system, based on placement guidance parameter set 1304, to automatically place a physical medical element (e.g., physical medical element 1502) onto an anatomical surface using one or more surgical instruments (e.g., surgical instrument 1504 and / or suture device 1604), without requiring user input to control the movement of one or more surgical instruments while the physical medical element is being placed onto the anatomical surface. For example, system 100 can instruct the computer-assisted surgical system to use surgical instrument 1504 to guide the physical medical element 1502 to the correct orientation and positioning above the anatomical surface. System 100 can then instruct the computer-assisted surgical system to use suture device 1604 to automatically suture the physical medical element 1502 to the anatomical surface.

[0112] In some examples, system 100 may track the relative posture of suturing device 1604 and / or surgical instrument 1504 relative to physical medical element 1502. System 100 may use the tracked posture to guide suturing device 1604 to properly perform suturing of physical medical element 1502 to anatomical surfaces and / or to guide surgical instrument 1504 to properly grasp and hold physical medical element 1502.

[0113] Figure 17 An exemplary method 1700 that can be performed by a medical component management system (e.g., system 100 and / or any implementation thereof) is illustrated. Although Figure 17 The illustration shows an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, and / or modify it. Figure 17 Any of the operations shown.

[0114] In operation 1702, the medical component management system instructs the display device to render a virtual medical component representing a physical medical component above a target location within an image of the patient's internal space, the target location depicting the anatomical surface to be covered by the physical medical component. Operation 1702 can be performed in any of the manner described herein.

[0115] In operation 1704, when the virtual medical element is rendered above the target location, the medical element management system receives user input that sets at least one of the pose and size of the virtual medical element within the image. Operation 1704 can be performed in any of the ways described herein.

[0116] In Operation 1706, the medical component management system determines the physical size of the physical medical component based on user input and depth data from a depth map representing the internal space. In some examples, the physical size may specify the surface area, volume, or any other suitable size of the physical medical component. Operation 1706 can be performed in any of the manner described herein.

[0117] Figure 18 Another exemplary method 1800 that can be performed by a medical component management system (e.g., system 100 and / or any implementation thereof) is illustrated. Although Figure 18 The illustration shows an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, and / or modify it. Figure 18 Any operation shown.

[0118] In operation 1802, the medical component management system accesses image data representing images acquired by the imaging device and depicting the patient's internal space. Operation 1802 can be performed in any of the manner described herein.

[0119] In operation 1804, the medical component management system acquires depth data, which represents a depth map of the interior space depicted in an image acquired by the imaging device. Operation 1804 can be performed in any of the manner described herein.

[0120] In operation 1806, the medical component management system identifies a target location within an image based on image data and depth data, which delineates the anatomical surface to be covered by a physical medical component. Operation 1806 can be performed in any of the manner described herein.

[0121] In operation 1808, the medical component management system instructs the display device to render a virtual medical component representing the physical medical component above the target area identified within the image. Operation 1808 can be performed in any of the manner described herein.

[0122] Figure 19 Another exemplary method 1900 that can be performed by a medical component management system (e.g., system 100 and / or any implementation thereof) is illustrated. Although Figure 19 The illustration shows an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, and / or modify it. Figure 19 Any operation shown.

[0123] In operation 1902, the medical component management system obtains anatomical characteristic data representing the characteristics associated with an anatomical surface to be covered by a physical medical component, which is located within the patient's internal space. Operation 1902 can be performed in any of the manner described herein.

[0124] In operation 1904, the medical device management system determines a placement guidance parameter set based on anatomical characteristic data. This placement guidance parameter set includes one or more parameters configured to guide the placement of a physical medical device on an anatomical surface using one or more surgical instruments controlled by a computer-assisted surgical system. Operation 1904 can be performed in any of the manner described herein.

[0125] Figure 20 Another exemplary method 2000 that can be performed by a medical component management system (e.g., system 100 and / or any implementation thereof) is illustrated. Although Figure 20 The illustration shows an exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, and / or modify it. Figure 20 Any of the operations shown.

[0126] In operation 2002, the medical component management system obtains anatomical characteristic data representing the characteristics associated with an anatomical surface to be covered by a physical medical component within the patient's internal space. Operation 2002 can be performed in any of the manner described herein.

[0127] In Operation 2004, as one or more surgical instruments controlled by a computer-assisted surgical system hold a physical medical element in place on an anatomical surface, the medical element management system determines a suturing plan for suturing the physical medical element to the anatomical surface based on anatomical characteristic data. Operation 2004 can be performed in any of the manner described herein.

[0128] In Operation 2006, the medical component management system graphically indicates the suture plan within an image of the interior space. Operation 2006 can be performed in any of the methods described herein.

[0129] The systems and methods described herein can be used in combination with and / or implemented by computer-aided surgical systems used to perform surgical procedures on patients. Figure 21 An exemplary computer-assisted surgical system 2100 (“surgical system 2100”) is illustrated. As shown, the surgical system 2100 may include a control system 2102, a user control system 2104, and an auxiliary system 2106 that are communicatively coupled to each other. The surgical system 2100 can be used by a surgical team to perform computer-assisted surgical procedures on a patient 2108. As shown, the surgical team may include a surgeon 2110-1, an assistant 2110-2, a nurse 2110-3, and an anesthesiologist 2110-4, all of whom may be collectively referred to as “surgical team members 2110”. Additional or alternative surgical team members, which may serve a particular implementation, may be present during the surgical procedure.

[0130] although Figure 21 The illustration depicts a minimally invasive surgical procedure in progress; however, it should be understood that the surgical system 2100 can be similarly used to perform open surgical procedures or other types of surgical procedures that can similarly benefit from the accuracy and convenience of the surgical system 2100. Furthermore, it should be understood that the entire surgical timeframe during which the surgical system 2100 can be used includes not only the operational timeframe of the surgical procedure, such as... Figure 21 As shown, this may also include preoperative, postoperative, and / or other appropriate periods for surgical procedures. Surgical procedures may include any procedure that uses manual and / or instrumental techniques on the patient to investigate or treat the patient's physical condition.

[0131] like Figure 21As shown, the manipulation system 2102 may include a plurality of manipulator arms 2112 (e.g., manipulator arms 2112-1 to 2112-4), and a plurality of surgical instruments may be coupled to the plurality of manipulator arms 2112. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool with tissue interaction capabilities), medical tool, imaging device (e.g., an endoscope), sensing device (e.g., a force-sensing surgical instrument), diagnostic instrument, etc., that can be used for computer-aided surgical procedures on patient 2108 (e.g., by being at least partially inserted into and manipulated to perform computer-aided surgical procedures on patient 2108). Although the manipulation system 2102 is depicted and described herein as including four manipulator arms 2112, it will be appreciated that the manipulation system 2102 may include only a single manipulator arm 2112 or any other number of manipulator arms that may serve a particular embodiment.

[0132] The manipulator arm 2112 and / or the surgical instruments attached to the manipulator arm 2112 may include one or more displacement transducers, orientation sensors, and / or positioning sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 2100 may be configured to use kinematic information to track (e.g., determine its orientation) and / or control the surgical instruments.

[0133] User control system 2104 can be configured to facilitate surgeon 2110-1's control of manipulator arm 2112 and surgical instruments attached to manipulator arm 2112. For example, surgeon 2110-1 can interact with user control system 2104 to remotely move or manipulate manipulator arm 2112 and surgical instruments. To this end, user control system 2104 can provide surgeon 2110-1 with images (e.g., high-resolution 3D images) of the surgical area associated with patient 2108 captured by an imaging system (e.g., any medical imaging system described herein). In some examples, user control system 2104 may include a stereoscopic viewer with two displays, wherein a stereoscopic image of the surgical area associated with patient 2108 and generated by a stereoscopic imaging system can be viewed by surgeon 2110-1. Surgeon 2110-1 can utilize the images to perform one or more procedures, wherein one or more surgical instruments are attached to manipulator arm 2112.

[0134] To facilitate control of surgical instruments, the user control system 2104 may include a set of master controls. These master controls can be manipulated by the surgeon 2110-1 to control the movement of the surgical instruments (e.g., by utilizing robotic and / or teleoperation technologies). The master controls can be configured to detect various hand, wrist, and finger movements of the surgeon 2110-1. In this way, the surgeon 2110-1 can intuitively perform procedures using one or more surgical instruments. In some examples, the user control system 2104 implements user control system 806.

[0135] The auxiliary system 2106 may include one or more computing devices configured to perform primary processing operations of the surgical system 2100. In this configuration, the one or more computing devices included in the auxiliary system 2106 may control and / or coordinate operations performed by various other components of the surgical system 2100, such as the manipulation system 2102 and the user control system 2104. For example, the computing device included in the user control system 2104 may transmit instructions to the manipulation system 2102 via one or more computing devices included in the auxiliary system 2106. As another example, the auxiliary system 2106 may receive and process image data representing images captured by an imaging device attached to one of the manipulator arms 2112 from the manipulation system 2102.

[0136] In some examples, the assistive system 2106 may be configured to present visual content to a surgical team member 2110 who may not have access to the images provided to the surgeon 2110-1 at the user control system 2104. For this purpose, the assistive system 2106 may include a display monitor 2114 configured to display one or more user interfaces, such as images of the surgical area (e.g., 2D images, 3D images), information associated with the patient 2108 and / or surgical procedures, and / or any other visual content that may serve a particular implementation. For example, the display monitor 2114 may display an image of the surgical area along with additional content displayed concurrently with the image (e.g., graphical content, contextual information, etc.). In some embodiments, the display monitor 2114 is implemented by a touchscreen display that the surgical team member 2110 can interact with (e.g., via touch gestures) to provide user input to the surgical system 2100.

[0137] The operating system 2102, the user control system 2104, and the auxiliary system 2106 can be communicatively coupled to each other in any suitable manner. For example, such as Figure 21As shown, the operating system 2102, the user control system 2104, and the auxiliary system 2106 can be communicatively coupled via a control line 2116, which can represent any wired or wireless communication link that can serve a particular implementation. Therefore, the operating system 2102, the user control system 2104, and the auxiliary system 2106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0138] In some examples, a non-transitory computer-readable medium may be provided for storing computer-readable instructions, based on the principles described herein. When executed by a processor of a computing device, the instructions may direct the processor and / or the computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0139] As used herein, a non-transitory computer-readable medium can include any non-transitory storage medium that contributes to providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by the processor of the computing device). For example, a non-transitory computer-readable medium can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0140] Figure 22 An exemplary computing device 2200 is illustrated, which may be specifically configured to perform one or more of the processes described herein. Any systems, computing devices, and / or other components described herein may be implemented by computing device 2200.

[0141] like Figure 22 As shown, computing device 2200 may include a communication interface 2202, a processor 2204, a storage device 2206, and an input / output (“I / O”) module 2208 that are communicatively connected to each other via communication infrastructure 2210. Although the exemplary computing device 2200... Figure 22 As shown in the text, but Figure 22 The components illustrated are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 22 The components of the computing device 2200 shown.

[0142] Communication interface 2202 can be configured to communicate with one or more computing devices. Examples of communication interface 2202 include, but are not limited to, wired network interfaces (e.g., network interface cards), wireless network interfaces (e.g., wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0143] Processor 2204 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 2204 can perform operations by executing computer-executable instructions 2212 (e.g., application programs, software, code, and / or other executable data instances) stored in storage device 2206.

[0144] Storage device 2206 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 2206 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including the data described herein) may be stored temporarily and / or permanently in storage device 2206. For example, data representing computer-executable instructions 2212 configured to direct processor 2204 to perform any of the operations described herein may be stored within storage device 2206. In some examples, data may be arranged in one or more databases residing within storage device 2206.

[0145] I / O module 2208 may include one or more I / O modules configured to receive user input and provide user output. I / O module 2208 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 2208 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0146] I / O module 2208 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 2208 is configured to provide graphical data to the display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0147] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made to the various exemplary embodiments, and additional embodiments can be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be considered illustrative rather than restrictive.

Claims

1. A system comprising: Non-temporary memory for storing instructions; as well as A processor, communicatively coupled to the memory and configured to execute the instructions to: Access image data, which represents images acquired by an imaging device targeting the patient's internal space; Obtain depth data, which represents a depth map of the interior space; The display device renders a virtual medical element above a target location within the image of the patient's internal space based on the image data and the depth data. The virtual medical element represents a physical medical element that will be introduced into the patient's body. The target location depicts an anatomical surface that will be covered by the physical medical element when it is introduced into the patient's body. When the virtual medical element is rendered above the target area, user input is received, wherein the user input sets at least one of the pose of the virtual medical element within the image or the size of the virtual medical element; and Based on the user input and the depth data representing the depth map of the internal space, the physical size is determined in order to determine the size of the physical medical device to be introduced into the patient's body.

2. The system according to claim 1, wherein: The processor is also configured to execute the instructions to access deformation data representing a deformation model of the physical medical component; and The determination of the physical size of the physical medical component is further based on the deformation data.

3. The system of claim 1, wherein determining the physical size of the physical medical element comprises: Based on the pose and size of the virtual medical element, identify multiple pixels within the image covered by the virtual medical element; Determine the two-dimensional pixel region of the plurality of pixels; The depth value of each of the plurality of pixels is determined based on the depth data; Based on the two-dimensional pixel region and the depth value, determine the surface area of ​​the anatomical surface to be covered by the physical medical element; as well as The physical size is determined based on the surface area of ​​the anatomical surface.

4. The system of claim 3, wherein the surface area of ​​the physical medical element defined by the physical size is greater than the physical area represented by the two-dimensional pixel region.

5. The system of claim 1, wherein the processor is further configured to execute the instructions to output data representing the physical size.

6. The system of claim 1, wherein the processor is further configured to execute the instructions to project the virtual medical element onto a physical material, the physical medical element being cut from the physical material.

7. The system of claim 1, wherein receiving the user input includes receiving user input that adjusts at least one of the pose of the virtual medical element within the image or the size of the virtual medical element.

8. The system of claim 7, wherein the processor is further configured to, in response to receiving user input that adjusts at least one of the pose of the virtual medical element within the image or the size of the virtual medical element, execute the instructions to dynamically adjust the rendering of the virtual medical element substantially in real time to depict at least one of the pose or size of the virtual medical element adjusted by the user input.

9. The system according to claim 1, wherein: The physical medical components include mesh patches, gauze, bandages, plates, prostheses, sample collection bags, fasteners, or bone slurry.

10. The system of claim 1, wherein the image is acquired by an imaging device targeting the internal space of the patient, and wherein the processor is further configured to execute the instructions to: Access represents image data representing the image acquired by the imaging device; Obtain the depth data; Based on the image data and the depth data, identify the locations within the image depicting tissue that needs to be covered by the physical medical device; and The location is designated as the target location.

11. The system of claim 10, wherein the imaging device comprises a visible light camera, and the image acquired by the imaging device comprises a two-dimensional visible light image acquired by the visible light camera.

12. The system of claim 11, wherein acquiring the depth data comprises acquiring the depth data by a depth sensor guided in the imaging device and separate from the visible light camera when the imaging device is aimed at the patient's internal space.

13. The system of claim 12, wherein the depth sensor comprises a time-of-flight sensor.

14. The system of claim 10, wherein obtaining the depth data comprises: The instructions include acquiring a first visible light image of the interior space using a first visible light camera in the imaging device; The instructions include acquiring a second visible light image of the interior space using a second visible light camera in the imaging device; and The depth data is generated based on the first visible light image and the second visible light image.

15. The system of claim 10, wherein obtaining the depth data includes accessing preoperative images registered to the image.

16. The system of claim 15, wherein the preoperative image comprises an image generated by at least one of a computer-aided computed tomography scanner, a magnetic resonance imaging device, or an ultrasound device.

17. The system of claim 15, wherein the preoperative image comprises an image generated by a three-dimensional scanning device.

18. The system of claim 10, wherein identifying the location of tissue depicted within the image that needs to be covered by the physical medical element comprises inputting the image data and the depth data into a machine learning model configured to identify tissue abnormalities.

19. The system of claim 10, wherein identifying the location within the image depicting tissue to be covered by the physical medical element comprises: The image is segmented based on the image data and the depth data; and The location is identified based on segmentation.

20. The system according to claim 10, wherein: The processor is also configured to execute the instructions to determine a phase within the surgical procedure being performed on the patient; and Identifying the location of the tissue depicted within the image that needs to be covered by the physical medical element is further based on the determined stage.

21. The system of claim 1, wherein the rendering of the virtual medical element is at least partially transparent to allow visualization of the target location by the user when the virtual medical element is rendered over the target location.

22. The system of claim 1, wherein the processor is further configured to execute the instructions to: Obtain anatomical characteristic data representing features associated with the anatomical surface; and Based on the anatomical characteristic data, a placement guidance parameter set is determined, the placement guidance parameter set comprising one or more parameters configured to guide the placement of the physical medical element on the anatomical surface using one or more surgical instruments controlled by a computer-assisted surgical system.

23. The system according to claim 22, wherein: The processor is also configured to execute the instructions to obtain component characteristic data representing characteristics associated with the physical medical component; and The determination of the placement guide parameter group is further based on the component characteristic data.

24. The system of claim 23, wherein the characteristic associated with the physical medical element represented by the element characteristic data includes at least one of the physical size of the physical medical element, the material type for the physical medical element, the tensile strength of the physical medical element, or the deformation characteristics of the physical medical element.

25. The system of claim 22, wherein the characteristic associated with the anatomical surface to be covered by the physical medical element, as represented by the anatomical characteristic data, includes at least one of the tissue type constituting the anatomical surface, the size of the anatomical surface, or the location of the anatomical surface within the patient.

26. The system of claim 22, wherein the placement guidance parameter set comprises one or more parameters configured to specify the optimal orientation of the physical medical element when the physical medical element is placed on the anatomical surface using the one or more surgical instruments controlled by the computer-assisted surgical system.

27. The system of claim 22, wherein the placement guidance parameter set comprises one or more parameters configured to define an optimal insertion path within the internal space, wherein the one or more surgical instruments will follow the optimal insertion path when the physical medical element is brought from outside the patient into contact with the anatomical surface.

28. The system of claim 22, wherein the placement guidance parameter set comprises one or more parameters configured to specify a suturing plan for suturing the physical medical element to the anatomical surface when the one or more surgical instruments controlled by the computer-assisted surgical system hold the physical medical element in place on the anatomical surface.

29. The system of claim 28, wherein the suture plan includes information specifying at least one of the following: the information specifying the suture pattern to be used for suturing the physical medical element to the anatomical surface, the spacing between the sutures to be used for suturing the physical medical element to the anatomical surface, the type of thread to be used for suturing the physical medical element to the anatomical surface, or the length of thread required to suturing the physical medical element to the anatomical surface.

30. A method comprising: Image data, representing images acquired by an imaging device targeting the patient's internal space, is accessed through a medical component management system. Depth data is obtained through the medical component management system, and the depth data represents a depth map of the internal space; The medical component management system instructs the display device to render a virtual medical component above a target location within the image based on the image data and the depth data. The virtual medical component represents a physical medical component to be introduced into the patient's body, and the target location depicts an anatomical surface that will be covered by the physical medical component when it is introduced into the patient's body. When the virtual medical element is rendered above the target area, user input is received through the medical element management system. The user input sets at least one of the pose of the virtual medical element within the image or the size of the virtual medical element. as well as Based on the user input and the depth data, the physical size is determined by the medical component management system in order to determine the size of the physical medical component that will be introduced into the patient's body.

31. The method of claim 30, further comprising: The deformation data representing the deformation model of the physical medical component is accessed through the medical component management system. The determination of the physical size of the physical medical element is further based on the deformation data.

32. The method of claim 30, wherein determining the physical size of the physical medical element comprises: Based on the pose and size of the virtual medical element, identify multiple pixels within the image covered by the virtual medical element; Determine the two-dimensional pixel region of the plurality of pixels; The depth value of each of the plurality of pixels is determined based on the depth data; Based on the two-dimensional pixel region and the depth value, determine the surface area of ​​the anatomical surface to be covered by the physical medical element; as well as The physical size is determined based on the surface area of ​​the anatomical surface.

33. The method of claim 32, wherein the surface area of ​​the physical medical element defined by the physical size is greater than the physical area represented by the two-dimensional pixel region.

34. The method of claim 30, further comprising outputting data representing the physical size by the medical component management system.

35. The method of claim 30, further comprising projecting the virtual medical element onto a physical material by the medical element management system, the physical medical element being cut from the physical material.

36. The method of claim 30, wherein receiving the user input includes receiving user input that adjusts at least one of the pose of the virtual medical element within the image or the size of the virtual medical element.

37. The method of claim 36, further comprising dynamically adjusting the rendering of the virtual medical element substantially in real time in response to receiving user input that adjusts at least one of the pose or the size of the virtual medical element within the image, to depict at least one of the pose or the size of the virtual medical element adjusted by the user input.

38. The method of claim 30, wherein: The physical medical element includes a mesh patch configured to cover gauze, bandage, plate, prosthesis, sample receiving bag, fastener, or bone paste.

39. The method of claim 30, wherein the image is acquired by an imaging device targeting the internal space of the patient, and wherein the method further comprises: The medical component management system accesses image data representing the image acquired by the imaging device; The depth data is obtained by the medical component management system; Based on the image data and the depth data, the medical device management system identifies the locations within the image that depict tissues that need to be covered by the physical medical device; and The medical component management system designates the location as the target location.

40. The method of claim 39, wherein identifying the location of tissue within the image that needs to be covered by the physical medical element comprises inputting the image data and the depth data into a machine learning model configured to identify tissue abnormalities.

41. The method of claim 39, wherein identifying the location within the image depicting tissue to be covered by the physical medical element comprises: The image is segmented based on the image data and the depth data; and The location is identified based on the segmentation.

42. The method of claim 30, further comprising: Anatomical characteristic data representing features associated with the anatomical surface are obtained through the medical component management system; and Based on the anatomical characteristic data, the medical device management system determines a placement guidance parameter set, which includes one or more parameters configured to guide the placement of the physical medical device on the anatomical surface using one or more surgical instruments controlled by a computer-assisted surgical system.

43. The method of claim 42, further comprising: The medical component management system obtains component characteristic data representing the characteristics associated with the physical medical component; The determination of the placement guide parameter group is further based on the component characteristic data.

44. A non-transitory computer-readable medium storing instructions, which, when executed, instruct a processor of a computing device to: Access image data, which represents images acquired by an imaging device targeting the patient's internal space; Obtain depth data, which represents a depth map of the interior space; The display device is instructed to render a virtual medical element over a target location within the image based on the image data and the depth data. The virtual medical element represents a physical medical element to be introduced into the patient's body, and the target location depicts an anatomical surface that will be covered by the physical medical element when it is introduced into the patient's body. When the virtual medical element is rendered above the target area, user input is received, wherein the user input sets at least one of the pose of the virtual medical element within the image or the size of the virtual medical element; and Based on the user input and the depth data, the physical size is determined in order to determine the dimensions of the physical medical device to be introduced into the patient's body.

Images