Systems and methods for registering visual representations of a surgical space

Abstract

An exemplary image management system accesses semantic information about anatomical objects in a surgical space and, at least in part, registers a first dataset of a first visual representation of the surgical space with a second dataset of a second visual representation of the surgical space based on the semantic information about the anatomical objects.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 993,587, filed March 23, 2020, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] During surgical procedures, various visual representations of the surgical space can be generated. For example, an endoscope can be used to acquire endoscopic images of the surgical site. These endoscopic images can be presented to the surgeon via a display device, allowing the surgeon to visualize the surgical site while performing the procedure.

[0004] In some scenarios, one or more imaging modalities may be used to acquire or present an additional visual representation of the surgical site to the surgeon. Such additional visual representations may be acquired preoperatively or intraoperatively and may be acquired, for example, via ultrasound scanning, computed tomography (“CT”) scanning, magnetic resonance imaging (“MRI”) scanning, fluorescence imaging scanning, and / or another suitable imaging modal configured to acquire images of the surgical site.

[0005] To integrate various imaging modalities and provide intuitive visualization, registering different visual representations of the surgical space can be useful. However, such registration is not always an easy process. Summary of the Invention

[0006] The following description presents a simplified overview of one or more aspects of the systems and methods described herein. This overview is not a comprehensive summary of all anticipated aspects and is neither intended to identify key or essential elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed descriptions that follow.

[0007] An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to access semantic information about anatomical objects in a surgical space; and to register a first dataset of a first visual representation of the surgical space with a second dataset of a second visual representation of the surgical space, based at least in part on the semantic information about the anatomical objects.

[0008] An exemplary method includes a processor (e.g., a processor of an image management system) that accesses scene segmentation information for a first visual representation of a surgical space including an anatomical object and a second visual representation of the surgical space; determines a set of reliable tissues based on the scene segmentation information to register a first dataset of the first visual representation with a second dataset of the second visual representation; determines a first region in an image of the first visual representation depicting one or more tissues from the set of reliable tissues; determines a first plurality of feature points in the first region depicting one or more tissues in an image of the first visual representation; determines a second region in an image of the second visual representation depicting one or more tissues; determines a second plurality of feature points in the second region depicting one or more tissues in an image of the second visual representation; and registers the first dataset of the first visual representation and the second dataset of the second visual representation at least in part based on the alignment of the first plurality of feature points with the second plurality of feature points.

[0009] Another exemplary method includes a processor (e.g., a processor of an image management system) that accesses information about the current stage of a procedure performed on an anatomical object in a surgical space; and selectively updates the initial registration of a dataset of a first visual representation of the surgical space with a dataset of a second visual representation of the surgical space based on the current stage of the procedure.

[0010] Another exemplary method includes a processor (e.g., a process of an image management system) that accesses force-sensing data of a computer-aided surgical system that performs a procedure on an anatomical object in a surgical space; detects an interaction with the anatomical object in a first dataset of a first visual representation of the surgical space; models the interaction in a second dataset of a second visual representation of the surgical space based on the force-sensing data; and registers the first dataset of the first visual representation with the second dataset of the second visual representation. Attached Figure Description

[0011] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0012] Figure 1 This document describes an exemplary image management system for managing imaging data based on the principles described herein.

[0013] Figure 2 An exemplary image is provided to illustrate registration for an image management system based on the principles described herein.

[0014] Figure 3 This document describes an exemplary configuration of an image management system for managing imaging data, based on the principles described herein.

[0015] Figure 4-5 Exemplary images illustrating the principles described herein for tracking anatomical objects.

[0016] Figure 6 Explaining one of the principles described in this article Figure 1 An example configuration for registering imaging data from different imaging modes in an image management system.

[0017] Figure 7 This document describes an exemplary computer-assisted surgical system based on the principles described herein.

[0018] Figure 8-10 This document describes an exemplary method for tracking anatomical objects in a video, based on the principles described herein.

[0019] Figure 11 An exemplary computing device is described in accordance with the principles described herein. Detailed Implementation

[0020] This document describes a dataset for registering a visual representation of a surgical space and a system and method for tracking anatomical objects within a surgical space. Various types of surgical procedures can be planned or performed in a surgical space, which may include anatomical objects of the body on which the surgical procedure is being (or will be) performed, anatomical structures adjacent to the anatomical objects on which the procedure is being (or will be) performed, and other areas adjacent to the anatomical objects (e.g., open spaces). The surgical space can be imaged and / or modeled using various imaging modalities, such as endoscopy, ultrasound scanning, preoperative imaging scans, etc. Imaging can produce various visual representations of the surgical space, such as videos of one or more surgical scenes depicting a scene of the surgical space, models of the surgical space, etc. The visual representation of the surgical space may include or be represented in any suitable format by data representing at least a portion of the surgical space and can be used to provide a visual representation of at least a portion of the surgical space. In some examples, data may be obtained by imaging the surgical space using one or more imaging modalities. Data can represent the surgical space in any suitable manner, such as 2D images, 3D images, 4D images (3D images with a time component), color images, depth data, texture data, etc. In some examples, the visual representation may include data representing a model of the surgical space, such as a 3D model of the surgical space. Data representing a visual representation of the surgical space, or a portion thereof, can be referred to as a dataset of the visual representation. Such datasets can be in any suitable format, such as a video dataset representing a video of the surgical space, a model dataset representing a model of the surgical space (e.g., a 2D or 3D model), or any other dataset suitable for representing the surgical space. The image management system can perform various operations to register datasets of various visual representations (e.g., identify, match, or align corresponding data points in the dataset). For example, the image management system may receive semantic information about anatomical objects from surgical instruments, such as scene segmentation information, procedural stage information, and / or force sensing information. The image management system can use the semantic information to accurately register datasets of visual representations.

[0021] The systems and methods described in this paper offer various advantages and benefits. For example, accurate registration of a dataset of visual representations of the surgical space enables image management systems to generate accurate synthetic images using multiple imaging modalities and / or models of anatomical objects. Furthermore, reliable registration of a dataset of visual representations of the surgical space facilitates accurate tracking of anatomical objects within the surgical space by the image management system. Such accurate imaging and tracking of anatomical objects can inform surgeons, which can be valuable for more efficient and / or effective execution of surgical procedures compared to the absence of such information.

[0022] Various embodiments will now be described in more detail with reference to the accompanying drawings. The disclosed systems and methods may provide one or more of the benefits described above and / or various additional and / or alternative benefits that will become apparent herein.

[0023] Figure 1 An exemplary image management system 100 (“System 100”) describes a dataset used for registering a visual representation of a surgical space. System 100 may be included in computer-assisted surgical systems (such as those described below in conjunction with...). Figure 6 The exemplary computer-assisted surgical system described herein is implemented in, by, or connected to one or more components of the computer-assisted surgical system. For example, system 100 may be implemented by one or more components of the computer-assisted surgical system, such as a manipulation system, a user control system, or an assistance system. As another example, system 100 may be implemented by a separate computing system communicatively coupled to the computer-assisted surgical system.

[0024] like Figure 1 As shown, system 100 may include, but is not limited to, storage facility 102 and processing facility 104 selectively and communicatively coupled to each other. Facilities 102 and 104 may each include or be implemented by one or more physical computing devices, which include hardware and / or software components such as processors, memory, memory drives, communication interfaces, instructions stored in memory for execution by the processor, etc. Although facilities 102 and 104 are... Figure 1 While facilities 102 and 104 are shown as separate facilities, they may be combined into fewer facilities, such as a single facility, or divided into more facilities as may be used in a particular implementation. In some examples, each of facilities 102 and 104 may be distributed among multiple devices and / or multiple locations as may be used in a particular implementation.

[0025] Storage facility 102 may hold (e.g., store) executable data used by processing facility 104 to perform any of the functions described herein. For example, storage facility 102 may store instructions 106, which may be implemented by processing facility 104 to perform one or more of the operations described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 102 may also hold any data received, generated, managed, used, and / or transmitted by processing facility 104.

[0026] Processing facility 104 may be configured to perform (e.g., execute instructions 106 stored in storage facility 102) various operations associated with a dataset of visual representations of the registered surgical space. For example, processing facility 104 may be configured to access a first dataset comprising a first visual representation of the surgical space of an anatomical object and a second dataset comprising a second visual representation of the surgical space of a second anatomical object. Processing facility 104 may also access semantic information about the anatomical object. Processing facility 104 may also register the first dataset of the first visual representation of the surgical space with the second dataset of the second visual representation of the surgical space, at least in part, based on the semantic information about the anatomical object.

[0027] This document describes these and other operations that can be performed by system 100 (e.g., processing facility 104). In the following description, any reference to functions performed by system 100 should be understood as being performed by processing facility 104 based on instructions 106 stored in storage facility 102.

[0028] Figure 2 Exemplary aspects of the registration of a first set of imaging data 200-1 and a second set of imaging data 200-2 are illustrated. In some examples, the registration of the first imaging data and the second imaging data may refer to a mapping from image data points in the first imaging data to corresponding image data points in the second imaging data, such that the registration allows the image datasets to be aligned from a particular viewpoint. For example, as... Figure 2 As shown, imaging data 200-1 represents a depiction 202-1 of an anatomical object (e.g., an internal organ or part thereof) in a surgical space. Imaging data 200-2 represents a depiction 202-2 of the same anatomical object (although acquired from slightly different viewpoints, making imaging data 200-1 and 200-2 similar but not identical). Registration of imaging data 200-1 and imaging data 200-2 may involve determining whether various features 204 (e.g., features 204-1 to 204-6) in depiction 202-1 correspond to similar features 206 (e.g., features 206-1 to 206-6) in description 202-2.

[0029] As shown in the figure, for example, features 204-1 and 206-1 can be identified as matches (i.e., representing the same physical feature), such as feature pairs 204-2 and 206-2, 204-3 and 206-3, 204-4 and 206-4, and 204-5 and 206-5. In this example, features in each depiction are also retrieved that do not correspond to similar features in another depiction. Specifically, no data point in depiction 202-2 of imaging data 200-2 corresponds to a data point representing feature 204-6 in depiction 202-1 of imaging data 200-1, and no data point in depiction 202-1 corresponds to a data point representing feature 206-6 in depiction 202-2. Imaging data 200-1 can be registered with imaging data 200-2 by identifying a sufficient number of corresponding data point pairs (e.g., data point pairs representing similar features 204 and 206) so that depiction 202-1 can be aligned with depiction 202-2 with respect to a specific viewpoint (e.g., the viewpoint from which imaging data 200-1 is acquired, the viewpoint from which imaging data 200-2 is acquired, or another suitable viewpoint).

[0030] exist Figure 2 In the example, depictions 202-1 and 202-2 may appear similar because the corresponding images of each imaging data 200-1 and 200-2 can come from a single visual representation acquired through the same imaging modality (e.g., in the same imaging manner, through the same acquisition device, using the same imaging technique, etc.). Furthermore, and due to visual similarity, registration of imaging data 200-1 with imaging data 200-2 can be performed in a relatively straightforward manner by identifying the corresponding data points by recognizing feature 204 in depiction 202-1, recognizing feature 206 in depiction 202-2, and matching features from each group.

[0031] However, although Figure 2 The examples illustrate aspects of registering imaging data acquired from a single imaging modality, but in some examples, it may be necessary to register imaging data from multiple visual representations. For instance, the dataset for a first visual representation may include imaging data from a first imaging modality (e.g., an endoscopic imaging modality), while the dataset for a second visual representation may include imaging data from a second imaging modality (e.g., an additional imaging modality, such as a CT scan, MRI scan, etc.) and / or imaging data from time points different from the first (or second) imaging modality.

[0032] As an example, different imaging modalities can acquire depictions or other representations of anatomical structures at different points in time. For instance, endoscopic imaging modalities can include intraoperative scans of anatomical structures within the surgical space and can therefore be performed in real time during surgery. Similarly, ultrasound or fluorescence imaging scans (where fluorescent dyes are injected into the body to facilitate imaging at specific frequencies at which the dyes exhibit fluorescence properties) can be similarly used during surgery, either in real time during surgery or during surgery, while active surgical procedures are temporarily suspended while imaging is being performed. In contrast, other types of imaging modalities can acquire depictions or other representations of anatomical structures at a point in time prior to the procedure in the surgical space (e.g., immediately before, a day earlier, or so on). For example, imaging modalities including CT scans, MRI scans, ultrasound scans, X-ray scans, 3D modeling based on data from any such scans, or other suitable imaging modalities can be performed at different times as the body is in different states. For example, the patient's position during the preoperative time period when using one imaging modality and during the intraoperative time period when using another imaging modality may differ (e.g., supine vs. lateral decubitus) or may have other significant differences (e.g., fasting vs. not fasting). As another example, images from different time points within an imaging modality can be used to track the movement of an anatomical object within the surgical space. In other examples, different modalities may be used simultaneously (e.g., both preoperatively, both intraoperatively, etc.) or at times that differ in other ways from this example (e.g., different preoperative times, different intraoperative times, preoperative and postoperative times, intraoperative and postoperative times, etc.).

[0033] System 100 can register multiple datasets of visual representations for various purposes. For example, as mentioned, one purpose of registering a first dataset including a first visual representation of first imaging data with a second dataset including a second visual representation of second imaging data is to align the first and second imaging data to allow system 100 to generate and provide a synthetic image of the surgical space for display on a display device. This synthetic image includes aspects of the first and second imaging data as viewed from a specific viewpoint (e.g., the viewpoint of a surgeon performing a surgical procedure in the surgical space). For example, such a synthetic image can be based on the registration of endoscopic imaging data from an endoscopic imaging mode with a second visual representation including additional imaging data from an additional imaging mode, and can allow both the endoscopic and additional imaging data aspects to be presented to the user in a conveniently customizable view to facilitate operations in the surgical space. Alternatively, the second dataset of the second visual representation may include other information, such as a model of the surgical space, a depth map of the surgical space, or graphical elements (e.g., annotations, virtual objects, etc.). Aligning such additional information with the endoscopic imaging data can also allow for an intuitive, convenient, and customizable view to facilitate operations in the surgical space. Alternatively, the second dataset for the second visual representation can be endoscopic imaging data with a time offset, allowing for the tracking of movement of anatomical structures depicted in the endoscopic imaging data. Other purposes for registering the visual representation dataset are described in this paper.

[0034] Figure 3 An exemplary configuration 300 is described, in which system 100 is configured to register a dataset of visual representations of a surgical space. As shown, system 100 accesses (e.g., receives, generates, retrieves, etc.) a first dataset comprising a first visual representation 302 of the surgical space of an anatomical object to which a procedure is performed. System 100 also accesses a second dataset of a second visual representation 304 of the surgical space. System 100 also accesses semantic information 306 about the anatomical object. Based at least in part on the semantic information 306, system 100 provides registration data 310, which can be used to register the dataset of the first visual representation 302 and the dataset of the second visual representation 304.

[0035] The first visual representation 302 can be any suitable visual representation of the surgical space. For example, the first visual representation 302 can include video images provided by an imaging device, such as an endoscope or other camera device configured to acquire images of the surgical space. In some examples, the imaging device can be configured to be attached to and controlled by a computer-assisted surgical system. In alternative examples, the imaging device can be handheld and manually operated by an operator (e.g., a surgeon). Additionally or alternatively, the first visual representation 302 can include any examples described with respect to the second visual representation 304. Any suitable dataset can represent the first visual representation 302.

[0036] The second visual representation 304 can also be any suitable visual representation of the surgical space. For example, the second visual representation 304 may include imaging data other than video images, such as ultrasound scans, CT scans, MRI scans, fluoroscopic imaging scans, etc., of the anatomical object and / or the surgical space. Alternatively or alternatively, the second visual representation 304 may include a model of the anatomical object, such as a model generated using any such suitable scan image. Alternatively or alternatively, the second visual representation 304 may include a depth map of the surgical space and / or the anatomical object defining the distances between points in the image from the viewpoint. Alternatively or alternatively, the second visual representation 304 may include graphical elements, such as virtual objects, annotations, user interface components, etc. Alternatively or alternatively, the second visual representation 304 may include an image of the first visual representation 302 with a time offset. For example, the first visual representation 302 may include video images. The second visual representation 304 may include the same video images offset by a certain amount of time and / or a certain number of video image frames. By registering the time-offset video images, movement of objects within the video can be detected, which may correspond to movement of anatomical structures (such as anatomical objects) in the surgical space. Alternatively or concurrently, the second visual representation 304 may include images similar to the first visual representation 302 but from a different viewpoint. For example, two images of a surgical space from two different viewpoints could provide information to generate a stereoscopic image of the surgical space and / or a depth map of the surgical space. While configuration 300 illustrates a system 100 for accessing the first and second visual representations, any suitable number of visual representations can be registered using the techniques described herein. Any suitable dataset can represent the second visual representation 304.

[0037] Semantic information 306 may include any suitable information that provides meaningful and / or contextual information for the visual representation of the surgical space. For example, semantic information 306 may include scene segmentation information (e.g., information indicating what is depicted in the image). Semantic information 306 may also include procedural stage information (e.g., information indicating the current stage of a surgical procedure performed on an anatomical object). As another example, semantic information 306 may include temporal information associated with the image. Semantic information 306 may also include force-sensing data (e.g., information indicating the amount of force sensed by the surgical instruments of the computer-assisted surgical system). Semantic information 306 may also include instrument tracking data (e.g., information indicating the posture of the surgical instruments of the computer-assisted surgical system). These and any other suitable exemplary types of semantic information 306 may be used individually or in any suitable combination. System 100 may use semantic information 306 to assist in registering the first visual representation 302 with the second visual representation 304 in any suitable manner (such as that described herein).

[0038] Registration data 310 can be any suitable data that provides information for registering the dataset of the first visual representation 302 and the dataset of the second visual representation 304. As described above, registration data 310 can include one or more feature pairs that are found to depict the same features in the images of the first visual representation 302 and the second visual representation 304. Alternatively or additionally, registration data 310 can include coordinate positions of the images of the first visual representation 302 and the second visual representation 304 for aligning the visual representations. Alternatively or additionally, registration data 310 can include the type of registration to be performed, such as between two different imaging modes and between time-shifted images of an imaging mode. In some examples, registration data 310 can include commands (or flags, or variables, or any other suitable indicators) for avoiding registration of the datasets of the first visual representation 302 and the second visual representation 304 (e.g., during certain stages of a procedure performed on an anatomical object, if scene segmentation information indicates that the anatomical object is not visible, etc.). Other examples of registration data 310 are described herein.

[0039] Figure 4An exemplary image 400 of a surgical space is shown. Image 400 may be an image from a first visual representation of the surgical space, such as a frame of video captured by an endoscope (or any suitable imaging device) depicting the surgical space. Image 400 shows an anatomical object 402, which may be an organ on which a surgeon is performing a surgical procedure. As shown, the anatomical object 402 is covered by a layer of fat 404, as some organs may be found to have this layer of fat. Image 400 also shows surgical instruments 406 (e.g., surgical instruments 406-1 and 406-2), each of which may be implemented by any suitable therapeutic instrument (e.g., a tool with tissue interaction capabilities), imaging device (e.g., an endoscope), diagnostic instrument, etc., that can be used to perform computer-assisted surgical procedures on a patient (e.g., by being at least partially inserted into and manipulated to perform computer-assisted surgical procedures on the patient). Image 400 may be generated and / or accessed by an image management system (e.g., system 100).

[0040] System 100 can access images from a second visual representation of the surgical space for registration with image 400. System 100 can also access semantic information and perform registration at least in part based on the semantic information.

[0041] For example, semantic information may include scene segmentation information. The scene segmentation information used for image 400 may indicate that pixel regions depicting anatomical object 402 correspond to anatomical object 402, pixel regions depicting fat layer 404 correspond to fat layer 404, and other pixel regions correspond to background tissue, vascular system, bone, surgical instruments 406, etc. Scene segmentation information may also include descriptive information, such as multiple labels for anatomical object 402 describing its state (e.g., covered, uncovered, moved, etc.). In this way, scene segmentation information can indicate what is depicted in image 400.

[0042] System 100 can perform registration of image 400 with an image from a second visual representation in any suitable manner and based on any suitable information. For example, based on scene segmentation information, system 100 can determine tissues that can be considered reliable for image registration and tissues that can be considered unreliable for registration. The reliability of a tissue can be based on at least one characteristic of the tissue, such as tissue stiffness, tissue rigidity, tissue mobility, and the typical amount of movement of the tissue during the anatomical object 402 procedure, or any other suitable feature. For example, tissues that typically move a lot during the procedure may not provide reliable feature points for registering the visual representation. Furthermore, some tissues, such as the fat layer 404, can be completely removed during the procedure. Registering the dataset of visual representations based on feature points on such tissues can also lead to unreliable results. Therefore, more rigid and / or relatively fixed tissues, such as bone, the vascular system, or anatomical object 402, can provide more reliable feature points for registration. Alternatively or additionally, system 100 can use a predefined list of tissues classified as reliable or unreliable, or classified using the amplitude / spectral line of reliability. The example predefined list of reliable tissues can include bones, the vascular system, anatomical objects 402, and any other suitable tissues predefined as reliable. This predefined list can be configured to be customized by the user.

[0043] System 100 can make registration based on the reliability of the organization in any suitable manner. For example, System 100 can use scene segmentation information to identify organizations in image 400 corresponding to reliable organizations and generate a binary mask on image 400, such that feature points are selected only on reliable organizations. In contrast, System 100 can identify organizations in image 400 corresponding to unreliable organizations and generate a similar binary mask, selecting feature points only at points outside of unreliable organizations. Alternatively, organizations can be classified according to the magnitude / spectral line of reliability, and feature points on each organization can be weighted accordingly. For example, feature points selected on highly reliable organizations may have high weights, feature points selected on unreliable organizations may have low weights or no weights at all, while other organizations may receive weights in between. System 100 can then use an algorithm to determine the registration using some weighted combination of feature points. For example, if a feature point found in one image has a low weight, that feature point may be more easily ignored if it is not found in another image with a corresponding feature point. Using one or more of these techniques, system 100 can determine multiple reliable feature points on image 400 and perform the same or similar determinations on images from the second visual representation. System 100 can then register the datasets of visual representations to each other, at least in part, based on the alignment of the multiple reliable feature points in each image.

[0044] System 100 may additionally or alternatively use scene segmentation information to align regions from the visual representation images to each other. For example, System 100 may use the boundaries of objects (e.g., anatomical object 402, surgical instrument 406) identified in the scene segmentation information to register the dataset of the visual representation. Alternatively or alternatively, System 100 may use the scene segmentation information in conjunction with any suitable image processing technique to register the dataset of the visual representation. For example, in addition to (or besides) feature points, System 100 may use image-based registration, optimized similarity measures (e.g., interaction information, cross-correlation, etc.), non-point features (e.g., edges, etc.), or any other suitable image processing technique.

[0045] Figure 5 An exemplary image 500 of the surgical space shown in image 400 is illustrated. Image 500 may be another image (e.g., a later image) from a first visual representation of the surgical space. Image 500 includes an anatomical object 402 after a fat layer 404 has been removed from the organ. This removal of fat from the organ is a typical step in a surgical procedure performed on the organ. As shown, removing the fat layer 404 exposes more of the anatomical object 402 in the image depicting the surgical space (e.g., image 500). Such exposure can allow more feature points on reliable tissue for registration of the visual representation. In contrast, discarding feature points on the fat layer 404 before removing it, as shown in image 400, can also allow for more reliable visual representation registration.

[0046] Furthermore, changes in the surgical space can be observed by registering images from different time points of the same imaging device and / or imaging mode. For example, movement of the anatomical object 402 can be tracked by registering a first visual representation of the surgical space with a second visual representation that is a time-offstream stream of the first visual representation. For example, the first visual representation could be a video of the surgical space including the anatomical object 402. The second visual representation could be the same video with any suitable time offset (e.g., one second, a fraction of a second, several frames of video, etc.). Any movement of the anatomical object 402 will be detected by registering the video at time T with the video at time (T-offset). Movement of the anatomical object 402 can be tracked by updating such registration at a specified rate. Therefore, any technique described herein can be used to perform tracking of the anatomical object 402 (or any other suitable anatomical object). For the purpose of tracking the anatomical object 402, feature points on the anatomical object 402 may be discarded and / or weighted differently when registering visual representations.

[0047] Furthermore, image 500 may depict a different stage of a surgical procedure than image 400. This stage information can be another example of semantic information accessible to system 100, upon which the registration of a dataset of visual representations (e.g., image 500 with an image from a second visual representation) is based. For example, the stage of the procedure can determine whether the registration should be updated from the initial registration. Such a determination can be based on the expected amount of change in the surgical space at the current stage of the procedure. If the surgical space changes significantly, registration completed before the significant change may no longer be accurate and / or more information may be available for more accurate registration of the visual representation. The removal of fat layer 404, as shown in image 500, can be an example instance where updated registration can be used for one or both of these reasons.

[0048] Alternatively, at certain stages of the procedure, registering the dataset of visual representations may not be worthwhile, for example, if registering the dataset of visual representations would be more difficult (e.g., due to a lack of reliable feature points, increased movement of the anatomical object 402, the surgical space and / or the viewpoint of the imaging device, etc.) or if multiple visual representations would not provide particularly useful information. In such stages, system 100 may avoid registering visual representations.

[0049] Alternatively, at some stages of the procedure, system 100 may update the registration at a higher or lower frequency and / or use more or fewer feature points or any other differences in parameters. For example, if a procedure stage is one in which the anatomical object 402 will move more and / or requires more accurate registration or tracking than in other stages, the registration may be updated at a higher specified rate and / or using more feature points. Alternatively, parameters that can be varied may include increasing or decreasing the amount of smoothing, increasing or decreasing the search space used for registration, switching registration methods (e.g., using image edges, depth maps, and / or feature points, etc.) to support higher or lower update rates, etc. Any other suitable parameter settings of this kind may be used based on procedure stage information.

[0050] Furthermore, system 100 can also access force-sensing data and / or instrument tracking data (e.g., from surgical instruments 406) as another example of semantic information to support the registration of the visual representation dataset. Registration of the visual representation dataset and tracking of the anatomical object 402 can become more difficult when the anatomical object 402 is manipulated (e.g., by surgical instruments 406). By accessing force-sensing data and / or instrument tracking data to determine at which points and with what force the surgical instruments 406 interact with the anatomical object 402, a model of the anatomical object 402 (e.g., a second visual representation) can be modified to correspond to the manipulation of the anatomical object 402, which can be depicted in a first visual representation (e.g., video). For example, the model of the anatomical object 402 can be modified based on the stiffness of the anatomical object 402. For a rigid anatomical object, the model can be modified by moving the model of the anatomical object 402 by a corresponding amount based on the force-sensing data and / or instrument tracking data. For anatomical objects with low rigidity, the model can be modified by deforming and moving the model of the anatomical object 402 by a corresponding amount based on the stiffness and force sensing data and / or instrument tracking data of the anatomical object 402.

[0051] Figure 6 An exemplary configuration 600 is illustrated, in which system 100 registers imaging data from first and second visual representations to generate a synthetic image of a surgical space. As shown, configuration 600 may include multiple imaging modes 602 (e.g., endoscopic imaging mode 602-1 and additional imaging mode 602-2) configured to acquire imaging data 604 of the surgical space 606 (e.g., endoscopic imaging data 604-1 acquired via endoscopic imaging mode 602-1 and additional imaging data 604-2 acquired via additional imaging mode 602-2).

[0052] Surgical space 606 may include any volumetric space associated with a surgical procedure. For example, surgical space 606 may include any part or more of the patient's body, such as the patient's anatomical structures 608 (e.g., tissues, etc.) within the space associated with the surgical procedure. In some examples, surgical space 606 may be entirely located within the patient's body and may include spaces within the patient's body adjacent to the location where a surgical procedure is planned, is being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue within the patient's body, surgical space 606 may include surface tissue, anatomical structures beneath the surface tissue, and space around the tissue where, for example, surgical instruments used to perform the surgical procedure are located. In other examples, surgical space 606 may be at least partially located outside the patient's body. For example, for an open surgical procedure performed on a patient, a portion of surgical space 606 (e.g., the tissue being manipulated) may be inside the patient, while another portion of surgical space 606 (e.g., space around the tissue where one or more surgical instruments may be located) may be outside the patient's body. Surgical space 606 may include a physical workspace in which surgical procedures are performed, such as a physical workspace associated with a patient and in which one or more surgical instruments are used to perform surgical procedures on the patient.

[0053] As used herein, a surgical procedure can include any medical procedure, including any diagnostic or therapeutic procedure in which manual and / or instrumental techniques are used on a patient to investigate or treat a patient's physical condition. A surgical procedure can refer to any stage of a medical procedure, such as the preoperative, surgical (i.e., intraoperative), and postoperative stages of a surgical procedure.

[0054] Imaging mode 602 can be configured to and / or used to acquire imaging data 604 representing the surgical space 606. Such acquisition is performed by... Figure 6 The dashed line 610 in the diagram represents this. Imaging modes 602 can each acquire imaging data 604 of the surgical space 606 in any suitable manner, and the imaging data 604 can take any suitable form. For example, imaging data 604 can be implemented as data representing, in one implementation, still frame images (e.g., grayscale images, color images, infrared images, etc.), videos (e.g., grayscale, color, infrared video, etc.), 3D models, depth maps, graphic elements, or any other type of visualization or depiction of the surgical space 606 that can be used to help a user visualize the surgical space 606. Imaging modes 602 can also each acquire imaging data 604 at any suitable time. For example, one or more imaging modes 602 can acquire imaging data 604 of the surgical space 606 during one or more preoperative, intraoperative, and / or postoperative stages of a surgical procedure.

[0055] Endoscopic imaging mode 602-1 is a mode involving the acquisition of imaging data via an endoscope (e.g., or another suitable type of endoscopic instrument) configured to project light (e.g., light at visible frequencies) onto the anatomical structures of a surgical space 606 and acquire radiographic images of the anatomical structures when the light reflects from the anatomical structures to one or more image sensors associated with the endoscope. In contrast, in some examples, supplementary imaging mode 602-2 may be a different type of imaging mode (i.e., a mode different from the endoscopic imaging mode). For example, as described above, supplementary imaging mode 602-2 may include or involve, but is not limited to, ultrasound imaging performed by an ultrasound module or machine, CT imaging performed by a CT machine, MRI imaging performed by an MRI machine, etc. Any other suitable supplementary imaging mode may be used in other examples.

[0056] In some examples, endoscopic imaging mode 602-1 can be configured to acquire images of surface anatomy structures included in the surgical space 606 (e.g., the outer surface of tissue included in the surgical space), and additional imaging mode 602-2 can be configured to acquire images of underlying anatomy structures included in the surgical space 606 (e.g., underlying tissue behind the outer surface of tissue included in the surgical space). For example, endoscopic imaging mode 602-1 can acquire images of surface tissue within a patient's body, and additional imaging mode 602-1 can include ultrasound, CT, or MRI imaging of underlying tissue that, from the endoscopic viewpoint, is behind and hidden outside the endoscopic field of view by the surface anatomy structures.

[0057] As described above, imaging modes 602 can each acquire imaging data 604 of the surgical scene 606 at any suitable time, such as during one or more stages of a surgical procedure or operation. In some examples, imaging modes 602 can simultaneously acquire imaging data 604 of the surgical space 606. For example, endoscopic imaging mode 602-1 can acquire endoscopic images during a surgical procedure (e.g., during the surgical phase of a surgical procedure), and additional imaging mode 602-1 can simultaneously acquire another type of image during a surgical procedure. In other examples, imaging modes 602 can acquire imaging data 604 of the surgical space 606 at different times and / or at different stages of a surgical procedure. For example, endoscopic imaging mode 602-1 can acquire endoscopic images during the surgical phase of a surgical procedure, while additional imaging mode 602-2 can acquire another type of image during the preoperative phase of a surgical procedure.

[0058] Imaging data 604 representing surgical space 606 may include images of surgical space 606 acquired through imaging mode 602. For example, imaging data 604 may include endoscopic images, ultrasound images, CT images, MRI images, and / or any other suitable form of image of surgical space 606. Alternatively or additionally, imaging data 604 may include one or more models of surgical space 606 generated based on imaging performed by imaging mode. For example, supplementary imaging data 604-2 may include a 3D model of surgical space 606 generated based on imaging performed by imaging mode (such as imaging performed by an ultrasound machine, CT machine, MRI machine, or other suitable imaging mode). The 3D model may be a complete volumetric model comprising voxels (i.e., volumetric pixels) having values ​​(e.g., color values, brightness values, etc.) representing the appearance of surgical space 606 at an intrinsic 3D point in the model. Such a volumetric model can facilitate the recognition and use of any slice of the 3D model by system 100 to generate images of slices of the 3D model.

[0059] Although Figure 6 Two imaging modes 602-1 and 602-2 are depicted, respectively acquiring imaging data 604-1 and 604-2 as input to system 100. However, other examples may include any suitable number and / or configuration of multiple different imaging modes acquiring images as input to system 100 to produce a composite image of surgical space 606. For example, three or more different imaging modes may acquire images input to system 100 to produce a composite image of surgical space 606.

[0060] System 100 can generate a synthetic image (e.g., including one or more synthetic images) of the surgical space 606 based on imaging data 604 acquired via imaging mode 602. System 100 can do this in any suitable manner to generate a synthetic image that includes a comprehensive representation of portions of the surgical space 606 acquired by different imaging modes 602.

[0061] System 100 may instruct display device 614 to display the composite image. For example, system 100 may provide data representing the composite image to display device 614, which may be configured to display the composite image for viewing by a user of the computer-assisted surgical system (e.g., a surgeon or other member of the surgical team performing a surgical procedure). Display device 614 may include any means capable of receiving and processing imaging data to display one or more images. For this purpose, display device 614 may include one or more display screens on which images can be displayed. In some examples, display device 614 may be a component of or communicatively connected to the computer-assisted surgical system, such as those described in more detail below.

[0062] As mentioned, system 100 can be implemented or communicatively coupled to a computer-assisted surgical system. System 100 can receive input from the computer-assisted surgical system and provide output to that system. For example, system 100 can access images of the surgical space and / or any information about the surgical space and / or the computer-assisted surgical system from the computer-assisted surgical system, use the accessed images and / or information to perform any of the processing described herein to generate a composite image of the surgical space, and provide data representing the composite image to the computer-assisted surgical system for display (e.g., via a display device associated with the computer-assisted surgical system).

[0063] To illustrate, Figure 7 An exemplary computer-assisted surgical system 700 (“surgical system 700”) is shown. System 100 may be implemented by, connected to, and / or used in conjunction with surgical system 700.

[0064] As shown in the figure, the surgical system 700 may include a control system 702, a user control system 704, and an auxiliary system 706 that are communicatively coupled to each other. The surgical system 700 can be used by a surgical team to perform computer-assisted surgical procedures on a patient 708. As shown in the figure, the surgical team may include a surgeon 710-1, an assistant 710-2, a nurse 710-3, and an anesthesiologist 710-4, all of whom can be collectively referred to as "surgical team members 710". Additional or alternative surgical team members may be present during the surgical procedure, depending on their intended use in a particular implementation.

[0065] Although Figure 7This describes an ongoing minimally invasive surgical procedure; however, it should be understood that the surgical system 700 can be similarly used to perform open surgical procedures or other types of surgical procedures that can similarly benefit from the accuracy and convenience of the surgical system 700. Furthermore, it should be understood that surgical activities that can utilize the surgical system 700 throughout can include not only the operational phases of surgical procedures, such as... Figure 7 This can also include the preoperative, postoperative, and / or other appropriate stages of a surgical procedure.

[0066] like Figure 7 As shown, the manipulation system 702 may include a plurality of manipulator arms 712 (e.g., manipulator arms 712-1 to 712-4) to which a plurality of surgical instruments (e.g., surgical instruments 406 as shown above) may be coupled. Each surgical instrument may be implemented by any suitable therapeutic instrument (e.g., a tool with tissue interaction capabilities), imaging device (e.g., an endoscope), diagnostic instrument, etc., which may be used for computer-assisted surgical procedures performed on patient 708 (e.g., by being at least partially inserted into and manipulated to perform computer-assisted surgical procedures on patient 708). In some examples, one or more of the surgical instruments may include force sensing and / or other sensing capabilities. While the manipulation system 702 is depicted and described herein as including four manipulator arms 712, it will be appreciated that the manipulation system 702 may include only a single manipulator arm 712 or any other number of manipulator arms that may be used for a particular implementation. Although the manipulation system 702 is depicted and described herein as comprising four manipulator arms 712, it will be appreciated that the manipulation system 702 may comprise only a single manipulator arm 712 or any other number of manipulator arms that may be used in a particular implementation.

[0067] The manipulator arm 712 and / or the surgical instruments attached to the manipulator arm 712 may include one or more displacement transducers, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the surgical system 700 may be configured to use kinematic information to track (e.g., determine position) and / or control surgical instruments (and any physical components attached to the instruments, such as an ultrasound module).

[0068] User control system 704 can be configured to facilitate control of robotic arm 712 and surgical instruments attached to robotic arm 712 by surgeon 710-1. For example, surgeon 710-1 can interact with user control system 704 to remotely move or manipulate robotic arm 712 and surgical instruments. To this end, user control system 704 can provide surgeon 710-1 with images of the surgical space associated with patient 708 acquired by an imaging system (e.g., any medical imaging system described herein). In some examples, user control system 704 may include a stereoscopic viewer with two displays, where surgeon 710-1 can view stereoscopic images of the surgical space associated with patient 708 and generated by a stereoscopic imaging system. In some examples, synthetic images generated by system 70 can be displayed by user control system 704. Surgeon 710-1 can use the images displayed by user control system 704 to perform one or more procedures, wherein one or more surgical instruments are attached to manipulator arm 712.

[0069] To facilitate control of surgical instruments, the user control system 704 may include a set of master controllers. These master controllers can be manipulated by the surgeon 710-1 to control the movement of surgical instruments (e.g., by utilizing robotic and / or remote operation technologies). The master controllers can be configured to detect various hand, wrist, and finger movements of the surgeon 710-1. In this way, the surgeon 710-1 can intuitively perform procedures using one or more surgical instruments.

[0070] The auxiliary system 706 may include one or more computing devices configured to perform primary processing operations of the surgical system 700. In such a configuration, the one or more computing devices included in the auxiliary system 706 may control and / or coordinate operations performed by various other components of the surgical system 700, such as the manipulation system 702 and the user control system 704. For example, the computing devices included in the user control system 704 may send instructions to the manipulation system 702 via one or more computing devices included in the auxiliary system. As another example, the auxiliary system 706 may receive (e.g., from the manipulation system 702) and may process image data representing images acquired by an imaging device attached to one of the manipulator arms 712.

[0071] In some examples, the assistive system 706 may be configured to present visual content to a surgical team member 710 who may not have access to images provided to the surgeon 710-1 at the user control system 704. To this end, the assistive system 706 may include a display monitor 714 configured to display one or more user interfaces, such as images of the surgical space (e.g., 2D images, 3D images, composite images, etc.), information associated with the patient 708 and / or surgical procedures, and / or any other visual content that may be used in a particular implementation. For example, the display monitor 714 may display images of the surgical space (e.g., composite images generated by system 100). In some embodiments, the display monitor 714 is implemented as a touchscreen display that the surgical team member 710 may interact with (e.g., via touch gestures) to provide user input to the surgical system 700.

[0072] The operating system 702, the user control system 704, and the auxiliary system 706 can be communicatively coupled to each other in any suitable manner. For example, as... Figure 7 As shown, the operating system 702, the user control system 704, and the auxiliary system 706 can be communicatively coupled via a control line 716, which can represent any wired or wireless communication link that may be used in a particular implementation. Therefore, the operating system 702, the user control system 704, and the auxiliary system 706 may each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0073] Figure 8 An exemplary method 800 for registering a dataset of visual representations of a surgical space is described. Although Figure 8 The illustration shows exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, combine and / or modify it. Figure 8 Any of the operations shown. Figure 8 One or more of the operations shown can be performed by an image management system such as System 100, any of its components and / or any implementation thereof.

[0074] In operation 802, the image management system may access first and second visual representations of the surgical space including the anatomical object (e.g., a dataset of the first and second visual representations). Operation 802 may be performed in any of the manner described herein.

[0075] In operation 804, the image management system can access scene segmentation information used for the first visual representation and the second visual representation. Operation 804 can be performed in any of the manner described herein.

[0076] In operation 806, the image management system may determine a reliable set of organizations based on scene segmentation information to register the dataset of the first visual representation with that of the second visual representation. Operation 806 may be performed in any of the manner described herein.

[0077] In operation 808, the image management system can determine a first region in the image of the first visual representation that depicts one or more tissues from a set of reliable tissues. Operation 808 can be performed in any of the manner described herein.

[0078] In operation 810, the image management system can determine a first plurality of feature points in a first region depicting one or more tissues in an image of a first visual representation. Operation 810 can be performed in any of the manner described herein.

[0079] In operation 812, the image management system can determine a second region depicting one or more tissues within the image of the second visual representation. Operation 812 can be performed in any of the manner described herein.

[0080] In operation 814, the image management system can determine a second plurality of feature points in a second region depicting one or more tissues within the image of the second visual representation. Operation 814 can be performed in any of the manner described herein.

[0081] In operation 816, the image management system may register the datasets of the first visual representation and the second visual representation based at least in part on the alignment of the first plurality of feature points with the second plurality of feature points. Operation 816 may be performed in any of the manner described herein.

[0082] Figure 9 Another exemplary method, method 900, is described to illustrate a dataset used for registering a visual representation of a surgical space. Although Figure 9 The illustration shows exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, combine and / or modify it. Figure 9 Any of the operations shown. Figure 9 One or more of the operations shown can be performed by an image management system such as System 100, any of its components and / or any implementation thereof.

[0083] In operation 902, the image management system may access a first visual representation of the surgical space including the anatomical object (e.g., a first dataset of the first visual representation). Operation 902 may be performed in any of the manner described herein.

[0084] In operation 904, the image management system may access a second visual representation of the surgical space (e.g., a second dataset of the second visual representation). Operation 1204 may be performed in any of the manner described herein.

[0085] In operation 906, the image management system can access information about the current stage of the procedure being performed on the anatomical object. Operation 906 can be performed in any of the manner described herein.

[0086] In operation 908, the image management system may selectively update the initial registration of the datasets of the first visual representation and the second visual representation based on the current stage of the program. Operation 908 may be performed in any of the manner described herein.

[0087] Figure 10 Another exemplary method, method 1000, is described for registering a dataset of visual representations of a surgical space. Although Figure 10 The illustration shows exemplary operation according to one embodiment, but other embodiments may omit, add, reorder, combine and / or modify it. Figure 10 Any of the operations shown. Figure 10 One or more of the operations shown can be performed by an image management system such as System 100, any of its components and / or any implementation thereof.

[0088] In operation 1002, the image management system may access a first visual representation of the surgical space including the anatomical object (e.g., a first dataset of the first visual representation). Operation 1002 may be performed in any of the manner described herein.

[0089] In operation 1004, the image management system may access a second visual representation of the surgical space (e.g., a second dataset of the second visual representation). Operation 1004 may be performed in any of the manner described herein.

[0090] In operation 1006, the image management system can access force-sensing data from the computer-aided surgical system performing procedures on the anatomical object. Operation 1006 can be performed in any of the manner described herein.

[0091] In operation 1008, the image management system can detect interactions with the anatomical object within the dataset of the first visual representation. Operation 1008 can be performed in any of the manner described herein.

[0092] In operation 1010, the image management system can model the interaction based on the force-sensing data within the dataset of the second visual representation. Operation 1010 can be performed in any of the manner described herein.

[0093] In operation 1012, the image management system can register the dataset of the first visual representation with the dataset of the second visual representation. Operation 1012 can be performed in any of the manner described herein.

[0094] In some examples, a non-transitory computer-readable medium may be provided for storing computer-readable instructions, based on the principles described herein. When executed by a processor of a computing device, the instructions may instruct the processor and / or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0095] As used herein, a non-transitory computer-readable medium can include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, a non-transitory computer-readable medium can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (“RAM”), and optical discs (e.g., compact discs, digital video discs, Blu-ray discs, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0096] Figure 11 An exemplary computing device 11000 is shown that can be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by computing device 1100.

[0097] like Figure 11 As shown, computing device 1100 may include a communication interface 1102, a processor 1104, a storage device 1106, and an input / output (“I / O”) module 1108 that are communicatively interconnected via communication infrastructure 1110. Although the exemplary computing device 1100... Figure 11 As shown in the text, but Figure 11 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 11 The components of the computing device 1100 shown.

[0098] Communication interface 1102 can be configured to communicate with one or more computing devices. Examples of communication interface 1102 include, but are not limited to, wired network interfaces (e.g., network interface cards), wireless network interfaces (e.g., wireless network interface cards), modems, audio / video connectors, and any other suitable interfaces.

[0099] Processor 1104 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or instructing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1104 may perform operations by executing computer-executable instructions 1112 (e.g., application programs, software, code, and / or other executable data instances) stored in storage device 1106.

[0100] Storage device 1106 may include one or more data storage media, devices, or configurations and may take the form of data storage media and / or devices of any type, form, and combination. For example, storage device 1106 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including the data described herein) may be stored temporarily and / or permanently in storage device 1106. For example, data representing computer-executable instructions 1112 configured to instruct processor 1104 to perform any of the operations described herein may be stored in storage device 1106. In some examples, data may be arranged in one or more databases residing within storage device 1106.

[0101] I / O module 1108 may include one or more I / O modules configured to receive user input and provide user output. I / O module 1108 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1108 may include hardware and / or software for acquiring user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0102] I / O module 1108 may include one or more means for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1108 is configured to provide graphical data to the display for presentation to a user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used in a particular implementation.

[0103] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing device 1100. For example, one or more applications 1112 residing in storage device 1106 may be configured to instruct the implementation of processor 1104 to perform one or more operations or functions associated with processing facility 104 of system 100. Similarly, storage facility 102 of system 100 may be implemented by or within storage device 1106.

[0104] Various exemplary embodiments have been described in the foregoing description with reference to the accompanying drawings. However, various modifications and alterations may be made thereto, and other embodiments may be practiced, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be considered illustrative rather than restrictive.

Claims

1. A system for registering a visual representation of a surgical space, the system comprising: The memory stores instructions; A processor, communicatively coupled to the memory and configured to execute the instructions to: Access semantic information about anatomical objects in a surgical scene; and Registering an image of the surgical scene with a model of the anatomical object, at least in part based on the semantic information about the anatomical object, wherein registering the image of the surgical scene with the model of the anatomical object, at least in part based on the semantic information, includes: The reliability of a set of tissues in the surgical scene is determined based on scene segmentation information; as well as The images of the surgical scene are registered with the model of the anatomical object, based at least in part on the reliability of the set of tissues.

2. The system of claim 1, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the scene segmentation information, a set of reliable tissues is determined to register the image of the surgical scene with the model of the anatomical object; In the image of the surgical scene, a first region of one or more tissues depicting the set of reliable tissues is identified; In the image of the surgical scene, first plurality of feature points in the first region depicting the one or more tissues are identified; In an image of the model of the anatomical object, a second region depicting the one or more tissues is identified; In the image of the model of the anatomical object, a second plurality of feature points depicting the one or more tissues are identified in the second region; and The image of the surgical scene and the model of the anatomical object are registered, at least in part, based on the alignment of the first plurality of feature points with the second plurality of feature points.

3. The system of claim 2, wherein determining the set of reliable organizations comprises selecting at least one organization from the scene segmentation information based on at least one characteristic of the selected at least one organization.

4. The system of claim 3, wherein the at least one characteristic of the selected at least one tissue includes at least one of the tissue stiffness, tissue rigidity, tissue mobility, or typical amount of tissue movement during the procedure performed on the anatomical object.

5. The system of claim 2, wherein determining the set of reliable organizations comprises selecting at least one organization from the scene segmentation information and comparing the selected at least one organization with a predefined list.

6. The system of claim 5, wherein the predefined list includes bones, vascular systems, and the anatomical objects.

7. The system of claim 1, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the scene segmentation information, a set of unreliable tissues is identified to register the image of the surgical scene with the model of the anatomical object; In the image of the surgical scene, a first region of one or more tissues depicting the set of unreliable tissues is identified; In the image of the surgical scene, determine a first plurality of feature points in a region other than the first region of the image of the surgical scene; In an image of the model of the anatomical object, a second region depicting the one or more tissues is identified; In the image of the model of the anatomical object, determine a second plurality of feature points in a region other than the second region of the image of the model of the anatomical object; and The image of the surgical scene is registered with the model of the anatomical object, based at least in part on the alignment of the first plurality of feature points with the second plurality of feature points.

8. The system of claim 1, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the scene segmentation information, a first group and a second group of tissues are determined in the group of tissues. Based on the reliability of the tissues, the first group and the second group are determined to register the image of the surgical scene with the model of the anatomical object. In the image of the surgical scene, a first region depicting one or more tissues of the first group of tissues is identified; In the image of the surgical scene, a second region depicting one or more tissues of the second group of tissues is identified; In the image of the surgical scene, a first plurality of feature points in the first region are determined; In the image of the surgical scene, a second plurality of feature points are determined in the second region; In an image of the model of the anatomical object, identify a third region depicting one or more tissues of the first group of tissues; In the image of the model of the anatomical object, a fourth region depicting one or more tissues of the second group of tissues is identified; In the image of the model of the anatomical object, a third plurality of feature points are determined in the third region; In the image of the model of the anatomical object, a fourth plurality of feature points are determined in the fourth region; The image of the surgical scene is registered with the model of the anatomical object based at least in part on the weighted alignment of the first plurality of feature points with the third plurality of feature points and the second plurality of feature points with the fourth plurality of feature points.

9. The system of claim 1, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the scene segmentation information, a first boundary between the first tissue and the second tissue is determined in the image of the surgical scene; Based on the scene segmentation information, a second boundary between the first tissue and the second tissue is determined in the image of the model of the anatomical object; and The image of the surgical scene is registered with the model of the anatomical object, based at least in part on the alignment of the first boundary and the second boundary.

10. The system of claim 1, wherein the semantic information includes the current stage of a program executed on the anatomical object.

11. The system of claim 10, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the current stage of the program, the initial registration of the image of the surgical scene with the model of the anatomical object is selectively updated.

12. The system of claim 10, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: A set of procedural stages that determine when the dissected object moves beyond a threshold amount; and When the current stage of the program is one of the set of program stages, selectively avoid registering the image of the surgical scene with the model of the anatomical object.

13. The system of claim 1, wherein the semantic information includes force-sensing data of a computer-aided surgical system performing a program on the anatomical object.

14. The system of claim 13, wherein registering the image of the surgical scene with the model of the anatomical object based at least in part on the semantic information comprises: Based on the force sensing data, interactions with the anatomical object are detected in the image of the surgical scene; and The interaction is modeled in the model of the anatomical object.

15. The system of claim 1, wherein the model of the anatomical object comprises one of the following: Ultrasound scan of the anatomical object or the surgical space; Computed tomography ("CT") scan of the anatomical object or the surgical space; Magnetic resonance imaging ("MRI") scan of the anatomical object or the surgical space; or Fluorescence imaging scan of the anatomical object or the surgical space.

16. The system of claim 1, wherein the model of the anatomical object comprises at least one of a virtual object or a annotation.

17. A method for registering a visual representation of a surgical space, the method comprising: The processor accesses scene segmentation information of images of a surgical scene including a surgical space containing anatomical objects and a model of the anatomical objects. The processor determines a set of reliable tissues based on the scene segmentation information to register the dataset of images of the surgical scene with the model of the anatomical object; The processor determines, in the image of the surgical scene, a first region of one or more tissues depicting the set of reliable tissues; The processor determines, in the image of the surgical scene, a first plurality of feature points in the first region depicting the one or more tissues; The processor determines a second region depicting the one or more tissues in an image of the model of the anatomical object; The processor determines, in the image of the model of the anatomical object, a second plurality of feature points in the second region depicting the one or more tissues; and The processor registers the image of the surgical scene with the model of the anatomical object based at least in part on the alignment of the first plurality of feature points and the second plurality of feature points.

18. A method for registering a visual representation of a surgical space, the method comprising: Force-sensing data of a computer-aided surgical system that accesses the processor to execute programs on anatomical objects in the surgical space; The processor detects interactions with the anatomical object in images of the surgical scene within the surgical space. The processor models the interaction in the model of the anatomical object based on the force sensing data; and The processor registers the image of the surgical scene with the model of the anatomical object.

Images