Weakly supervised generation of operating room semantics from 3D scene representations
Patent Information
- Application Number
- JP2026520375
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-10-04
- Filing Date
- 2024-10-04
- Publication Date
- 2026-10-13
Smart Images
Figure 2026533911000001_ABST
Abstract
Description
[Technical Field]
[0001] (Cross-reference of related applications) This application claims priority to U.S. Provisional Application No. 63 / 587,990, filed on 4 October 2023, which is incorporated herein by reference.
[0002] (Field of Invention) This disclosure relates to medical image processing, and more specifically to semantic segmentation of operating room scenes using weakly supervised learning. [Background technology]
[0003] The digitalization of operating rooms has gained significant momentum in recent years, driven by its potential to improve patient outcomes and enhance surgical procedures. Robot-assisted surgery has demonstrated advantages such as reduced blood loss, shorter recovery times, and shorter hospital stays. [Overview of the project] [Problems that the invention aims to solve]
[0004] However, for robotic systems to autonomously interact with hospital staff, surgical tools, and patients, it is desirable that they achieve a high level of detailed understanding of the highly complex operating room environment. [Means for solving the problem]
[0005] According to one aspect of this disclosure, a method for semantic segmentation of an operating room scene is provided. The method includes receiving a sequence of three-dimensional (3D) representations (e.g., point cloud or voxel-based representations) of an operating room scene from a set of sensors. A machine learning model is applied to the sequence of 3D representations to predict semantic segmentation labels for points in the sequence of 3D representations. The machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on the spatial and temporal relationships between supervoxel features. Semantic segmentation labels are output.
[0006] According to other aspects of this disclosure, a surgical robot in robot-assisted surgery may be controlled at least in part on semantic segmentation labels. A surgical phase prediction model may also be applied to semantic segmentation labels to predict the surgical phase of a surgical procedure. A surgical robot in robot-assisted surgery may be controlled at least in part on the surgical phase.
[0007] A weakly supervised semantic segmentation method may involve obtaining a set of training 3D representations. The set of training 3D representations is divided into supervoxels based on geometric boundaries. Sparse annotations are obtained for a sparse subset of the supervoxels. In each training iteration, labels are propagated from the subset of supervoxels to the unannotated supervoxels based on spatial and temporal relationships. The model parameters of the machine learning model are updated based on the loss function. The machine learning model may include a segmentation network that models the semantic class of each point in the sequence of 3D representations, and a relational network that models the embedding of each class of semantic segmentation labels.
[0008] Label propagation may involve constructing a graph representing the similarities between supervoxels and performing graph-based label propagation to assign labels to unannotated supervoxels. A matching matrix can be applied to model the temporal consistency between corresponding supervoxels in consecutive 3D representations within a set of training 3D representations. The matching matrix may be applied using at least one of nearest neighbor matching, optical flow matching, or scene flow matching. The matching matrix may be iteratively updated during training of the machine learning model to refine the temporal consistency between corresponding supervoxels.
[0009] The sensor set may include multiple depth cameras positioned at different viewpoints within the operating room. The 3D representation sequence may include a temporal sequence of RGB-D data (e.g., point cloud or voxel-based representation) captured at different points in time during the surgical procedure. Semantic segmentation labels may correspond to different objects or classes of objects present within the operating room scene.
[0010] According to another aspect of this disclosure, a method is provided for training a machine learning model for semantic segmentation of an operating room scene. A set of training 3D representations of the operating room scene is obtained. The set of training 3D representations is divided into supervoxels based on geometric boundaries. Sparse annotations are obtained for at least a subset of the supervoxels. In each training iteration, labels are propagated from the subset of supervoxels to the unannotated supervoxels based on spatial and temporal relationships. The model parameters of the machine learning model are updated based on a loss function. The machine learning model is output.
[0011] According to another aspect of the present disclosure, the machine learning model may include a segmentation network that models the semantic class of each point in the sequence of 3D representations, and a relationship network that models the embedding of each class of semantic segmentation labels. Propagating labels may include constructing a graph representing the similarity between supervoxels, and performing graph-based label propagation to assign labels to unannotated supervoxels. The matching matrix can be applied to model temporal consistency between corresponding supervoxels in successive 3D representations within the set of training 3D representations. The matching matrix may be applied using at least one of nearest neighbor matching, optical flow matching, or scene flow matching. The matching matrix may be iteratively updated during training of the machine learning model to refine the temporal consistency between corresponding supervoxels.
[0012] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing instructions for semantic segmentation of an operating room scene. The instructions, when executed by a processor, cause the processor to perform steps including receiving a sequence of 3D representations of the operating room scene from a set of sensors. A machine learning model is applied to the sequence of 3D representations to predict semantic segmentation labels for points within the sequence of 3D representations. The machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on spatial and temporal relationships between supervoxel features. The semantic segmentation labels are output. [Brief Description of the Drawings]
[0013] [Figure 1] Fig. 2 illustrates a block diagram of a computing system associated with generating semantic information related to a medical environment, in accordance with aspects of the present disclosure. [Figure 2]Shows a block diagram of a processing system for training and applying one or more machine learning models for inferring semantic information associated with a medical environment, according to one embodiment. [Figure 3] Shows a block diagram of an inference module and an output system for generating semantic information associated with a medical environment, in accordance with aspects of the present disclosure. [Figure 4] Shows a flowchart of a method for training a machine learning model for semantic segmentation of an operating room scene, according to one embodiment. [Figure 5] Shows a flowchart for a semantic segmentation process, in accordance with aspects of the present disclosure. [Figure 6] Shows a flowchart of a method for semantic segmentation of a 3D representation sequence, according to one embodiment. [Figure 7] Shows a set of images representing labeling of segmentation techniques associated with the disclosed embodiments. [Figure 8] This is a table representing performance data associated with a first experiment related to the operation of the disclosed embodiments. [Figure 9] This is a table representing performance data associated with a second experiment related to the operation of the disclosed embodiments. [Figure 10] This is a table representing performance data associated with a third experiment related to the operation of the disclosed embodiments. DESCRIPTION OF EMBODIMENTS
[0014] The figures (FIGS.) and the following description illustrate specific embodiments for illustrative purposes only. Those skilled in the art will readily understand from the following description that alternative embodiments of the structures and methods illustrated herein can be adopted without departing from the principles described herein. Herein, several embodiments are referenced, and these embodiments are illustrated in the accompanying figures. Where possible, similar or identical reference numerals may be used in the drawings, and these reference numerals may indicate similar or identical elements.
[0015] This disclosure relates to a method and system for semantic learning associated with an operating room scene. Using various machine learning techniques, the embodiments described may infer segmentation data (e.g., the location and / or characteristics of objects in a medical environment) in real time or substantially in real time based on sensor data from a set of cameras. This segmentation process may include acquiring video footage of the operating room and dividing the frame into regions labeled as specific objects or classes of objects. The segmentation data may be applied as input to another machine learning model that infers surgical phases (e.g., steps of a medical procedure) from the segmentation data. Semantic information (including segmentation data, surgical phase data, or a combination thereof) may be output to various systems associated with the medical environment. For example, semantic information may be applied as a control and / or configuration input to enable a surgical robot to adapt its actions in real time based on detected states. The semantic information may also be output to various medical support and / or analysis systems that can provide medical staff with information related to the medical procedure (in real time or otherwise). Semantic information can significantly improve efficiency, enhance safety, and improve patient outcomes in the performance of medical procedures.
[0016] The embodiments described include techniques for applying weakly supervised semantic segmentation to 3D representation sequences (i.e., volumetric images or point cloud sequences) captured from operating room sensors. For example, training data may be labeled using an approach in which an annotationer provides a single label for each identifiable object in the scene. The learning algorithm infers labels for unlabeled regions by applying a pseudo-labeling technique that leverages the temporal and spatial consistency of the 3D representation sequences. This approach significantly reduces the annotation burden while maintaining high segmentation accuracy compared to conventional fully supervised learning techniques.
[0017] Figure 1 shows a computing environment 100 for semantic segmentation of an operating room scene. The computing environment 100 includes a processing system 104, client devices 106, a network 108, a training data store 110, and a medical environment 120.
[0018] The medical environment 120 represents a medical environment such as an operating room for performing surgery or other medical procedures. The medical environment 120 includes a patient 116, medical staff 118, a medical robot 114, and one or more sensors 112. The medical environment 120 may also include various other elements typically found in an operating room scene, such as an operating table, medical instruments, tool benches, lighting systems, monitoring systems, or other equipment. At least the medical robot 114 and the sensors 112 may have connected devices that can communicate via a network 108 and / or communicate directly with other devices via wired or wireless communication mechanisms.
[0019] The medical robot 114 may include various elements to assist in surgery or other medical procedures under the direction of a physician. For example, the medical robot 114 may have one or more robotic arms equipped with specialized surgical instruments. These arms may be controlled by the surgeon via a head-mounted display device and connected controllers, or through other control mechanisms, using a console with three-dimensional visualization capabilities. The medical robot 114 may incorporate technologies such as motion scaling technology, which enables the surgeon's hand movements to be translated into smaller and more precise actions by the robotic instruments. The medical robot 114 may also feature a force feedback mechanism, which can provide the surgeon with tactile feedback to improve control and tissue manipulation. In addition, the robot may be equipped with an advanced imaging system, enabling real-time visualization of the surgical site and integration with preoperative imaging data. The medical robot 114 may be designed to have multiple degrees of freedom, enabling complex movements and access to hard-to-reach areas within the patient's body. Different types of medical robots 114 can operate with varying levels of autonomy, and in some cases, may be fully autonomous.
[0020] The medical robot 114 may be controlled and / or configured based in part on semantic information of the medical environment 120. Semantic information may include, for example, segmentation information describing the position, movement, and / or other characteristics of objects or classes of objects within the medical environment 120. Semantic information may also include surgical phase information indicating the current and / or expected steps in a medical procedure. Surgical phase information may be partially derived from segmentation data or obtained independently. Semantic information can improve the control capabilities and safe operation capabilities of the medical robot 114. By leveraging detailed information describing the spatial layout, object positions, and dynamic movements within the medical environment 120, the medical robot 114 can adapt its actions to detected conditions in real time. For example, the robot may use semantic data to identify and avoid obstacles, adjust its trajectory to adapt to the movements of medical staff 118, and precisely position surgical instruments on a nearby tool bench. The medical robot 114 can further anticipate and prepare for subsequent tasks to enable the efficient and accurate completion of medical procedures. Semantic information can also be employed to ensure safe human-robot interaction by enabling the medical robot 114 to predict and respond to the intentions and movements of medical staff 118. In addition, the medical robot 114 can use semantic information to monitor the overall state of the operating room and warn staff of potential safety hazards or deviations from standard protocols. By continuously updating its understanding of the dynamic medical environment 120, the medical robot 114 can adapt to changing conditions and thereby improve the efficiency and safety of surgical procedures.
[0021] A set of sensors 112 captures a temporal sequence of the 3D representation of the operating room scene. The 3D representation may include a 3D point cloud or a voxel-based representation. Here, the 3D point cloud may include color and depth information for a discrete set of scattered points that do not necessarily follow a regular grid structure. In a voxel-based representation, each voxel represents a volume element within a regular grid structure.
[0022] In exemplary embodiments, the sensor set 112 may include multiple color and depth cameras (e.g., RGB-D cameras), each capturing a sequence of 3D point clouds. For example, the cameras may capture red, green, blue, and depth information in the point cloud representation. The sensors 112 may be positioned at various locations within the operating room to capture different viewpoints of the medical environment. The point clouds captured by individual sensors 112 may be aggregated into a voxel-based representation of the medical environment. This aggregation may be performed using various image registration and image processing techniques that combine and align point cloud data from different sensors 112. Alternatively, the sensors 112 may directly capture the voxel-based representation without necessarily converting from the point cloud representation.
[0023] Sensor 112 may capture and output such data at regular frame intervals (e.g., 15 frames per second, 30 frames per second, 60 frames per second, or other intervals) to generate a 3D representation sequence. Each frame of the 3D representation sequence contains a 3D representation of a medical environment in which each point is represented by color data (e.g., red, green, and blue values) and positional information (e.g., x, y, and z coordinates).
[0024] In other embodiments, different types of sensors 112 may encode color information using different color space representations that are not necessarily RGB. Alternatively, the medical environment 120 may use a grayscale sensor that does not contain color information. In other implementations, the sensor 112 may not necessarily capture depth information, which may instead be inferred based on various multi-view image processing techniques. Alternatively, a 3D representation may be captured from a single sensor 112.
[0025] The processing system 104 infers semantic information about the medical environment 120 based on sensor data from sensor 112. The processing system 104 may employ one or more machine learning models trained to output segmentation data associated with a 3D representation sequence (e.g., a 3D point cloud sequence or a voxel-based representation sequence). This segmentation data may indicate the inferred object class for each point in the 3D representation sequence. The processing system 104 may also employ one or more additional machine learning models trained to infer surgical phase information based on the segmentation data, either alone or in combination with various other inputs. The processing system 104 can infer semantic information in substantially real-time or near real-time to generate output relevant to the ongoing medical procedure. For example, as described above, the semantic information may be output to a medical robot 114, which may operate at least partially on this semantic information. In further embodiments, semantic information may be output to an analysis system that can output support information (e.g., warnings, recommendations, etc.) to medical staff 118 during a medical procedure (e.g., via a display screen, speaker, or other output device). For example, a virtual surgical assistant can provide step-by-step guidance related to different steps of a medical procedure, notify medical staff of any abnormalities that may occur during the course of the medical procedure, or generate other information from inferred semantic data that may be useful to medical staff during the procedure. In yet another embodiment, the processing system 104 may output the semantic information to an analysis system that can aggregate such semantic information across the course of multiple medical procedures and derive various statistical information that may be relevant to improving future medical procedures. For example, such statistical information may be relevant to evaluating the performance of medical staff, evaluating patient outcomes, or evaluating the use of specific medical devices.
[0026] The processing system 104 can also facilitate the training of one or more machine learning models in the training process based on training data in the training data store 110. Here, the training data in the training data store 110 may include a sequence of 3D representations captured from an operating room environment, along with sparse annotations indicating object classes for subsets of points or regions in the 3D representation. The training process may include a weakly supervised approach, in which the machine learning model is trained to propagate sparse annotations to unlabeled regions of the 3D representation based on spatial and temporal relationships. This approach may enable effective training with reduced annotation effort compared to a fully supervised method. The training process may utilize various techniques such as graph-based label propagation, spatiotemporal consistency constraints, and iterative refinement of model parameters. Further details of the training process are described in relation to Figures 4 and 5.
[0027] The processing system 104 may be implemented using on-site computing or storage systems, cloud computing or storage systems, or a combination thereof, and may be implemented using local or cloud-based servers, which may include physical or virtual machines, or a combination thereof. Cloud-based servers may include private cloud systems, public cloud systems, hybrid public / private cloud systems, or a combination thereof. Thus, the processing system 104 can be local, remote, and / or distributed to the medical environment 120. Furthermore, different parts of the processing system 104 may run on different remote servers, and the various system elements of the processing system 104 may be connected in a communicative manner via the network 108.
[0028] Network 108 includes a communication path for communication between the medical imaging device 102, the processing system 104, and the client device 106. Network 108 may include one or more local area networks and / or one or more wide area networks (including the Internet). Network 108 may also include one or more direct wired or wireless connections (e.g., Ethernet®, WiFi, Cellular Protocol, WiFi Direct, Bluetooth®, Universal Serial Bus (USB), or other communication links).
[0029] The client device 106 may include any computing device for inputting information and / or control commands, browsing various information, and / or interacting with various data. The client device 106 may run one or more applications that include a user interface for performing various functions described herein. The applications may include web-based applications accessible by a web browser or locally installed applications. The applications may enable the client device 106 to facilitate the management functions of the processing system 104, such as configuring machine learning training parameters and adding or removing training data. One or more client devices 106 may be further employed in the annotation process to obtain sparse labels used in the training data from one or more expert labelers. The client device 106 may be further configured to enable browsing and / or interacting with semantic information and / or analytical data derived from semantic information generated by the processing system 104.
[0030] The client device 106 may include, for example, a mobile phone, tablet, laptop or desktop computer, or other computing device. The client device 106 may include conventional computer hardware such as a display, an input device (e.g., a touchscreen), memory, a processor, and a non-temporary computer-readable storage medium that stores instructions for execution by the processor in order to perform the functions described herein.
[0031] Figure 2 is a block diagram showing an example of a processing system 104. The processing system 104 includes one or more processors 202 and one or more storage media 204. The one or more storage media 204 includes various functional modules (implemented as instructions executable by one or more processors 202), including a user interface module 206, a management interface module 208, a data acquisition module 210, a training module 212, an inference module 214, a model store 216, an analysis module 218, and an output module 220. In alternative embodiments, the processing system 104 may include different or additional modules. The one or more processors 202 and the one or more storage media 204 do not necessarily have to be located in the same place and may be distributed (for example, in a cloud architecture).
[0032] The user interface module 206 facilitates the server-side functionality of the user interface accessible on the client device 106. In exemplary operation, the user may input various information via the user interface on the client device 106, such as configuration or control commands, which are communicated to the user interface module 206. The user interface module 206 can further facilitate the presentation of various outputs from the processing system 104. For example, the user interface module 206 may output semantic information inferred by the processing system 104 (text, images, videos, or other forms of representation), or various analytical data derived therefrom (e.g., graphs, charts, text, etc.).
[0033] Although the user interface module 206 is shown as a component of the processing server 104 in Figure 2, all or a subset of the functions of the user interface module 206 may instead be executed on the client device 106. For example, the client device 106 may download an application from the processing system 104 that includes all or some of the functions of the user interface module 206. The client device 106 may execute the instructions associated with these functions locally.
[0034] The management interface module 208 facilitates various management functions associated with the operation of the processing system 104. For example, the management interface module 208 may provide an interface that enables the configuration of various machine learning parameters associated with the training module 212 and the inference module 214. Furthermore, the management interface module 208 may provide an interface for controlling the application version and / or access of the client device 106, or perform other management functions. The management interface module 208 can further access, aggregate, and present various statistical data associated with the operation of the training module 212 and / or the inference module 214 in order to enable the machine learning model to be evaluated and improved.
[0035] The data acquisition module 210 facilitates the acquisition of training data from the training data store 110 and / or other data sources and can perform various preprocessing tasks. For example, the data acquisition module 210 can interpret data from native image and / or video formats and convert it to a standardized format. The standardized format may include standard 3D data representations, encoding formats, resolutions, etc. The data acquisition module 210 can further encode metadata (which may include labels) associated with the training data into a standard format for use by the training module 212 and the inference module 214. During real-time operation, the data acquisition module 210 can similarly process data received from the sensor 112 to generate 3D representation sequences in a standard format.
[0036] The training module 212 trains one or more machine learning models based on a training dataset for inferring semantic information about the medical environment 120. As further described below, the training module 212 may train one or more segmentation models using a weakly supervised training approach in which the machine learning models are derived from sparsely labeled 3D representations. The training process may include a pseudo-labeling process for generating additional labels, where ground truth labels are propagated to unlabeled regions of the 3D representation based on spatial and temporal relationships. The training module 212 may further train one or more surgical phase models to generate surgical phase information from the segmentation data (e.g., using a supervised learning approach).
[0037] The inference module 214 applies one or more trained models to an input 3D representation sequence (captured by sensors 112 in the medical environment 120) to generate inferences for semantic classification. Here, the models output inferred labels for each point in the 3D representation sequence, thereby dividing the 3D representation sequence into respective object classes representing different objects in the medical environment 120. The inference module 214 can further infer surgical phases by applying one or more surgical phase models to the segmentation data. Inferences can be generated substantially in real time during the course of medical procedures. As described above, the inferred semantic information can be output to the robot control system of the medical robot 114, to a surgical support device that outputs information via a display screen, speaker, or other output device, to an analysis engine, or to other output systems.
[0038] The analysis module 218 can generate various analyses associated with the inferences and / or other relevant input data (e.g., patient data, physician data, facility data, medical device data, etc.) generated from the inference module 214. For example, the analysis system can generate statistical data characterizing medical procedures derived from semantic inference, which can be used to improve medical procedures, evaluate medical staff, predict patient outcomes, or for other purposes. The analysis module 218 may employ various rule-based models and / or machine learning models to generate the various types of analytical outputs described herein.
[0039] The model store 216 stores one or more machine learning models generated by the training module 212. The model store 216 can be accessed by the inference module 214 to apply one or more models during the inference process. The model store 216 may further store various analytical models that can be applied by the analysis module 218 to generate various analytical data, as described herein.
[0040] The output module 220 may output semantic information or other information derived therefrom to one or more output systems. For example, the output module 220 may interface with the medical robot 114 to generate commands and / or configuration data for controlling the robot 114 based in part on the semantic information. The output module 220 may further output the semantic information to a real-time surgical support system, an analysis system, or other system that processes semantic information.
[0041] Figure 3 shows an exemplary data flow diagram associated with the processing system 104. In this example, the inference module 214 includes a segmentation inference module 304 and a surgical phase inference module 312. The segmentation inference module 304 receives a 3D representation 302 and applies a segmentation model 306 to infer segmentation data 308. The surgical phase inference module 312 receives the segmentation data 308 and applies a surgical phase model 310 to infer surgical phase data 314. In an alternative embodiment, the surgical phase inference module 312 may operate on a latent representation of the segmentation data 308 that is not necessarily in a form of segmentation data that is interpretable by humans. In a further embodiment, a single inference module 214 may apply a model that outputs both the segmentation data 308 and the surgical phase data 314 together.
[0042] The output system 330 may include one or more subsystems that utilize segmentation data 308, surgical phase data 314, or a combination thereof. These subsystems may include, for example, a robot control system 316, an analysis system 318, and a surgical support system 320. In further embodiments, different or additional types of output systems 330 may be included.
[0043] The robot control system 316 can use segmentation data 308 and surgical phase data 314 to guide and control the actions of the medical robot 114 during a medical procedure. For example, the robot control system 316 can interpret semantic information to obtain information about the current state of the medical environment 120, including the positions of medical staff 118, instruments, and the patient 116. Using this information, the robot control system 316 can plan and execute precise movements, avoid obstacles, and adapt to changes in the surgical environment. In some cases, the system may anticipate the next surgical phase and prepare the robot 114 for a specific task or instrument change.
[0044] The analysis system 318 can process and analyze segmentation data 308 and surgical phase data 314 to generate insights, performance metrics, or other analytical data related to surgical procedures. The system can track various aspects of surgery, such as instrument use, staff movements, and procedure duration. The analysis system 318 can compile this information across multiple surgeries to identify trends, optimize workflows, and improve overall surgical efficiency. In some implementations, the system may generate reports or visualizations to help medical teams and hospital administrators make data-driven decisions regarding resource allocation, training needs, or process improvements.
[0045] The surgical support system 320 can utilize segmentation data 308 and surgical phase data 314 to provide real-time support and guidance to the surgical team. For example, the surgical support system 320 may include a virtual assistant that can display relevant information on a monitor in the operating room, highlighting key elements of the current surgical phase or alerting staff to potential problems. The surgical support system 320 may also provide voice cues or commands to help coordinate the team's actions and ensure compliance with surgical protocols. In some embodiments, the system can integrate with other hospital systems to retrieve and display patient-specific information or recommended action steps based on the current phase of the surgery.
[0046] In various embodiments, functions associated with one or more output systems 330 may be implemented within the processing system 104 or as a separate control system. For example, in some embodiments, the processing system 104 operates to derive control signals for a medical robot 114 based on semantic information. In another embodiment, the processing system 104 may output semantic information to an independent robot control system that derives control signals. Similarly, the processing system 104 may directly generate surgical assistance and / or analytical data, or it may output semantic information that enables an independent system to perform the functions described herein.
[0047] Figure 4 is a flowchart showing a method for training a machine learning model for semantic segmentation of an operating room scene. Temporal sequence
[0048]
number
[0049]
number
[0050]
number
[0051] The method begins in step 402, when training data is received. The training data may include a sequence of 3D representations acquired from a set of sensors 112, such as RGB-D cameras, positioned at various locations within the operating room. Each 3D representation in the sequence may represent a snapshot of the operating room scene at a specific point in time, with each point in the cloud encoding color and depth information. The sequence of 3D representations may be captured at regular intervals, such as every few milliseconds, to provide a temporal sequence of the operating room scene. Furthermore, sets of 3D representations from different sensors 112 may be aggregated to generate a single 3D representation. Alternative implementations may use different types of 3D representations, such as colorless (e.g., grayscale) 3D representations. In some implementations, depth information may be derived indirectly from multi-view 2D images using various image processing techniques.
[0052] In step 404, sparse ground truth labels are obtained for a sequence of 3D representations. These labels may be manually annotated by an expert and may identify a subset of points in the 3D representation belonging to a specific object or class of objects within the operating room scene. For example, the labels may identify points corresponding to surgical instruments, medical staff, patients, or other objects in the scene. The labels can be sparse in the sense that not all points in the 3D representation are labeled. Instead, only a small fraction of the points may be labeled, such as points randomly selected for each class. This approach reduces the annotation burden and enables weakly supervised training of machine learning models. In exemplary implementations, annotations can generally follow a "one thing, one click" approach, where the annotator is asked to provide one label for each individually identified object and then select any point on an object (e.g., by clicking with a pointing device) to provide a label that identifies the object.
[0053] In step 406, the sequence of 3D representations is 3D representation X t Supervoxels based on geometric boundaries in
[0054]
number
[0055]
number
[0056]
number
[0057] At step 408, an iterative process is executed to iteratively update model parameters and propagate labels to unlabeled supervoxels. By propagating learned information to unlabeled supervoxels, the level of supervision can be significantly increased through pseudo-labeling. This process may include constructing a graph representing the similarity between supervoxels, and performing graph-based label propagation to assign labels to unlabeled supervoxels. Label propagation may be guided by a matching matrix that models both spatial and temporal consistency between corresponding supervoxels within a 3D representation and within consecutive 3D representations. The matching matrix may be initialized using nearest neighbor matching, optical flow matching, or scene flow matching, and may be iteratively updated during the training process to refine the spatiotemporal consistency between corresponding supervoxels. The iterative process may continue until a convergence criterion such as a predetermined number of iterations or a threshold change in model parameters is satisfied.
[0058] At step 410, the trained machine learning model is output. The trained model may be stored in the model store 216 for later use in semantic segmentation of new 3D representations. The model is capable of predicting a label for every point in the 3D representation, thereby enabling semantic segmentation of the entire operating room scene.
[0059] Figure 5 is a flowchart showing an example of a semantic segmentation process 408 employing the principle described above. In step 502 of the iterative process, the semantic segmentation network
[0060]
number
[0061]
number
[0062]
number
[0063]
number
[0064] In step 504, the relational network R Ψ 3D representation X t Applied to, control loss
[0065]
number
[0066]
number
[0067] In step 506, the labels are segmented into a network.
[0068]
number
[0069]
number
[0070]
number
[0071]
number
[0072]
number
[0073]
number
[0074]
number
[0075] Instead of considering all supervoxel pairs as candidates in a single static acquisition during this extension, graph propagation considers a pair of frames X t and X t+1 The time dimension can be extended by pooling all supervoxels over time and creating a fully connected graph across both sets of supervoxels. This general approach can be further optimized in light of the observation that the nearest supervoxels in adjacent time steps are likely to describe similar regions in the 3D representation. Thus, the training process is supervoxel matching matrix
[0076]
number
[0077]
number
[0078]
number
[0079]
number
[0080]
number
[0081] Various approaches can be employed to initialize the matching matrix and improve temporal pseudo-label propagation, such as nearest neighbor, unsupervised optics, and scene flow prior information. After initialization through one of these prior information sources, the matching can be iteratively updated during training to establish a link between temporal consistency and semantic understanding. To enhance this dynamic matching and account for potentially inaccurate matching, the process involves a relational net R for refining the supervoxel matching matrix M. Ψ Features can be incorporated additionally. This technique takes into account potentially inaccurate matching and allows the network to use semantics during training to improve temporal correspondence.
[0082] Formally, a matching update for a single entry can be defined as follows:
[0083]
number
[0084] The updated matching is based on the supervoxel with the highest matching probability, i.e.,
[0085]
number
[0086]
number
[0087] It can be further regularized using the updated matching matrix.
[0088]
number
[0089]
number
[0090]
number
[0091]
number
[0092]
number
[0093]
number
[0094] In step 508, the loss function
[0095]
number
[0096]
number
[0097]
number
[0098]
number
[0099]
number
[0100] In step 510, the loss function
[0101]
number
[0102]
number
[0103]
number
[0104] The aforementioned semantic segmentation process 408 provides a weakly supervised learning approach for semantic segmentation of operating room scenes. By leveraging sparse annotations and spatiotemporal information, process 408 reduces annotation effort and enables effective training of semantic segmentation networks and relational networks. Process 408 also enables the generation of accurate semantic segmentation labels for new 3D representations, thereby facilitating the understanding and analysis of complex surgical environments.
[0105] Referring to Figure 6, the flowchart illustrates a method for applying a machine learning model trained to infer semantic segmentation of a 3D representation. The method begins in step 602, when a sequence of 3D representations representing the operating room scene is received. The sequence of 3D representations may be captured by a set of sensors 112, such as RGB-D cameras, positioned at various locations within the operating room. Each 3D representation in the sequence may represent a snapshot of the operating room scene at a specific point in time, with each point in the cloud encoding color (or grayscale intensity) and depth information. The sequence of 3D representations may be captured at regular intervals, such as every few milliseconds, to provide a temporal sequence of the operating room scene.
[0106] In step 604, the inference model is applied to a sequence of 3D representations. The inference model may be a machine learning model trained using the weakly supervised semantic segmentation method described above. The inference model can predict a semantic segmentation label for each point in the 3D representation based on the point's color and depth information, as well as the spatial and temporal relationships between the points. The inference model outputs a set of predicted labels for each point in the 3D representation, thereby segmenting the 3D representation into different regions corresponding to different objects or classes of objects within the operating room scene.
[0107] Finally, in step 606, the method generates an output representing a segmented 3D representation sequence based on the predicted labels. In one embodiment, the output may be a sequence of segmented 3D representations, where each point in the 3D representation is assigned a predicted label. The output can be used to provide a detailed understanding of the operating room scene, including the positions and movements of various objects and individuals in the scene. This information can be used to control the medical robot 114, to provide input to a surgical phase prediction model as described above, or to generate surgical assistance and / or analytical data associated with the medical environment.
[0108] Figure 7 shows exemplary images representing the semantic segmentation techniques described herein. Figure 702 shows an exemplary image of a motion scene that may be represented by a 3D representation. Figure 704 shows a sparse annotation technique in which each object is provided with a label. Each label can be associated with any point within the object. Therefore, the labels can be very sparse, and there are significantly fewer labels than there are points in the 3D representation. Figure 706 shows an exemplary representation of a segmentation output in which objects in the scene are segmented into different object classes. Figure 7 represents only a simplified conceptual example of the techniques described herein and does not represent actual 3D representations or segmentation outputs from exemplary real-world applications.
[0109] Figures 8–10 are tables showing performance results from experiments conducted using the techniques described herein. In one such experiment, the overall impact of temporal guidance in a weakly supervised setting was quantified through three-fold cross-validation by comparing the described approach with a baseline technique described in Liu, Z., Qi, X., Fu, CW, “One thing one click: A self-training approach for weakly supervised 3D semantic segmentation” (CVPR, pp. 1726–1736 (2021), referred herein as “one-thing-one-click” or “OTOC”). Due to privacy concerns associated with medical settings, color information may not be available in some training datasets; therefore, additional experiments were performed without RGB features to contextualize performance in such settings. Furthermore, different unsupervised prior information was explored to improve semantic predictions, namely (i) nearest neighbor matching, (ii) optical flow matching, and (iii) scene flow matching. For each piece of prior information, the matching matrix M[t,t+1] was then initialized based on the acquired flow. Using any single piece of prior information resulted in an improvement over OTOC.
[0110] The results in Figure 8 (Table 1) show that the described embodiments consistently outperformed the baseline, with and without RGB features, when initialized with nearest-neighbor temporal prior information and learned updates. As shown in Table 1, the described technique outperformed OTOC by approximately 6% mIoU for non-color point clouds and improved to approximately 10% for color point clouds, demonstrating that the described learned temporal matching can more effectively leverage color similarity across time steps.
[0111] In another experiment, the impact of adding up to three additional clicks on OTOC performance was quantified. This experiment was performed on the first training and validation splits using a varying number of click annotations for each class.
[0112] As shown in Figure 9 (Table 2), the results indicated that a 3x or 4x increase in annotations resulted in an improvement of approximately 8% in mIoU. Performance saturated at 3 clicks. The baseline did not reach 73% mIoU of the proposed method, even with increased monitoring. This suggests that temporal consistency not only increases the monitored signal but also enables a more robust overall feature representation.
[0113] To further evaluate the quality of semantic predictions, we assessed their impact on surgical workflow analysis. Using a ResNet50 backbone, we performed quadrant cross-validation with random splits across different surgeries on a larger, complete RGB-D video dataset. Semantic predictions were inferred for each of the 582k fused point clouds (inference at 12.5 fps) using the best-performing segmentation model, and these were reprojected onto two cameras. The performance of raw RGB, depth, and semantic label inputs was then compared to fusing the latter two via late fusion.
[0114] As shown in Figure 10 (Table 3), RGB achieved the best performance across both cameras. Semantic prediction alone performed significantly lower than RGB or depth. However, the fact that semantic prediction from the network (achieving 73% mIoU) can be used for this challenging task demonstrates the potential advantages of segmentation output for surgical scene understanding. Furthermore, when combined with depth through late fusion, the results improved accuracy by approximately 6% and 0.6% across depth, respectively, for surgical and workflow cameras. This suggests that segmentation maps can replace RGB features when they are unavailable for privacy reasons.
[0115] This disclosure presents a semantic segmentation method for surgical scene understanding that significantly reduces the annotation burden by leveraging the temporal consistency of point cloud sequences. The results demonstrate the effectiveness of this approach for point clouds from surgical phase recognition datasets, enhanced with manual 3D annotations. By incorporating self-supervised temporal prior information, the method achieves a high segmentation mIoU of 73.10% using only 0.005% of annotated points. Furthermore, the results establish a formal link between semantic segmentation and workflow analysis by demonstrating that semantic predictions are useful for downstream surgical phase recognition methods.
[0116] The results presented above are based on specific experiments and implementations of the described techniques. It should be understood that these results may vary depending on factors such as dataset characteristics, hardware configuration, and specific implementation details. The performance metrics and improvements described are illustrative examples and may not represent the full range of possible outcomes achievable using the disclosed methods. In some cases, different implementations or variations of the described techniques may yield further improved results. The techniques described herein can be applied in various ways and combined with other methods to potentially achieve different or improved performance in semantic segmentation tasks. Therefore, the scope of the invention should not be limited to the specific experimental results presented, but rather encompasses the broader principles and techniques disclosed herein.
[0117] The disclosed embodiments include significant technological improvements in the field of automatically segmenting operating room scenes during medical procedures. Conventional approaches to scene segmentation often rely on fully supervised learning methods, where training a machine learning model involves obtaining densely annotated training datasets with labels for every point. This approach requires a massive amount of data and imposes a massive annotation burden. Such granular labeling is often impractical for widespread implementation in real-world surgical settings due to its time-consuming nature. Using weakly supervised learning approaches is particularly challenging in operating room environments due to factors such as human-object interactions that blur lines between objects and create occlusion, inaccuracies in 3D alignment between multiple views that can create artifacts, and substantial dynamic movement within the scene.
[0118] The disclosed embodiments represent further significant technological advancements in the field of robot-assisted surgery. As described above, the described semantic reasoning approach generates outputs that can be employed in robot control schemes to enable the robot to operate based in part on precise information about its surroundings and the context of the medical procedure being performed. This allows the robot to adapt its trajectory or other control functions to improve its efficiency, performance, and safety.
[0119] The disclosed embodiments also represent technological improvements in the management of medical procedures. For example, by facilitating improvements in the performance of surgical robots, the disclosed technologies can directly lead to more efficient performance of medical procedures and better patient outcomes. Furthermore, the inferred semantic information may be utilized by a virtual surgical assistant that can directly provide recommendations, feedback, or other information to medical staff, which can similarly improve the efficiency of procedures and patient outcomes. In addition, the inferred semantic information may be utilized by an analytical system that aggregates statistical data, which may be used by medical staff to identify improvements in medical technology in future procedures.
[0120] The foregoing description of embodiments is provided for illustrative purposes only and is not intended to be exhaustive or to limit embodiments to the exact forms disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.
[0121] Some parts of this description describe embodiments of algorithms and symbolic representations of operations on information. These operations are described functionally, computationally, or logically, but are understood to be implemented by computer programs or equivalent electrical circuits, microcode, etc. Furthermore, without loss of generality, it is sometimes convenient to refer to the configurations of these operations as modules. The described operations and their associated modules can be embodied in software, firmware, hardware, or any combination thereof.
[0122] Any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, either alone or in combination with other devices. Embodiments may also relate to devices for performing the operations described herein. Such devices may comprise general-purpose computer devices that are specifically constructed for a required purpose and / or selectively invoked or reconfigured by computer programs stored in a computer. Such computer programs may be stored in tangible, non-temporary computer-readable storage media or any type of medium suitable for storing electronic instructions and coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing multiple processor designs to enhance computing power.
[0123] Finally, the language used herein has been selected primarily for readability and teaching purposes and may not be selected to delineate or limit the subject matter of the invention. Accordingly, the scope is not intended to be limited by this detailed description, but rather by any claims issued in connection with an application based herein. Accordingly, the disclosure of embodiments is intended to be illustrative of, and not limiting, the scope of the invention as set out in the following claims.
[0124] [Implementation Method] (1) A method for semantic segmentation of an operating room scene, Receiving a sequence of 3D representations of the operating room scene from a set of sensors, The method involves applying a machine learning model to a sequence of 3D representations to predict the semantic segmentation labels of points within the sequence of 3D representations, wherein the machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on the spatial and temporal relationships between supervoxel features. A method comprising outputting the aforementioned semantic segmentation labels. (2) The method according to Embodiment 1, further comprising controlling a surgical robot in robot-assisted surgery based at least in part on the semantic segmentation labels. (3) The method according to Embodiment 1, further comprising applying a surgical phase prediction model to the semantic segmentation labels to predict the surgical phase of a surgical procedure. (4) The method according to Embodiment 3, further comprising controlling a surgical robot in robot-assisted surgery, at least in part, based on the surgical phase. (5) The weakly supervised semantic segmentation method described above is Obtaining a set of training 3D representations, Based on geometric boundaries, the set of training 3D representations is divided into supervoxels, Obtaining sparse annotations for the sparse subset of the aforementioned supervoxels, The method according to Embodiment 1, comprising: propagating labels from the subset of supervoxels to unannotated supervoxels based on spatial and temporal relationships in each training iteration; and updating the model parameters of the machine learning model based on a loss function.
[0125] (6) The method according to Embodiment 5, wherein the machine learning model includes a segmentation network that models the semantic class of each point in the sequence of the 3D representation, and a relational network that models the embedding of each class of the semantic segmentation label. (7) Propagating the label means To construct a graph representing the similarity between the supervoxels, The method according to Embodiment 5, comprising performing graph-based label propagation to assign the labels to the unannotated supervoxels. (8) Performing the graph-based label propagation described above means The method according to Embodiment 7, comprising applying a matching matrix to model the temporal consistency between corresponding supervoxels in consecutive 3D representations within the set of training 3D representations. (9) The method according to embodiment 8, wherein applying the matching matrix includes applying at least one of nearest neighbor matching, optical flow matching, or scene flow matching. (10) Applying the matching matrix means that The method according to Embodiment 8, comprising iteratively updating the matching matrix during the training of the machine learning model to refine the temporal consistency between the corresponding supervoxels.
[0126] (11) The method according to Embodiment 1, wherein the set of sensors comprises a plurality of depth cameras positioned at different viewpoints within the operating room. (12) The method according to Embodiment 1, wherein the sequence of 3D representations includes a temporal sequence of RGB-D point clouds captured at different points in time during the surgical procedure. (13) The method according to Embodiment 1, wherein the semantic segmentation labels correspond to different objects or classes of objects present in the operating room scene. (14) A method for training a machine learning model for semantic segmentation of operating room scenes, To obtain a set of training 3D representations of operating room scenes, Based on geometric boundaries, the set of training 3D representations is divided into supervoxels, Obtaining sparse annotations for at least a subset of the supervoxels, In each training iteration, semantic segmentation labels are propagated from the subset of supervoxels to unannotated supervoxels based on spatial and temporal relationships, and the model parameters of the machine learning model are updated based on the loss function. A method including outputting the aforementioned machine learning model. (15) The method according to Embodiment 14, wherein the machine learning model includes a segmentation network that models the semantic class of each point in the set of training 3D representations, and a relational network that models the embedding of each class of the semantic segmentation labels.
[0127] (16) Propagating the label means To construct a graph representing the similarity between the supervoxels, The method according to Embodiment 14, comprising performing graph-based label propagation to assign the labels to the unannotated supervoxels. (17) Performing the graph-based label propagation described above means The method according to embodiment 16, comprising applying a matching matrix to model the temporal consistency between corresponding supervoxels in consecutive 3D representations within the set of training 3D representations. (18) The method according to Embodiment 17, wherein applying the matching matrix includes applying at least one of nearest neighbor matching, optical flow matching, or scene flow matching. (19) The method of Embodiment 17, further comprising iteratively updating the matching matrix during the training of the machine learning model to refine the temporal consistency between the corresponding supervoxels. (20) A non-temporary computer-readable storage medium for storing instructions for semantic segmentation of an operating room scene, wherein, when the instructions are executed by the processor, the processor is provided with Receiving a sequence of 3D representations of the operating room scene from a set of sensors, The method involves applying a machine learning model to a sequence of 3D representations to predict the semantic segmentation labels of points within the sequence of 3D representations, wherein the machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on the spatial and temporal relationships between supervoxel features. A non-temporary computer-readable storage medium that causes a step to be performed, which includes outputting the aforementioned semantic segmentation labels.
Claims
1. A method for semantic segmentation of operating room scenes, Receiving a sequence of 3D representations of the operating room scene from a sensor set, The method involves applying a machine learning model to a sequence of 3D representations to predict the semantic segmentation labels of points within the sequence of 3D representations, wherein the machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on the spatial and temporal relationships between supervoxel features. A method comprising outputting the aforementioned semantic segmentation labels.
2. The method according to claim 1, further comprising controlling a surgical robot in robot-assisted surgery based at least in part on the semantic segmentation labels.
3. The method according to claim 1, further comprising applying a surgical phase prediction model to the semantic segmentation labels to predict the surgical phase of a surgical procedure.
4. The method according to claim 3, further comprising controlling a surgical robot in robot-assisted surgery based at least in part on the surgical phase.
5. The aforementioned unsupervised semantic segmentation method is: To obtain a set of training 3D representations, Based on geometric boundaries, the set of training 3D representations is divided into supervoxels, Obtaining sparse annotations for the sparse subset of the aforementioned supervoxels, The method according to claim 1, comprising: in each training iteration, propagating labels from the subset of supervoxels to unannotated supervoxels based on spatial and temporal relationships; and updating the model parameters of the machine learning model based on a loss function.
6. The method according to claim 5, wherein the machine learning model includes a segmentation network that models the semantic class of each point in the sequence of the 3D representation, and a relational network that models the embedding of each class of the semantic segmentation label.
7. Propagating the aforementioned label means To construct a graph representing the similarity between the supervoxels, The method according to claim 5, comprising performing graph-based label propagation to assign the labels to the unannotated supervoxels.
8. Performing the aforementioned graph-based label propagation means that The method according to claim 7, comprising applying a matching matrix to model the temporal consistency between corresponding supervoxels in consecutive 3D representations within the set of training 3D representations.
9. The method according to claim 8, wherein applying the matching matrix includes applying at least one of nearest neighbor matching, optical flow matching, or scene flow matching.
10. Applying the aforementioned matching matrix means that The method according to claim 8, comprising iteratively updating the matching matrix during the training of the machine learning model to refine the temporal consistency between the corresponding supervoxels.
11. The method according to claim 1, wherein the set of sensors comprises a plurality of depth cameras positioned at different viewpoints within the operating room.
12. The method according to claim 1, wherein the sequence of 3D representations includes a temporal sequence of RGB-D point clouds captured at different points in time during a surgical procedure.
13. The method according to claim 1, wherein the semantic segmentation labels correspond to different objects or classes of objects present in the operating room scene.
14. A method for training a machine learning model for semantic segmentation of operating room scenes, To obtain a set of training 3D representations of operating room scenes, Based on geometric boundaries, the set of training 3D representations is divided into supervoxels, Obtaining sparse annotations for at least a subset of the supervoxels, In each training iteration, semantic segmentation labels are propagated from the subset of supervoxels to unannotated supervoxels based on spatial and temporal relationships, and the model parameters of the machine learning model are updated based on the loss function. A method including outputting the aforementioned machine learning model.
15. The method according to claim 14, wherein the machine learning model includes a segmentation network that models the semantic class of each point in the set of training 3D representations, and a relational network that models the embedding of each class of the semantic segmentation labels.
16. Propagating the aforementioned label means To construct a graph representing the similarity between the supervoxels, The method according to claim 14, comprising performing graph-based label propagation to assign the labels to the unannotated supervoxels.
17. Performing the aforementioned graph-based label propagation means that The method according to claim 16, comprising applying a matching matrix to model the temporal consistency between corresponding supervoxels in consecutive 3D representations within the set of training 3D representations.
18. The method according to claim 17, wherein applying the matching matrix includes applying at least one of nearest neighbor matching, optical flow matching, or scene flow matching.
19. The method according to claim 17, further comprising iteratively updating the matching matrix during the training of the machine learning model to refine the temporal consistency between the corresponding supervoxels.
20. A non-temporary, computer-readable storage medium for storing instructions for semantic segmentation of an operating room scene, wherein, when the instructions are executed by a processor, the processor receives, Receiving a sequence of 3D representations of the operating room scene from a sensor set, The method involves applying a machine learning model to a sequence of 3D representations to predict the semantic segmentation labels of points within the sequence of 3D representations, wherein the machine learning model is trained using a weakly supervised semantic segmentation method in which sparse labels of supervoxels are iteratively propagated to unannotated supervoxels based on the spatial and temporal relationships between supervoxel features. A non-temporary computer-readable storage medium that causes a step to be performed, which includes outputting the aforementioned semantic segmentation labels.