Surgical procedure guidance system

Abstract

Methods for guiding surgical procedures include: accessing information related to the surgical procedure; acquiring at least one image of a surgical site captured by an endoscope during the surgical procedure; identifying an instrument captured in the at least one image using a machine learning system; determining whether the instrument should be replaced based on a comparison of the information related to the surgical procedure and the instrument identified by the machine learning system; and providing an instruction when the determination indicates that the instrument should be replaced.

Description

Technical Field

[0001] This technology generally relates to assisting surgical procedures, and more specifically, to surgical procedure guidance systems and methods, such as those in robotic surgical procedures. Background Technology

[0002] In laparoscopic surgical procedures, endoscopes are used to visualize the surgical site. Especially in minimally invasive surgery (MIS) involving robotic surgery, image sensors have been used to allow surgeons to visualize the surgical site.

[0003] Surgeons performing laparoscopic surgery have a limited field of view of the surgical site via monitor. The correct or appropriate selection and positioning of non-mechanical, mechanical, or laparoscopic tools for the surgical procedure relies on the clinician's judgment and experience. For example, the treatment outcome may be suboptimal when inappropriate tools are used or improperly positioned. There is interest in developing systems that supplement the clinician's experience and judgment during surgical procedures. Summary of the Invention

[0004] The technology disclosed herein generally relates to systems and methods for guiding surgical procedures by supplementing clinician judgment and experience.

[0005] In one aspect, a method for guiding a surgical procedure includes: accessing information related to the surgical procedure; acquiring at least one image of a surgical site captured by an endoscope during the surgical procedure; identifying an instrument captured in the at least one image using a machine learning system; determining whether the instrument should be replaced based on a comparison of the information related to the surgical procedure and the instrument identified by the machine learning system; and providing an instruction when the determination indicates that the instrument should be replaced.

[0006] In various embodiments of the method, information relating to the surgical procedure indicates that a tool used in another surgical procedure is of the same type as the surgical procedure, and determining whether the tool should be replaced includes determining whether the tool belongs to the category of tools already used in the other surgical procedure.

[0007] In various implementations of the surgical guidance system, determining whether the tool should be replaced includes determining that the tool should be replaced when the tool is not one that has been used in the other surgical procedures.

[0008] In various embodiments of the invention, the endoscope is a stereoscopic endoscope, the at least one image includes at least one stereoscopic image, and the at least one stereoscopic image includes depth information relating to the instrument and the tissue of the surgical site.

[0009] In various embodiments of the present invention, the method includes the machine learning system determining the orientation information of the tool based on the at least one image.

[0010] In various embodiments of the invention, the machine learning system is trained using tool orientation information from other surgical procedures of the same type as the surgical procedure.

[0011] In various embodiments of the present invention, the orientation information of the tool determined by the machine learning system indicates whether the tool should be reoriented.

[0012] In various embodiments of the present invention, the method includes the machine learning system determining the location information of the tool based on the at least one image.

[0013] In various embodiments of the invention, the machine learning system is trained using tool location information from other surgical procedures of the same type as the surgical procedure in question.

[0014] In various embodiments of the present invention, the location information of the tool determined by the machine learning system indicates whether the tool should be repositioned.

[0015] In one aspect, a surgical guidance system for guiding surgical procedures includes: a memory configured to store instructions; and a processor coupled to the memory and configured to execute the instructions. The processor is configured to execute instructions to cause the surgical guidance system to: access information related to the surgical procedure; acquire at least one image of a surgical site captured by an endoscope during the surgical procedure; identify an instrument captured in the at least one image using a machine learning system; determine whether the instrument should be replaced based on a comparison of the information related to the surgical procedure and the instrument identified by the machine learning system; and provide an instruction when the determination indicates that the instrument should be replaced.

[0016] In various implementations of the surgical guidance system, information related to the surgical procedure indicates that a tool used in another surgical procedure is of the same type as the surgical procedure, and when determining whether the tool should be replaced, the instruction, when executed, causes the surgical guidance system to determine whether the tool belongs to the category of tools already used in the other surgical procedure.

[0017] In various implementations of the surgical guidance system, when determining whether the tool should be replaced, the instruction, when executed, causes the surgical guidance system to determine that the tool should be replaced if it is not one of the tools already used in the other surgical procedures.

[0018] In various embodiments of the surgical guidance system, the endoscope is a stereoscopic endoscope, the at least one image includes at least one stereoscopic image, and the at least one stereoscopic image includes depth information relating to the instrument and the tissue at the surgical site.

[0019] In various implementations of the surgical guidance system, the instructions, when executed, further enable the surgical guidance system to determine the orientation information of the tool based on the at least one image using the machine learning system.

[0020] In various implementations of the surgical guidance system, the machine learning system is trained using tool-oriented information from other surgical procedures of the same type as the surgical procedure in question.

[0021] In various implementations of the surgical guidance system, the orientation information of the tool determined by the machine learning system indicates whether the tool should be reoriented.

[0022] In various implementations of the surgical guidance system, the instructions, when executed, further enable the surgical guidance system to determine the position information of the tool based on the at least one image using the machine learning system.

[0023] In various implementations of the surgical guidance system, the machine learning system is trained using tool location information from other surgical procedures of the same type as the surgical procedure in question.

[0024] In various implementations of the surgical guidance system, the tool's location information, determined by the machine learning system, indicates whether the tool should be repositioned.

[0025] Details of one or more aspects of the invention are set forth in the following drawings and description. Other features, objects, and advantages of the technology described in this disclosure will become apparent from the detailed description, the drawings, and the claims. Attached Figure Description

[0026] Figure 1A This is a perspective view of a surgical system according to an embodiment of this disclosure;

[0027] Figure 1B It is based on the implementation scheme of this disclosure. Figure 1A Functional block diagram of the surgical system;

[0028] Figure 2A This is a functional block diagram of a computing device according to an embodiment of this disclosure;

[0029] Figure 2B This is a block diagram of a machine learning system according to an embodiment of the present disclosure;

[0030] Figure 3 This is an illustration of a surgical image processed by a machine learning system according to an embodiment of this disclosure; and

[0031] Figure 4 It is a flowchart illustrating a method for checking whether tools should be moved or changed during surgery, in accordance with the contents of this disclosure. Detailed Implementation

[0032] Laparoscopic surgery utilizes surgical instruments of various sizes, types, shapes, and kinds. If inappropriate instruments are used during laparoscopic surgery, the surgical outcome may be incomplete or unsatisfactory. Furthermore, similar unsatisfactory results may occur if the instruments are positioned too close or too far from the target tissue, or if their orientation is incorrect. This disclosure provides clinicians with guidance on the instruments used during surgical procedures and can provide guidance regarding whether instruments should be changed, repositioned, and / or redirected. As described in more detail below, various aspects of this disclosure relate to machine learning systems that have utilized data from other surgical procedures similar to the one being performed. Such machine learning systems can complement clinicians' experience and judgment based on data from other similar surgical procedures.

[0033] Reference Figure 1A and 1B This illustration shows a surgical system or robotic surgical system 100 according to various aspects of this disclosure, including a surgical robot 110, a processor 140, and a user console 150. The surgical system 100 alone may not be able to perform the surgery entirely and may be supplemented by non-mechanical tools 170. The surgical robot 110 includes one or more mechanical kinematic chains or robotic arms 120, and a robot base 130 supporting the respective mechanical kinematic chain 120. Each mechanical kinematic chain 120 movably supports an end effector or tool 126 configured to act on a target of interest. Each mechanical kinematic chain 120 has an end 122 supporting the end effector or tool 126. Furthermore, the end 122 of the mechanical kinematic chain 120 may include an imaging device 124 for imaging the surgical site “S”.

[0034] The user console 150 communicates with the robot base 130 via the processor 140. Additionally, as... Figure 1B As shown, each robot base 130 may include a controller 132 communicating with a processor 140 and an arm or robotic arm motor 134. A mechanical kinematic chain 120 may include one or more arms and joints between two adjacent arms. The arm motor 134 may be configured to drive each joint of the mechanical kinematic chain 120 to move the end effector 126 to the appropriate position.

[0035] A non-mechanical tool 170 or end effector 126 may be inserted into the surgical site "S" to assist or perform the surgery during the procedure. According to various aspects of this disclosure, to reduce the occurrence of improper tool use during surgery, the processor 140 may determine whether the inserted non-mechanical tool 170 or end effector 126 is appropriate. When the tool is determined to be inappropriate, the processor 140 may display a pop-up window on the display device 156 of the user console 150 to provide an indication that the tool may be unsuitable. This instruction is given in a manner that does not interfere with the surgery.

[0036] Based on various aspects of this disclosure, and as described in more detail below, processor 140 can determine when the tool is not correctly positioned, such as being too far or too close to the target organ during surgery; when the tool is not correctly oriented, such as being oriented at an inappropriate angle relative to the target; or when the tool is moving too fast toward the target. Indications can be presented to bring these determinations to the attention of the clinician. In various embodiments, such indications may include, but are not limited to, sound, pop-ups on the display 156 screen, and / or vibrations to the input handle 152 of the user console 150.

[0037] Now for reference Figure 1B The processor 140 can be similar to Figure 2A The processor 140 may be a standalone computing device of the computing device 200, or a computing device integrated into one or more components of the surgical system 100 (e.g., in the robot base 130 or user console 150). The processor 140 of the surgical system 100 may also be distributed across multiple components of the surgical system 100 (e.g., in multiple robot bases 130). The processor 140 of the surgical system 100 generally includes a processing unit 142, a memory 149, a robot base interface 146, a console interface 144, and an imaging device interface 148. The robot base interface 146, the console interface 144, and the imaging device interface 148 communicate with the robot base 130, the user console 150, and the imaging device 162, respectively, via wireless configuration (e.g., Wi-Fi, Bluetooth, LTE) and / or wired configuration. Although depicted as separate modules, in various embodiments, the console interface 144, the robot base interface 146, and the imaging device interface 148 may be a single component.

[0038] The user console 150 also includes input handles 152 supported on the control arm 154, allowing clinicians to manipulate the surgical robot 110 (e.g., move the mechanical kinematic chain 120, its end effector 122, and / or the tool 126). Each of the input handles 152 communicates with the processor 140 to transmit control signals to and receive feedback signals from it. Alternatively, each of the input handles 152 may include an input device (not explicitly shown) that allows the surgeon to manipulate the tool 126 supported at the end effector 122 of the mechanical kinematic chain (e.g., clamp, grasp, fire, open, close, rotate, advance, slice, etc.).

[0039] Each of the input handles 152 is movable through a predefined workspace to move the end 122 of the mechanical kinematic chain 120, such as tool 126, within the surgical site "S". As the input handles 152 move, tool 126 moves within the surgical site "S", as described below. The movement of tool 126 may also include the movement of the end 122 of the mechanical kinematic chain 120 supporting tool 126.

[0040] User console 150 further includes computer 158, which includes a processing unit or processor and memory, and includes data, instructions and / or information related to various components, algorithms and / or operations of robot base 130, and is similar in many respects to Figure 2A The computing device 200. The user console 150 can be operated using any suitable electronic service, database, platform, cloud, etc. The user console 150 communicates with input handles 152 and a display 156. When engaged by a clinician, each input handle 152 can provide an input signal to the computer 158 corresponding to the movement of the input handle 152. Based on the received input signals, the computer 158 can process the signals and transmit them to a processor 140, which in turn transmits control signals to the robot base 118 and the means of the robot base 118 to achieve movement at least in part based on the signals transmitted from the computer 158. In various embodiments, the input handles 152 can be implemented by another mechanism, such as a handle, pedal, or computer accessory (e.g., keyboard, joystick, mouse, button, touchscreen, switch, trackball, etc.).

[0041] User console 150 includes a display device 156 configured to display two-dimensional and / or three-dimensional images of the surgical site “S,” which may include data captured by an imaging device 124 located at the end 122 of the mechanical motion chain 120. In various embodiments, the imaging device 124 may capture stereoscopic images, visual images, infrared images, ultrasound images, X-ray images, thermal images, and / or other images of the surgical site “S.” The imaging device 124 transmits the captured imaging data to a processor 140, which creates a display screen of the surgical site “S” from the imaging data and transmits the display screen to the display device 156 for display.

[0042] Display device 156 can be connected to an endoscope mounted on the end effector 122 of robotic arm 120 to display a real-world image from the endoscope on display device 156. Furthermore, as described above, potential notifications can be displayed on the real-world image in an overlay or superimposed manner. The endoscope can capture images of non-mechanical tools 170 or end effector 126. Such captured images of tools 170 or 126 can be transmitted to and processed by processor 140, which acts as or coordinates with a machine learning system, to identify tools 170 or 126. According to various aspects of this disclosure, this information can be used to determine whether surgery should be performed appropriately.

[0043] Tool recognition can be performed by a machine learning system based on one or more images. The recognized tool can be compared with information related to the surgical procedure to determine whether the tool is suitable for the surgery. In other words, it determines whether the identified tool should be moved or replaced. For example, the machine learning system can identify the tool, as well as aspects of the tool, such as size and shape. If the size, shape, and / or type of the tool is determined to be unsuitable for the surgery, an indicator can suggest that the tool should be replaced. Furthermore, in various aspects of this disclosure, the machine learning system can predict whether the tool is correctly positioned or oriented. In these cases, an indicator can suggest whether the tool should be moved to a different location or orientation. The machine learning system can be trained based on training data obtained from previously performed surgeries. Previous surgeries may be relevant to the current surgery. For example, training data may include frame images and labeling information used to train the machine learning system to recognize tools, determine whether the tool's orientation is appropriate, and / or whether the tool's positioning is appropriate. In one aspect, the training labels can be manually entered by a physician, specialist, or medical professional who performed previous surgeries.

[0044] The labeling information can identify the tools captured in the frame images, and / or indicate whether the tools captured in frame images of previous surgeries were properly positioned or oriented, such as whether the tools were too far or too close to the target tissue, and / or whether the angle relative to the target tissue was appropriate or incorrect, etc. Machine learning systems can process frame images with labeling information to train themselves to determine which images match the labeling information. Furthermore, machine learning systems can be trained to identify the progress stage in the current surgery based on frame images and / or labeling information from previous surgeries related to the current surgery.

[0045] Regarding machine learning systems, previous surgical videos or frame images can be used to train them. For example, doctors or medical professionals can label each video or frame image with information about tools, organs, and surgical progress. Specifically, medical personnel can label tools and target organs in each frame image or video. Furthermore, they can label frame images as "too close," meaning the tool is too close to the target organ in the image, or "too far," meaning the tool is too far from the target organ in the image. This labeling information can be used to train the machine learning system to determine whether the tools captured in the frame images should be moved to different positions or orientations.

[0046] In addition, medical personnel can mark tools as "inappropriate" on image frames, meaning that the type, size, or shape of the tool is not suitable for the corresponding stage or progress of surgery. This labeling information can be used to train machine learning systems to predict whether the tool captured in the frame image should be replaced.

[0047] In addition, medical personnel can mark non-target but important organs in the image frame to determine if these organs are too close to the tool. This can reduce accidental contact with important but non-target organs.

[0048] On one hand, machine learning systems can process images or videos of previously performed surgeries and add labeling information about tools and progression stages. This labeled information can then be reviewed by experts, doctors, or medical professionals to confirm or modify the labeling.

[0049] Now for reference Figure 2A The accompanying drawings illustrate a functional block diagram of a computing device, which generally refers to computing device 200. Although not explicitly shown in the corresponding figures of this application, computing device 200 or one or more components thereof may represent one or more components of surgical system 100 (e.g., processor 140 or computer 158). Computing device 200 may include one or more processors 210, memory 220, input interface 230, output interface 240, network interface 250, or any desired subset thereof.

[0050] Memory 220 includes a non-transitory computer-readable storage medium for storing data and / or software, which includes instructions executable by one or more processors 210. When executed, these instructions can cause processor 210 to control the operation of computing device 200, such as, but not limited to, receiving, analyzing, and transmitting sensor signals received in response to movement and / or actuation of one or more input handles 152. Memory 220 may include one or more solid-state storage devices, such as flash memory chips. Alternatively or additionally, memory 220 may include one or more mass storage devices communicating with processor 210 via a mass storage controller and a communication bus (not shown). While the description of computer-readable medium in this disclosure refers to solid-state storage devices, those skilled in the art will understand that computer-readable medium may include any available medium accessible to processor 210. More specifically, computer-readable storage medium may include, but is not limited to, non-transitory, volatile, non-removable, removable, or non-transitory media implemented using any technical method for storing information, such as computer-readable instructions, data structures, program modules, or other suitable data access and management systems. Examples of computer-readable storage media include: RAM, ROM, EPROM, EEPROM, flash memory or other known solid-state memory technologies; CD-ROM, DVD, Blu-ray or other such optical storage; magnetic tape cassette, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to store information and can be accessed by computing device 200.

[0051] In this implementation, memory 220 stores data 222 and / or one or more applications 224. Such applications 224 may include instructions that execute on one or more processors 210 of computing device 200. Data 222 may include surgical care standards, which may include the progress stages of each surgery and the appropriate tools for each progress stage. Furthermore, the care standards stored in data 222 may be updated or improved by future surgeries. Additionally, the care standards may be updated for each surgery by a team of expert clinicians.

[0052] Application 224 may include instructions that cause input interface 230 and / or output interface 240 to receive sensor signals from and transmit sensor signals to various components of surgical system 100, respectively. Alternatively, computing device 200 may send signals for analysis and / or display via output interface 240. For example, memory 220 may include instructions that, when executed, generate depth maps or point clouds of objects within the surgical environment based on real-time image data received from imaging equipment of surgical system 100. The depth maps or point clouds may be stored in memory 220 over multiple iterations for subsequent cumulative analysis of the depth maps or point clouds.

[0053] In addition, application 224 may include machine learning algorithms, and computing device 200 may serve as a machine learning system trained using previous surgical video / frame images with relevant labeling information.

[0054] Now for reference Figure 2B It provides what can be provided by Figure 2A A block diagram of a machine learning system 260 implemented by a computing device 200. The machine learning system 260 can be trained using multiple data records 270a-270n. Videos and frame images from previous surgeries can constitute a set of data records. For example, data record 270a may include frame images / videos 272a, labeling information 274a associated with the frame images 272a, and (if relevant) control parameters 276a of a generator that provides surgical energy to the surgery. On one hand, a surgery may be divided into several stages. In this case, data record 270a may include multiple sets of data records 270a, each set of data records including frame images, labeling information, and corresponding generator control parameters for one stage. On the other hand, each stage can be considered separate from other stages. Therefore, a surgery may result in two or more sets of data records.

[0055] For simplicity, the letter appended to the end of the number (e.g., an) may be omitted below unless necessary. For example, tag information 274 may represent one or more tag information 274a-274n. Tag information 274 may be added to or embedded in frame image 272 manually or automatically. For example, a medical professional may manually tag information in frame image 272, or a tagging algorithm may process frame image 272 and automatically tag information in frame image 272.

[0056] On the other hand, frame image 272, marker information 274, generator control parameters 276, and patient parameters 278 from the previously performed surgery generate a data record 270. A data record 270 can be separate and independent of another of the multiple data records 270a-270n generated from other surgeries.

[0057] The machine learning system 260 can be trained using multiple data records 270a-270n. On one hand, the machine learning system 260 can be trained using data records generated from surgeries similar to the current surgery. In this case, the machine learning system 260 can be trained using supervised or reinforcement learning methods. If the multiple data records 270a-270n are generated from various surgeries, the machine learning system 260 can be trained using unsupervised learning. On the other hand, the machine learning system 260 can utilize data science and artificial science techniques, including but not limited to convolutional neural networks, recurrent neural networks (RNNs), Bayesian regression, Nay Bayes, nearest neighbor, least squares, mean squares, and support vector regression.

[0058] The labeling information 274 may have one or more layers. The first layer is a global layer, meaning that the labeling information at the first layer is valid throughout the entire video or image frame. The second layer is a local layer, meaning that the labeling information at the second layer is valid for a portion of the video or frame image. The first layer information may include the type of surgery, the target organ, the location of the target organ, and the surgical plan, including an appropriate range of surgical angles. The second layer information may include tool information and progress information. Progress information can indicate what stage the surgery is in in the corresponding frame image. Tool information can provide information on whether the size, shape, and type of the tools are appropriate during the surgical procedure, whether the tools are too close or too far from the target organ, whether the tools are approaching the target organ at an appropriate angle, and whether the tools are approaching the target organ too quickly. In another aspect, tool information can provide information on whether the tools are too close to important non-target organs during the surgical procedure.

[0059] Doctors, specialists, or medical professionals can add labeling information 274 to frame image 272. For example, tools can be labeled with tool boundaries in frame image 272. Target organs and non-target vital organs can be labeled in the same way as tools. Positional and / or orientation information about the tool, such as "too far," "too close," "wrong angle," "too fast," etc., can be added to frame image 272. Machine learning system 260 can process image 272 with relevant or corresponding labeling information 274, adjust, update, and revise the internal control parameters of machine learning system 260, and store the internal control parameters in a configuration file.

[0060] In an implementation, the marking information 274 may further include surgical procedure information related to the surgical operation. The surgical procedure information may indicate the relationship between the tool and the tissue. For example, the tool may include two gripper members, and the surgical procedure information may indicate how much tissue is grasped between the two gripper members. Furthermore, the surgical procedure information may include the force with which the two gripper members compress the tissue.

[0061] When the tool is a scalpel, surgical procedure information can indicate how deep the incision made by the scalpel is.

[0062] Surgical procedure information may further include hemodynamics during the procedure. Bleeding may occur during tissue anatomy or access to tissues. Surgical procedure information may indicate whether bleeding occurred or how much bleeding occurred.

[0063] In addition, other information related to the surgical procedure can be labeled, allowing the machine learning system 260 to use this labeled information for training.

[0064] The generator control parameters 276 can be parameters of the generator that provides surgical energy for the operation, including but not limited to, for example, duration, power, slope, frequency, or other surgical generator parameters. The generator control parameters 276 can be stored in a database or memory, since it is not possible to obtain or retrieve the generator control parameters 276 from the processed frame image 272.

[0065] Data record 270 may further include patient parameters 278. Patient parameters 276 may include the patient's age, tissue moisture, hydration level, and / or the location of tissues within the patient's body, as well as other patient characteristics. In various embodiments, data relating to patient parameters 278 may be manually or automatically entered into data record 270 from the patient's medical records. Since patient parameters 278 may not be available from image processing of image 272, patient parameters 278 may be stored as generator control parameters 276 in a database or memory.

[0066] After processing and learning from the data records 270 generated from previous surgeries, the machine learning system 260 can process the real-time frame images / videos of the current surgery and provide notifications based on the results. For example, when one or more real-time frame image display tools are too close to the target organ, the machine learning system 260 can present an indication that the tool is too close to the target organ. Or when one or more real-time frame image display tools are approaching the target organ at the wrong angle, the machine learning system 260 can present an indication that the tool is approaching the target from the wrong angle. Similarly, when one or more frame image display tools are approaching the target organ too quickly, the machine learning system 260 can present such an indication.

[0067] Now refer to Figure 2A The output interface 240 can further transmit and / or receive data via network interface 250 through one or more wireless configurations, such as radio frequency, optical, etc. (An open wireless protocol used for exchanging data over short distances, using short radio waves, from fixed and mobile devices, and creating Personal Area Networks (PANs)) (A method for using a personal wireless local area network (WPAN)) The 802.15.4-2003 standard is a specification for a set of advanced communication protocols using small, low-power digital radios. Although depicted as a separate component, network interface 250 can be integrated into input interface 230 and / or output interface 240.

[0068] See again Figure 3 , Figure 1A The surgical system 100 may include an endoscope 310 inserted through a body cavity of the patient “P” and configured to provide an optical view or frame image of the surgical site “S” and transmit the frame image to a display device 156. The endoscope 310 includes a camera 320 to capture images of the surgical site “S” and instruments 340 during the surgical procedure, as detailed below. The camera 320 may be an ultrasound imaging device, a laser imaging device, a fluorescence imaging device, or any other imaging device capable of producing real-time frame images. In various embodiments, the endoscope 310 may be a stereoscopic endoscope that captures stereoscopic images with depth information.

[0069] Endoscope 310 is inserted through an opening (natural opening or incision) to position camera 320 within a body cavity adjacent to the surgical site "S," allowing camera 320 to capture images of the surgical site "S" and the instrument 340. Camera 320 then transmits the captured images to... Figure 2B The machine learning system 260 receives images of the surgical site "S" captured by the camera 320 and displays the received images on a display device, allowing the clinician to visually see the surgical site "S" and the instrument 340. The endoscope 310 may include a sensor 330 that captures the pose of the camera 320 while capturing images of the surgical site "S". The sensor 330 communicates with the machine learning system 260, enabling the machine learning system 260 to receive the pose of the camera 320 from the sensor 330 and correlate the pose of the camera 320 with the images captured by the camera 320.

[0070] On one hand, the machine learning system 260 can process images and can identify the type, shape, and size of the tool 340, taking into account the camera pose.

[0071] The machine learning system 260 can also be trained to identify the progress stages of surgery from real-time frame images, which may have been augmented by multiple images from previous surgeries related to the current surgery.

[0072] The identified stages of progress may include navigation to the target organ, identification of the target region, preparation of surgical instruments for entry into the target organ, performance of the surgery, confirmation of the integrity of the surgery, and withdrawal of all instruments from the target organ. The list of stages of progress is provided as an example, but is not limited thereto.

[0073] Based on the progress stage of the identification, the machine learning system 260 can determine whether the tool 340 is suitable in terms of type, shape, and size. If the tool 340 does not have a suitable type, shape, or size, the machine learning system 260 can provide an indication that the tool 340 is unsuitable in terms of type, shape, or size based on the progress stage of the identification. In various embodiments, the tool identified by the machine learning system 260 can be compared with a database of tools used in other similar surgeries. If the tool identified by the machine learning system 260 is not among the tools used in other similar surgeries, a notification can be provided indicating that the tool should be replaced.

[0074] On the one hand, a notification may indicate that the type of tool is suitable for the surgery, but its size is too large or too small to be properly operated on at the identified stage of progression, or that the tool is unsuitable for the identified stage of progression. Notifications can usually be given without interfering with the surgery. If the potential harm from using tool 340 appears urgent or severe, the notification may present an enhanced alarm, such as a flashing red light on the screen, tactile vibration on the input handle 152 of Figure 1, or an audible alarm.

[0075] Furthermore, based on frame images, the machine learning system 260 can identify whether the pose or orientation of the tool 340 is appropriate or incorrect. If the machine learning system 260 determines that the orientation of the tool 340 is incorrect or inappropriate, a notification can indicate that the tool should be redirected or repositioned.

[0076] The machine learning system 260 can record judgments and notifications during surgery to further refine its internal control parameters in the future. For example, the surgeon or expert group performing the surgery can label information on frame images, taking notifications into account. They can discuss the effectiveness of the tools used in the surgery and, based on their positive effectiveness during the surgery, refine or update the tool labeling from "inappropriate" to "appropriate," or vice versa. The machine learning system 260 can use these updates for training to further refine the internal control parameters stored in the configuration file.

[0077] Figure 4This is a flowchart illustrating method 400 for checking the suitability of a tool captured in an image during surgery, according to an embodiment of this disclosure. When a surgical tool or instrument is inserted into the patient's orifice, one or more cameras (e.g., a stereoscopic endoscope) can capture images of the tool or instrument and the surgical site. Method 400 begins at step 405 with receiving surgical information. The surgical information includes the type of surgery, the target organ, and the location of the target organ. The surgical information can be retrieved manually or automatically from a database stored in memory, which is entered by a physician or medical professional.

[0078] A machine learning system can be configured for the current surgery based on information from the current procedure. The machine learning system can maintain separate configuration files for different types of surgical procedures. In one aspect, the machine learning system can retrieve configuration files from the images used to process the current surgery.

[0079] In step 410, the machine learning system receives an image of the surgical site from the endoscope. The image is processed to determine whether the tool is captured in the image in step 415. This determination can be performed by an image processing algorithm or a machine learning system.

[0080] If it is determined that the tool was not captured in the image in step 410, method 400 returns to step 410 until the tool is captured in the image.

[0081] When an tool is detected in the image, the machine learning system identifies its information in step 420. This information may include the tool's size, shape, location, and orientation. Alternatively, the orientation of the endoscope or camera used to capture the image may be considered to adjust the image and enable the machine learning system to accurately identify the tool. In step 420, the machine learning system may also identify the target organ.

[0082] In the implementation scheme, hemodynamics can also be determined in step 420. For example, it can be determined whether bleeding has occurred, and if so, the amount of bleeding. Furthermore, the relationship between the tool and the tissue can also be determined in step 420. The amount of tissue grasped by the two gripper members of the tool can be determined. Additionally, the magnitude of the pressure applied by the two gripper members can be determined. When such determination is found in step 425 to be beyond the range suitable for surgical operation, a warning regarding the surgical operation can be displayed in step 430.

[0083] In step 425, the machine learning system determines whether the tool should be replaced, repositioned, or reoriented based on a configuration file generated, modified, and updated by the machine learning system from previous surgical images. If the machine learning system determines that the tool should be moved or replaced, method 400 proceeds to step 440.

[0084] When it is determined in step 425 that a tool should be replaced, i.e., the shape, size, or type of the tool is unsuitable for the surgery, appropriate instructions are displayed to the surgeon in step 430. In particular, this notification draws the surgeon's attention to the fact that the type, size, or shape of the tool is unsuitable for the surgery and should be replaced or substituted.

[0085] When it is determined in step 425 that the tool should be repositioned, that is, the tool is not oriented correctly relative to the target organ of the surgery, an instruction will be displayed to reorient the tool.

[0086] The determination in step 425 that the tool should be moved may also mean that the tool is mispositioned relative to the target organ being operated on. In this case, instructions are given to the surgeon to reposition the tool.

[0087] Furthermore, the determination in step 425 that the tool should be moved may also mean that the tool is approaching the target organ too quickly. In this case, instructions are given to the surgeon to slow the tool's movement toward the target organ.

[0088] On the one hand, warnings can be conveyed to clinicians through tactile vibrations or auditory reminders. If the level of inappropriateness is high enough to offset the effects of the surgery, the surgery can be abruptly stopped to mitigate potential harm.

[0089] In step 435, the detected tools and images can be recorded for future reference. For example, an expert committee can gather to input, refine, or update the labeling information for the procedure based on warning records, and the machine learning system can use this new data to train itself. In this way, the machine learning system can update and refine internal control parameters and save them in a configuration file.

[0090] In step 440, it is determined whether the surgery has been completed. If the surgery has not been completed, method 400 repeats steps 410-440. Otherwise, method 400 terminates after the surgery has been completed.

[0091] It should be understood that the various aspects disclosed herein can be combined with combinations different from those specifically presented in the specification and drawings. It should also be understood that, by way of example, the actions or events of any process or method described herein may be performed in a different order, added, combined, or excluded entirely (e.g., all described actions and events may not be necessary for performing the technique). Furthermore, although certain aspects of this disclosure are described for clarity as being performed by a single module or unit, it should be understood that the techniques of this disclosure can be performed by a combination of units or modules associated with, for example, a medical device.

[0092] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which correspond to tangible media such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and is accessible by a computer).

[0093] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other physical structure suitable for implementing the technology. Furthermore, the technology can be entirely implemented in one or more circuit or logic elements.

Claims

1. A storage medium storing instructions, said instructions, when executed by a processor, causing the processor to perform a method for guiding a surgical procedure, said method comprising: Obtain information related to the surgical procedure; Acquire at least one image of the surgical site captured by the endoscope during the surgical procedure; The tool captured in the at least one image is identified using a machine learning system; The decision on whether the tool should be replaced is made by comparing information related to the surgical procedure with the tool identified by the machine learning system. and Provide instructions when the determination indicates that the tool should be replaced; The movement speed of the tool toward the target organ tissue is determined based on the acquired image of the surgical site, and an indication is provided when the tool moves toward the target organ tissue at a speed greater than a predetermined speed.

2. The storage medium of claim 1, wherein information relating to the surgical procedure indicates that tools used in other surgical procedures are of the same type as the surgical procedure, and in, Determining whether the tool should be replaced includes determining whether the tool is one that has already been used in the other surgical procedure.

3. The storage medium of claim 2, wherein determining whether the tool should be replaced includes determining that the tool should be replaced when the tool is not one of the tools already used in the other surgical procedure.

4. The storage medium of claim 1, wherein the endoscope is a stereoscopic endoscope, the at least one image includes at least one stereoscopic image, and the at least one stereoscopic image includes depth information relating to the instrument and the tissue of the surgical site.

5. The storage medium according to claim 4, further comprising orientation information of the tool determined by the machine learning system based on the at least one image.

6. The storage medium of claim 5, wherein the machine learning system is trained using tool orientation information from other surgical procedures of the same type as the surgical procedure.

7. The storage medium of claim 6, wherein the orientation information of the tool determined by the machine learning system indicates whether the tool should be reoriented.

8. The storage medium of claim 4, further comprising the location information of the tool determined by the machine learning system based on the at least one image.

9. The storage medium of claim 8, wherein the machine learning system is trained using tool location information from other surgical procedures of the same type as the surgical procedure.

10. The storage medium of claim 9, wherein the location information of the tool determined by the machine learning system indicates whether the tool should be repositioned.

11. A surgical guidance system for guiding surgical procedures, the system comprising: The memory is configured to store instructions; as well as A processor, coupled to the memory, is configured to execute the instructions to enable the surgical guidance system: Obtain information related to the surgical procedure; Acquire at least one image of the surgical site captured by the endoscope during the surgical procedure; The tool captured in the at least one image is identified using a machine learning system; The decision on whether the tool should be replaced is made by comparing information related to the surgical procedure with the tool identified by the machine learning system. and Provide instructions when the determination indicates that the tool should be replaced; The movement speed of the tool toward the target organ tissue is determined based on the acquired image of the surgical site, and an indication is provided when the tool moves toward the target organ tissue at a speed greater than a predetermined speed.

12. The surgical guidance system of claim 11, wherein information relating to the surgical procedure indicates that tools used in other surgical procedures are of the same type as those used in the surgical procedure, and in, When determining whether the tool should be replaced, the instruction, when executed, causes the surgical guidance system to determine whether the tool is one that has already been used in the other surgical procedure.

13. The surgical guidance system of claim 12, wherein when determining whether the tool should be replaced, the instruction, when executed, causes the surgical guidance system to determine that the tool should be replaced if it is not a tool already used in the other surgical procedure.

14. The surgical guidance system of claim 11, wherein the endoscope is a stereoscopic endoscope, the at least one image includes at least one stereoscopic image, and the at least one stereoscopic image includes depth information relating to the instrument and the tissue of the surgical site.

15. The surgical guidance system of claim 14, wherein the instruction, when executed, further causes the surgical guidance system to determine the orientation information of the tool based on the at least one image through the machine learning system.

16. The surgical guidance system of claim 15, wherein the machine learning system is trained using tool orientation information from other surgical procedures of the same type as the surgical procedure.

17. The surgical guidance system of claim 16, wherein the orientation information of the tool determined by the machine learning system indicates whether the tool should be reoriented.

18. The surgical guidance system of claim 14, wherein the instruction, when executed, further causes the surgical guidance system to determine the position information of the tool based on the at least one image through the machine learning system.

19. The surgical guidance system of claim 18, wherein the machine learning system is trained using tool location information from other surgical procedures of the same type as the surgical procedure.

20. The surgical guidance system of claim 19, wherein the tool position information determined by the machine learning system indicates whether the tool should be repositioned.

Images