Computer assisted stomach volume reduction procedures

Abstract

The present disclosure relates to computer assisted volume reduction procedures. The computer system includes a memory and a processor. The processor determines, based on a first video of a stomach and a patient profile, a plan for a volume reduction procedure on a stomach. The processor also generates, based on a second video of the stomach, a map of the stomach and presents, on a display during the volume reduction procedure, at least one of the second video or the map. The processor further determines, based on the map and the plan, a location where a suture should be applied to the stomach, generates, based on the plan, an overlay that indicates the location, and presents the overlay on the display to indicate, on at least one of the second video or the map, the location in the stomach to place the suture.

Description

COMPUTER ASSISTED STOMACH VOLUME REDUCTION PROCEDURESTECHNICAL FIELD

[0001] The present disclosure relates generally to stomach volume reduction procedures (e.g., endoscopic sleeve gastroplasty (ESG) and endoscopic revision procedures). Specifically, the present disclosure relates to a computer system used during stomach volume reduction procedures.BACKGROUND

[0002] Volume reduction procedures (e.g., ESG and endoscopic revision) are trans-oral endoscopic procedures that reduce the volume of a patient’s stomach. During these procedures, sutures are applied to the inner wall of the stomach to tie the stomach together and reduce its volume. In the case of an ESG, the procedure is performed primarily for weight loss and a secondary metabolic effect. In the case of endoscopic revision, a prior bariatric surgery was performed and an endoscopic revision is done with suture plications to reduce the volume of the sleeve or pouch to achieve additional weight loss and metabolic benefit for the patient. In both cases, the stomach (or pouch) holds a reduced volume of food, which may reduce the patient’s caloric intake. Thus, ESG may be used to treat obesity and obesity related comorbidities (e.g., cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovarian syndrome, dyslipidemia, etc.). Endoscopic revision may be used to achieve additional weight loss in patients with failed prior bariatric surgery (e.g., gastric bypass or sleeve gastrectomy) or for maintenance of a prior ESG itself.SUMMARY

[0003] The present disclosure relates to computer assisted volume reduction procedures. According to an embodiment, a computer system for a volume reduction procedure includes a memory and a processor communicatively coupled to the memory. The processor determines, based on a first video of a stomach and a patient profile, a plan for a volume reduction procedure on a stomach. The plan includes a volume reduction for the stomach. The processor also generates, based on a second video of the stomach, a map of the stomach and presents, on a display during the volume reduction procedure, at least one of the second video or the map. Theprocessor further determines, based on the map and the plan, a location where a suture should be applied to the stomach, generates, based on the plan, an overlay that indicates the location, and presents the overlay on the display to indicate, on at least one of the second video or the map, the location in the stomach to place the suture.

[0004] According to another embodiment, a method of performing a volume reduction procedure includes determining, based on a first video of a stomach and a patient profile, a plan for a volume reduction procedure on a stomach. The plan includes a volume reduction for the stomach. The method also includes generating, based on a second video of the stomach, a map of the stomach and presenting, on a display during the volume reduction procedure, at least one of the second video or the map. The method further includes determining, based on the map and the plan, a location where a suture should be applied to the stomach, generating, based on the plan, an overlay that indicates the location, and presenting the overlay on the display to indicate, on at least one of the second video or the map, the location in the stomach to place the suture. Other embodiments includes a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0005] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 illustrates an example ESG system.

[0007] Figure 2 illustrates an example tube in the system of Figure 1 .

[0008] Figure 3 illustrates an example tube in the system of Figure 1 .

[0009] Figure 4A illustrates an example tool in the system of Figure 1 .

[0010] Figure 4B illustrates an example tool in the system of Figure 1 .

[0011] Figure 4C illustrates an example tool in the system of Figure 1.

[0012] Figure 4D illustrates an example tool in the system of Figure 1.

[0013] Figure 5 illustrates example stages of an ESG procedure using the system of Figure 1 .

[0014] Figure 6 illustrates an example computer system in the system of Figure 1 .

[0015] Figure 7 illustrates an example computer system in the system of Figure 1 .

[0016] Figure 8 illustrates an example computer system in the system of Figure 1 .

[0017] Figure 9 is a flowchart of an example method performed in the system ofFigure 1.

[0018] Figure 10 illustrates an example computer system in the system of Figure 1.

[0019] Figure 11 illustrates an example computer system in the system of Figure 1.

[0020] Figure 12 illustrates an example computer system in the system of Figure 1.

[0021] Figure 13 is a flowchart of an example method performed in the system of Figure 1.

[0022] Figure 14 illustrates an example computer system in the system of Figure 1.

[0023] Figure 15 illustrates an example computer system in the system of Figure 1.

[0024] Figure 16 illustrates an example computer system in the system of Figure 1.

[0025] Figure 17 is a flowchart of an example method performed in the system of Figure 1.DETAILED DESCRIPTION

[0001] Volume reduction procedures (e.g., endoscopic sleeve gastroplasty (ESG) or endoscopic revision) may help treat certain conditions related to obesity including obesity-related co-morbidities (cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovarian syndrome, dyslipidemia, etc.). Endoscopic revision may be used to achieve additional weight loss in patients with failed prior bariatric surgery (e.g., gastric bypass or sleeve gastrectomy) or for maintenance of a prior ESG itself.

[0002] During a volume reduction procedure, an endoscope is inserted into a patient’s stomach (or pouch). A healthcare provider (e.g., a doctor, physician, nurse, medical technician, surgeon, endoscopist, etc.) views the video obtained via the endoscope while operating a tool that applies sutures (e.g., full-thickness suture plications) to anterior, lateral, and / or posterior walls of the stomach to reduce the volume of the stomach. In the case of ESG, the reduction in volume reduces the amount of food the patient eats before feeling full, which reduces the patient’s caloric intake and puts the patient in a negative caloric balance to achieve weight loss. In the case of an endoscopic revision, the prior sleeve or pouch is reduced in volume.

[0003] Certain technical challenges, however, may negatively impact these volume reduction procedures. For example, the healthcare provider may be limited to operating using only the gross direct view provided by the endoscope. As the endoscope moves in the stomach of a patient during the procedure, the endoscope may rotate, turn, or twist, which causes the view to rotate, turn, or twist. This movement may disorient the healthcare provider and may make it more difficult for the healthcare provider to determine or track the position of the endoscope and the tool within the stomach. This disorientation may also make it difficult for the healthcare provider to know where to apply a suture and in what direction to align the suture on the anterior, lateral, and / or posterior wall of the stomach, especially as the volume is reduced. As another example, it may be difficult for the healthcare provider to determine by how much the volume of the stomach has been reduced and therefore rely on guesswork. The healthcare provider may not know the overall shape and volume of the stomach prior to starting the ESG procedure. Additionally, during the ESG procedure, the healthcare provider may estimate or evaluate the volumereduction of the stomach based on gross visualization, which may be inaccurate. Finally, the healthcare provider may not be aware of anatomical variations which might prevent volume reduction in specific areas (fundus, antrum, etc.).

[0004] These challenges may limit the success or effectiveness of the volume reduction procedure. For example, the volume reduction procedure may not achieve a desired or adequate volume reduction, which results in an insufficient reduction in caloric intake and unsuccessful treatment of the patient’s obesity and obesity-related co-morbidity condition(s). As another example, when the sutures are applied in incorrect locations or with the incorrect direction or alignment, the sutures may detach, pull-through, or cheese wire through the stomach wall during or after the procedure, which results in the volume of the stomach increasing back to baseline and negatively impacting outcomes.

[0005] The present disclosure describes a computer system that assists or guides a volume reduction procedure. Generally, the system uses artificial intelligence (e.g., machine learning) during the procedure to provide information to the healthcare provider. For example, during a pre-operative stage of the procedure, the computer system may use artificial intelligence (e.g., a neural network) to analyze a video (which may include medical images such as computerized tomography images and magnetic resonance images) from the inside of a patient’s stomach along with a medical profile for the patient to determine a plan for the procedure. The plan may indicate whether the patient is a good candidate for the procedure (e.g., a primary ESG for weight loss or a revision procedure for a failed prior bariatric procedure). The plan may also indicate a volume reduction for the stomach that may successfully treat the patient’s medical condition.

[0006] As another example, during an intra-operative stage of the volume reduction procedure, the computer system may use a simultaneous location and mapping (SLAM) process (or other process) and / or a neural network to generate a map of the patient’s stomach from a video (which may include medical images such as computerized tomography images and magnetic resonance images) of the inside of the stomach and to track the location of the endoscope and the suturing tool within the map. The computer system may display the map and the location of the endoscopeand tool within the map to prevent the healthcare provider from becoming disoriented during the procedure.

[0007] The computer system may also calculate a volume of the stomach using the map (e.g., in real-time or post-procedure). The computer system may update the map along with the volume calculation as sutures are applied to the stomach during the volume reduction procedure. For example, the computer system may use the neural network to analyze the video to determine when and where a suture had been applied and to determine a change in the shape of the stomach. The computer system may then update the map of the stomach to account for the change in shape. The computer system may update the volume calculation using the updated map. In this manner, the computer system provides a real-time volume calculation, so the healthcare provider can more easily determine the progress of the procedure and when to stop or continue the procedure.

[0008] As another example, during the procedure, the computer system may generate an overlay that indicates the positioning and direction of sutures that should be applied to the stomach. For example, the computer system may use the neural network to determine where sutures should be applied in the stomach to be consistent with existing best medical practices and to achieve the volume reduction indicated in the pre-operative plan. The computer system then generates the overlay that indicates the positioning of the sutures. The computer system may then position the overlay (e.g., on a display) over a video captured from inside the stomach and / or a map of the stomach so the healthcare provider can see where the sutures should be applied on the display in a specific orientation, pattern, or even step-to-step layout. For example, the overlay may present visual indicators on the video and / or map to indicate where the sutures should be applied. The computer system may also present audio and textual messages or indicators that inform the healthcare provider where to place the sutures. The healthcare provider may then operate a tool to apply a suture at a position indicated in the overlay. In some embodiments, the computer system may use the neural network to analyze the video to determine a position and direction of the applied suture. The computer system may then update the locations and directions of subsequent sutures in the overlay to account for the changes caused by the applied suture. In this manner, the computer system guides the suturing process,which may reduce the number of sutures that detach from the stomach after the procedure is complete.

[0009] During a post-operative stage of the procedure, the computer system collects data about the procedure. For example, the computer system may track the number of sutures applied during the procedure and the locations and directions of the sutures. As another example, the computer system may collect images or pictures (e.g., a pre-operative picture of the map of the stomach, an intra-operative picture of the map, and a post-operative picture of the map) that show the effect of the procedure on the stomach. As another example, the computer system may collect follow-up data for the patient that shows the effectiveness of the procedure (e.g., weight of the patient, number of detached sutures, volume reduction of the stomach, etc.). In some embodiments, the computer system uses the collected data to train or update the artificial intelligence (e.g., the neural network) used by the computer system during the pre-operative and intra-operative stages. In this manner, the computer system uses the information from the procedure to inform subsequent procedures.

[0010] In some embodiments, the computer system provides several technical advantages. For example, the computer system provides a map of the stomach and a location of the endoscope and tool within the stomach, which helps the healthcare provider stay oriented as the endoscope moves, rotates, or twists during the procedure. As another example, the computer system provides a real-time volume calculation for the stomach, which may be more accurate than the healthcare provider evaluating the volume reduction visually. The computer system may also provide an overlay that guides the healthcare provider when applying sutures to the stomach to achieve a desired volume reduction, which may reduce the likelihood of a suture subsequently detaching from the stomach. The computer system may also collect data during and after the procedure and use that data to further train and update the artificial intelligence used during the pre-operative and intra-operative stages, which may further improve the diagnostic and analytical capabilities of the artificial intelligence. In this manner, the computer system may increase the success rate of volume reduction procedures, and the computer system may improve the consistency of volume reduction procedures performed by different healthcare providers.

[0011] Figure 1 illustrates an example volume reduction system 100, which may be used during the pre-operative, intra-operative, and post-operative stages. As seen in Figure 1 , the system 100 includes a surgery cart 102, a control station 104, and a computer system 106. Generally, the system 100 may be used during a volume reduction procedure to generate a video from the inside of a patient’s stomach and / or to apply sutures to the stomach. The sutures may tie portions of the stomach together, which reduces the volume of the stomach. As a result of the volume reduction, the patient may feel full after eating a smaller amount of food, which reduces caloric intake. Thus, the system 100 and the procedure may be helpful in treating certain conditions related to the patient’s obesity and obesity related co-morbidities (e.g., cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovarian syndrome, dyslipidemia, etc.)

[0012] The surgery cart 102 may include the tools and devices that perform the volume reduction procedure. As seen in Figure 1 , the surgery cart 102 includes an actuator box 108, a tube 110, a camera 112, a tool 114, and a display 116. Generally, the tube 110, camera 112, and tool 114 are controlled by the actuator box 108. The actuator box 108 receives instructions for controlling the tube 110, camera 112, and tool 14 from the computer system 106. The surgery cart 102 may be positioned or moved next to a subject or patient. A guide wire may then be inserted or positioned within the subject’s or patient’s body and into an organ. The actuator box 108 may then insert the tube 110 (and the camera 112) along the guide wire and into the organ. The tool 114 may also be inserted through the tube 110 and into the organ. The volume reduction procedure may then be performed using the camera 112 and the tool 114. After the procedure is complete, the actuator box 108 may retract the tool 114 and the tube 110.

[0013] The camera 112 may be positioned at a distal end of the tube 110 (e.g. , an end opposite the actuator box 108). The camera 112 may provide a video feed of the environment within the stomach when the tube 110 is inserted into the stomach. The video feed may show the movement of the tube 110 or the tool 114 along with the progress of the procedure.

[0014] The tool 114 may be inserted through the tube 110 and into the stomach. The tool 114 may be used to apply sutures. For example, the tool 114 may be movedtowards the wall of the stomach. The tool 114 may grab and pinch the wall of the stomach (e.g., along a fold). The tool 114 may then apply a suture to tie the pinched part of the wall together. This process of pinching and suturing the wall of the stomach reduces the volume of the stomach.

[0015] The display 116 may display the video feed from the camera 112 during the volume reduction procedure. An operator of the system 100 (e.g., a healthcare provider) may view the display 116 to inspect the progress of the procedure. The display 116 may also present other vital information about the subject or patient. In some embodiments, the display 116 also presents a map of the stomach, which indicates a position or location of the tube 110 and the tool 114 in the map. Additionally, the display 116 may present an overlay on the displayed video feed to indicate where and how sutures should be applied to the stomach. For example, the overlay may show the locations on the stomach where the sutures should be applied and the directions of those sutures. Moreover, the display 116 may present messages or images that indicate a progress of the procedure. For example, the display 116 may present messages or a progress bar indicating a percentage change in the volume of the stomach, which may inform the healthcare provider whether to continue or stop the procedure. This information may assist or guide the procedure, which may improve the consistency of the results of the procedure.

[0016] The operator of the system 100 may use the control station 104 to control the surgery cart 102. As seen in Figure 1 , the control station 104 includes a display 118 and a control 120. The display 118 may provide information about the procedure, similar to the display 116. For example, the display 118 may present the video feed from the camera 112 to the operator of the system 100, the map of the stomach, the overlay indicating the suture locations and directions, and / or messages or images indicating the progress of the procedure. The operator of the system 100 may use the control 120 to control the movement of the surgery cart 102 or the operation of the actuator box 108. For example, the operator of the system 100 may use the control 120 to control the movement of the surgery cart 102, the movement of the tube 110, or the movement of the tool 114.

[0017] The computer system 106 uses artificial intelligence to assist the operator of the system 100 during the procedure. In certain embodiments, the computer system106 is separate from the surgery cart 102 and the control station 104. In some embodiments, the computer system 106 is partially or fully embodied within the surgery cart 102 and / or the control station 104. The computer system 106 may include any number of computers distributed across different locations. Different computers of the computer system 106 may be used during different stages of the procedure. As seen in Figure 1 , the computer system 106 includes a processor 122 and a memory 124, which perform the actions or functions of the computer system 106 described herein. The computer system 106 may include any number of processors 122 and memories 124.

[0018] During a pre-operative stage of the procedure, the computer system 106 may use artificial intelligence to classify the patient and to determine a treatment plan for the patient. For example, the tube 110 and the camera 112 is inserted into the patient’s stomach to capture video of the inside of the stomach. The computer system 106 may use a neural network to analyze the video to determine one or more physical characteristics of the stomach. Examples of physical characteristics of the stomach include one or more of size, topography, orientation, and shape of the stomach. The computer system 106 may also compare a health profile of the patient (along with one or more physical characteristics of the stomach) with health profiles (e.g., metabolic health profiles) of other patients to determine a treatment plan for the patient. The health profile may indicate the patient’s medical conditions (e.g., metabolic conditions). The health profile of the patient and the health profile of the other patients may include anatomical information, genetic information, and / or responder information for the patient and the other patients. The computer system 106 may compare the patient’s health profile with the health profiles of the other patients to determine whether volume reduction of the stomach is a suitable treatment for the patient given one or more physical characteristics of the patient’s stomach and given the patient’s existing medical conditions. If the computer system 106 determines that a volume reduction procedure should be performed, the computer system 106 may also determine an appropriate volume reduction for the stomach to treat the patient’s medical or health condition.

[0019] In some embodiments, the computer system 106 uses the video from the pre-operative stage and the plan developed in the pre-operative stage to generate asimulation of the desired volume reduction procedure. For example, the simulation may be a virtual reality or augmented reality simulation of the procedure. The simulation may simulate one or more physical characteristics of the stomach determined during the pre-operative stage. By performing the simulation, the healthcare provider may practice the procedure on a simulation of the patient’s stomach prior to performing the actual procedure on the patient. As a result, the healthcare provider may become more familiar with the patient’s stomach and with the maneuvers to be performed during the procedure, which may reduce errors during the actual procedure.

[0020] During the intra-operative stage, the computer system 106 may use artificial intelligence to assist the healthcare provider in implementing the plan that was developed during the pre-operative stage. For example, the computer system 106 may use a SLAM process (or other process) and a neural network to generate a map of the patient’s stomach based on a video captured by the camera 112 during the intraoperative stage. The computer system 106 may also determine the location of the camera 112 and the tool 114 in the stomach in the map. The displays 116 and 118 may present the map and the location of the camera 112 and the tool 114 in the map so that the healthcare provider does not become disoriented during the procedure and / or has real-time information regarding progress of the procedure.

[0021] Additionally, the computer system 106 may use outputs from sensors on the tube 110 to determine measurements for the stomach. The computer system 106 may use these measurements along with the map of the stomach to calculate a volume of the stomach. As the procedure progresses and sutures are applied, the computer system 106 may determine changes in the shape of the stomach and update the map accordingly. The computer system 106 may also update the volume calculation. In this manner, the computer system 106 provides a real-time volume calculation so that the healthcare provider does not need to visually estimate the volume reduction. Additionally, the computer system 106 may inform the healthcare provider when a desired or optimal volume reduction has been achieved and when to stop the procedure.

[0022] Furthermore, the computer system 106 may use the map to generate an overlay that indicates where sutures should be applied on the stomach. The computersystem 106 may determine from the map where sutures should be applied in the stomach. For example, the computer system 106 may determine where folds and bends are in the stomach (e.g., stomach topography) and determine that sutures should be applied along or consistent with the bends and the folds in a manner consistent with standard medical practice. The computer system 106 may then generate the overlay and display the overlay over the displayed video of the stomach or as a separate image. The overlay may indicate on the video where the sutures should be applied. Additionally, the overlay may indicate directions for the sutures. For example, these directions may be indicated visually using lines or marks drawn in a particular direction. These lines or marks may be presented relative to the topography of the stomach or other coordinate system displayed to the healthcare provider. In some embodiments, the overlay may also include a guide that shows where the tool 114 should be maneuvered and directed (e.g., an orientation of the tool 114) to apply the sutures. The guide may indicate a distance that the tool 114 should be moved between bites of suture. The guide may also indicate a velocity of the tool 114. The healthcare provider may move the tool 114 according to the guide to apply the sutures at the locations indicated in the overlay. The guide may include visual indicators displayed on the display, audible instructions, or haptic feedback through the tool controllers, among others. In this manner, the computer system 106 instructs the healthcare provider where and how to apply sutures, which reduces the chances that sutures will subsequently detach from the stomach walls.

[0023] During the post-operative stage, the computer system 106 collects data about the procedure (e.g., the number of sutures applied, where the sutures were applied, images of the stomach and / or the map during the pre-operative, intraoperative, and post-operative stages, etc.). The computer system 106 may also collect information indicating the results of the procedure during any follow-ups to the procedure. For example, the computer system 106 may collect information indicating whether any sutures detached from the stomach, any improvements to the medical conditions of the patient (e.g., weight loss, blood sugar levels, snoring, etc.), or any unintended side effects of the procedure. The computer system 106 may also collect information about remedial or follow-up procedures based on the pattern of applied sutures, final stomach geometry, and / or post-operative information. The computer system 106 may then use the collected data and information to update or train theartificial intelligence that the computer system 106 uses during the pre-operative and intra-operative stages. For example, the computer system 106 may store the collected data and information in the patient profile. The computer system 106 may then train or update the artificial intelligence using the patient profile. In this manner, the artificial intelligence may be improved for subsequent volume reduction procedures.

[0024] The processor 122 is any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and / or state machines, that communicatively couples to memory 124 and controls the operation of the computer system 106. The processor 122 may be 8-bit, 16-bit, 32- bit, 64-bit or of any other suitable architecture. The processor 122 may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory and executes them by directing the coordinated operations of the ALU, registers and other components. The processor 122 may include other hardware that operates software to control and process information. The processor 122 executes software stored on the memory 124 to perform any of the functions described herein. The processor 122 controls the operation and administration of the computer system 106 by processing information (e.g., information received from the surgery cart 102, control station 104, and memory 124). The processor 122 is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor 122 is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.

[0025] The memory 124 may store, either permanently or temporarily, data, operational software, or other information for the processor 122. The memory 124 may include any one or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, the memory 124 may include random access memory (RAM), read only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage device or a combination ofthese devices. The software represents any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, the software may be embodied in the memory 124, a disk, a CD, or a flash drive. In particular embodiments, the software may include an application executable by the processor 122 to perform one or more of the functions described herein. The memory 124 is not limited to a single memory and may encompass multiple memories contained in the same device or computer or distributed across multiple devices or computers. The memory 124 is considered to store a set of data, operational software, or information if the multiple memories collectively store the set of data, operational software, or information, even if different memories store different portions of the data, operational software, or information in the set.

[0026] Figure 2 illustrates an example tube 110 in the system 100 of Figure 1 . As seen in Figure 2, the tube 110 may be a flexible tube that houses the camera 112. Additionally, the tube 110 includes one or more channels (which may also be referred to as lumens). In the example of Figure 2, the tube 110 includes channels 202, 204, and 206. Various instruments may be inserted through the tube 110 and through the channels 202, 204, and 206. For example, the tool 114 or a guide wire may be inserted through the channels 202, 204, and 206. Additionally, the tube 110 may house or include lights 208. The lights 208 may illuminate the area in front of the tube 110 so that the camera 112 may capture video footage of the region in front of the tube 110.

[0027] Figure 3 illustrates an example tube 110 in the system 100 of Figure 1 . As seen in Figure 3, the tube 110 may be a flexible tube that includes a distal end 302 (e.g., an end inserted into the organ) and a proximal end 304 (e.g., an end closest to the surgery cart 102). The distal end 302 may include the camera 112 and the light 208 that illuminates the region in front of the tube 110 in the organ. The proximal end 304 may be connected to the actuator box 108. The tube 110 may be formed using segments 310. Each segment 310 may be connected so as to allow the tube 110 to bend or fold along the segments 310. In some instances, the tube 110 and the camera 112 may collectively be referred to as an endoscope.

[0028] Additionally, the tube 110 may include multiple sensors that monitor or measure various aspects of the tube 110. As seen in Figure 3, the tube 110 includes a position sensor 306, a kinematic sensor 307, and a shape sensor 308. Each ofthese sensors may be positioned on or within the tube 110, and are coupled to one or both of the actuator box 108 and the computer system 106. The position sensor 306 provide information to the computer system 106 that may be used track or measure a position of the tube 110. For example, information from the position sensor 306 may be used to determine coordinates representing a position or location of the tube 110. The kinematic sensor 307 may detect or measure movement or motion of the tube 110. For example, information from the kinematic sensor 307 may be used to measure an acceleration or velocity of the tube 110. The kinematic sensor 307 and the position sensor 306 may include accelerometers that detect the movement or positioning of the tube 110. The shape sensor 308 may detect or measure a shape of the tube 110. For example, the shape sensor 308 may include an optical fiber that is used to detect when the tube 110 bends or folds. The tube 110 may also include other types of sensors such as visual sensors, stereo visual sensors, depth sensors, etc.

[0029] Figures 4A through 4D illustrate different types of volume reduction procedures for which the tool 114 may be used. Figure 4A illustrates an example tool 114 in the system 100 of Figure 1. Specifically, Figure 4A shows the tool 114 being used during an ESG procedure. The tool 114 may be inserted through the tube 110 and into the stomach 402. The tool 114 may emerge from the tube 110 and into the stomach. When the tool 114 is positioned by or near a wall of the stomach, the tool 114 may grab or pinch a portion of the wall of the stomach. The tool 114 may then apply a suture to the grabbed or pinched wall to tie that portion of the wall together, which reduces the volume of the stomach. The tool 114 may continue suture plicating other portions of the stomach to further reduce the volume of the stomach.

[0030] Figure 4B illustrates an example tool 114 in the system 100 of Figure 1. Specifically, Figure 4B shows the tool 114 being used during an endoscopic revision procedure. As seen in (a), a gastric bypass pouch and outlet have dilated. In (b), the tool 114 is used to apply an interrupted suture to narrow and reduce the outlet. In (c), the tool 114 is used to reduce the volume of the dilated pouch. As seen in (d), the total volume of the outlet and pouch is reduced.

[0031] Figure 4C illustrates an example tool 114 in the system 100 of Figure 1. Specifically, Figure 4C shows the tool 114 being used during an endoscopic revision procedure. As seen in (a), there is a dilated stomach remnant. In (b), the tool 114 isused to apply additional suture to reduce the volume of the stomach. As seen in (c), the total volume of the stomach is reduced.

[0032] Figure 4D illustrates an example tool 114 in the system 100 of Figure 1. Specifically, Figure 4D shows the tool 114 being used during an endoscopic revision procedure. As seen in (a), a prior sleeve gastroplasty has dilated. The tool 114 is used to apply sutures to reduce the volume. As seen in (b), the total volume of the stomach is reduced.

[0033] Figure 5 illustrates example stages of a volume reduction procedure (an ESG procedure or a revision procedure) using the system 100 of Figure 1. The computer system 106 may perform certain actions or functions during each stage of the procedure. As seen in Figure 5, the procedure is divided into three different stages: the pre-operative stage, the intra-operative stage, and the post-operative stage. The procedure may alternatively have other stages. The pre-operative stage occurs before the intra-operative stage, and the post-operative stage occurs after the intra-operative stage. Generally, the pre-operative stage includes a screening to determine whether a patient is a good candidate for the volume reduction procedure, and if so, to develop a plan for the procedure. During the intra-operative stage, the volume reduction procedure is performed to reduce the volume of the patient’s stomach. During the post-operative stage, follow-up care and testing is performed to assess whether the procedure was successful or not successful. The computer system 106 may implement certain features during each of these stages to assist the healthcare provider and to improve the chances of the ESG procedure being successful (e.g., based on optimal technique, suture pattern, suture density, distance travel between sutures, etc.).

[0034] During the pre-operative stage, the computer system 106 may use artificial intelligence to evaluate the patient and to develop a plan for the procedure. The computer system 106 may analyze a video of the inside of a patient’s stomach and a health or medical profile of the patient to determine the plan for the procedure. As seen in Figure 5, the computer system 106 receives the video 502 of the inside of the patient’s stomach. The video 502 may be generated by the camera 112 when the tube 110 is inserted into the patient’s stomach during the pre-operative stage. The video 502 may be captured for screening purposes. In some embodiments, thecomputer system 106 may also generate recommendations about suture locations, distance traveled between suture bites, orientation of suture plications, and / or directions to achieve the desired volume reductions.

[0035] The computer system 106 also receives or retrieves the profile 504 for the patient. The profile 504 may include health or medical information about the patient. For example, the profile 504 may indicate the height, weight, body composition, or body mass index of the patient. The profile 504 may also indicate any health or metabolic conditions of the patient (e.g., health conditions related to obesity). For example, the profile 504 may indicate whether the patient has hyperlipidemia, hypertension, diabetes, sleep apnea, arthritis, heart disease, etc. The profile 504 may also indicate data collected by a wearable of the patient, such as heartrate, caloric intake, blood sugar levels, snoring levels, etc. The profile 504 may also indicate data collected using blue-tooth remote patient monitoring such as smart scales that provide weight and body composition.

[0036] The computer system 106 may use a neural network 506 (which may be one or more neural networks) to analyze the video 502 and the profile 504 to determine whether the patient is a candidate for the volume reduction procedure, and if so, generate the plan 508 for the procedure. The neural network 506 may be trained to determine any suitable information from the video 502 and the profile 504. For example, the neural network 506 may be trained to determine landmarks and features of the stomach in a video. The neural network 506 may analyze the video 502 to identify or detect these landmarks or features. Based on the detected landmarks or features, the neural network 506 may determine a size, orientation, or shape of the patient’s stomach from the video 502.

[0037] As another example, the neural network 506 may be trained to analyze information in the profile 504. The neural network 506 may be trained using profiles of different patients stored in a database. The neural network 506 may analyze these profiles to detect patterns or trends in the profiles and to learn whether these patterns or trends indicate that the patient is a candidate for ESG. For example, the neural network 506 may learn from these profiles what types of medical conditions are treatable using the volume reduction procedure and what types of medical conditions may suggest that the volume reduction procedure should not be performed. Asanother example, the neural network 506 may learn from these profiles which combinations of health or medical conditions make a patient a suitable candidate for the volume reduction procedure and which combinations make a patient less suitable of a candidate for the volume reduction procedure.

[0038] The neural network 506 may also learn how to use the information in the profiles to determine a plan for the volume reduction procedure. For example, the neural network 506 may learn what types of tools were successfully used to perform volume reduction for stomachs with different physical characteristics (e.g., stomach shapes, sizes, orientations, and typographies). As another example, the neural network 506 may learn what amount of volume reduction successfully treated certain medical conditions.

[0039] The trained neural network 506 may then analyze the profile 504 for the patient to determine whether the patient is a candidate for the volume reduction procedure. For example, the neural network 506 may examine the health and medical conditions of the patient indicated in the profile 504 to determine whether the patient has health and medical conditions that may be treated with volume reduction. As another example, the neural network 506 may determine whether the profile 504 indicates health or medical conditions that would indicate the benefit level (e.g., adverse, neutral, significant) that the patient is likely to obtain from receiving the volume reduction procedure. In this manner, the neural network 506 effectively compares the profile 504 with the other profiles used to train the neural network 506 to determine whether the volume reduction procedure will help or be detrimental to the patient.

[0040] The trained neural network 506 may also use the information in the profile 504 and the information gleaned from the video 502 to determine a plan 508 for the volume reduction procedure. As discussed previously, the neural network 506 may analyze the video 502 to determine a size, topography, orientation, and / or shape of the stomach. The neural network 506 may then determine, from one or more physical characteristics of the stomach (e.g., size, topography, orientation, and shape) and the health and medical conditions indicated in the profile 504, the plan 508 for the procedure. For example, the neural network 506 may determine an amount of volume reduction for the stomach that would beneficially treat the medical condition of thepatient. As another example, the neural network 506 may determine the types of tools that should be used to perform the volume reduction procedure to treat the medical condition of the patient. This information may be included in the plan 508. For example, the plan 508 may indicate whether the patient is a candidate for volume reduction, one or more physical characteristics of the stomach (e.g., size, topography, orientation, and shape), and what medical conditions are being treated by the procedure. As another example, the plan 508 may indicate the desired volume reduction for the patient and the types of tools that should be used to perform the procedure.

[0041] In certain embodiments, the computer system 106 generates a simulation 510 using the plan 508. The simulation 510 may simulate the procedure on the patient’s stomach according to the plan 508. For example, the computer system 106 may use the one or more physical characteristics of the stomach indicated in the plan 508 to generate a virtual environment (e.g., a virtual reality or augmented reality environment) that simulates the patient’s stomach. The computer system 106 may use a SLAM process or other three-dimensional reconstruction technique to build the virtual environment for the simulation 510. The healthcare provider may execute or perform the simulation 510 to practice the procedure on the stomach prior to actually performing the procedure on the patient. In this manner, the healthcare provider may practice the procedure, which reduces chances of error occurring during the actual procedure.

[0042] The computer system 106 may record the maneuvers performed during the simulation 510. The recorded maneuvers may be displayed as ghost images or footage during the intra-operative stage to guide the healthcare provider. The healthcare provider may then view and mimic the recorded maneuvers during the intra-operative stage. In this manner, the healthcare provider may practice the procedure during the simulation and prepare a ghost image or footage that guides the healthcare provider during the intra-operative stage.

[0043] In some embodiments, the plan 508 includes an overlay in which a model or image of the stomach is overlaid with a model or image of the estimated, predicted, or approximated stomach after the procedure is performed. The overlay shows thechange in size, shape, and / or orientation of the stomach that may be expected after performing the procedure.

[0044] During the intra-operative stage, the computer system 106 uses artificial intelligence to assist the healthcare provider to implement the plan 508 that was developed during the pre-operative stage. As seen in Figure 5, the computer system 106 receives the video 512 during the intra-operative stage. The video 512 may be captured by the camera 112 when the tube 110 and the camera 112 are inserted into the patient’s stomach. The video 512 may be a separate video from the video 502 that was captured during the pre-operative stage. In some embodiments, the preoperative stage occurs immediately before the intra-operative stage. The healthcare provider need not remove the tube 110 or the camera 112 from the patient’s stomach. The camera 112 captures both the video 502 and the video 512. The video 512 is then a continuation of the video 502.

[0045] The computer system 106 uses a neural network 514 (which may be one or more neural networks) to analyze the video 512. The neural network 514 may be the same as the neural network 506, or the neural network 514 may be separate from the neural network 506. The neural network 514 may be trained to identify landmarks or features in videos of the stomach. For example, the neural network 514 may be trained using different videos of stomachs. The neural network 514 may learn to identify different features (e.g., transitions, bends, folds, etc.) that appear in these videos. The trained neural network 514 may then analyze the video 512 to identify landmarks 516 in the patient’s stomach. For example, the neural network 514 may identify landmarks 516 that identify transitions from the stomach to other organs (e.g., the esophagus or the duodenum). The neural network 514 may use these landmarks 516 to identify or mark the boundaries of the patient’s stomach. As another example, the neural network 514 may identify landmarks 516 that indicate bends or folds in the stomach including the ruggae which may serve as an anatomical fingerprint of that patient’s anatomy.

[0046] The neural network 514 may identify the landmarks 516 in different frames of the video 512. For example, the neural network 514 may identify a landmark 516 appearing in a frame of the video 512, and the neural network 514 may identify the same landmark 516 appearing in a subsequent frame of the video 512. The neuralnetwork 514 may determine that the landmark 516 identified in the different frames is the same landmark 516 (e.g., based on the size and shape of the landmark 516). The neural network 514 may then tie these landmark 516 identifications in the different frames together. The computer system 106 may then analyze the frames of the video 512 to see how these landmarks 516 move to different regions of the different frames of the video 512.

[0047] The computer system 106 may use a SLAM process and the landmarks 516 to generate a map 518 of the patient’s stomach. The SLAM process may determine the boundaries of the stomach from the video 512. The computer system 106 may use these boundaries to generate the map 518, which may be a two-dimensional or a three-dimensional map of the stomach. The landmarks 516 may inform the boundaries of the stomach in the map 518. For example, the landmarks 516 may identify the transitions to the duodenum and the esophagus. The computer system 106 may exclude the regions beyond the transitions from the map 518. As a result, the map 518 may omit the esophagus and the duodenum. The computer system 106 may present the map 518 of the stomach on the display 116 or 118.

[0048] In an example process, the computer system 106 may use the landmarks 516 and other measurements from the sensors on the tube 110 to generate the map 518 of the stomach. For example, the computer system 106 may locate a landmark 516 in several frames of the video 512. The computer system 106 may analyze each of the frames to identify or match the landmark 516 in each of the frames. The landmark 516 may move to a different position in the frames due to movement of the tube 110 in the organ. The measurements from the sensors on the tube 110 may indicate the movement of the tube 110 that occurred between frames. Using this information, the computer system 106 may determine how the frames correspond to one another in a three-dimensional space (e.g., the depth of one frame relative to another frame). The computer system 106 may then stitch the frames together according to the positioning of the landmarks 516 and according to the sensor measurements to generate the map 518, which may be a three-dimensional map.

[0049] In some embodiments, the computer system 106 also uses the neural network 514 to identify the tube 110, the camera 112, and / or the tool 114 (e.g., selected according to the plan 508) appearing in the video 512. For example, theneural network 514 may be trained to detect these items in videos. The neural network 514 may identify these items when analyzing the video 512. The computer system 106 may then use the SLAM process to determine the location of these items in the map 518 of the stomach. The computer system 106 may present the map 518 and the location of the tube 110, the camera 112, or the tool 114 in the map 518 on the display 116 or 118 to prevent the healthcare provider from becoming disoriented during the volume reduction procedure.

[0050] The computer system 106 may also generate an overlay 520 that guides the application of sutures. For example, the computer system 106 may determine from the map 518 of the stomach and / or from one or more physical characteristics of the stomach (e.g., size, topography, orientation, and / or shape) indicated in the plan 508 where sutures should be applied to the stomach. For example, the computer system 106 may determine where bends and folds are in the stomach and determine that sutures should be applied along or consistent with these bends and folds to reduce the chances that the sutures injure the stomach or subsequently detach from the stomach. Additionally, the computer system 106 may determine the pattern or arrangement of the sutures and their directions. The determined locations, arrangement, and directions of the sutures may be consistent with best medical practices. The computer system 106 may then generate the overlay 520 that indicates the locations, arrangement, and directions of the sutures.

[0051] In some embodiments, the computer system 106 also considers the desired volume reduction indicated in the plan 508 when determining the locations, arrangement, or directions of the sutures. For example, the computer system 106 may determine the locations, arrangement, or directions of the sutures that will achieve the desired volume reduction. The computer system 106 may stop adding indications of sutures into the overlay 520 when the computer system 106 determines that the desired volume reduction should be achieved with the sutures indicated in the overlay 520.

[0052] The computer system 106 may overlay the overlay 520 onto the video 512 and / or the map 518 to indicate where and in what direction the sutures should be applied to the stomach. As the healthcare provider maneuvers the tube 110, the camera 112, and / or the tool 114 through the stomach, the healthcare provider mayview the video 512 and / or the map 518 on the display 116 or 118 along with the overlay 520 to understand where and in what direction to apply sutures to the stomach. The healthcare provider may apply sutures consistent with the overlay 520.

[0053] In some embodiments, the overlay 520 may include a guide that indicates to the healthcare provider how to position or direct the tool 114 to apply the sutures indicated in the overlay 520. For example, the guide may include a virtual representation of the tool pointed in a particular direction. The healthcare provider may maneuver the tool 114 according to the guide to position the tool 114 near a suture location and in the proper orientation to apply a suture in the direction indicated by the overlay 520. For example, by following the guide, the healthcare provider may maneuver the tool 114 near the suture location and direct the tool 114 such that the tool 114 is perpendicular or orthogonal to the stomach wall at that location. The tool 114 may then grab and pinch the wall, and apply the suture. In this manner, the computer system 106 further assists the healthcare provider in applying sutures properly, which may reduce the chances that the sutures detach from the stomach. The guide may also provide other information that helps guide the tool 114. For example, the guide may indicate a velocity of the tool 114 or a distance that the tool 114 should be moved.

[0054] In particular embodiments, the computer system 106 may use the neural network 514 to update the map 518 and / or the plan 508 when a suture is applied to the stomach. For example, the neural network 514 may be trained to detect the presence of sutures in the video 512. Additionally, the neural network 514 may be trained to detect motions of the stomach in the video 512. The computer system 106 may translate these motions into changes in the shape, orientation, and / or size (e.g., volume) of the stomach. The neural network 514 may analyze the video 512 to detect when a suture has been applied to the stomach of the patient. The neural network 514 may also detect motions in the walls of the stomach in the video 512 when the sutures are applied. The computer system 106 may determine changes in the shape, orientation, and / or size of the stomach based on these motions. The computer system 106 may then update the map 518 to reflect the change in size, orientation, and / or shape of the stomach. In this manner, the computer system 106 shows a real-time size or shape of the stomach as the procedure progresses.

[0055] As another example, the computer system 106 may compare the suture detected by the neural network 514 with the indications in the overlay 520 to determine whether the suture is aligned or misaligned with the overlay 520. Based on the alignment or misalignment of the suture with the overlay 520, the computer system 106 may adjust the locations or directions of subsequent sutures in the overlay 520. The computer system 106 may update the overlay 520 to show the healthcare provider the locations and directions of the subsequent sutures that will achieve the desired volume reduction. In this manner, the computer system 106 provides real-time instruction for the positioning of the sutures during the procedure. When the healthcare provider applies subsequent sutures according to the updated locations or directions, the healthcare provider may achieve the desired volume reduction in the stomach. In some instances, the computer system 106 may determine that a misaligned, crossed, or incomplete full-thickness suture should be removed, rather than leaving that suture in the stomach.

[0056] In some embodiments, the plan 508 may indicate different types of tools 114 to be used during different portions of the intra-operative stage. The computer system 106 may indicate to the healthcare provider when it is appropriate to switch the tool 114 the healthcare provider is using during the intra-operative stage. For example, the computer system 106 may determine when the healthcare provider has reached a portion of the stomach that the plan 508 indicates should be sutured using a different type of tool 114. The computer system 106 may present a message or indicator to the healthcare provider to switch tools 114.

[0057] During the post-operative stage, the computer system 106 collects data and information about the procedure. For example, the computer system 106 may generate images of the stomach during the different stages of the procedure. The computer system 106 may generate or collect a pre-operative image 522 of the stomach and a post-operative image 524 of the stomach. The pre-operative image 522 may be an image of the stomach during the pre-operative stage, before sutures were applied to the stomach. The post-operative image 524 of the stomach may be an image of the stomach after the sutures were applied. By comparing the preoperative image 522 with the post-operative image 524, the computer system 106 may determine a percentage change in the size of the stomach as a result of the procedure.In some embodiments, the computer system 106 may also collect images of the stomach or of the map 518 during the intra-operative stage. These images may show the progress of the procedure between the time when the pre-operative image 522 and the post-operative image 524 were taken. The computer system 106 also analyze the pre-operative image 522 and / or the post-operative image 524 to determine whether the desired volume reduction was achieved.

[0058] The computer system 106 may also collect operation statistics 526 for the procedure. The operations statistics 526 may include any information pertinent to the procedure. For example, the computer system 106 may collect or log the number of sutures that were applied to the stomach, the location of those sutures, and the directions of those sutures. Additionally, the computer system 106 may log the percentage change in the size of the stomach as a result of the procedure.

[0059] The computer system 106 may also collect results 528 of the procedure. The results 528 may be collected or gathered during the patient’s follow-up visits after the procedure and may indicate a patient response to the procedure. The results 528 may indicate how successful the procedure was in treating the patient’s health or medication conditions. For example, the results 528 may include weight lost by the patient at various time points, the blood sugar levels of the patient, change in medications, and / or whether the patient continues to snore. Additionally, the results 528 may include a number of sutures that have detached from the stomach, failed, or loosened after the ESG procedure.

[0060] The computer system 106 may use the information and data collected during the post-operative stage to further train the artificial intelligence that was used during the pre-operative and / or intra-operative stages. For example, the computer system 106 may store the collected data and information in the profile 504 of the patient. In the example of Figure 5, the computer system 106 includes the preoperative image 522, the post-operative image 524, the operation statistics 526, and the results 528 in the profile 504 of the patient. The computer system 106 then uses the updated profile 504 to train the neural network 506 or 514. The neural network 506 or 514 may analyze the profile 504 to learn whether the ESG procedure successfully treated the medical or health condition of the patient. The neural network 506 or 514 may also learn whether the locations and directions of the sutures achieveda desired reduction in the volume of the stomach. The neural network 506 or 514 may then use this new information when analyzing future videos and profiles for subsequent ESG procedures. In this manner, the computer system 106 continues to improve the artificial intelligence used during the ESG procedure.

[0061] Figure 6 illustrates an example computer system 106 in the system 100 of Figure 1. Generally, Figure 6 shows a feature that the computer system 106 implements to assist the healthcare provider during the intra-operative stage. As seen in Figure 6, the computer system 106 has the plan 508 that was developed during the pre-operative stage. The plan 508 may indicate any suitable information for the ESG procedure, including the desired volume reduction for the stomach, one or more physical characteristics of the stomach (e.g., size, topography, orientation, and / or shape), and the types of tools 114 to be used for the procedure. The computer system 106 may also have the video 512 captured by the camera 112 in the stomach during the intra-operative stage.

[0062] The computer system 106 generates the overlay 520 that includes indications that show where the sutures should be applied in the stomach and the directions of those sutures. In some embodiments, the overlay 520 includes a guide 602. The guide 602 may provide an indication of the positioning and orientation (e.g., direction) of the tool 114 to properly apply the sutures at the locations and in the directions indicated by the overlay 520. For example, the guide 602 may direct the tool 114 to be positioned orthogonal to a wall of the stomach so that the tool may more securely grab and pinch the stomach wall and so that the suture may be more securely applied to the stomach wall. The guide 602 may also indicate other information about the tool 114 such as the velocity of the tool 114 and a distance the tool 114 should be moved. The computer system 106 may present the overlay 520 and the guide 602 on the display 116 or 118. The overlay 520 and the guide 602 may be positioned on the video 512 presented on the display 115 or 118. The healthcare provider may view the overlay 520 and the guide 602 on the video 512 to determine how to maneuver the tool 114 and where to apply the sutures. In some embodiments, the guide 602 may indicate how to maneuver the tool 114 to reduce or minimize tissue trauma. For example, the guide 602 may direct the healthcare provider how to direct or angle thetool 114 when inserting the tool 114 into the stomach to minimize or reduce tissue trauma.

[0063] The computer system 106 may use the neural network 514 to analyze the video 512 to detect when a suture 604 has been applied to the stomach wall. For example, the neural network 514 may be trained using many videos taken from the inside of stomachs. The neural network 514 may be trained to detect sutures that were applied to those stomachs. The trained neural network 514 may then analyze the video 512 to determine when the suture 604 has been applied to the stomach. The neural network 514 may detect the suture 604 along with any corresponding motion or movement in the stomach wall as a result of the suture 604 being applied. The computer system 106 may use the outputs of the neural network 514 to determine the change in physical characteristics (e.g., shape, orientation, and / or size) of the stomach caused by the suture 604. The computer system 106 may then update the map 518 of the stomach to account for that change in the physical characteristics of the stomach. For example, if the suture 604 caused a portion of the stomach to fold in, then the computer system 106 may update the map 518 to show that portion of the stomach folding in. The computer system 106 may then present the updated map 518 on the display 116 or 118 so the healthcare provider can see the result of applying the suture 604.

[0064] Figure 7 illustrates an example computer system 106 in the system 100 of Figure 1. Generally, Figure 7 shows a feature that the computer system 106 implements to assist the healthcare provider during the intra-operative stage. As see in Figure 7, the computer system 106 receives the video 512 captured by the camera 112 positioned in the stomach of the patient. The computer system 106 uses the neural network 514 to analyze the video 512 to detect the tool 114. The neural network 514 may be trained using different videos from inside the stomach during volume reduction procedures. The neural network 514 may be trained to detect the tool 114 in those videos. The computer system 106 may then use the neural network 514 to analyze the video 512 to detect the tool 114 in the video 512. The neural network 514 may also detect an occlusion 704 caused by the tool 114. The occlusion 704 may be a portion of the stomach in the video 512 that is blocked or occluded from view by the tool 114.

[0065] The computer system 106 may adjust the video 512 to improve the visibility of the occluded region. For example, the computer system 106 may segment out the tool 114 from the video 512 so that the occluded region becomes visible. As another example, the computer system 106 may increase a transparency 706 of the tool 114 in the video 512 so that the occlusion 704 becomes more visible through the tool 114. In this manner, the computer system 106 makes the occlusion 704 more visible to the healthcare provider during the procedure without requiring the healthcare provider to move the tool 114 to reveal the portion of the stomach blocked by the occlusion 704. As a result, the computer system 106 provides increased visibility of the stomach during the procedure. In some embodiments, the healthcare provider may control when the computer system 106 segments or removes the tool 114 from view or when the computer system 106 increases the transparency 706 of the tool 114. For example, the computer system 106 may provide a setting or option for the healthcare provider to activate and deactivate the segmentation or transparency.

[0066] Figure 8 illustrates an example computer system 106 in the system 100 of Figure 1 . Generally, Figure 8 shows a feature implemented by the computer system 106 to assist the healthcare provider during the intra-operative stage. As seen in Figure 8, the computer system 106 receives the plan 508 that was developed during the pre-operative stage. The plan 508 may indicate one or more physical characteristics of the stomach (e.g., a size, topography, orientation, and / or shape) and / or a desired reduction in the volume of the stomach. The computer system 106 also generates the map 518 of the stomach (e.g., by analyzing the video 512 of the stomach using the neural network 514).

[0067] The computer system 106 may generate the overlay 520 using information in the plan 508 and the map 518. For example, the computer system 106 may locate bends and folds in the stomach using the map 518. The computer system 106 may also determine where sutures should be applied to the stomach to achieve a volume reduction indicated in the plan 508. For example, the computer system 106 may determine that a certain number of sutures should be applied to achieve the desired volume reduction. Additionally, the computer system 106 may determine that the sutures should be positioned consistent with or along the bends or folds in the stomach to reduce the chances that the sutures detach from the stomach. In someembodiments, the computer system 106 may also determine the directions of the sutures to reduce the chances that the sutures detach, lock, or cross.

[0068] As seen in Figure 8, the overlay 520 may include the locations 802, 804, and 806 of the sutures that should be applied to the stomach. In some embodiments, the overlay 520 may also include the directions for these sutures. The computer system 106 may overlay the overlay 520 onto the video 512 captured by the camera 112 in the stomach of the patient. By overlaying the overlay 520 onto the video 512, the computer system 106 may introduce into the video 512 markings or indications at the locations 802, 804, and 806 where the sutures should be applied. The healthcare provider may view the video 512 on the display 116 or 118 to see the overlay 520 and the indicators at the locations 802, 804, and 806 to understand where to apply the sutures. The healthcare provider may also understand in what direction to align or run the sutures.

[0069] The neural network 514 may also identify locations where sutures should be avoided and include those locations in the overlay 520. For example, the neural network 514 may be trained to identify scars, polyps, neoplasms, or ulcers in many videos captured from other stomachs. The trained neural network 514 may then analyze the video 512 to identify scars, polyps, neoplasms, or ulcers in the stomach. The computer system 106 may then include the locations of the scars, polyps, neoplasms, or ulcers as landmarks 516 for the overlay 520. The overlay 520 may then indicate these locations as areas of the stomach where sutures should not be applied. The healthcare provider may view the overlay 520 during the volume reduction procedure to understand where on the stomach to avoid applying sutures.

[0070] The computer system 106 may use the neural network 514 to analyze the video 512 to detect when a suture 808 has been applied to the stomach. The neural network 514 may be trained using many videos of volume reduction procedures to detect a suture appearing in these videos. The trained neural network 514 may then be used to analyze the video 512 to detect when the suture 808 has been applied to the stomach of the patient.

[0071] The computer system 106 may use the output of the neural network 514 to determine a location or direction of the suture 808. Specifically, the computer system106 may determine how aligned the suture 808 is with an indicator at one of the locations 802, 804, or 806 included in the overlay 520. The computer system 106 may then update the overlay 520 depending on how aligned the suture 808 is with the indicators. For example, if the suture 808 is aligned with one of the indicators in the overlay 520, then the computer system 106 may not adjust or change the overlay 520 much for subsequent sutures. However, if the suture 808 is misaligned with one of the indicators in the overlay 520, then the computer system 106 may adjust or change the overlay 520 to account for the misaligned suture 808. For example, the computer system 106 may adjust the locations or directions in the overlay 520 for subsequent sutures to strengthen or reinforce the suture 808. As another example, the computer system 106 may add additional locations or directions to the overlay 520 so that additional sutures may be indicated in the overlay 520. These additional sutures may support or reinforce the suture 808. In this manner, the computer system 106 may adjust the overlay 520 during the procedure to accommodate for the sutures that the healthcare provider applies to the stomach. As a result, the computer system 106 may improve the probability that the procedure successfully treats the health or medical condition of the patient.

[0072] Figure 9 is a flowchart of an example method 900 performed in the system 100 of Figure 1. In particular embodiments, the computer system 106 performs the method 900. By performing the method 900, the computer system 106 implements certain features that assist the healthcare provider during the volume reduction procedures. These features may improve the chances that the procedure successfully treats the health or medical condition of the patient.

[0073] In block 902, the computer system 106 determines the plan 508 for the volume reduction procedure during the pre-operative stage. For example, the computer system 106 may analyze the video 502 of the inside of the patient’s stomach using the neural network 506 to determine one more physical characteristics of the stomach (e.g., a size, topography, orientation, and / or shape). Additionally, the computer system 106 may analyze the profile 504 of the patient that indicates health or medical conditions of the patient. The computer system 106 may use the neural network 506, which compares the profile 504 with other profiles of past patients to determine whether the patient is a good candidate for the volume reduction procedure,and if so, a desired volume reduction for the stomach given the patient’s health and medical conditions.

[0074] In block 904, the computer system 106 generates the map 518 of the stomach during the intra-operative stage. The computer system 106 may use the neural network 514 to analyze the video 512 captured from the inside of the patient’s stomach. The neural network 514 may detect landmarks 516 appearing in the video 512. These landmarks 516 may indicate the boundaries of the stomach, as well as specific locations within the stomach. The computer system 106 may use these landmarks 516 to generate the map 518 of the stomach. The map 518 may be a two- dimensional or three-dimensional map of the stomach.

[0075] In block 906, the computer system 106 displays the video 512 and / or the map 518. For example, the computer system 106 may communicate the video 512 and the map 518 to the display 116 or 118. The healthcare provider may view the video 512 and the map 518 on the display 116 or 118 to understand where the healthcare provider is operating and to reduce the chances of becoming disoriented.

[0076] In block 908, the computer system 106 generates the overlay 520 using the map 518 and the plan 508 that was developed during the pre-operative stage. The overlay 520 may include indicators at the locations where sutures should be applied to the stomach. The indicators may also indicate the directions of these sutures.

[0077] In block 910, the computer system 106 presents the overlay 520 on the video 512 and / or the map 518 on the display 116 or 118. For example, the overlay 520 may present indicators or markings on the video 512 or the map 518 to show the healthcare provider where in the stomach the sutures should be applied and the directions of those sutures. The healthcare provider may then maneuver the tool 114 to apply sutures at these indicated locations. In this manner, the computer system 106 increases the chances that the procedure will successfully treat the health and medical conditions of the patient, in certain embodiments.

[0078] In some embodiments, the overlay 520 also includes the guide 602 that shows the healthcare provider how to maneuver the tool 114 that applies the sutures. For example, the guide 602 may indicate a position and orientation (e.g., direction) of the tool 114. The healthcare provider may maneuver the tool 114 to align with theguide 602 to apply a suture to a location indicated in the overlay 520 and in a direction indicated in the overlay 520.

[0079] During the post-operative stage, the computer system 106 may collect information and data about the procedure that is used to train the neural network 506 and / or the neural network 514 for subsequent volume reduction procedures. For example, the computer system 106 may collect the pre-operative images 522 of the stomach, the post-operative image 524 of the stomach, the operation statistics 526, and the results 528. The computer system 106 may update the patient profile 504 with the collected information and data. The computer system 106 may then update or train the neural network 506 and / or the neural network 514 using the updated patient profile 504. Thus, the computer system 106 continues to update and train the artificial intelligence used during the ESG procedure with the results of completed ESG procedures, which may improve the diagnostic capabilities and other capabilities of the neural network 506 and / or the neural network 514.

[0080] Figure 10 illustrates an example computer system 106 in the system 100 of Figure 1. Generally, Figure 10 shows the computer system 106 performing a volume calculation during the intra-operative stage. In particular embodiments, the computer system 106 may calculate the volume of the stomach during the intra-operative stage so that the healthcare provider does not need to visually evaluate the volume of the stomach, which may be an inaccurate way to determine volume.

[0081] The computer system 106 receives the video 512, which may be captured by the camera 112 positioned in the stomach during the intra-operative stage. The computer system 106 may use the neural network 514 to analyze the video 512 to detect landmarks 516 in the stomach. The neural network 514 may be trained to detect different landmarks in many videos captured inside different stomachs. The trained neural network 514 may analyze the video 512 to detect these landmarks 516 when they appear in the video 512.

[0082] The landmarks 516 may include transitions 1002 and features 1004. The transitions 1002 may indicate the boundaries or entrances and exits of the stomach. For example, a transition 1002 may indicate the boundary between the stomach and the esophagus. Another transition 1002 may indicate the boundary between thestomach and the duodenum. The neural network 514 may be trained to recognize these transitions 1002 when they appear in the video 512. The features 1004 may indicate parts or structures of the stomach. For example, the features 1004 may indicate folds or bends in the wall of the stomach. As another example, the features 1004 may include scars or polyps on the wall of the stomach. The features 1004 may also include the walls and geometry of the stomach.

[0083] The computer system 106 uses the landmarks 516 to generate the map 518 of the stomach. For example, the landmarks 516 may indicate the boundaries, folds, and bends in the walls of the stomach. The computer system 106 may generate the map 518 consistent with the detected landmarks 516. As a result, the map 518 may be an accurate representation of one or more physical characteristics of the stomach (e.g., the size, topography, orientation, and / or shape) in the video 512. In some embodiments, the computer system 106 utilizes the transitions 1002 to determine some of the boundaries of the map 518. The computer system 106 may omit regions outside the transitions 1002 from the map 518. For example, if the transitions 1002 are boundaries between the stomach and the esophagus or duodenum, then the computer system 106 may omit the esophagus and duodenum from the map 518. In this manner, the computer system 106 may limit the map 518 to the stomach.

[0084] The computer system 106 may then calculate the volume 1006 of the stomach from the map 518. For example, the computer system 106 may calculate the volume 1006 of the stomach from one or more physical characteristics of the stomach (e.g., the size, topography, orientation, and.ir shape) appearing in the map 518. In some embodiments, the computer system 106 uses measurements 1008 to calculate the volume 1006. The measurements 1008 may be measurements of length or size of different portions of the stomach. These measurements 1008 may be determined from sensor output 1010. The sensor output 1010 may be produced by one or more sensors on the tube 110. For example, the sensor output 1010 may be produced by one or more of the position sensor 306, the kinematic sensor 307, and the shape sensor 308 positioned on the tube 110. As the tube 110 is moved through the stomach, the sensor output 1010 may indicate the distance moved by the tube 110. The sensor output 1010 may thus provide the measurements 1008 of different areas of the stomach. The computer system 106 may use these measurements 1008 whencalculating the volume 1006 of the stomach. For example, the computer system 106 may determine that the tube 110 has moved from one area of the stomach to another area of the stomach in the map 518. The computer system 106 may also determine the measurements 1008 that measure the distance traveled by the tube 110. Using the measurements 1008, the computer system 106 may determine the distance between the two points in the map 518. The computer system 106 may then extrapolate the size, topography, orientation, and / or shape of the stomach in the map 518 and calculate the volume 1006 from that size, topography, orientation, and / or shape.

[0085] As an example, the computer system 106 and / or the neural network 514 may identify a feature 1004 in the video 512 and track how that feature 1004 transitions to different pixels in the different frames of the video 512 as the tube 110 moves in the stomach. For example, the computer system 106 and / or the neural network 514 may identify the feature in a first frame of the video 512. The tube 110 may then move in the stomach, and the computer system 106 and / or the neural network 514 may identify the feature in a second frame of the video 512. Due to the movement of the tube 110, the feature may appear in different pixels of the first and second frames. The computer system 106 may determine, from the distance between the pixels in which the feature 1004 appears in the first and second frames, a number of units that the tube 110 moved in the map 518. The computer system 106 may also determine, from one or more sensors on the tube 110, the distance that the tube 110 moved. The computer system 106 may then determine, from the distance moved by the tube 110, the measurement 1008 for the stomach. The computer system 106 may then extrapolate a dimension of the stomach from the measurement 1008 and the number of units the tube 110 moved in the map 518. The computer system 106 may calculate the volume 1006 of the stomach using the dimension.

[0086] Figure 11 illustrates an example computer system 106 in the system 100 of Figure 1 . Generally, Figure 11 shows the computer system 106 updating the volume 1006 based on sutures that were applied to the stomach. As sutures are applied to the stomach, the computer system 106 may detect a change in the volume 1006 of the stomach caused by the sutures. The computer system 106 may then update the volume calculation. In this manner, the computer system 106 provides the surgeon areal-time volume of the stomach as sutures are applied during the intra-operative stage.

[0087] The computer system 106 receives the video 512. The computer system 106 uses the neural network 514 to analyze the video 512 to detect when a suture 1102 has been applied to the stomach. The neural network 514 may be trained using many different videos of volume reduction procedures. The neural network 514 may be trained to detect sutures appearing in those videos. The trained neural network 514 may then be used to analyze the video 512 to detect when the suture 1102 has been applied to the stomach. For example, the neural network 514 may detect a location and direction of the suture 1102. The neural network 514 may also be trained to detect motion 1104 in the walls of the stomach when the suture 1102 is applied. The neural network 514 may also detect a direction and distance for the detected motion 1104.

[0088] The computer system 106 may use the detected suture 1102 and the detected motion 1104 to update the map 518 to produce the updated map 1105. For example, the computer system 106 may determine, based on the location and direction of the suture 1102 and the direction of the motion 1104, that a certain portion of the stomach has been tied together by the suture 1102. The computer system 106 may then update the map 518 so that that portion of the stomach in the map 518 is tied together and moved according to the suture 1102 and the motion 1104. This update produces the updated map 1105.

[0089] The computer system 106 may then recalculate the volume 1006 of the stomach in the updated map 1105. For example, the suture 1102 and the motion 1104 may reduce the volume of the stomach. As a result, the computer system 106 may calculate a reduced volume 1006. In some embodiments, the computer system 106 also calculates a change 1106 in the volume 1006. For example, the change 1106 may be a percentage change in the volume 1006 caused by the suture 1102 and the motion 1104.

[0090] In some embodiments, the computer system 106 overlays the updated map 1105 on the map 518 that was generated prior to applying the suture 1102 (or prior to applying any sutures). By overlaying the updated map 1105 on the original map 518and presenting that overlay on the display 116 or 118, the healthcare provider may compare the size of the stomach after applying the suture 1102 and before applying the suture 1102. The healthcare provider may then understand the amount of change resulting from the suture 1102. The computer system 106 may continue updating the overlay as the healthcare provider applies more sutures to the stomach.

[0091] Figure 12 illustrates an example computer system 106 in the system 100 of Figure 1. Generally, Figure 12 shows the computer system 106 using the determined change 1106 in the volume 1006 of the stomach to determine a progress of the volume reduction procedure. As seen in Figure 12, the computer system 106 compares the change 1106 to a threshold 1202. The threshold 1202 may be the desired volume reduction in the plan 508 that was developed during the pre-operative stage. If the change 1106 equals or exceeds the threshold 1202, then the computer system 106 may determine that the procedure is complete and that the healthcare provider should stop. If the change 1106 does not exceed the threshold 1202, then the computer system 106 may determine that procedure may continue to further reduce the volume 1006 of the stomach. For example, if the threshold 1202 indicates a desired 70% reduction in the volume 1006 of the stomach, then the computer system 106 may allow the procedure to continue until the change 1106 meets or exceeds the 70-80% threshold 1202.

[0092] In some embodiments, the computer system 106 provides a progress bar 1204 on the display 116 or 118. The progress bar 1204 may indicate the change 1106 and how close the change 1106 is to the threshold 1202. As the volume 1006 of the stomach is reduced during the intra-operative stage, the change 1106 may increase and the computer system 106 may increase the progress shown by the progress bar 1204. In this manner, the computer system 106 provides a visual indicator to the healthcare provider of the volume reduction of the stomach during the intra-operative stage.

[0093] In certain embodiments, the computer system 106 provides an indication 1206 to the healthcare provider to explain to the healthcare provider whether the procedure should continue or stop. The indication 1206 may be a visual or audible indication. For example, the computer system 106 may present a displayed message or an audible message that tells the healthcare provider whether to continue or to stopreducing the volume of the stomach. When the change 1106 exceeds the threshold 1202, the computer system 106 may present a visual indication 1206 or an audible indication 1206 that the healthcare provider should stop, because the desired volume reduction has been achieved.

[0094] Figure 13 is a flowchart of an example method 1300 performed in the system 100 of Figure 1. In particular embodiments, the computer system 106 performs the method 1300. By performing the method 1300, the computer system 106 provides a real-time volume calculation for the stomach during the intra-operative stage, which may be more accurate than having the healthcare provider visually evaluate the volume reduction of the stomach.

[0095] In block 1302, the computer system 106 receives the video 512. The video 512 may be captured by the camera 112 positioned in the stomach of the patient during the intra-operative stage. In block 1304, the computer system 106 generates the map 518 based on the video 512. For example, the computer system 106 may use a SLAM process to determine the boundaries of the stomach. The computer system 106 may also use the neural network 514 to analyze the video 512 to detect landmarks 516 appearing in the video 512. The landmarks 516 may also indicate the boundaries, size, or shape of the stomach. The computer system 106 may use the information from the SLAM process and the neural network 514 to generate the map 518 of the stomach. The map 518 may be a two-dimensional or three-dimensional map of the stomach.

[0096] In block 1306, the computer system 106 calculates the volume 1006 of the stomach based on the map 518 of the stomach. In some embodiments, the computer system 106 may calculate the volume 1006 using the map 518 and the measurements 1008 derived from the sensor outputs 1010. For example, the computer system 106 may determine or calculate the volume of the stomach by measuring the size, topography, orientation, and / or shape of the stomach in the map 518. The computer system 106 may then present the volume 1006 on the display 116 or 118 for the healthcare provider during the intra-operative stage.

[0097] In block 1308, the computer system 106 detects the suture 1102 applied to the stomach during the intra-operative stage. For example, the computer system 106may use the neural network 514 to analyze the video 512 to detect when and where the suture 1102 was applied to the stomach. In some embodiments, the computer system 106 may also use the neural network 514 to analyze the video 512 to detect the motion 1104 in the walls of the stomach caused by the application of the suture 1102. In block 1310, the computer system 106 updates the map 518 to account for the suture 1102 and the motion 1104 to produce the updated map 1105. For example, the suture 1102 and the motion 1104 may indicate that a portion of the stomach has been tied together. The computer system 106 may then update the map 518 to produce the updated map 1105 that shows that portion of the stomach being tied together, which may cause the volume 1006 of the stomach to be reduced.

[0098] In block 1312, the computer system 106 updates or recalculates the volume 1006 of the stomach based on the updated map 1105. For example, the computer system 106 may calculate the updated volume 1006 based on the change to the shape, orientation, and / or size of the stomach in the updated map 1105. The computer system 106 may then display the updated volume 1006 on the display 116 or 118 for the healthcare provider. In this manner, the computer system 106 provides the healthcare provider with a real-time calculation of the volume 1006 of the stomach during the intra-operative stage, which may be more accurate than the healthcare provider visually evaluating the volume of the stomach when viewing the video 512.

[0099] Figure 14 illustrates an example computer system 106 in the system 100 of Figure 1. Generally, Figure 14 shows the computer system 106 generating the overlay 520, which may indicate the locations and directions of sutures that should be applied to the stomach to achieve a desired volume reduction.

[0100] The computer system 106 receives the video 512, which may be captured by the camera 112 positioned in the stomach during the intra-operative stage. The computer system 106 uses the neural network 514 to analyze the video 512 to identify landmarks 516 that appear in the video 512. The landmarks 516 may include features 1004 that appear in the stomach. For example, the features 1004 may include folds, bends, scars, or polyps that appear in the stomach. In some embodiments, the neural network 514 also identifies the tool 114 that appears in the video 512.

[0101] The computer system 106 may determine locations 1402 based on the features 1004. The locations 1402 may be the locations of particular features 1004. For example, the locations 1402 may be the locations of bends or folds in the stomach. As another example, the locations 1402 may be the locations of scars, neoplasms, or polyps in the stomach. The computer system 106 may determine that the locations 1402 are locations where sutures should be applied or locations where sutures should not be applied. For example, the computer system 106 may determine that sutures should be applied in locations 1402 of bends or folds so that the sutures are applied long the bends or folds consistent with the shape of the stomach, which may reduce the chances that the sutures will subsequently detach from the stomach. As another example, the computer system 106 may determine that sutures should not be applied at locations 1402 of scars, neoplasms, or polyps, which may reduce the chances that the sutures injure or damage the stomach.

[0102] The computer system 106 determines the map 518 based on the landmarks 516. The map 518 may be a two-dimensional or a three-dimensional map of the stomach. The computer system 106 may determine, from the landmarks 516, the shape, topography, orientation, and / or size of the stomach. The computer system 106 may also determine from the locations 1402, the positioning, shape, and arrangement of the walls of the stomach. The computer system 106 may then generate the map 518 of the stomach consistent with the determined size, topography, orientation, and / or shape.

[0103] The computer system 106 may then generate the overlay 520 using the map 518 and the plan 508 developed during the pre-operative stage. For example, the computer system 106 may determine, from the plan 508, a desired volume reduction to be achieved. The computer system 106 may also determine from the map 518 where sutures should be applied (e.g., at some of the locations 1402) and how many sutures should be applied to achieve the volume reduction. The computer system 106 may then generate the overlay 520 that may be positioned on the video 512 and / or the map 518. The overlay 520 may indicate the locations and directions of the sutures to be applied to the stomach. The computer system 106 may display the video 512 and / or the map 518 with the overlay 520 on the display 116 or 118. The healthcareprovider may view the overlay 520 on the video 512 or the map 518 to determine where to place the sutures.

[0104] Figure 15 illustrates an example computer system 106 in the system 100 of Figure 1 . Generally, Figure 15 shows the computer system 106 generating the overlay 520. As seen in Figure 15, the computer system 106 may receive the video 512 captured by the camera 112 in the stomach during the intra-operative stage. The computer system 106 may analyze the video 512 using the neural network 514 to determine landmarks 516 that appear in the stomach. The computer system 106 may also receive the plan 508 that was developed during the pre-operative stage. The plan 508 may indicate the desired volume reduction. The computer system 106 may also receive sensor outputs 1010 from the sensors on the tube 110 and the measurements 1008 taken based on those sensor outputs.

[0105] The computer system 106 may consider the outputs of the neural network 514, the plan 508, and the measurements 1008 to determine the locations 1502 and the directions 1504 of sutures to be applied to the stomach during the intra-operative stage. For example, the computer system 106 may determine certain locations 1502 for the sutures that are consistent with the locations 1402 of the bends and folds in the stomach. The computer system 106 may also determine the directions 1504 that are consistent with the bends and folds in the stomach. The locations 1502 of the sutures and the directions 1504 of the sutures may lock or cross causing the stomach to tie together along the folds and bends in the stomach, which may reduce the chances that the sutures will detach from the stomach in the future. As another example, the computer system 106 may determine the locations 1502 such that the sutures are not applied on scars, neoplasms, or polyps in the stomach. In this manner, the computer system 106 avoids damaging or injuring the stomach.

[0106] In some embodiments, the computer system 106 may determine (e.g., from the plan 508 determined during the pre-operative stage or the map 518) the position or location of other organs next to the stomach. The computer system 106 may determine the locations 1502 such that the sutures are not applied near these adjacent organs. In this manner, the computer system 106 may avoid the tool 114 grabbing the stomach wall and portions of the adjacent organ and unintentionally applying the suture to both the stomach and the adjacent organ(s) (e.g., gallbladder, colon, etc.).

[0107] In some embodiments, the computer system 106 or the neural network 514 may also determine an angle of the stomach or rigidity of the stomach from the video 512 or the plan 508. The computer system 106 may determine the locations 1502 and the directions 1504 based on the angle or the rigidity of the stomach. For example, the computer system 106 may determine the directions 1504 of the sutures such that the sutures align with the angle of the stomach, which may reduce the chances that the sutures detach from the stomach. As another example, the computer system 106 may determine the locations 1502 such that the sutures are applied in less rigid sections of the stomach, as opposed to more rigid sections of the stomach. In this manner, the computer system 106 reduces the chances that the sutures detach from the stomach and reduces the chances of the stomach being damaged or injured by the sutures.

[0108] The com puter system 106 generates the overlay 520 based on the locations 1502 and the directions 1504. The overlay 520 may indicate the locations 1502 where the sutures should be applied and the directions 1504 of those sutures. The computer system 106 may then present the overlay 520 on the video 512 or the map 518 of the stomach. The healthcare provider may view the overlay 520 to determine where to apply sutures and in what direction the sutures should be applied.

[0109] Figure 16 illustrates an example computer system 106 in the system 100 of Figure 1 . Generally, Figure 16 shows the computer system 106 updating the overlay 520 when sutures are applied to the stomach. As seen in Figure 16, the computer system 106 receives the video 512 captured by the camera 112 in the stomach during the intra-operative stage. The computer system 106 uses the neural network 514 to analyze the video 512. The neural network 514 may detect the appearance of a suture 1602 in the video 512 when the suture 1602 has been applied by the healthcare provider. For example, the healthcare provider may use the tool 114 to apply the suture 1602 consistent with the overlay 520. The neural network 514 may detect the presence of the suture 1602 after the suture 1602 is applied.

[0110] In certain embodiments, the neural network 514 may also detect a motion 1604 of the stomach caused by the application of the suture 1602. For example, the suture 1602 may tie together portions of the stomach. The motion 1604 may be the detected movement of these portions of the stomach as they are tied together.

[0111] The computer system 106 may determine the locations 1502 and the directions 1504 of subsequent sutures based on the detected suture 1602 and motion 1604. These locations 1502 and directions 1504 may have been determined prior to the suture 1602 being applied. The computer system 106 may change or adjust these locations 1502 and directions 1504 based on the suture 1602 and the motion 1604. For example, the computer system 106 may determine the suture 1602 was slightly misaligned from the indicator in the overlay 520. In response, the computer system 106 may adjust the locations 1502 and directions 1504 of subsequent sutures to accommodate for the misalignment of the suture 1602 (e.g., to better support the suture 1602 to prevent future detachment).

[0112] The computer system 106 may then update the overlay 520 with the updated locations 1502 and directions 1504. The overlay 520 may then include indicators that show the updated locations 1502 and directions 1504 of subsequent sutures. The healthcare provider may view the updated overlay 520 to determine where subsequent sutures should be applied to the stomach and in what directions. In this manner, the computer system 106 updates the healthcare provider, as to where and how to apply subsequent sutures, to increase the likelihood of a successful treatment.

[0113] Figure 17 is a flowchart of an example method 1700 performed in the system 100 of Figure 1. In particular embodiments, the computer system 106 performs the method 1700. By performing the method 1700, the computer system 106 determines where sutures should be applied to the stomach and generates an overlay 520 indicating the locations 1502 of those sutures.

[0114] In block 1702, the computer system 106 receives the video 512 captured by the camera 112 positioned in the stomach during the intra-operative stage. In block 1704, the computer system 106 determines locations 1502 for sutures. For example, the computer system 106 may analyze the video 512 using the neural network 514 to determine landmarks 516 in the stomach. These landmarks 516 may indicate the locations 1402 of bends or folds in the stomach, where sutures should be applied to reduce the chances that the sutures detach from the stomach. Additionally, these landmarks 516 may indicate the locations 1402 of scars or polyps, where sutures should not be applied to avoid damaging or injuring the stomach. The computersystem 106 may also analyze the plan 508 developed during the pre-operative stage and sensor outputs 1010 and measurements 1008 derived from those sensor outputs 1010 to determine the locations 1502 for the sutures. For example, the plan 508 may indicate a desired volume reduction. The computer system 106 may determine the number of sutures and the locations 1502 for the sutures (e.g., consistent with or along the determined locations 1402 of folds or bends in the stomach) to achieve the volume reduction.

[0115] In block 1706, the computer system 106 generates the overlay 520. The overlay 520 may indicate the locations 1502 where the sutures should be applied in the stomach. In some embodiments, the overlay 520 also indicates the directions 1504 of the sutures. In block 1708, the computer system 106 presents the overlay 520 on the video 512. As a result, the display 116 or 118 may present the overlay 520 on the video 512. The healthcare provider may view the overlay 520 on the video 512 to understand where to apply the sutures to the stomach to achieve a desired volume reduction during the ESG procedure.

[0116] In summary, the computer system 106 assists or guides a volume reduction procedure (e.g., an ESG procedure or a revision procedure). Generally, the computer system 106 uses artificial intelligence (e.g., machine learning) during different stages of the procedure to provide information to the healthcare provider. For example, during a pre-operative stage, the computer system 106 may use artificial intelligence (e.g., a neural network) to analyze a video 502 from the inside of a patient’s stomach along with a medical profile 504 for the patient to determine a plan 508 for the procedure. The plan 508 may indicate whether the patient is a good candidate for the volume reduction procedure. The plan 508 may also indicate a volume reduction for the stomach that may successfully treat the patient’s medical condition.

[0117] As another example, during an intra-operative stage, the computer system 106 may use a SLAM process and a neural network 514 to generate a map 518 of the patient’s stomach from a video 512 of the inside of the stomach and to track the location of the endoscope and the suturing tool 114 within the map 518. The computer system 106 may display the map 518 and the location of the endoscope and tool 114 within the map 518 to prevent the healthcare provider from becoming disoriented during the procedure.

[0118] The computer system 106 may also calculate a volume 1006 of the stomach using the map 518. The computer system 106 may update the map 518 along with the calculation of the volume 1006 as sutures are applied to the stomach during the procedure. For example, the computer system 106 may use the neural network 514 to analyze the video 512 to determine when and where a suture 1102 had been applied and to determine a change in the shape, size, and / or orientation of the stomach. The computer system 106 may then update the map 518 of the stomach to account for the change in shape. The computer system 106 may update the calculation of the volume 1006 using the updated map 518. In this manner, the computer system 106 provides a real-time volume calculation, so the healthcare provider can more easily determine the progress of the procedure and when to stop or continue the procedure.

[0119] As another example, during the procedure, the computer system 106 may generate an overlay 520 that indicates the positioning and direction of sutures that should be applied to the stomach. For example, the computer system 106 may use the neural network 514 to determine where sutures should be applied in the stomach to be consistent with existing best medical practices and to achieve the volume reduction indicated in the pre-operative plan 508. The computer system 106 then generates the overlay 520 that indicates the positioning of the sutures. The computer system 106 may then position the overlay 520 (e.g., on a display 116 or 118) over a video 512 captured from inside the stomach and / or a map 518 of the stomach so the healthcare provider can see where the sutures should be applied on the display 116 or 118. For example, the overlay 520 may present visual indicators on the video 512 and / or map 518 to indicate where the sutures should be applied. The healthcare provider may then operate a tool 114 to apply a suture at a position indicated in the overlay 520. In some embodiments, the computer system 106 may use the neural network 514 to analyze the video 512 to determine a position and / or direction of the applied suture. The computer system 106 may then update the locations and directions of subsequent sutures in the overlay 520 to account for the changes caused by the applied suture. In this manner, the computer system 106 guides the suturing process, which may beneficially reduce the number of sutures that detach from the stomach after the procedure is complete.

[0120] During a post-operative stage, the computer system 106 collects data about the procedure. For example, the computer system 106 may track the number of sutures applied during the procedure and the locations and directions of the sutures. As another example, the computer system 106 may collect images or pictures (e.g., a pre-operative picture of the map of the stomach, an intra-operative picture of the map, and a post-operative picture of the map) that show the effect of the procedure on the stomach. As another example, the computer system 106 may collect follow-up data for the patient that shows the effectiveness of the procedure (e.g., weight of the patient, number of detached sutures, volume reduction of the stomach, etc.). In some embodiments, the computer system 106 uses the collected data to train or update the artificial intelligence (e.g., the neural network) used by the computer system 106 during the pre-operative and intra-operative stages. In this manner, the computer system 106 uses the information from the procedure to inform subsequent volume reduction procedures.

[0121] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0122] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0123] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0124] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspects unless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0125] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0126] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0127] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human or animal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human oranimal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0128] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. A computer system for a volume reduction procedure, the computer system comprising: a memory; and a processor communicatively coupled to the memory, the processor configured to: determine, based on a first video of a stomach and a patient profile, a plan for a volume reduction procedure on a stomach, wherein the plan comprises a volume reduction for the stomach; generate, based on a second video of the stomach, a map of the stomach; present, on a display during the volume reduction procedure, at least one of the second video or the map; determine, based on the map and the plan, a location where a suture should be applied to the stomach; generate, based on the plan, an overlay that indicates the location; and present the overlay on the display to indicate, on at least one of the second video or the map, the location in the stomach to place the suture.

2. The computer system of Claim 1 , wherein determining the plan comprises comparing the patient profile to a plurality of patient profiles.

3. The computer system of Claim 2, wherein the plurality of patient profiles are determined based on at least one of anatomical information, responder information, or genetic information in the plurality of patient profiles.

4. The computer system of any of Claims 1 through 3, wherein the processor is further configured to determine, based on the first video, at least one of a shape or an orientation of the stomach, and wherein determining the plan is further based on at least one of the shape and the orientation of the stomach.

5. The computer system of any of Claims 1 through 4, wherein the patient profile indicates at least one of a body mass index, a body composition, a metabolic condition, or a medical condition related to obesity.

6. The computer system of any of Claims 1 through 5, wherein the processor is further configured to determine a direction of the suture, and wherein the overlay indicates the direction of the suture.

7. The computer system of any of Claims 1 through 6, wherein the processor is further configured to generate, based on the plan, a simulation that trains for the volume reduction procedure on the stomach.

8. The computer system of any of Claims 1 through 7, wherein the overlay comprises a guide that directs a tool that applies the suture during the volume reduction procedure.

9. The computer system of Claim 8, wherein the guide indicates at least one of a direction for the tool, an orientation for the tool, a velocity for the tool, or a distance that the tool should be moved.

10. The computer system of any of Claims 1 through 9, wherein the processor is further configured to update the map of the stomach after the suture is applied to show how the suture changed at least one of a shape, a volume, or an orientation of the stomach.

11. The computer system of any of Claims 1 through 10, wherein the processor is further configured to: determine a location in the stomach of a tool for applying the suture during the volume reduction procedure; and present, on the display and in the map, an indicator of the location of the tool.

12. The computer system of any of Claims 1 through 11 , wherein the processor is further configured to: determine a second location in the stomach where a suture should not be applied; and present, on the display and in the second video, an indicator of the second location in the stomach.

13. The computer system of any of Claims 1 through 12, wherein the processor is further configured to:determine, during the volume reduction procedure, that a portion of the stomach in the second video is being occluded by an instrument for applying the suture; and increase a transparency of the instrument in the second video so that the portion of the stomach becomes more visible in the second video.

14. The computer system of any of Claims 1 through 13, wherein the processor is further configured to: determine, based on the second video, a location in the stomach where the suture was applied; determine, based on the location in the stomach where the suture was applied, a location in the stomach where a second suture should be applied; and update the overlay to indicate the location in the stomach where the second suture should be applied.

15. The computer system of any of Claims 1 through 14, wherein the processor is further configured to generate an image of the stomach before the volume reduction procedure and an image of the stomach after the volume reduction procedure.

16. The computer system of any of Claims 1 through 15, wherein the processor is further configured to track at least one of a number of sutures or a suture pattern applied during the volume reduction procedure.

17. The computer system of any of Claims 1 through 16, wherein the processor is further configured to determine a patient response to the volume reduction procedure.

18. The computer system of any of Claims 1 through 17, wherein the processor is further configured to update the patient profile with results of the volume reduction procedure.

19. The computer system of Claim 18, wherein the processor is further configured to use the updated patient profile when determining a plan for a subsequent volume reduction procedure.

20. The computer system of Claim 19, wherein the processor is further configured to use a plurality of patient profiles in combination with the updated patient profile when determining the plan for the subsequent volume reduction procedure, wherein the plurality of patient profiles indicate results of other volume reduction procedures.

21. The computer system of any of Claims 1 through 20, wherein the plan comprises at least one of a model or image of the stomach overlaid with a predicted model or image of the stomach after the volume reduction procedure is performed.

22. A method of performing a volume reduction procedure, the method comprising: determining, based on a first video of a stomach and a patient profile, a plan for a volume reduction procedure on a stomach, wherein the plan comprises a volume reduction for the stomach; generating, based on a second video of the stomach, a map of the stomach; presenting, on a display during the volume reduction procedure, at least one of the second video or the map; determining, based on the map and the plan, a location where a suture should be applied to the stomach; generating, based on the plan, an overlay that indicates the location; and presenting the overlay on the display to indicate, on at least one of the second video or the map, the location in the stomach to place the suture.

23. The method of Claim 22, wherein determining the plan comprises comparing the patient profile to a plurality of patient profiles.

24. The method of Claim 23, wherein the plurality of patient profiles are determined based on at least one of anatomical information, responder information, or genetic information in the plurality of patient profiles.

25. The method of any of Claims 22 through 24, further comprising determining, based on the first video, at least one of a shape or an orientation of the stomach, and wherein determining the plan is further based on at least one of the shape and the orientation of the stomach.

26. The method of any of Claims 22 through 25, wherein the patient profile indicates at least one of a body mass index, a body composition, a metabolic condition, or a medical condition related to obesity.

27. The method of any of Claims 22 through 26, further comprising determining a direction of the suture, and wherein the overlay indicates the direction of the suture.

28. The method of any of Claims 22 through 27, further comprising generating, based on the plan, a simulation that trains for the volume reduction procedure on the stomach.

29. The method of any of Claims 22 through 28, wherein the overlay comprises a guide that directs a tool that applies the suture during the volume reduction procedure.

30. The method of Claim 29, wherein the guide indicates at least one of a direction for the tool, an orientation for the tool, a velocity for the tool, or a distance that the tool should be moved.31 . The method of any of Claims 22 through 30, further comprising updating the map of the stomach after the suture is applied to show how the suture changed at least one of a shape, a volume, or an orientation of the stomach.

32. The method of any of Claims 22 through 31 , further comprising: determining a location in the stomach of a tool for applying the suture during the volume reduction procedure; and presenting, on the display and in the map, an indicator of the location of the tool.

33. The method of any of Claims 22 through 32, further comprising: determining a second location in the stomach where a suture should not be applied; and presenting, on the display and in the second video, an indicator of the second location in the stomach.

34. The method of any of Claims 22 through 33, further comprising:determining, during the volume reduction procedure, that a portion of the stomach in the second video is being occluded by an instrument for applying the suture; and increasing a transparency of the instrument in the second video so that the portion of the stomach becomes more visible in the second video.

35. The method of any of Claims 22 through 34, further comprising: determining, based on the second video, a location in the stomach where the suture was applied; determining, based on the location in the stomach where the suture was applied, a location in the stomach where a second suture should be applied; and update the overlay to indicate the location in the stomach where the second suture should be applied.

36. The method of any of Claims 22 through 35, further comprising generating an image of the stomach before the volume reduction procedure and an image of the stomach after the volume reduction procedure.

37. The method of any of Claims 22 through 36, further comprising tracking at least one of a number of sutures or a suture pattern applied during the volume reduction procedure.

38. The method of any of Claims 22 through 37, further comprising determining a patient response to the volume reduction procedure.

39. The method of any of Claims 22 through 38, further comprising updating the patient profile with results of the volume reduction procedure.

40. The method of Claim 39, further comprising using the updated patient profile when determining a plan for a subsequent volume reduction procedure.

41. The method of Claim 40, further comprising using a plurality of patient profiles in combination with the updated patient profile when determining the plan for the subsequent volume reduction procedure, wherein the plurality of patient profiles indicate results of other volume reduction procedures.

42. The method of any of Claims 22 through 41 , wherein the plan comprises at least one of a model or image of the stomach overlaid with a predicted model or image of the stomach after the volume reduction procedure is performed.

43. A non-transitory machine-readable medium storing instructions for a volume reduction procedure that, when executed by a processor, cause the processor to: perform the method of any of Claims 22 through 42.

Images