Computer-assisted gastric volume reduction process

Abstract

The present disclosure relates to computer systems and methods for estimating stomach volume. A computer system includes a memory and a processor communicatively coupled to the memory. A processor receives a video of the interior of the stomach and generates a map of the stomach based on the video. The processor also calculates a volume of the stomach based on the map of the stomach, and detects a suture applied to the stomach during the endoscopic cuff stoplasty procedure based on the video and using a neural network, the suture causing a change in shape of the stomach. The processor also updates a map of the stomach based on the change in the shape of the stomach, and updates the calculated volume of the stomach based on the update of the map of the stomach.

Description

Technical Field

[0001] This disclosure generally relates to gastric volume reduction procedures (e.g., endoscopic sleeve gastrectomy (ESG) and endoscopic revision procedures). Specifically, this disclosure relates to computer systems used during gastric volume reduction procedures. Background Technology

[0002] Volume reduction procedures (e.g., ESG and endoscopic revision) are transoral endoscopic procedures that reduce the volume of a patient's stomach. During these procedures, sutures are applied to the inner wall of the stomach to bind it together and reduce its volume. In the case of ESG, the procedure is performed primarily for weight loss and secondary metabolic effects. In the case of endoscopic revision, a previous weight loss surgery has been performed, and the revision utilizes suture folds to reduce the volume of the sleeve or pouch stomach, thereby achieving additional weight loss and metabolic benefits for the patient. In both cases, the stomach (or pouch stomach) holds a reduced volume of food, which may reduce the patient's calorie intake. Therefore, ESG can be used to treat obesity and obesity-related comorbidities (e.g., cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovary syndrome, dyslipidemia, etc.). Endoscopic revision can be used to achieve additional weight loss in patients who have failed previous weight loss surgeries (e.g., gastric bypass or sleeve gastrectomy) or to maintain the previous ESG itself. Summary of the Invention

[0003] This disclosure relates to a computer system and method for estimating gastric volume. According to one embodiment, the computer system includes a memory and a processor communicatively coupled to the memory. The processor receives video of the interior of the stomach and generates a map of the stomach based on the video. The processor also calculates the gastric volume based on the gastric map and detects sutures applied to the stomach during an endoscopic sleeve gastrectomy procedure, using a neural network, based on the video, that cause changes in the shape of the stomach. The processor also updates the gastric map based on changes in the stomach's shape and updates the calculated gastric volume based on the updated gastric map.

[0004] According to another embodiment, a method includes receiving a video of the interior of the stomach and generating a map of the stomach based on the video. The method further includes calculating the stomach volume based on the stomach map, and detecting sutures applied to the stomach during an endoscopic sleeve gastrectomy procedure, caused by these sutures, based on the video and using a neural network, that cause changes in the shape of the stomach. The method also includes updating the stomach map based on changes in stomach shape, and updating the calculated stomach volume based on the updated stomach map. Other embodiments include a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0005] The foregoing general description and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of this disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0006] Figure 1 An example ESG system is shown.

[0007] Figure 2 It shows Figure 1 Example tube in the system.

[0008] Figure 3 It shows Figure 1 Example tube in the system.

[0009] Figure 4A It shows Figure 1 Example tools in the system.

[0010] Figure 4B It shows Figure 1 Example tools in the system.

[0011] Figure 4C It shows Figure 1 Example tools in the system.

[0012] Figure 4D It shows Figure 1 Example tools in the system.

[0013] Figure 5 It shows the use of Figure 1 Example phase of the ESG process of the system.

[0014] Figure 6 It shows Figure 1 Example computer systems in the system.

[0015] Figure 7 It shows Figure 1 Example computer systems in the system.

[0016] Figure 8 It shows Figure 1 Example computer systems in the system.

[0017] Figure 9 Is Figure 1 The flowchart shows an example method executed in the system.

[0018] Figure 10 It shows Figure 1 Example computer systems in the system.

[0019] Figure 11 It shows Figure 1 Example computer systems in the system.

[0020] Figure 12 It shows Figure 1 Example computer systems in the system.

[0021] Figure 13 Is Figure 1 The flowchart shows an example method executed in the system.

[0022] Figure 14 It shows Figure 1 Example computer systems in the system.

[0023] Figure 15 It shows Figure 1 Example computer systems in the system.

[0024] Figure 16 It shows Figure 1 Example computer systems in the system.

[0025] Figure 17 Is Figure 1 The flowchart shows an example method executed in the system. Detailed Implementation

[0026] Volume reduction procedures (such as endoscopic sleeve gastrectomy (ESG) or endoscopic revision surgery) can help treat certain conditions associated with obesity, including obesity-related comorbidities (cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovary syndrome, dyslipidemia, etc.). Endoscopic revision surgery can be used to achieve additional weight loss in patients who have failed previous weight loss surgery (such as gastric bypass or sleeve gastrectomy) or to maintain the previous ESG itself.

[0027] During the volume reduction procedure, an endoscope is inserted into the patient's stomach (or pouch stomach). Healthcare providers (e.g., doctors, physicians, nurses, medical technicians, surgeons, endoscopists, etc.) view video obtained via the endoscope while manipulating instruments that apply sutures (e.g., full-thickness suture folds) to the anterior, lateral, and / or posterior walls of the stomach to reduce its volume. In the case of ESG, the volume reduction decreases the amount of food the patient eats before feeling full, which reduces the patient's calorie intake and puts the patient in a negative calorie balance to achieve weight loss. In the case of endoscopic revision, the volume of the previously sleeve stomach or pouch stomach is reduced.

[0028] However, certain technical challenges can negatively impact these volume reduction procedures. For example, healthcare providers may be limited to operating only with the gross direct view provided by the endoscope. As the endoscope moves through the patient's stomach during the procedure, it can rotate, turn, or twist, causing the view to rotate, turn, or twist. This movement can disorient the healthcare provider and make it more difficult to determine or track the position of the endoscope and instruments within the stomach. This disorientation can also make it difficult for the healthcare provider to know where to apply sutures and in what direction to align them on the anterior, lateral, and / or posterior walls of the stomach, especially when volume reduction occurs. As another example, healthcare providers may have difficulty determining how much the stomach's volume has decreased and therefore rely on guesswork. The healthcare provider may not know the overall shape and volume of the stomach before initiating the ESG procedure. Additionally, during the ESG procedure, the healthcare provider may estimate or assess the stomach's volume reduction based on gross visualization, which can be inaccurate. Finally, healthcare providers may not be aware of anatomical variations that could prevent volume reduction in specific areas (fundus, antrum, etc.).

[0029] These challenges can limit the success or effectiveness of a volume reduction procedure. For example, a volume reduction procedure may fail to achieve the desired or sufficient volume reduction, leading to insufficient reduction in calorie intake and unsuccessful treatment of the patient's obesity and obesity-related comorbidities. As another example, when sutures are applied in the wrong place or with incorrect orientation or alignment, the sutures may detach, tug through, or cut into the stomach wall like cheese threads during or after the procedure, causing the stomach volume to return to baseline and negatively impacting the outcome.

[0030] This disclosure describes a computer system for assisting or guiding a volume reduction process. Typically, the system uses artificial intelligence (e.g., machine learning) to provide information to the healthcare provider during the process. For example, during the pre-procedure phase, the computer system can use artificial intelligence (e.g., neural networks) to analyze video from inside the patient's stomach (which may include medical images such as computed tomography (CT) and magnetic resonance imaging (MRI) images) along with the patient's medical profile to determine a plan for the process. This plan can indicate whether the patient is a good candidate for the process (e.g., a primary ESG for weight loss or a revision procedure for a failed previous weight loss program). The plan can also direct the reduction of stomach volume, which can successfully treat the patient's medical condition.

[0031] As another example, during the intraoperative phase of a volume reduction procedure, a computer system can use Simultaneous Localization and Mapping (SLAM) processing (or other processing) and / or neural networks to generate a map of the patient's stomach based on video of the stomach's interior (which may include medical images such as computed tomography and magnetic resonance imaging), and track the position of the endoscope and suture tools within the map. The computer system can display the map along with the positions of the endoscope and tools within it to prevent healthcare providers from becoming disoriented during the procedure.

[0032] Computer systems can also use maps to calculate the volume of the stomach (e.g., in real time or after the procedure). When sutures are applied to the stomach during the volume reduction process, the computer system can update the map along with the volume calculation. For example, the computer system can use neural networks to analyze video to determine when and where sutures have been applied and to identify changes in the shape of the stomach. The computer system can then update the map of the stomach to account for these shape changes. The computer system can use the updated map to update the volume calculation. In this way, the computer system provides real-time volume calculations, making it easier for healthcare providers to determine the progress of the procedure and when to stop or continue it.

[0033] As another example, during the procedure, the computer system can generate an overlay indicating the location and orientation of sutures that should be applied to the stomach. For example, the computer system can use a neural network to determine where the sutures should be applied in the stomach to align with existing best medical practices and achieve the volume reduction indicated in preoperative planning. The computer system then generates an overlay indicating the location of the sutures. The computer system can then position the overlay (e.g., on a display) over video captured from inside the stomach and / or a map of the stomach, allowing the healthcare provider to see on the display where the sutures should be applied in a specific orientation, pattern, or even a step-by-step layout. For example, the overlay can present visual indicators on the video and / or map to indicate where the sutures should be applied. The computer system can also present audio and text messages or indicators informing the healthcare provider where to place the sutures. The healthcare provider can then operate tools to apply the sutures at the location indicated in the overlay. In some implementations, the computer system can use a neural network to analyze video to determine the location and orientation of the applied sutures. The computer system can then update the position and orientation of subsequent sutures in the overlay to account for variations caused by the applied sutures. In this way, the computer system guides the suturing process, which can reduce the number of sutures that detach from the stomach after the procedure is complete.

[0034] During the postoperative phase of the procedure, the computer system collects data about the procedure. For example, the computer system may track the number of sutures applied during the procedure, as well as the location and orientation of the sutures. As another example, the computer system may collect images or pictures showing the effect of the procedure on the stomach (e.g., preoperative images, intraoperative images, and postoperative images of the stomach map). As yet another example, the computer system may collect follow-up data of the patient showing the effectiveness of the procedure (e.g., patient weight, number of detached sutures, reduction in stomach volume, etc.). In some implementations, the computer system uses the collected data to train or update artificial intelligence (e.g., neural networks) used by the computer system during the preoperative and intraoperative phases. In this way, the computer system uses information from the procedure to inform subsequent procedures.

[0035] In some implementations, the computer system offers several technical advantages. For example, the computer system provides a map of the stomach and the location of the endoscope and instruments within it, which helps the healthcare provider maintain orientation as the endoscope moves, rotates, or twists during the procedure. As another example, the computer system provides real-time volume calculations of the stomach, which can be more accurate than a healthcare provider visually assessing volume reduction. The computer system can also provide overlay maps that guide the healthcare provider to achieve the desired volume reduction when sutures are applied to the stomach, which can reduce the likelihood of sutures subsequently detaching from the stomach. The computer system can also collect data during and after the procedure and use this data to further train and update the artificial intelligence used during the preoperative and intraoperative phases, which can further improve the diagnostic and analytical capabilities of the AI. In this way, the computer system can increase the success rate of volume reduction procedures and improve the consistency of volume reduction procedures performed by different healthcare providers.

[0036] Figure 1 An exemplary volume reduction system 100 is shown that can be used during the preoperative, intraoperative, and postoperative stages. (See example...) Figure 1 As shown, system 100 includes a surgical cart 102, a control station 104, and a computer system 106. Typically, system 100 can be used during a volume reduction procedure to generate video from inside the patient's stomach and / or apply sutures to the stomach. Sutures can bind the different parts of the stomach together, reducing its volume. Due to the volume reduction, the patient may feel full after eating a smaller amount of food, thus reducing calorie intake. Therefore, system 100 and the procedure can help treat certain conditions and comorbidities associated with obesity in patients (e.g., cardiovascular disease, diabetes, sleep apnea, osteoarthritis, fatty liver, polycystic ovary syndrome, dyslipidemia, etc.).

[0037] The surgical trolley 102 may include tools and devices for performing the volume reduction process. For example... Figure 1 As shown, the surgical trolley 102 includes an actuator box 108, a tube 110, a camera device 112, tools 114, and a display 116. Typically, the tube 110, camera device 112, and tools 114 are controlled by the actuator box 108. The actuator box 108 receives instructions from a computer system 106 for controlling the tube 110, camera device 112, and tools 14. The surgical trolley 102 can be positioned or moved next to the subject or patient. A guidewire can then be inserted or positioned within the subject or patient's body and advanced into an organ. The actuator box 108 can then insert the tube 110 (and camera device 112) along the guidewire into the organ. The tools 114 can also be inserted through the tube 110 and advanced into the organ. The camera device 112 and tools 114 can then be used to perform a volume reduction procedure. After the procedure is complete, the actuator box 108 can retract the tools 114 and the tube 110.

[0038] The camera device 112 can be positioned at the distal end of the tube 110 (e.g., the end opposite the actuator housing 108). When the tube 110 is inserted into the stomach, the camera device 112 can provide video feeds of the intragastric environment. The video feeds can show the movement of the tube 110 or tool 114 along with the progress of the process.

[0039] Tool 114 can be inserted through tube 110 and enter the stomach. Tool 114 can be used to apply sutures. For example, tool 114 can be moved toward the stomach wall. Tool 114 can grasp and pinch the stomach wall (e.g., along folds). Tool 114 can then apply sutures to bind the pinched portions of the wall together. This pinching and suturing of the stomach wall reduces the volume of the stomach.

[0040] Display 116 can show video feeds from camera device 112 during the volume reduction process. An operator of system 100 (e.g., a healthcare provider) can view display 116 to check the progress of the process. Display 116 can also present other important information about the subject or patient. In some embodiments, display 116 also presents a map of the stomach, indicating the location or position of tube 110 and tool 114 within the map. Additionally, display 116 can present an overlay over the displayed video feed to indicate where and how sutures should be applied to the stomach. For example, the overlay can show where sutures should be applied to the stomach and the direction of these sutures. Furthermore, display 116 can present messages or images indicating the progress of the process. For example, display 116 can present messages or progress bars indicating a percentage change in stomach volume, which can inform the healthcare provider whether to continue or stop the process. This information can assist or guide the process, which can improve the consistency of the process outcome.

[0041] The operator of system 100 can use control station 104 to control surgical cart 102. For example... Figure 1 As shown, control station 104 includes a display 118 and a controller 120. Similar to display 116, display 118 can provide information about the process. For example, display 118 can present the operator of system 100 with video feeds from camera device 112, a map of the stomach, an overlay indicating the position and orientation of sutures, and / or messages or images indicating the progress of the process. The operator of system 100 can use controller 120 to control the movement of surgical cart 102 or the operation of actuator box 108. For example, the operator of system 100 can use controller 120 to control the movement of surgical cart 102, tube 110, or tool 114.

[0042] Computer system 106 uses artificial intelligence to assist the operator of system 100 during the process. In some embodiments, computer system 106 is separate from surgical cart 102 and control station 104. In some embodiments, computer system 106 is partially or completely embodied within surgical cart 102 and / or control station 104. Computer system 106 may include any number of computers distributed in different locations. Different computers of computer system 106 may be used during different stages of the process. Figure 1 As shown, computer system 106 includes processor 122 and memory 124, which perform the actions or functions of computer system 106 described herein. Computer system 106 may include any number of processors 122 and memory 124.

[0043] During the preoperative phase of the procedure, computer system 106 can use artificial intelligence to classify patients and determine treatment plans for them. For example, tube 110 and camera device 112 are inserted into the patient's stomach to capture video of the stomach's interior. Computer system 106 can use neural networks to analyze the video to determine one or more physical features of the stomach. Examples of physical features of the stomach include one or more of the stomach's size, shape, orientation, and form. Computer system 106 can also compare the patient's health profile (along with one or more physical features of the stomach) with health profiles of other patients (e.g., metabolic health profiles) to determine treatment plans for the patient. Health profiles may indicate the patient's medical condition (e.g., metabolic status). The patient's health profile and other patients' health profiles may include anatomical information, genetic information, and / or responder information for the patient and other patients. Computer system 106 can compare the patient's health profile with other patients' health profiles to determine whether stomach volume reduction is an appropriate treatment for the patient, given one or more physical features of the patient's stomach and the patient's existing medical condition. If the computer system 106 determines that a volume reduction process should be performed, the computer system 106 can also determine an appropriate reduction in the stomach volume to treat the patient's medical or health condition.

[0044] In some implementations, computer system 106 uses video from the preoperative phase and the planning developed during the preoperative phase to generate a simulation of the desired volume reduction process. For example, the simulation could be a virtual reality or augmented reality simulation of the process. The simulation could model one or more physical features of the stomach determined during the preoperative phase. By performing the simulation, the healthcare provider can practice the procedure on a simulation of the patient's stomach before performing the actual procedure on the patient. Therefore, the healthcare provider can become more familiar with the patient's stomach and the manipulations to be performed during the procedure, which can reduce errors during the actual procedure.

[0045] During the intraoperative phase, computer system 106 can use artificial intelligence to assist the healthcare provider in implementing the plans developed during the preoperative phase. For example, computer system 106 can use SLAM processing (or other processing) and neural networks to generate a map of the patient's stomach based on video captured by camera device 112 during the intraoperative phase. Computer system 106 can also determine the location of camera device 112 and tools 114 within the stomach on the map. Displays 116 and 118 can present the map and the location of camera device 112 and tools 114 on the map, enabling the healthcare provider to remain disoriented during the procedure and / or have real-time information about the progress of the procedure.

[0046] Additionally, the computer system 106 can use the output from sensors on tube 110 to determine measurements of the stomach. The computer system 106 can use these measurements, along with a map of the stomach, to calculate the stomach's volume. As the procedure progresses and sutures are applied, the computer system 106 can determine changes in the stomach's shape and update the map accordingly. The computer system 106 can also update the volume calculation. In this way, the computer system 106 provides real-time volume calculation, eliminating the need for healthcare providers to visually estimate volume reduction. Additionally, the computer system 106 can inform healthcare providers when the desired or optimal volume reduction has been achieved and when to stop the procedure.

[0047] Furthermore, computer system 106 can use a map to generate an overlay indicating where sutures should be applied to the stomach. Computer system 106 can determine, based on the map, where sutures should be applied in the stomach. For example, computer system 106 can determine where folds and bends are located in the stomach (e.g., stomach morphology) and determine that the sutures should be applied along or in accordance with the bends and folds in a manner consistent with standard medical practice. Computer system 106 can then generate the overlay and display it over a video of the stomach or as a separate image. The overlay can indicate where sutures should be applied on the video. Additionally, the overlay can indicate the direction of the sutures. For example, lines or markers drawn in a particular direction can be used to visually indicate these directions. These lines or markers can be presented relative to the morphology of the stomach shown to a healthcare provider or other coordinate system. In some embodiments, the overlay may also include a guide indicating where the tool 114 should be manipulated and guided (e.g., the orientation of the tool 114) to apply the sutures. The guide can indicate the distance the tool 114 should move between the sutures. The guide can also indicate the speed of the tool 114. The healthcare provider can move the tool 114 according to the guide to apply the suture at the location indicated in the overlay diagram. The guide may include visual indicators displayed on a monitor, auditory instructions, or tactile feedback via a tool controller. In this way, the computer system 106 instructs the healthcare provider on where and how to apply the suture, which reduces the chance of the suture subsequently detaching from the stomach wall.

[0048] During the postoperative phase, computer system 106 collects data about the procedure (e.g., the number of sutures applied, where the sutures were applied, and images and / or maps of the stomach during the preoperative, intraoperative, and postoperative phases). Computer system 106 may also collect information indicating the outcome of the procedure during any follow-up visits. For example, computer system 106 may collect information indicating whether any sutures have detached from the stomach, whether the patient's medical condition has improved (e.g., weight loss, blood sugar levels, snoring, etc.), or whether the procedure has any unexpected side effects. Computer system 106 may also collect information about the remedial or follow-up procedure based on the pattern of sutures applied, the final stomach geometry, and / or postoperative information. Computer system 106 can then use the collected data and information to update or train the artificial intelligence used by computer system 106 during the preoperative and intraoperative phases. For example, computer system 106 may store the collected data and information in a patient profile. Computer system 106 can then use the patient profile to train or update the artificial intelligence. In this way, the artificial intelligence can be improved for subsequent volume reduction procedures.

[0049] Processor 122 is any electronic circuit system communicatively coupled to memory 124 and controlling the operation of computer system 106, including but not limited to one or a combination of the following: microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine. Processor 122 may be 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. Processor 122 may include an arithmetic logic unit (ALU) for performing arithmetic and logical operations, processor registers for supplying operands to the ALU and storing the results of ALU operations, and a control unit for fetching instructions from memory and executing instructions by coordinating the operation of the ALU, registers, and other components. Processor 122 may include other hardware that operates software to control and process information. Processor 122 executes software stored on memory 124 to perform any of the functions described herein. Processor 122 controls the operation and management of computer system 106 by processing information, such as information received from surgical cart 102, control station 104, and memory 124. Processor 122 is not limited to a single processing device, and may include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, processor 122 is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions within that set.

[0050] Memory 124 may permanently or temporarily store data, operating software, or other information of processor 122. Memory 124 may include any or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 124 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage devices or combinations thereof. Software refers to any suitable set of instructions, logic, or code embodied in a computer-readable storage medium. For example, software may be embodied in memory 124, a disk, CD, or flash drive. In certain embodiments, software may include an application executable by processor 122 to perform one or more of the functions described herein. Memory 124 is not limited to a single memory and may include multiple memories contained in the same device or computer or distributed across multiple devices or computers. Memory 124 is considered to store the set of data, operating software, or information if multiple memories jointly store the set, even if different memories store different portions of the set of data, operating software, or information.

[0051] Figure 2 It shows Figure 1 Example tube 110 in system 100. Figure 2 As shown, tube 110 can be a flexible tube housing the camera device 112. Additionally, tube 110 includes one or more channels (which may also be referred to as lumens). Figure 2 In the example, tube 110 includes channels 202, 204, and 206. Various instruments can be inserted through tube 110 and channels 202, 204, and 206. For example, tool 114 or guidewire can be inserted through channels 202, 204, and 206. Additionally, tube 110 may accommodate or include a lamp 208. Lamp 208 can illuminate the area in front of tube 110, allowing camera device 112 to capture video footage of the area in front of tube 110.

[0052] Figure 3 It shows Figure 1 Example tube 110 in system 100. Figure 3 As shown, tube 110 may be a flexible tube comprising a distal end 302 (e.g., the end inserted into an organ) and a proximal end 304 (e.g., the end closest to surgical trolley 102). The distal end 302 may include an imaging device 112 and a light 208 illuminating an area in front of tube 110 within the organ. The proximal end 304 may be connected to actuator housing 108. Tube 110 may be formed using segments 310. Each segment 310 may be connected such that tube 110 can be bent or folded along the segment 310. In some cases, tube 110 and imaging device 112 may be collectively referred to as an endoscope.

[0053] Additionally, tube 110 may include multiple sensors for monitoring or measuring various aspects of tube 110. For example... Figure 3 As shown, tube 110 includes a positioning sensor 306, a kinematic sensor 307, and a shape sensor 308. Each of these sensors may be positioned on or within tube 110 and coupled to one or both of actuator housing 108 and computer system 106. Positioning sensor 306 provides computer system 106 with information that can be used to track or measure the position of tube 110. For example, information from positioning sensor 306 may be used to determine coordinates representing the positioning or location of tube 110. Kinematic sensor 307 may detect or measure movement or motion of tube 110. For example, information from kinematic sensor 307 may be used to measure acceleration or velocity of tube 110. Kinematic sensor 307 and positioning sensor 306 may include accelerometers for detecting movement or positioning of tube 110. Shape sensor 308 may detect or measure the shape of tube 110. For example, shape sensor 308 may include optical fiber for detecting when tube 110 bends or folds. The tube 110 may also include other types of sensors, such as vision sensors, stereo vision sensors, depth sensors, etc.

[0054] Figures 4A to 4D The different types of volume reduction processes that can be performed using tool 114 are shown. Figure 4A It shows Figure 1 Example tool 114 in system 100. Specifically, Figure 4A A tool 114 used during an ESG procedure is shown. Tool 114 can be inserted through tube 110 and enter into stomach 402. Tool 114 can exit from tube 110 and enter into the stomach. When tool 114 is positioned next to or near the stomach wall, tool 114 can grasp or pinch a portion of the stomach wall. Tool 114 can then apply sutures to the grasped or pinched portion of the wall to tie that portion of the wall together, which reduces the volume of the stomach. Tool 114 can continue suturing other portions of the stomach to further reduce the volume of the stomach.

[0055] Figure 4B It shows Figure 1 Example tool 114 in system 100. Specifically, Figure 4B The tool 114 used during an endoscopic revision procedure is shown. As shown in (a), the pouch stomach and outlet of the gastric bypass procedure have been dilated. In (b), tool 114 is used to apply interrupted sutures to narrow and reduce the outlet. In (c), tool 114 is used to reduce the volume of the dilated pouch stomach. As shown in (d), the total volume of the outlet and pouch stomach is reduced.

[0056] Figure 4C It shows Figure 1Example tool 114 in system 100. Specifically, Figure 4C The tool 114 used during an endoscopic revision procedure is shown. As shown in (a), there is a distended gastric remnant. In (b), tool 114 is used to apply additional sutures to reduce the gastric volume. As shown in (c), the total gastric volume is reduced.

[0057] Figure 4D It shows Figure 1 Example tool 114 in system 100. Specifically, Figure 4D The tool 114 used during an endoscopic revision procedure is shown. As shown in (a), the site of the previous sleeve gastrectomy has been expanded. Tool 114 is used to apply sutures to reduce the volume. As shown in (b), the total volume of the stomach is reduced.

[0058] Figure 5 It shows the use of Figure 1 This is an example phase of the volume reduction process (ESG process or overhaul process) of system 100. Computer system 106 can perform certain actions or functions during each phase of the process. For example... Figure 5 As shown, the procedure is divided into three distinct phases: a preoperative phase, an intraoperative phase, and a postoperative phase. The procedure may alternatively have other phases. The preoperative phase occurs before the intraoperative phase, and the postoperative phase occurs after the intraoperative phase. Typically, the preoperative phase involves screening to determine if a patient is a good candidate for the volume reduction procedure, and if so, developing a plan for the procedure. During the intraoperative phase, the volume reduction procedure is performed to reduce the volume of the patient's stomach. During the postoperative phase, follow-up care and testing are performed to assess whether the procedure was successful or unsuccessful. The computer system 106 may implement certain features during each of these phases to assist healthcare providers and improve the chances of success for the ESG procedure (e.g., based on optimal technique, suture pattern, suture density, distance traveled between sutures, etc.).

[0059] During the preoperative phase, computer system 106 can use artificial intelligence to assess the patient and develop a procedure plan. Computer system 106 can analyze video footage of the patient's stomach interior and the patient's health or medical records to determine the procedure plan. For example... Figure 5 As shown, computer system 106 receives video 502 of the inside of the patient's stomach. Video 502 can be generated by camera device 112 when tube 110 is inserted into the patient's stomach during the preoperative phase. Video 502 can be captured for screening purposes. In some embodiments, computer system 106 can also generate recommendations regarding suture position, the distance traveled between suture occlusions, the orientation of suture folds, and / or the direction for achieving the desired volume reduction.

[0060] The computer system 106 also receives or retrieves a patient profile 504. The profile 504 may include health or medical information about the patient. For example, the profile 504 may indicate the patient's height, weight, body composition, or body mass index. The profile 504 may also indicate any health or metabolic conditions of the patient (e.g., obesity-related health conditions). For example, the profile 504 may indicate whether the patient has hyperlipidemia, hypertension, diabetes, sleep apnea, arthritis, heart disease, etc. The profile 504 may also indicate data collected by the patient's wearable devices, such as heart rate, calorie intake, blood sugar levels, snoring levels, etc. The profile 504 may also indicate data collected using Bluetooth remote patient monitoring, such as a smart scale that provides weight and body composition.

[0061] Computer system 106 can use neural network 506 (which may be one or more neural networks) to analyze video 502 and profile 504 to determine whether the patient is a candidate for the volume reduction procedure, and if so, to generate a plan 508 for the procedure. Neural network 506 can be trained to determine any suitable information based on video 502 and profile 504. For example, neural network 506 can be trained to determine landmarks and features of the stomach in the video. Neural network 506 can analyze video 502 to identify or detect these landmarks or features. Based on the detected landmarks or features, neural network 506 can determine the size, orientation, or shape of the patient's stomach based on video 502.

[0062] As another example, neural network 506 can be trained to analyze information in profile 504. Neural network 506 can be trained using profiles of different patients stored in a database. Neural network 506 can analyze these profiles to detect patterns or trends in the profiles and learn whether these patterns or trends indicate that the patient is a candidate for ESG. For example, neural network 506 can learn from these profiles which types of medical conditions are treatable using a volume reduction procedure and which types of medical conditions would suggest that a volume reduction procedure should not be performed. As another example, neural network 506 can learn from these profiles which combinations of health or medical conditions make the patient a suitable candidate for a volume reduction procedure and which combinations make the patient less suitable as a candidate for a volume reduction procedure.

[0063] The neural network 506 can also learn how to use information from the profile to determine the planning for the volume reduction process. For example, the neural network 506 can learn what types of tools are successfully used to perform volume reduction on stomachs with different physical characteristics (e.g., the shape, size, orientation, and morphology of the stomach). As another example, the neural network 506 can learn what amount of volume reduction is successfully used to treat certain medical conditions.

[0064] The trained neural network 506 can then analyze the patient's profile 504 to determine whether the patient is a candidate for the volume reduction procedure. For example, the neural network 506 can examine the patient's health and medical conditions indicated in the profile 504 to determine if the patient has health and medical conditions that can be treated with volume reduction. As another example, the neural network 506 can determine whether the profile 504 indicates a health or medical condition that would indicate the level of benefit (e.g., adverse, neutral, significant) the patient is likely to gain from undergoing the volume reduction procedure. In this way, the neural network 506 effectively compares the profile 504 with other profiles used to train the neural network 506 to determine whether the volume reduction procedure would be helpful or harmful to the patient.

[0065] The trained neural network 506 can also use information from the profile 504 and information collected from the video 502 to determine a plan 508 for the volume reduction process. As previously described, the neural network 506 can analyze the video 502 to determine the size, shape, orientation, and / or shape of the stomach. The neural network 506 can then determine a plan 508 for the process based on one or more physical characteristics of the stomach (e.g., size, shape, orientation, and shape) and the health and medical condition indicated in the profile 504. For example, the neural network 506 can determine the amount of stomach volume reduction that would be beneficial in treating the patient's medical condition. As another example, the neural network 506 can determine the type of tool that should be used to perform the volume reduction process to treat the patient's medical condition. This information can be included in the plan 508. For example, the plan 508 can indicate whether the patient is a candidate for volume reduction, one or more physical characteristics of the stomach (e.g., size, shape, orientation, and shape), and what medical condition is being treated by the process. As another example, the plan 508 can indicate the patient's desired volume reduction and the type of tool that should be used to perform the process.

[0066] In some implementations, computer system 106 uses planning 508 to generate simulation 510. Simulation 510 can simulate a procedure on a patient's stomach according to planning 508. For example, computer system 106 can use one or more physical features of the stomach indicated in planning 508 to generate a virtual environment (e.g., virtual reality or augmented reality environment) simulating the patient's stomach. Computer system 106 can use SLAM processing or other 3D reconstruction techniques to construct the virtual environment for simulation 510. Healthcare providers can perform or execute simulation 510 to practice the procedure on the stomach before actually performing the procedure on the patient. In this way, healthcare providers can practice the procedure, which reduces the chance of errors occurring during the actual procedure.

[0067] Computer system 106 can record manipulations performed during simulation 510. The recorded manipulations can be displayed as ghost images or fragments during the intraoperative phase to guide the healthcare provider. The healthcare provider can then view and simulate the recorded manipulations during the intraoperative phase. In this way, the healthcare provider can practice the procedure during simulation and prepare ghost images or fragments to guide the healthcare provider during the intraoperative phase.

[0068] In some implementations, planning 508 includes an overlay plot, in which a model or image of the stomach is superimposed with an estimated, predicted, or approximate model or image of the stomach after the execution process. The overlay plot illustrates changes in the size, shape, and / or orientation of the stomach that may be expected after the execution process.

[0069] During the intraoperative phase, computer system 106 uses artificial intelligence to assist healthcare providers in implementing the plans 508 developed during the preoperative phase. For example... Figure 5 As shown, computer system 106 receives video 512 during the intraoperative phase. Video 512 can be captured by camera device 112 when tube 110 and camera device 112 are inserted into the patient's stomach. Video 512 can be a separate video from video 502 captured during the preoperative phase. In some embodiments, the preoperative phase occurs immediately preceding the intraoperative phase. The healthcare provider does not need to remove tube 110 or camera device 112 from the patient's stomach. Camera device 112 captures both video 502 and video 512. Video 512 is then a continuation of video 502.

[0070] Computer system 106 uses neural network 514 (which may be one or more neural networks) to analyze video 512. Neural network 514 may be the same as neural network 506, or neural network 514 may be separate from neural network 506. Neural network 514 can be trained to recognize landmarks or features in the video of the stomach. For example, different videos of the stomach can be used to train neural network 514. Neural network 514 can learn to recognize different features (e.g., transitions, bends, folds, etc.) appearing in these videos. The trained neural network 514 can then analyze video 512 to identify landmarks 516 in the patient's stomach. For example, neural network 514 can identify landmarks 516 that mark transitions from the stomach to other organs (e.g., the esophagus or duodenum). Neural network 514 can use these landmarks 516 to identify or mark the boundaries of the patient's stomach. As another example, neural network 514 can identify landmarks 516 that indicate bends or folds in the stomach (including folds that can serve as anatomical fingerprints of the patient's anatomy).

[0071] The neural network 514 can identify marker points 516 in different frames of video 512. For example, the neural network 514 can identify a marker point 516 appearing in one frame of video 512, and the neural network 514 can identify the same marker point 516 appearing in subsequent frames of video 512. The neural network 514 can determine that the marker point 516 identified in different frames is the same marker point 516 (e.g., based on the size and shape of the marker point 516). The neural network 514 can then bundle the identification results of these marker points 516 in different frames together. The computer system 106 can then analyze the frames of video 512 to see how these marker points 516 move to different regions in different frames of video 512.

[0072] Computer system 106 can use SLAM processing and marker points 516 to generate a map 518 of the patient's stomach. SLAM processing can determine the boundaries of the stomach based on video 512. Computer system 106 can use these boundaries to generate map 518, which can be a two-dimensional or three-dimensional map of the stomach. Marker points 516 can indicate the boundaries of the stomach in map 518. For example, marker points 516 can identify the transition between the duodenum and esophagus. Computer system 106 can exclude areas beyond the transition from map 518. Therefore, map 518 can omit the esophagus and duodenum. Computer system 106 can display the map 518 of the stomach on display 116 or 118.

[0073] In the example processing, computer system 106 can use marker point 516 and other measurements from sensors on tube 110 to generate a map 518 of the stomach. For example, computer system 106 can locate marker point 516 in several frames of video 512. Computer system 106 can analyze each frame to identify or match marker point 516 in each frame. Due to the movement of tube 110 within the organ, marker point 516 may move to different locations within the frame. Measurements from sensors on tube 110 can indicate the movement of tube 110 that occurs between frames. Using this information, computer system 106 can determine how the frames correspond to each other in three-dimensional space (e.g., the depth of one frame relative to another). Computer system 106 can then stitch the frames together based on the location of marker point 516 and the sensor measurements to generate a map 518, which can be a three-dimensional map.

[0074] In some implementations, computer system 106 also uses neural network 514 to identify tubes 110, camera devices 112, and / or tools 114 appearing in video 512 (e.g., selected according to planning 508). For example, neural network 514 can be trained to detect these items in the video. Neural network 514 can identify these items while analyzing video 512. Computer system 106 can then use SLAM processing to determine the location of these items in a map 518 of the stomach. Computer system 106 can present map 518 and the location of tubes 110, camera devices 112, or tools 114 in map 518 on displays 116 or 118 to prevent healthcare providers from becoming disoriented during volume reduction procedures.

[0075] The computer system 106 can also generate an overlay map 520 to guide the application of the sutures. For example, the computer system 106 can determine where the sutures should be applied to the stomach based on a map 518 of the stomach and / or based on one or more physical features of the stomach (e.g., size, shape, orientation, and / or form) indicated in the plan 508. For example, the computer system 106 can determine where bends and folds are located in the stomach and determine that the sutures should be applied along or consistent with these bends and folds to reduce the chance of the sutures injuring the stomach or subsequently detaching from it. Additionally, the computer system 106 can determine the pattern or arrangement of the sutures and their orientation. The determined location, arrangement, and orientation of the sutures can be consistent with best medical practice. The computer system 106 can then generate an overlay map indicating the location, arrangement, and orientation of the sutures.

[0076] In some implementations, when determining the location, arrangement, or orientation of the sutures, computer system 106 also considers the desired volume reduction indicated in planning 508. For example, computer system 106 may determine the location, arrangement, or orientation of the sutures that will achieve the desired volume reduction. When computer system 106 determines that the desired volume reduction should be achieved using the sutures indicated in overlay drawing 520, computer system 106 may stop adding suture indications to overlay drawing 520.

[0077] Computer system 106 can overlay overlay diagram 520 onto video 512 and / or map 518 to indicate where and in what direction the suture should be applied to the stomach. When the healthcare provider manipulates tube 110, camera device 112, and / or tool 114 through the stomach, the healthcare provider can view video 512 and / or map 518 along with overlay diagram 520 on monitors 116 or 118 to understand where and in what direction the suture should be applied to the stomach. The healthcare provider can apply a suture consistent with overlay diagram 520.

[0078] In some implementations, overlay diagram 520 may include a guide instructing a healthcare provider on how to position or guide tool 114 to apply the sutures indicated in overlay diagram 520. For example, the guide may include a virtual representation of the tool pointing in a specific direction. The healthcare provider can manipulate tool 114 according to the guide to position it near the suture location and in the appropriate orientation to apply the sutures in the direction indicated by overlay diagram 520. For example, by following the guide, the healthcare provider can manipulate tool 114 near the suture location and guide it such that tool 114 is perpendicular to orthogonal to the stomach wall at that location. Tool 114 can then grasp and pinch the wall and apply the sutures. In this way, computer system 106 further assists the healthcare provider in properly applying the sutures, which can reduce the chance of the sutures detaching from the stomach. The guide may also provide other information to help guide tool 114. For example, the guide may indicate the speed of tool 114 or the distance tool 114 should move.

[0079] In a particular implementation, when sutures are applied to the stomach, computer system 106 can use neural network 514 to update map 518 and / or plan 508. For example, neural network 514 can be trained to detect the presence of sutures in video 512. Additionally, neural network 514 can be trained to detect stomach movement in video 512. Computer system 106 can translate these movements into changes in the shape, orientation, and / or size (e.g., volume) of the stomach. Neural network 514 can analyze video 512 to detect when sutures have been applied to the patient's stomach. When sutures are applied, neural network 514 can also detect movement in the stomach wall in video 512. Computer system 106 can determine changes in the shape, orientation, and / or size of the stomach based on these movements. Computer system 106 can then update map 518 to reflect changes in the size, orientation, and / or shape of the stomach. In this way, as the process progresses, computer system 106 displays the real-time size or shape of the stomach.

[0080] As another example, computer system 106 can compare the suture detected by neural network 514 with indications in overlay map 520 to determine whether the suture is aligned or misaligned with overlay map 520. Based on the alignment or misalignment of the suture with overlay map 520, computer system 106 can adjust the position or orientation of subsequent sutures in overlay map 520. Computer system 106 can update the overlay map to show the healthcare provider the position and orientation of subsequent sutures that will achieve the desired volume reduction. In this way, computer system 106 provides real-time instructions for locating sutures during the procedure. When the healthcare provider applies subsequent sutures according to the updated position or orientation, the healthcare provider can achieve the desired volume reduction in the stomach. In some cases, computer system 106 can determine that a misaligned, crossed, or incomplete full-thickness suture should be removed, rather than left in the stomach.

[0081] In some implementations, planning 508 may indicate different types of tools 114 to be used during different parts of the intraoperative phase. Computer system 106 may instruct the healthcare provider when it is appropriate to switch the tools 114 being used by the healthcare provider during the intraoperative phase. For example, computer system 106 may determine when the healthcare provider has arrived at a part of the stomach that planning 508 indicates should be sutured using different types of tools 114. Computer system 106 may present the healthcare provider with messages or instructions to switch tools 114.

[0082] During the postoperative phase, computer system 106 collects data and information about the procedure. For example, computer system 106 may generate images of the stomach during different phases of the procedure. Computer system 106 may generate or collect preoperative image 522 and postoperative image 524 of the stomach. Preoperative image 522 may be an image of the stomach during the preoperative phase before sutures are applied to the stomach. Postoperative image 524 may be an image of the stomach after sutures are applied. By comparing preoperative image 522 with postoperative image 524, computer system 106 can determine the percentage change in stomach size as a result of the procedure. In some embodiments, computer system 106 may also collect images of the stomach or images of map 518 during the intraoperative phase. These images may show the progress of the procedure between the time when preoperative image 522 and postoperative image 524 were taken. Computer system 106 also analyzes preoperative image 522 and / or postoperative image 524 to determine whether the desired volume reduction has been achieved.

[0083] The computer system 106 can also collect operational statistics 526 for the process. Operational statistics 526 may include any information related to the process. For example, the computer system 106 may collect or record the number of sutures applied to the stomach, the location of these sutures, and the orientation of these sutures. Additionally, the computer system 106 may record the percentage change in stomach size as a result of the process.

[0084] The computer system 106 can also collect the results 528 of the procedure. Results 528 can be collected or summarized during follow-up visits to the patient after the procedure and can indicate the patient's response to the procedure. Results 528 can indicate how successful the procedure has been in treating the patient's health or medication status. For example, results 528 may include the patient's weight loss at various time points, the patient's blood glucose levels, changes in medications, and / or whether the patient continues to snore. Additionally, results 528 may include the number of sutures that have detached from the stomach, failed, or loosened after the ESG procedure.

[0085] Computer system 106 can use information and data collected during the postoperative phase to further train the artificial intelligence used during the preoperative and / or intraoperative phases. For example, computer system 106 can store the collected data and information in the patient's file 504. Figure 5 In the example, computer system 106 includes preoperative images 522, postoperative images 524, operational statistics 526, and results 528 from a patient profile 504. Computer system 106 then uses the updated profile 504 to train neural network 506 or 514. Neural network 506 or 514 can analyze profile 504 to determine whether the ESG procedure successfully treated the patient's medical or health condition. Neural network 506 or 514 can also determine whether the location and orientation of the sutures achieved the desired reduction in gastric volume. Neural network 506 or 514 can then use this new information when analyzing future videos and profiles for subsequent ESG procedures. In this way, computer system 106 continues to improve the artificial intelligence used during ESG procedures.

[0086] Figure 6 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 6 The computer system 106 is illustrated with features designed to assist healthcare providers during the intraoperative phase. (Example) Figure 6As shown, computer system 106 has a plan 508 developed during the preoperative phase. Plan 508 may indicate any suitable information for the ESG procedure, including the expected reduction in stomach volume, one or more physical characteristics of the stomach (e.g., size, shape, orientation, and / or form), and the type of instrument 114 to be used in the procedure. Computer system 106 may also have video 512 captured by a camera device 112 in the stomach during the intraoperative phase.

[0087] Computer system 106 generates an overlay diagram 520 that includes indications showing where sutures should be applied in the stomach and the direction of these sutures. In some embodiments, overlay diagram 520 includes a guide 602. Guide 602 can provide indications of the positioning and orientation (e.g., direction) of tool 114 to properly apply sutures at the location and in the direction indicated by overlay diagram 520. For example, guide 602 can guide tool 114 to be positioned orthogonal to the stomach wall, allowing the tool to grip and pinch the stomach wall more firmly and allowing sutures to be applied more firmly to the stomach wall. Guide 602 can also indicate other information about tool 114, such as the speed of tool 114 and the distance tool 114 should travel. Computer system 106 can display overlay diagram 520 and guide 602 on display 116 or 118. Overlay diagram 520 and guide 602 can be positioned on video 512 displayed on display 115 or 118. Healthcare providers can view overlay diagram 520 and guide 602 on video 512 to determine how to manipulate tool 114 and where to apply sutures. In some embodiments, guide 602 can instruct on how to manipulate tool 114 to reduce or minimize tissue trauma. For example, guide 602 can guide a healthcare provider on how to guide or tilt tool 114 to minimize or reduce tissue trauma when inserting it into the stomach.

[0088] Computer system 106 can use neural network 514 to analyze video 512 to detect when suture 604 has been applied to the stomach wall. For example, neural network 514 can be trained using numerous videos taken from inside the stomach. Neural network 514 can be trained to detect sutures applied to these stomachs. The trained neural network 514 can then analyze video 512 to determine when suture 604 has been applied to the stomach. Neural network 514 can detect suture 604 along with any corresponding movement or motion of the stomach wall as a result of the application of suture 604. Computer system 106 can use the output of neural network 514 to determine changes in the physical characteristics of the stomach (e.g., shape, orientation, and / or size) caused by suture 604. Computer system 106 can then update map 518 of the stomach to account for this change in the physical characteristics of the stomach. For example, if suture 604 causes a portion of the stomach to fold inward, computer system 106 can update map 518 to show that portion of the stomach folded inward. The computer system 106 can then display an updated map 518 on a monitor 116 or 118, allowing healthcare providers to see the results of applying the sutures 604.

[0089] Figure 7 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 7 The computer system 106 is illustrated with features designed to assist healthcare providers during the intraoperative phase. (Example) Figure 7 As shown, computer system 106 receives video 512 captured by camera device 112 positioned in the patient's stomach. Computer system 106 uses neural network 514 to analyze video 512 to detect tool 114. Neural network 514 can be trained using different videos from inside the stomach during the volume reduction process. Neural network 514 can be trained to detect tool 114 in these videos. Computer system 106 can then use neural network 514 to analyze video 512 to detect tool 114 in video 512. Neural network 514 can also detect occlusion 704 caused by tool 114. Occlusion 704 can be a part of the stomach that is blocked or obstructed from view in video 512 by tool 114.

[0090] Computer system 106 can adjust video 512 to improve the visibility of occluded areas. For example, computer system 106 can segment tool 114 from video 512, making the occluded area visible. As another example, computer system 106 can increase the transparency 706 of tool 114 in video 512, making the occluded portion 704 more visible through tool 114. In this way, computer system 106 makes the occluded portion 704 more visible to the healthcare provider during the procedure without requiring the healthcare provider to move tool 114 to expose the portion of the stomach occluded by the occluded portion 704. Therefore, computer system 106 provides increased visibility of the stomach during the procedure. In some implementations, the healthcare provider can control when computer system 106 segments or removes tool 114 from the field of view, or when computer system 106 increases the transparency 706 of tool 114. For example, computer system 106 can provide the healthcare provider with settings or options for activating and deactivating segmentation or transparency.

[0091] Figure 8 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 8 Features implemented by computer system 106 to assist healthcare providers during the intraoperative phase are illustrated. For example... Figure 8 As shown, computer system 106 receives a plan 508 developed during the preoperative phase. Plan 508 may indicate a desired reduction in one or more physical features of the stomach (e.g., size, shape, orientation, and / or form) and / or stomach volume. Computer system 106 also generates a map 518 of the stomach (e.g., by analyzing video 512 of the stomach using a neural network 514).

[0092] Computer system 106 can use information from plan 508 and map 518 to generate overlay map 520. For example, computer system 106 can use map 518 to locate bends and folds in the stomach. Computer system 106 can also determine where sutures should be applied in the stomach to achieve the volume reduction indicated in plan 508. For example, computer system 106 can determine a certain number of sutures should be applied to achieve the desired volume reduction. Additionally, computer system 106 can determine that sutures should be positioned aligned with or along bends or folds in the stomach to reduce the chance of sutures detaching from the stomach. In some embodiments, computer system 106 can also determine the direction of the sutures to reduce the chance of suture detachment, knotting, or crossing.

[0093] like Figure 8As shown, overlay diagram 520 may include locations 802, 804, and 806 where sutures should be applied to the stomach. In some embodiments, overlay diagram 520 may also include the orientation of these sutures. Computer system 106 may overlay overlay diagram 520 onto video 512 captured by camera device 112 in the patient's stomach. By overlaying overlay diagram 520 onto video 512, computer system 106 may introduce markings or indications at locations 802, 804, and 806 where sutures should be applied into video 512. Healthcare providers may view video 512 on displays 116 or 118 to see overlay diagram 520 and indications at locations 802, 804, and 806 to understand where sutures should be applied. Healthcare providers may also understand the orientation or path of the sutures.

[0094] The neural network 514 can also identify locations where sutures should be avoided and include these locations in the overlay map 520. For example, the neural network 514 can be trained to identify scars, polyps, growths, or ulcers in numerous videos captured from other stomachs. The trained neural network 514 can then analyze video 512 to identify scars, polyps, growths, or ulcers in the stomach. The computer system 106 can then include the locations of scars, polyps, growths, or ulcers as marker points 516 in the overlay map 520. The overlay map 520 can then indicate these locations as areas in the stomach where sutures should not be applied. Healthcare providers can review the overlay map 520 during a volume reduction process to understand where sutures should be avoided on the stomach.

[0095] Computer system 106 can use neural network 514 to analyze video 512 to detect when suture 808 has been applied to the stomach. Numerous videos of the volume reduction process can be used to train neural network 514 to detect sutures appearing in these videos. The trained neural network 514 can then be used to analyze video 512 to detect when suture 808 has been applied to the patient's stomach.

[0096] Computer system 106 can use the output of neural network 514 to determine the position or orientation of suture 808. Specifically, computer system 106 can determine the degree of alignment between suture 808 and an indicator included at one of positions 802, 804, or 806 in overlay map 520. Computer system 106 can then update overlay map 520 based on the degree of alignment between suture 808 and the indicator. For example, if suture 808 is aligned with one of the indicators in overlay map 520, computer system 106 may not adjust or change overlay map 520 much for subsequent sutures. However, if suture 808 is misaligned with one of the indicators in overlay map 520, computer system 106 can adjust or change overlay map 520 to account for the misaligned suture 808. For example, computer system 106 can adjust the position or orientation in overlay map 520 for subsequent sutures to strengthen or reinforce suture 808. As another example, computer system 106 can add additional positions or orientations to overlay 520, allowing additional sutures to be indicated in overlay 520. These additional sutures can support or reinforce suture 808. In this way, computer system 106 can adjust overlay 520 during the procedure to accommodate sutures applied to the stomach by a healthcare provider. Therefore, computer system 106 can increase the probability that the procedure will successfully treat the patient's health or medical condition.

[0097] Figure 9 Is Figure 1 The flowchart illustrates an example method 900 executed in system 100. In a particular embodiment, computer system 106 executes method 900. By executing method 900, computer system 106 implements certain features that assist healthcare providers during a volume reduction process. These features can increase the chances of the process successfully treating a patient's health or medical condition.

[0098] In box 902, computer system 106 determines a plan 508 for the volume reduction procedure during the preoperative phase. For example, computer system 106 may use neural network 506 to analyze video 502 of the patient's stomach interior to determine one or more physical features of the stomach (e.g., size, shape, orientation, and / or form). Additionally, computer system 106 may analyze a patient profile 504 indicating the patient's health or medical condition. Computer system 106 may use neural network 506 to compare profile 504 with other past patient profiles to determine if the patient is a good candidate for the volume reduction procedure, and if so, to determine the desired volume reduction of the stomach given the patient's health and medical condition.

[0099] In box 904, computer system 106 generates a map 518 of the stomach during the intraoperative phase. Computer system 106 can use neural network 514 to analyze video 512 captured from inside the patient's stomach. Neural network 514 can detect landmarks 516 appearing in video 512. These landmarks 516 can indicate the boundaries of the stomach and specific locations within the stomach. Computer system 106 can use these landmarks 516 to generate a map 518 of the stomach. Map 518 can be a two-dimensional or three-dimensional map of the stomach.

[0100] In box 906, computer system 106 displays video 512 and / or map 518. For example, computer system 106 may transmit video 512 and map 518 to display 116 or 118. Healthcare providers can view video 512 and map 518 on display 116 or 118 to understand where they are operating and reduce the chance of getting lost.

[0101] In box 908, computer system 106 uses map 518 and a plan 508 developed during the preoperative phase to generate overlay map 520. Overlay map 520 may include indicators indicating the locations where sutures should be applied to the stomach. The indicators may also indicate the direction of these sutures.

[0102] In box 910, computer system 106 presents an overlay map 520 on display 116 or 118 on video 512 and / or map 518. For example, overlay map 520 may present indicators or markers on video 512 or map 518 to show a healthcare provider where sutures should be applied in the stomach and the direction of these sutures. The healthcare provider can then manipulate tool 114 to apply sutures at these indicated locations. In this way, in some implementations, computer system 106 increases the chances that the procedure will successfully treat the patient's health and medical condition.

[0103] In some embodiments, overlay diagram 520 also includes a guide 602 that shows a healthcare provider how to manipulate the tool 114 for applying sutures. For example, guide 602 may indicate the location and orientation (e.g., direction) of tool 114. The healthcare provider may manipulate tool 114 to align with guide 602 to apply sutures to the location indicated in overlay diagram 520 and in the direction indicated in overlay diagram 520.

[0104] During the postoperative phase, computer system 106 can collect information and data regarding the process used to train neural networks 506 and / or 514 for subsequent volume reduction procedures. For example, computer system 106 can collect preoperative images 522 of the stomach, postoperative images 524 of the stomach, operational statistics 526, and results 528. Computer system 106 can use the collected information and data to update patient profile 504. Computer system 106 can then use the updated patient profile 504 to update or train neural networks 506 and / or 514. Therefore, computer system 106 continues to use the results of the completed ESG procedure to update and train the artificial intelligence used during the ESG procedure, which can improve the diagnostic and other capabilities of neural networks 506 and / or 514.

[0105] Figure 10 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 10 The illustration shows a computer system 106 performing volume calculations during the intraoperative phase. In a particular embodiment, the computer system 106 can calculate the stomach's volume during the intraoperative phase, eliminating the need for healthcare providers to visually assess the stomach's volume, which can be an inaccurate method for determining volume.

[0106] Computer system 106 receives video 512, which can be captured by camera device 112 positioned in the stomach during the surgical phase. Computer system 106 can use neural network 514 to analyze video 512 to detect landmarks 516 in the stomach. Neural network 514 can be trained to detect different landmarks in many videos captured inside different parts of the stomach. The trained neural network 514 can analyze video 512 to detect these landmarks 516 when they appear in video 512.

[0107] The marker 516 may include transitions 1002 and features 1004. Transitions 1002 may indicate the boundaries or inlet and outlet of the stomach. For example, transition 1002 may indicate the boundary between the stomach and the esophagus. Another transition 1002 may indicate the boundary between the stomach and the duodenum. The neural network 514 may be trained to recognize these transitions 1002 when they appear in video 512. Features 1004 may indicate the location or structure of the stomach. For example, feature 1004 may indicate a fold or bend in the stomach wall. As another example, feature 1004 may include scars or polyps on the stomach wall. Feature 1004 may also include the geometry of the stomach wall and the stomach.

[0108] Computer system 106 uses marker points 516 to generate a map 518 of the stomach. For example, marker points 516 may indicate boundaries, folds, and bends in the stomach wall. Computer system 106 can generate a map 518 that corresponds to the detected marker points 516. Therefore, map 518 can be an accurate representation of one or more physical features of the stomach (e.g., size, shape, orientation, and / or form) in video 512. In some embodiments, computer system 106 utilizes transitions 1002 to determine some of the boundaries of map 518. Computer system 106 can omit areas outside transitions 1002 from map 518. For example, if transition 1002 is the boundary between the stomach and the esophagus or duodenum, computer system 106 can omit the esophagus and duodenum from map 518. In this way, computer system 106 can limit map 518 to the stomach.

[0109] Computer system 106 can then calculate the volume 1006 of the stomach based on map 518. For example, computer system 106 can calculate the volume 1006 of the stomach based on one or more physical features of the stomach appearing in map 518 (e.g., size, shape, orientation, and / or form). In some embodiments, computer system 106 uses measurement results 1008 to calculate the volume 1006. Measurement results 1008 may be measurements of the length or size of different parts of the stomach. These measurement results 1008 can be determined based on sensor outputs 1010. Sensor outputs 1010 may be generated by one or more sensors on tube 110. For example, sensor outputs 1010 may be generated by one or more of a positioning sensor 306, a kinematic sensor 307, and a shape sensor 308 positioned on tube 110. As tube 110 moves through the stomach, sensor outputs 1010 can indicate the distance traveled by tube 110. Thus, sensor outputs 1010 can provide measurement results 1008 for different regions of the stomach. Computer system 106 can use these measurements 1008 when calculating the volume 1006 of the stomach. For example, computer system 106 can determine that tube 110 has moved from one region of the stomach to another in map 518. Computer system 106 can also determine the measurement 1008 of the distance traveled by tube 110. Using the measurement 1008, computer system 106 can determine the distance between two points in map 518. Computer system 106 can then extrapolate the size, shape, orientation, and / or form of the stomach in map 518 and calculate the volume 1006 based on this size, shape, orientation, and / or form.

[0110] As an example, computer system 106 and / or neural network 514 can identify feature 1004 in video 512 and track how feature 1004 transitions to different pixels in different frames of video 512 as tube 110 moves through the stomach. For example, computer system 106 and / or neural network 514 can identify a feature in the first frame of video 512. Tube 110 can then move through the stomach, and computer system 106 and / or neural network 514 can identify a feature in the second frame of video 512. Due to the movement of tube 110, the feature can appear in different pixels in the first and second frames. Computer system 106 can determine the number of units that tube 110 has moved in map 518 based on the distance between the pixels where feature 1004 appears in the first and second frames. Computer system 106 can also determine the distance that tube 110 has moved from one or more sensors on tube 110. Computer system 106 can then determine a measurement 1008 of the stomach based on the distance that tube 110 has moved. The computer system 106 can then extrapolate the size of the stomach based on the measurement results 1008 and the number of units the tube 110 moves in the map 518. The computer system 106 can use this size to calculate the volume 1006 of the stomach.

[0111] Figure 11 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 11 The computer system 106 is shown updating the volume 1006 based on the sutures applied to the stomach. When sutures are applied to the stomach, the computer system 106 can detect changes in the stomach volume 1006 caused by the sutures. The computer system 106 can then update the volume calculation. In this way, when sutures are applied during the intraoperative phase, the computer system 106 provides the surgeon with the real-time volume of the stomach.

[0112] Computer system 106 receives video 512. Computer system 106 uses neural network 514 to analyze video 512 to detect when suture 1102 has been applied to the stomach. Neural network 514 can be trained using many different videos of the volume reduction process. Neural network 514 can be trained to detect sutures appearing in these videos. The trained neural network 514 can then be used to analyze video 512 to detect when suture 1102 has been applied to the stomach. For example, neural network 514 can detect the position and orientation of suture 1102. Neural network 514 can also be trained to detect movement 1104 in the stomach wall when suture 1102 is applied. Neural network 514 can also detect the direction and distance of the detected movement 1104.

[0113] Computer system 106 can use the detected suture 1102 and the detected movement 1104 to update map 518 to produce an updated map 1105. For example, computer system 106 can determine that a portion of the stomach has been bound together by suture 1102 based on the position and orientation of suture 1102 and the direction of movement 1104. Computer system 106 can then update map 518 such that the portion of the stomach in map 518 is bound together and moves according to suture 1102 and movement 1104. This update produces an updated map 1105.

[0114] The computer system 106 can then recalculate the stomach volume 1006 in the updated map 1105. For example, the sutures 1102 and movement 1104 can reduce the stomach volume. Therefore, the computer system 106 can calculate the reduced volume 1006. In some embodiments, the computer system 106 also calculates the change 1106 in the volume 1006. For example, the change 1106 can be a percentage change in the volume 1006 caused by the sutures 1102 and movement 1104.

[0115] In some implementations, computer system 106 overlays an updated map 1105 onto a map 518 generated before the application of suture 1102 (or before the application of any sutures). By overlaying the updated map 1105 onto the original map 518 and presenting the overlay on display 116 or 118, the healthcare provider can compare the size of the stomach before and after the application of suture 1102. The healthcare provider can then understand the amount of change caused by suture 1102. As the healthcare provider applies more sutures to the stomach, computer system 106 can continue to update the overlay.

[0116] Figure 12 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 12 The diagram illustrates how computer system 106 uses a determined change 1106 in the stomach's volume 1006 to determine the progress of the volume reduction process. (As shown...) Figure 12As shown, computer system 106 compares change 1106 with threshold 1202. Threshold 1202 may be the expected volume reduction in the plan 508 developed during the preoperative phase. If change 1106 is equal to or exceeds threshold 1202, computer system 106 may determine that the procedure is complete and the healthcare provider should stop. If change 1106 does not exceed threshold 1202, computer system 106 may determine that the procedure can continue to further reduce the stomach volume 1006. For example, if threshold 1202 indicates an expected 70% reduction in stomach volume 1006, computer system 106 may allow the procedure to continue until change 1106 meets or exceeds the 70% to 80% threshold 1202.

[0117] In some implementations, the computer system 106 provides a progress bar 1204 on a display 116 or 118. The progress bar 1204 can indicate a change 1106 and how close the change 1106 is to a threshold 1202. As the stomach volume 1006 shrinks during the intraoperative phase, the change 1106 can increase, and the computer system 106 can increase the progress indicated by the progress bar 1204. In this way, the computer system 106 provides a visual indicator of the reduction in stomach volume during the intraoperative phase to the healthcare provider.

[0118] In some implementations, computer system 106 provides instruction 1206 to a healthcare provider, informing them whether the process should continue or stop. Instruction 1206 may be a visual or auditory instruction. For example, computer system 106 may present a visual or textual message informing the healthcare provider whether to continue or stop reducing the stomach's volume. When the change 1106 exceeds a threshold 1202, computer system 106 may present a visual or auditory instruction 1206 to the healthcare provider that they should stop, because the desired volume reduction has been achieved.

[0119] Figure 13 Is Figure 1 The flowchart illustrates an example method 1300 executed in system 100. In a particular implementation, computer system 106 executes method 1300. By executing method 1300, computer system 106 provides real-time calculation of the stomach's volume during the intraoperative phase, which can be more accurate than having a healthcare provider visually assess the reduction in stomach volume.

[0120] In box 1302, computer system 106 receives video 512. Video 512 may be captured by camera device 112 positioned in the patient's stomach during the surgical phase. In box 1304, computer system 106 generates map 518 based on video 512. For example, computer system 106 may use SLAM processing to determine the boundaries of the stomach. Computer system 106 may also use neural network 514 to analyze video 512 to detect landmarks 516 appearing in video 512. Landmarks 516 may also indicate the boundaries, size, or shape of the stomach. Computer system 106 may use information from SLAM processing and neural network 514 to generate map 518 of the stomach. Map 518 may be a two-dimensional or three-dimensional map of the stomach.

[0121] In block 1306, computer system 106 calculates the volume 1006 of the stomach based on a map 518 of the stomach. In some embodiments, computer system 106 may use map 518 and measurements 1008 derived from sensor output 1010 to calculate the volume 1006. For example, computer system 106 may determine or calculate the volume of the stomach by measuring its size, shape, orientation, and / or form in map 518. Computer system 106 may then present the volume 1006 to a healthcare provider on display 116 or 118 during the intraoperative phase.

[0122] In block 1308, computer system 106 detects sutures 1102 applied to the stomach during the surgical phase. For example, computer system 106 may use neural network 514 to analyze video 512 to detect when and where sutures 1102 are applied to the stomach. In some embodiments, computer system 106 may also use neural network 514 to analyze video 512 to detect movement 1104 in the stomach wall caused by the application of sutures 1102. In block 1310, computer system 106 updates map 518 to account for sutures 1102 and movement 1104, thereby producing an updated map 1105. For example, sutures 1102 and movement 1104 may indicate that a portion of the stomach has been ligated together. Computer system 106 may then update map 518 to produce an updated map 1105 showing that this portion of the stomach is ligated together, which may cause a reduction in stomach volume 1006.

[0123] In box 1312, computer system 106 updates or recalculates the stomach volume 1006 based on the updated map 1105. For example, computer system 106 may calculate the updated volume 1006 based on changes in the shape, orientation, and / or size of the stomach in the updated map 1105. Computer system 106 can then display the updated volume 1006 to the healthcare provider on display 116 or 118. In this way, computer system 106 provides the healthcare provider with a real-time calculation of the stomach volume 1006 during the intraoperative phase, which can be more accurate than the healthcare provider visually assessing the stomach volume while viewing video 512.

[0124] Figure 14 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 14 The computer system 106 generates an overlay map 520, which can indicate the position and orientation of sutures that should be applied to the stomach to achieve a desired volume reduction.

[0125] Computer system 106 receives video 512, which may be captured by camera device 112 positioned in the stomach during the surgical phase. Computer system 106 uses neural network 514 to analyze video 512 to identify landmarks 516 appearing in video 512. Landmarks 516 may include features 1004 appearing in the stomach. For example, feature 1004 may include folds, bends, scars, or polyps appearing in the stomach. In some embodiments, neural network 514 also identifies tools 114 appearing in video 512.

[0126] Computer system 106 can determine location 1402 based on feature 1004. Location 1402 can be the location of a specific feature 1004. For example, location 1402 can be the location of a bend or fold in the stomach. As another example, location 1402 can be the location of a scar, growth, or polyp in the stomach. Computer system 106 can determine whether location 1402 is a location where sutures should be applied or where sutures should not be applied. For example, computer system 106 can determine that sutures should be applied at location 1402 of a bend or fold such that the sutures are applied along the bend or fold that conforms to the shape of the stomach, which can reduce the chance that the sutures will subsequently detach from the stomach. As another example, computer system 106 can determine that sutures should not be applied at location 1402 of a scar, growth, or polyp, which can reduce the chance that the sutures will damage or harm the stomach.

[0127] Computer system 106 determines map 518 based on landmark points 516. Map 518 can be a two-dimensional or three-dimensional map of the stomach. Computer system 106 can determine the shape, form, orientation, and / or size of the stomach based on landmark points 516. Computer system 106 can also determine the location, shape, and arrangement of the stomach wall based on location 1402. Computer system 106 can then generate map 518 of the stomach that corresponds to the determined size, form, orientation, and / or shape.

[0128] Computer system 106 can then use map 518 and planning 508 developed during the preoperative phase to generate overlay map 520. For example, computer system 106 can determine the desired volume reduction to be achieved based on planning 508. Computer system 106 can also determine, based on map 518, where sutures should be applied (e.g., at some locations in location 1402) and how many sutures should be applied to achieve volume reduction. Computer system 106 can then generate an overlay map that can be positioned on video 512 and / or map 518. Overlay map 520 can indicate the location and orientation of sutures to be applied to the stomach. Computer system 106 can display video 512 and / or map 518 with the overlay map on display 116 or 118. Healthcare providers can view the overlay map on video 512 or map 518 to determine where to place sutures.

[0129] Figure 15 It shows Figure 1 Example computer system 106 in system 100. Typically, Figure 15 The overlay image generated by computer system 106 is shown. For example... Figure 15 As shown, computer system 106 can receive video 512 captured by camera device 112 in the stomach during the intraoperative phase. Computer system 106 can analyze video 512 using neural network 514 to determine landmarks 516 appearing in the stomach. Computer system 106 can also receive a plan 508 developed during the preoperative phase. Plan 508 can indicate the desired volume reduction. Computer system 106 can also receive sensor outputs 1010 from sensors on tube 110 and measurement results 1008 obtained based on these sensor outputs.

[0130] Computer system 106 can consider the output of neural network 514, planning 508, and measurement results 1008 to determine the location 1502 and direction 1504 of sutures to be applied to the stomach during the intraoperative phase. For example, computer system 106 can determine certain locations 1502 of sutures that correspond to locations 1402 of bends and folds in the stomach. Computer system 106 can also determine a direction 1504 that corresponds to bends and folds in the stomach. The locations 1502 and directions 1504 of the sutures can be interlocked or crossed, causing the stomach to be bound together along the folds and bends in the stomach, which can reduce the chance of the sutures detaching from the stomach in the future. As another example, computer system 106 can determine locations 1502 such that the sutures are not applied to scars, growths, or polyps in the stomach. In this way, computer system 106 avoids damage or injury to the stomach.

[0131] In some implementations, computer system 106 can (e.g., based on planning 508 or map 518 determined during the preoperative phase) determine the location or position of other organs adjacent to the stomach. Computer system 106 can determine position 1502 such that sutures are not applied near these adjacent organs. In this way, computer system 106 can avoid tool 114 gripping portions of the stomach wall and adjacent organs, and unintentionally applying sutures to both the stomach and adjacent organs (e.g., gallbladder, colon, etc.).

[0132] In some implementations, the computer system 106 or neural network 514 can also determine the angle or rigidity of the stomach based on the video 512 or the planning 508. The computer system 106 can determine the position 1502 and orientation 1504 based on the stomach's angle or rigidity. For example, the computer system 106 can determine the orientation 1504 of the suture so that the suture is aligned with the angle of the stomach, which can reduce the chance of the suture detaching from the stomach. As another example, the computer system 106 can determine the position 1502 so that the suture is applied to a less rigid portion of the stomach rather than a more rigid portion. In this way, the computer system 106 reduces the chance of the suture detaching from the stomach and reduces the chance of the stomach being damaged or injured by the suture.

[0133] Computer system 106 generates an overlay map 520 based on location 1502 and orientation 1504. The overlay map 520 can indicate the location 1502 where sutures should be applied and the orientation 1504 of these sutures. Computer system 106 can then display the overlay map 520 on a video 512 or map 518 of the stomach. Healthcare providers can view the overlay map 520 to determine where to apply sutures and in what direction they should be applied.

[0134] Figure 16 It shows Figure 1Example computer system 106 in system 100. Typically, Figure 16 The overlay diagram 520 shows the computer system 106 updating when sutures are applied to the stomach. (See Figure 520.) Figure 16 As shown, computer system 106 receives video 512 captured by camera device 112 in the stomach during the intraoperative phase. Computer system 106 uses neural network 514 to analyze video 512. Neural network 514 can detect the appearance of suture 1602 in video 512 when a healthcare provider has applied suture 1602. For example, a healthcare provider can use tool 114 to apply suture 1602 consistent with overlay image 520. Neural network 514 can detect the presence of suture 1602 after it has been applied.

[0135] In some embodiments, the neural network 514 can also detect stomach movement 1604 caused by the application of the suture 1602. For example, the suture 1602 can tie parts of the stomach together. Movement 1604 can be the movement of these parts of the stomach detected when these parts of the stomach are tied together.

[0136] Computer system 106 can determine the position 1502 and orientation 1504 of subsequent sutures based on detected suture 1602 and movement 1604. These positions 1502 and orientations 1504 may have been determined before the suture 1602 is applied. Computer system 106 can change or adjust these positions 1502 and orientations 1504 based on the suture 1602 and movement 1604. For example, computer system 106 can determine that the suture 1602 is slightly misaligned with an indicator in overlay figure 520. In response, computer system 106 can adjust the position 1502 and orientation 1504 of subsequent sutures to accommodate the misalignment of the suture 1602 (e.g., to better support the suture 1602 to prevent future detachment).

[0137] Computer system 106 can then update the overlay map using the updated position 1502 and orientation 1504. Overlay map 520 can then include indicators showing the updated position 1502 and orientation 1504 of the subsequent suture. The healthcare provider can review the updated overlay map 520 to determine where and in what direction the subsequent suture should be applied to the stomach. In this way, computer system 106 updates the healthcare provider on where and how to apply the subsequent suture to increase the likelihood of successful treatment.

[0138] Figure 17 Is Figure 1The flowchart illustrates an example method 1700 executed in system 100. In a particular embodiment, computer system 106 executes method 1700. By executing method 1700, computer system 106 determines where sutures should be applied to the stomach and generates an overlay diagram 520 indicating the locations 1502 of these sutures.

[0139] In box 1702, computer system 106 receives video 512 captured during the intraoperative phase by a camera device 112 positioned in the stomach. In box 1704, computer system 106 determines the location 1502 of the sutures. For example, computer system 106 may use neural network 514 to analyze video 512 to determine landmarks 516 in the stomach. These landmarks 516 may indicate the location 1402 of bends or folds in the stomach where sutures should be applied to reduce the chance of suture detachment from the stomach. Additionally, these landmarks 516 may indicate the location 1402 of scars or polyps where sutures should not be applied to avoid damage or injury to the stomach. Computer system 106 may also analyze a plan 508 developed during the preoperative phase, as well as sensor outputs 1010 and measurements 1008 derived from these sensor outputs 1010, to determine the location 1502 of the sutures. For example, plan 508 may indicate a desired volume reduction. The computer system 106 can determine the number of sutures and the location 1502 of the sutures (e.g., consistent with or along the determined location 1402 of the folds or bends in the stomach) to achieve volume reduction.

[0140] In block 1706, computer system 106 generates overlay image 520. Overlay image 520 may indicate the location 1502 where sutures should be applied in the stomach. In some embodiments, overlay image 520 also indicates the direction of the sutures 1504. In block 1708, computer system 106 presents overlay image 520 on video 512. Therefore, display 116 or 118 may present the overlay image on video 512. Healthcare providers can view overlay image 520 on video 512 to understand where sutures should be applied to the stomach during an ESG procedure to achieve the desired volume reduction.

[0141] In summary, computer system 106 assists or guides a volume reduction process (e.g., an ESG procedure or a refractory procedure). Typically, computer system 106 uses artificial intelligence (e.g., machine learning) to provide information to the healthcare provider during different phases of the process. For example, during the pre-operative phase, computer system 106 may use artificial intelligence (e.g., neural networks) to analyze video 502 from inside the patient's stomach along with the patient's medical profile 504 to determine a plan 508 for the procedure. Plan 508 may indicate whether the patient is a good candidate for a volume reduction procedure. Plan 508 may also indicate the volume reduction of the stomach that can successfully treat the patient's medical condition.

[0142] As another example, during the intraoperative phase, computer system 106 can use SLAM processing and neural network 514 to generate a map 518 of the patient's stomach based on video 512 of the stomach's interior, and track the positions of the endoscope and suture tools 114 within the map 518. Computer system 106 can display the map 518 and the positions of the endoscope and tools 114 within it to prevent healthcare providers from becoming disoriented during the procedure.

[0143] Computer system 106 can also use map 518 to calculate the volume 1006 of the stomach. As sutures are applied to the stomach during the procedure, computer system 106 can update map 518 along with the volume 1006 calculation. For example, computer system 106 can use neural network 514 to analyze video 512 to determine when and where sutures 1102 have been applied and to identify changes in the shape, size, and / or orientation of the stomach. Computer system 106 can then update the stomach map 518 to account for these shape changes. Computer system 106 can use the updated map 518 to update the volume 1006 calculation. In this way, computer system 106 provides real-time volume calculation, making it easier for healthcare providers to determine the progress of the procedure and when to stop or continue it.

[0144] As another example, during the procedure, computer system 106 can generate an overlay map 520 indicating the location and orientation of sutures that should be applied to the stomach. For example, computer system 106 can use neural network 514 to determine where in the stomach the sutures should be applied to align with existing best medical practices and achieve the volume reduction indicated in preoperative planning 508. Computer system 106 then generates an overlay map indicating the location of the sutures. Computer system 106 can then position the overlay map 520 (e.g., on display 116 or 118) over video 512 captured from inside the stomach and / or a map 518 of the stomach, so that a healthcare provider can see on display 116 or 118 where the sutures should be applied. For example, overlay map 520 can present visual indicators on video 512 and / or map 518 to indicate where the sutures should be applied. The healthcare provider can then manipulate tool 114 to apply the sutures at the location indicated in overlay map 520. In some implementations, computer system 106 may use neural network 514 to analyze video 512 to determine the position and / or orientation of the applied sutures. Computer system 106 can then update the position and orientation of subsequent sutures in overlay 520 to account for variations caused by the applied sutures. In this way, computer system 106 guides the suturing process, which can advantageously reduce the number of sutures detached from the stomach after the procedure is complete.

[0145] During the postoperative phase, computer system 106 collects data about the procedure. For example, computer system 106 may track the number of sutures applied during the procedure, as well as the location and orientation of the sutures. As another example, computer system 106 may collect images or pictures illustrating the effect of the procedure on the stomach (e.g., preoperative images, intraoperative images, and postoperative images of the stomach map). As another example, computer system 106 may collect follow-up data of the patient illustrating the effectiveness of the procedure (e.g., patient weight, number of detached sutures, reduction in stomach volume, etc.). In some implementations, computer system 106 uses the collected data to train or update artificial intelligence (e.g., neural networks) used by computer system 106 during the preoperative and intraoperative phases. In this way, computer system 106 uses information from the procedure to inform subsequent volume reduction procedures.

[0146] The aspects, embodiments, or modules illustrated in this specification and the accompanying drawings should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure other features. Similar reference numerals in two or more figures denote the same or similar elements.

[0147] In this specification, specific details are set forth in relation to some embodiments consistent with this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Other elements within the scope and spirit of this disclosure may be implemented by those skilled in the art, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0148] Furthermore, the terminology used in this specification is not intended to be limiting. For example, spatially related terms such as “below,” “under,” “lower,” “above,” “upper,” “near,” “far”, etc., may be used to describe the relationship between one element or feature and another element or feature as shown in the figures. In addition to the positions and orientations shown in the figures, these spatially related terms are also intended to include different positions (i.e., positioning) and orientations (i.e., rotational placement) of elements or their operations. For example, if one of the contents in the figures is flipped, the element described as being “below” or “under” other elements or features would then be “above” or “on” other elements or features. Thus, the exemplary term “below” can include both above and below positions and orientations. Devices may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially related descriptors used herein are interpreted accordingly. Similarly, descriptions of movement along and about various axes include various specific element positions and orientations. Additionally, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. Furthermore, the terms "comprises," "comprising," and "includes" specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0149] Where feasible, elements described in detail with reference to one embodiment or module may be included in other embodiments or modules in which such elements are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, the element may still be claimed as included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment or application may be incorporated into other embodiments or aspects unless otherwise specifically described, unless one or more elements would render one or more embodiments inoperable, or unless two or more of the elements provide conflicting functionality.

[0150] In some cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the implementation.

[0151] This disclosure describes the various devices, elements, and portions of computer-aided devices and elements in terms of their state in three-dimensional space. As used herein, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along the element. As used herein, and for devices with repositionable arms, the term "proximal" refers to a direction along its kinematic chain toward the base of the computer-aided device, and "distal" refers to a direction along the kinematic chain away from the base.

[0152] Various aspects of this disclosure are described with reference to computer-aided systems and apparatuses, which may include teleoperated, remotely controlled, autonomous, semi-autonomous, robotic, and other such systems and apparatuses. Furthermore, various aspects of this disclosure are described according to implementations using medical systems, such as the da Vinci Surgical System or ion systems commercially available from Intuitive Surgical Inc. of Sunnyvale, California. However, those skilled in the art will understand that the aspects disclosed herein can be implemented and carried out in various ways, including robotic and non-robotic implementations (where applicable). The techniques described with reference to surgical instruments and methods can be used in other contexts. Therefore, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomy, industrial systems, general-purpose robotics, or teleoperation systems. As another example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general-purpose robotic applications, sensing or manipulating non-tissue artifacts, cosmetic enhancements, imaging of human or animal anatomy, summarizing data from human or animal anatomy, installing or disassembling systems, training medical or non-medical personnel, etc. Additional example applications include procedures for removing tissue from human or animal anatomy (with or without returning it to the anatomy) and procedures for human or animal cadavers. Furthermore, these techniques can also be used in medical treatments or diagnostic procedures, with or without surgical involvement.

[0153] Although illustrative embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, certain features of the embodiments may be employed without the need for corresponding use of other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of this disclosure should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A computer system for estimating gastric volume, the computer system comprising: Memory; as well as A processor, communicatively coupled to the memory, is configured to: Receive video of the inside of the stomach; A map of the stomach is generated based on the video; The volume of the stomach is calculated based on the map of the stomach. Based on the video, a neural network was used to detect sutures applied to the stomach during an endoscopic sleeve gastrectomy procedure, which caused changes in the shape of the stomach. The map of the stomach is updated based on the changes in the shape of the stomach; as well as The calculated volume of the stomach is updated based on the update of the map of the stomach.

2. The computer system according to claim 1, wherein, Generating the map of the stomach includes using the neural network to detect the transition between the esophagus and the stomach in the video, wherein the map omits the esophagus.

3. The computer system according to any one of claims 1 to 2, wherein, Generating the map of the stomach includes using the neural network to detect the transition between the stomach and the duodenum in the video, wherein the map omits the duodenum.

4. The computer system according to any one of claims 1 to 3, wherein, Calculating the volume includes using the neural network to detect features of the stomach in a first frame of the video and to detect the location where the features appear in a second frame of the video.

5. The computer system according to claim 4, wherein, The processor is also configured to determine the measurement result of the stomach based on the position of the feature in the second frame of the video.

6. The computer system according to claim 5, wherein, The volume of the stomach is calculated based on the measurement results.

7. The computer system according to any one of claims 5 to 6, wherein, The measurement results are also determined based on the output of the sensors from the endoscope that generated the video.

8. The computer system according to any one of claims 1 to 7, wherein, The processor is also configured to use the neural network in the video to detect the tool used to apply the suture to the stomach.

9. The computer system according to claim 8, wherein, The processor is also configured to segment the tool from the video.

10. The computer system according to any one of claims 1 to 9, wherein, The processor is also configured to detect movement in the stomach wall when the suture is applied to the stomach in the video.

11. The computer system according to claim 10, wherein, The map of the stomach is also updated based on detected movement in the walls of the stomach.

12. The computer system according to any one of claims 1 to 11, wherein, The processor is also configured to calculate the percentage change in the volume of the stomach after the suture is applied.

13. The computer system according to claim 12, wherein, The processor is also configured to compare the percentage change with a threshold.

14. The computer system according to any one of claims 12 to 13, wherein, The processor is also configured to display a progress bar on the monitor that indicates the progress of the endoscopic sleeve gastrectomy procedure.

15. The computer system according to claim 14, wherein, The processor is also configured to update the progress bar based on a calculated percentage change in the stomach's volume.

16. The computer system according to any one of claims 1 to 15, wherein, Updating the map of the stomach includes changing the shape of the map to represent the change in the shape of the stomach.

17. The computer system according to any one of claims 1 to 16, wherein, The processor is also configured to overlay an updated map of the stomach onto a map of the stomach generated prior to the endoscopic sleeve gastrectomy procedure on a display.

18. A method for estimating gastric volume, the method comprising: Receive video of the inside of the stomach; A map of the stomach is generated based on the video; The volume of the stomach is calculated based on the map of the stomach. Based on the video, a neural network was used to detect sutures applied to the stomach during an endoscopic sleeve gastrectomy procedure, which caused changes in the shape of the stomach. The map of the stomach is updated based on the changes in the shape of the stomach; as well as The calculated volume of the stomach is updated based on the update of the map of the stomach.

19. The method according to claim 18, wherein, Generating the map of the stomach includes using the neural network to detect the transition between the esophagus and the stomach in the video, wherein the map omits the esophagus.

20. The method according to any one of claims 18 to 19, wherein, Generating the map of the stomach includes using the neural network to detect the transition between the stomach and the duodenum in the video, wherein the map omits the duodenum.

21. The method according to any one of claims 18 to 20, wherein, Calculating the volume includes using the neural network to detect features of the stomach in a first frame of the video and to detect the location where the features appear in a second frame of the video.

22. The method of claim 21, further comprising determining the measurement result of the stomach based on the position of the feature in the second frame of the video.

23. The method according to claim 22, wherein, The volume of the stomach is calculated based on the measurement results.

24. The method according to any one of claims 22 to 23, wherein, The measurement results are also determined based on the output of the sensors from the endoscope that generated the video.

25. The method of any one of claims 18 to 24, further comprising using the neural network in the video to detect the tool used to apply the suture to the stomach.

26. The method of claim 25, further comprising the tool for segmenting the video.

27. The method of any one of claims 18 to 26, further comprising detecting, in the video, movement in the wall of the stomach when the suture is applied to the stomach.

28. The method according to claim 27, wherein, The map of the stomach is also updated based on detected movement in the walls of the stomach.

29. The method according to any one of claims 18 to 28, further comprising calculating the percentage change in the volume of the stomach after the suture is applied.

30. The method of claim 29, further comprising comparing the percentage change with a threshold.

31. The method according to any one of claims 29 to 30, further comprising displaying a progress bar on a display indicating the progress of the endoscopic sleeve gastrectomy procedure.

32. The method of claim 31, further comprising updating the progress bar based on a calculated percentage change in the volume of the stomach.

33. The method according to any one of claims 18 to 32, wherein, Updating the map of the stomach includes changing the shape of the map to represent a change in the shape of the stomach.

34. The method according to any one of claims 18 to 33, further comprising overlaying an updated map of the stomach onto a map of the stomach generated prior to the endoscopic sleeve gastrectomy procedure on a display.

35. A non-transitory machine-readable medium storing instructions for estimating gastric volume, the instructions causing the processor, when executed by a processor, to: Perform the method according to any one of claims 18 to 34.

Images