Surgical system and control method

Abstract

The surgical control system communicates with multiple surgical devices. During operation, the surgical devices broadcast device data via a communication bus that communicates with multiple device controllers of the multiple surgical devices, including a message indicating at least one of the control states and detected states associated with the operation of each of the multiple surgical devices. Image data indicating the surgical site is captured, and the surgical state is identified in response to the image data. Aggregated data containing the surgical state and device data is generated in a packet. The aggregated data is communicated via the communication bus.

Description

[Technical Field]

[0001] This disclosure relates, in general, to control systems for surgical devices, and more specifically, to distributed control systems and communication interfaces configured to communicate device data for automatic or assisted operation. Modern operating rooms can incorporate a wide range of medical or surgical devices that can be used to assist surgical procedures, monitor patient health statistics and vital signs, and control various lighting, ventilation, sterilization, and additional devices associated with the medical room. In various implementations, this disclosure provides systems and methods that can assist the operation of such devices by improving the interoperability of medical or surgical devices used in various combinations. [Overview of the project] [Means for solving the problem]

[0002] This disclosure provides a surgical control system that facilitates distributed communication of device data, including control states and detected states related to the operation of multiple medical or surgical devices. In various implementations, the device data may be communicated via a device network or communication bus, through which each of the multiple surgical devices may broadcast or report updates. Updates may be communicated at standard intervals and / or in response to changes in corresponding control states and / or detected states. The device data may be accessed by multiple device controllers corresponding to each of the multiple surgical devices. During operation, the device controllers may independently initiate one or more automatic control settings or control prompts based on programming or control structures associated with a particular surgical device. In this way, this disclosure may provide improved operation of various interconnected surgical or medical devices via a distributed control platform.

[0003] In some implementations, the control system may include a camera device, which may be in the form of an endoscope, arthroscopic device, surgical camera, or similar imaging device communicating with multiple medical devices. During operation, the camera device may be implemented to capture image data indicating the surgical site, and based on the image data, the camera device's control unit or system controller may determine the surgical status of the surgical site. Such determination may include determining a wide range of patient conditions, procedure steps, surgical device status, surgical site status, or various characteristics that can be determined from the image data captured by the camera device. In addition to image data, the camera device may be configured to further interpret device data associated with each of multiple surgical devices communicated via a device network to further improve the interpretation of the image data and determine the surgical or surgical site status.

[0004] In some implementations, the system controller of a camera device may access and generate aggregated data within packets. Each packet of aggregated data may contain surgical status, device data, and various control or status data associated with multiple surgical devices. The aggregated data may be arranged or formatted in a standard that is easily accessible and processed by the device controllers of the surgical devices via the device network. In this way, the surgical control system may provide multiple surgical devices, including a camera device, to automatically respond to device data, surgical status, and / or aggregated data, or, in some cases, may provide proposed control settings to medical professionals to assist in the operation of the surgical control system.

[0005] In various implementations, the comprehensive operation of the system controller can provide improved operation and control of fluid inflow and / or outflow to the surgical site, as controlled by the surgical pump. During operation, the system controller may be configured to identify a blood indicator indicating the level of blood in the surgical site, and an image data visibility indicator independent of the blood indicator. In response to the blood indicator and visibility indicator, the system controller may be configured to identify control settings for the surgical pump. Furthermore, the control settings or settings for the surgical pump may differ based on the blood level identified by the blood indicator, even in situations where the visibility indicator identifies similar visibility results in the image data. Thus, the blood and visibility indicators may be monitored and applied individually or in combination to adjust the control settings for the surgical pump.

[0006] These and other features, purposes, and advantages of this disclosure will become apparent upon reading the following description with reference to the accompanying drawings. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a diagram of a surgical control system illustrating exemplary operation of a control routine for distributed control. [Figure 2] Figure 2 is a schematic diagram of a surgical control system for multiple surgical devices. [Figure 3] Figure 3 is a simplified block diagram showing the device network for a surgical control system. [Figure 4] Figure 4 is a process diagram illustrating an exemplary control routine for a device network for a surgical control system. [Figure 5A] Figure 5A is a flowchart showing a method for distributed control of multiple surgical devices. [Figure 5B] Figure 5B is a flowchart following Figure 5A, illustrating a distributed control method for multiple surgical devices. [Figure 6]Figure 6 is an illustrative process diagram showing the cleaning process related to the operation of a surgical pump. [Figure 7] Figure 7 is a graphic representation of a visual processing method for a surgical control system. [Figure 8] Figure 8 is a graphic representation of an exemplary visual processing method. [Figure 9A] Figure 9A is a graphic representation of an exemplary visual processing method. [Figure 9B] Figure 9B is a process diagram showing the visual processing method of Figure 9A. [Figure 10] Figure 10 is a flowchart showing a method for operating a visual processing module for a surgical control system. [Figure 11] Figure 11 is a plot illustrating an exemplary process for tracking the operating characteristics associated with surgical pumps. [Figure 12] Figure 12 is a typical diagram of a user interface for a surgical control system. [Figure 13] Figure 13 is an exemplary process diagram illustrating various use cases of a surgical control system in relation to multiple surgical devices. [Figure 14] Figure 14 is a flowchart showing a method for controlling the operation of a surgical device based on the patient's condition or user preferences. [Modes for carrying out the invention]

[0008] In the following description of preferred implementations, references are made to the accompanying drawings illustrating specific possible implementations. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Naturally, other implementations may be utilized and structural and functional modifications may be made without departing from the scope of this disclosure.

[0009] Referring to FIGS. 1 and 2, an exemplary diagram of a control system 10 for a plurality of surgical devices 12 is shown. As shown in FIG. 2, the surgical devices 12 are communicatively coupled via a device network 14. In various implementations, each of the surgical devices 12 may be configured to incorporate a corresponding device controller 16 or to communicate with a corresponding device controller 16. As shown in FIG. 1, each of the device controllers 16 may be configured to be incorporated as an integral component of a corresponding surgical device 12. However, in some implementations, one or more of the surgical devices 12 may be configured to share a common device controller 16. In this configuration, each of the surgical devices 12 may be configured to report and receive device data via the device network 14 in the form of a message or packet indicating one or more control states or detected states associated with the operation of each of the surgical devices 12. Communication of device data between the device controllers 16 of the surgical devices 12 may be configured to provide independent control of the operation of each of the plurality of surgical devices 12 to facilitate the coordinated operation of the control system 10.

[0010] In the exemplary implementation shown, the surgical device 12 may be configured to include a video console 12a, an electrosurgical console 12b, an arthroscopic pump controller 12c, an excision console 12d, an insufflation console 12e, and an additional device 12f. As shown, the device 12f or console may be configured to include various surgical or medical devices, including an anesthesia control device, an infusion pump controller, a patient monitor, a control or interface peripheral device, a tourniquet or restraint control, and / or various medical or surgical devices. Thus, while specific devices are contemplated and demonstrated in the various examples presented in this application, the scope of this application is not intended to be limited to the examples provided.

[0011] In addition to the medical or surgical device 12, various computerized devices including the display 18 and the tablet or computer 20 may be configured to be communicably connected to the device network 14 via a wired or wireless communication interface. As shown in FIG. 1, the display 18 may be connected to the video console 12a and may be configured to present the image data 26 captured by the camera device 22 exemplified as an endoscope or arthroscope. During operation, the system controller 24 (e.g., a camera control unit [CCU]) may be implemented as the device controller 16 connected to the video console 12a. The system controller 24 may function as a common communication hub of the control system 10 and may process and aggregate the communication with each of the surgical devices 12 on the device network 14. As will be discussed later with reference to specific embodiments, the video console 12a may be configured to process the image data 26 from the camera device 22 to identify the surgical state or features of the surgical site via a vision-based specific technology that can correspond to forms of computer vision such as object detection, image classification, sequence classification, convolutional neural network classification, spectrum recognition, etc. The surgical state may be incorporated into the aggregated data from each of the surgical devices 12 and reported by the system controller 24 via the device network 14.

[0012] The surgical status identified by the system controller 24 (e.g., CCU) may include patient status, procedure step, surgical device status, site status, or various features, which can be identified based on image data 26 captured by the camera device 22. For example, if the video console 12a monitors the image data 26 and identifies increased blood presence at the surgical site, it may be configured to report the corresponding surgical status via the device network 14 through the system controller 24. In this configuration, each of the device controllers 16 of the surgical device 12 may adjust their operation in response to increased blood detection or any other device data reported via the device network 14. In a particular example of blood detection, the arthroscopy pump controller 12c may be configured to respond by increasing the fluid circulation rate to and from the surgical site, as illustrated in the display 18, to clear and improve the visibility of the surgical site in the image data 26. Furthermore, if the video console 12a recognizes a clear field of view within the surgical site by monitoring the image data 26, the arthroscopy pump controller 12c may respond by adjusting its operation by continuously decreasing the pump pressure, in which case it may be configured to respond down to a previously determined preset minimum pressure value. In this way, the system 10 can provide each of the surgical devices 12 with a distribution of device data or status information, as well as a visual-based surgical state, so that the control of the system 10 can be coordinated and performed independently by each of the device controllers 16.

[0013] In addition to processing image data 26 to determine the visual-based surgical state, the system controller 24 may be further configured to receive, process, and aggregate device data reported by each of the surgical devices 12. Once the device data is combined, the system controller 24 may be configured to report the combined device data as aggregated data to the device network 14. In some implementations, the aggregated data reported by the system controller 24 may include the surgical state or visually detected features of the surgical site 28 having the aggregated data. If the aggregated data includes the visual-based surgical state, the device controller 16 of the surgical device 12 may be configured to monitor and respond by adjusting one or more settings according to the visual-based state associated with the surgical state reported in the aggregated data. Thus, the device controller 16 of the surgical device 12 may control various settings or prompt the user to approve settings suggested based on the aggregated data.

[0014] Referring further to Figures 1 and 2, in some implementations, the system controller 24 or CCU may access the device data reported by the surgical device 12 to assist in or process factors in determining the vision-based state or surgical state. For example, the system controller 24 may be configured to use the device data to notify or filter potential statuses or procedures identified in the image data 26 using computer vision. During operation, the device data may be processed by the system controller 24 in combination with the image data 26 to classify or filter several potential vision states, or to modify the state based on the device data. In this way, the system 10 can process the device data accessed via the device network 14 to assist in or improve the determination of the surgical state or computer vision evaluation of the image data. As will be further discussed in the detailed embodiments below, the device data may be processed by one or more processors or controllers of the video console 12a to filter several potential surgical states associated with the state of the surgical site 28. In this way, the control system 10 may be configured to provide coordinated operation of each connected surgical device 12 while maintaining independent or distributed control of each device 12, as provided by the device controller 16.

[0015] Referring particularly to Figure 1, the system 10 may be configured to provide customized or user-specific control schemes 30 or preferences that can be detected and defined for multiple users and accessed via corresponding user profiles 32. The control schemes 30 may be configured to be attributed to the user profiles 32 via manual programming or machine learning algorithms, which may be called artificial intelligence. For example, in response to the detection of device data and / or surgical status, the system controller 24 may be configured to identify one or more user-specific control settings or control prompts associated with the active profile 34 of the user profile 32. The system controller 24 may be configured to detect such user-specific control settings or control prompts and store them in a server or memory 50. The control settings of each user profile 32 may be identified in response to inputs to each of the surgical devices 12 via corresponding interfaces, computers or tablets 20, and / or various peripheral devices or inputs to the system 10.

[0016] Throughout the operation, the system controller 24 may be configured to detect control settings as a combination of device data and / or surgical status reported via the device network 14. These control settings may be stored in a control scheme 30 and associated with a user profile 32 so that the system controller 24 can learn user preferences associated with each of the surgical devices 12 over time. In this way, the system 10 can program or learn control settings and provide customized control for each of the surgical devices 12. As discussed herein, the control scheme 30 may be configured to include a variety of control settings, including display settings, device settings, user input or control mapping preferences, and various automatic control prompts or settings that can be initiated in response to device data, surgical status, and / or aggregated data.

[0017] Referring here to Figure 3, a schematic diagram of the device network 14 of the control system 10 shows an example of a distributed control implementation of system 10. As previously mentioned, each surgical device 12 may be configured to communicate with a corresponding device controller 16. During operation, each device controller 16 may be configured to report device data associated with the current operation or control state of the corresponding surgical device 12. For example, an electrosurgical console 12b may report its control state in the form of the current electrosurgical treatment settings and update the settings via one or more messages communicated through the device network 14 throughout the operation of the electrosurgical console 12b. Furthermore, the device controller 16 of the electrosurgical console 12b may be configured to report one or more detected or monitored states related to the operation of the electrosurgical console 12b. For example, in some implementations, an electrosurgical device connected to or controlled by an electrosurgical console 12b may include one or more sensors (e.g., temperature sensors, pressure sensors, etc.) or feedback devices (e.g., current sensors, load sensors, fault sensors, etc.), the status of which may be monitored and reported as messages on the device network 14 by the corresponding device controller 16. Such communication may also be provided by the corresponding device controller 16 of each surgical device 12 that communicates with the device network 14. In this way, the device controller 16 may be configured to distribute device data associated with each operation of the surgical device 12 so that the device data 42 is accessible for independent control of the surgical device 12 by the corresponding device controller 16. As mentioned above, the device data 42 may also be combined with surgical status or image-based status information by the system controller 24 of the video console 12a to form aggregated data 44.

[0018] Referring further to the diagram shown in Figure 3, the device network 14 is shown between a plurality of exemplary surgical devices 12, called device nodes 40. Each of the device nodes 40 is connected via dashed lines representing shared communication of device data 42 and aggregated data 44, which are communicated between a plurality of corresponding surgical devices 12 linked to the device node 40. Thus, the device data 42 reported by each of the device controllers 16 of the surgical devices 12 may be configured to be accessible via the communication lines shown by dashed lines in Figure 3. Furthermore, the aggregated data 44 may be broadcast from the system controller 24 to each of the device controllers 16, as shown by the dashed lines. While the communication of device data 42 is discussed with reference to specific surgical devices 12 and nodes 40, the device data 42 and aggregated data 44 may be broadcast over the device network 14 so that each of the connected devices 12 can access and respond independently according to the received messages and / or packets.

[0019] In addition to general communication between surgical instruments 12 provided by the instrument network 14, the diagram shown in Figure 3 further illustrates multiple arrows representing instrument data 42 and aggregate data 44, which are communicated to all connected instruments but are further processed and implemented by some of the surgical instruments 12 to control or update corresponding operations. Instruments that act in response to instrument data 42 and aggregate data 44 are distinguished from the rest of the surgical instruments 12 in Figure 3 by arrows associated with instrument data 42 and aggregate data 44 communicated to the corresponding or responsive surgical instrument 46. For clarity, the responsive surgical instrument 46 represents a subset of the surgical instruments 12 shown in relation to the instrument network 14, indicating that each of the surgical instruments 12 can be programmed to respond independently to instrument data 42 and aggregate data 44 available to all instruments 12 via the instrument network 14.

[0020] In the example shown, the device node 40 corresponds to a first device node 40a, a second device node 40b, a third device node 40c, and a fourth device node 40d. The system controller 24 of the video console 12a may be configured to respond to communications from the first device node 40a, which may correspond to patient data associated with blood pressure, heart rate, blood / oxygen levels, or various other features that can be monitored via the patient monitor 12g. Furthermore, device data 42 may be reported from the second device node 40b, which may correspond to device settings or ablation intensity of the electrosurgical console 12b. In response to the device data 42 reported by the first node 40a and the second node 40b, the system controller 24 may be configured to generate aggregated data 44 and report it to each of the device nodes 40 and the corresponding surgical devices 12. In addition to the device data 42, the system controller 24 may be configured to further supplement or modify the aggregated data 44 based on the visual-based state or surgical state detected in the image data, as described above. Therefore, the aggregated data 44 may be combined, modified, and distributed by the system controller 24 to each of the device nodes 40 that communicate via the device network 14. Although various embodiments have been discussed with reference to the system controller 24, the aggregated data 44 may also be combined and processed by one or more additional or alternative device controllers 16 generally associated with the surgical apparatus 12.

[0021] Referring further to Figure 3, a portion of the responsive surgical device 46, represented by the second device node 40b and the third device node 40c, may be configured to adjust one or more operating settings in response to device data 42 and / or aggregated data 44. As shown, the electrosurgical console 12b associated with the second node 40b may be configured to respond to aggregated data 44 by adjusting the ablation setting or ablation intensity. As an additional or alternative configuration, as will be discussed later in the embodiment shown in Figure 5B, the electrosurgical console 12b, or another surgical device 12 connected to the system 10, may be configured to identify and communicate the proposed control settings. The proposed control settings may be presented to the user of the system 10 and may be configured to request an update of one or more operations of the surgical device 12 rather than providing automatic control. Thus, the system 10 may be configured to provide semi-automatic control of various settings or control commands for the surgical device 12.

[0022] Referring again to Figure 3, in addition to controlling the electrosurgical console 12b, the arthroscopy pump controller 12c associated with the third node 40c may be configured to respond to aggregate data 44 by adjusting the pressure, flow rate, or circulation velocity of the fluid circulating through the surgical site 28 (e.g., the joint cavity or patient cavity). In addition to the response behavior shown by the devices associated with each of the second node 40b and the third node 40c, the patient monitor connected to the first node 40a, for example, the resection console 12d connected to the fourth device node 40d, may be configured not to respond to or propose a control response to the device data 42 or aggregate data 44 reported via the device network 14. Thus, the control system 10 may be configured to provide reporting of device data 42 among each of the surgical devices 12 so that the device controllers 16 of the surgical devices 12 process the device data 42 and aggregate data 44 and respond independently to reported device and / or visually-based surgical condition conditions accessed via the device network 14.

[0023] Referring further to Figure 3, a schematic diagram of the video console 12a shows the connection to the camera device 22 and the implementation of one or more image processors 48 that may be implemented to facilitate the computer vision technology discussed herein. The image processor may include one or more digital processing units, such as a central processing unit (CPU) having one or more processing cores, a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some configurations, the multiple processing units are combined into a system-on-a-chip (SoC) configuration, while in other configurations, the multiple processing units may correspond to individual components. During operation, one or more processors 48 may execute program instructions stored in memory 50 to perform the operations described herein. As shown in the figure, memory 50 is referred to as multiple modules representing corresponding control routines stored therein.

[0024] During operation, one or more processors 48 may be configured to review or process the image data 26 according to routines established by one or more computer vision routines, exemplified as detection module 50a and classification module 50b. Thus, one or more computer vision routines may be implemented by processor 48 according to various image processing techniques after acquisition of the image data 26, which may include preprocessing, filtering, segmentation, and feature extraction, as well as various additional processes that may be commanded by detection module 50a. In some embodiments, features identified by detection module 50a may be interpreted by classification module 50b as identifiable features or characteristics of the image data 26 to generate or output associated classifications or descriptions referred to as vision-based states reported in this disclosure. In some cases, detection module 50a may be omitted, and vision-based states may be identified in response to one or more features of the image data 26 that can be inferred based on the operation of classification module 50b or similar vision-based detection or classification routines, as discussed herein. Vision-based states may include, as described above, surgical states, surgical stages or steps, or features of surgical sites 28. In this way, the system controller 24 can provide the application of computer vision technology to modify the aggregated data 44 reported to the surgical apparatus 12 and improve the coordinated operation of the control system 10, which responds not only to apparatus data 42 but also to current or expected conditions at the surgical site 28, such as potential bleeding events.

[0025] Referring here to Figure 4, the process diagram illustrates an exemplary communication architecture and operating procedure of the control system 10. As shown, the detection module 50a and the classification module 50b may generally be referred to as the vision module 60. In the illustrated embodiment, the visual states identified by the vision module 60 may be configured to serve as inputs to a state module 62 or state machine configured to determine and track various combined states of the surgical apparatus 12 based on communications received from each of the corresponding apparatus controllers 16. Throughout the operation, each of the apparatus controllers 16 may be configured to report individual apparatus data 42, including operating states, settings, and / or measured operating values, via the apparatus network 14. The apparatus data 42 may then be accessible via the apparatus network 14 by each of the apparatus controllers 16, as well as by the system controller 24 or camera control unit (CCU). In this configuration, the control system 10 may be configured to provide distributed control of each of the surgical apparatuses 12 communicating with the apparatus network 14.

[0026] In response to receiving device data 42, the system controller 24 may be configured to generate aggregate data 44 by processing and packaging combined state information, including settings, operating states, and visual states, as aggregate data 44 reported via the device network 14. In addition to device data 42, the system controller 24 may be configured to additionally include a procedure tracking module 64 that can track and communicate the current step of a predetermined set of steps associated with a particular surgical procedure to a state module 62. As will be provided in more detail in the various embodiments of the following description, the visual state, as well as one or more combinations of each device state and / or setting of the surgical apparatus 12, may be configured to allow the state module 62 to automatically invoke various inferences regarding proposed operating settings or to propose for invoking one or more of the surgical apparatus 12 of the control system 10. Such inferences and / or contextual information are again reported to the device network 14 as aggregate data 44, providing meaningful input to each of the surgical apparatus 12 and improving the operation of the system 10 by enabling intuitively operating suggestions and / or automatic control.

[0027] Figures 5A and 5B illustrate an exemplary control routine or method 70 that provides independent control of each surgical device 12 across the device network 14. As illustrated, method 70 may be referred to as a distributed control routine based on the independent nature of the control of each surgical device 12 via the device controller 16. As previously mentioned, aggregated data 44 may be used alone or in combination with device data 42. In the particular example illustrated, method steps associated with the surgical device 12 are shown in the left column 70a, and steps associated with the system controller 24 are shown in the right column 70b. However, it should be understood that the steps associated with the surgical device 12 in the first column 70a may be similarly implemented by the system controller 24. Thus, method 70 may be adapted to fit almost any number of surgical devices and may be implemented with or without the video console 12a if the tasks associated with the system controller 24 do not include image acquisition operations.

[0028] Referring to Figure 5A, Method 70 may generally be configured to be initiated in response to the activation of one or more surgical devices 12 of the control system 10 (72). After activation, the surgical devices 12 may be configured to be manually or automatically controlled by one or more users to provide conventional applications (i.e., fluid injection, delivery of electrosurgical energy, measurement of patient vital signs or sensor data, etc.) (74). In some cases, the operation of the surgical devices 12 may be configured to be controlled by one or more external devices, such as a computer or tablet 20, via communication over the device network 14. However, generally, the surgical devices 12 may be configured to respond to input to each of their dedicated or connected user interfaces, which may include one or more foot pedals, switches, remote controls, touchscreens, etc. Thus, Method 70 may provide a combination of manual device operation and recommended or automatic control updates, depending on the specific application.

[0029] After step 74, each associated device controller 16 may be configured to report device data 42 to each of the surgical devices 12 connected via the device network 14, based on the operation of each of the surgical devices 12 (76). After, or simultaneously with, the reporting of the device data 42, the device controller 16 may be configured to further monitor the device data 42 and / or aggregated data 44 reported by other surgical devices 12 communicated via the device network 14 (78). Following step 78, each device controller 16 of the surgical devices 12 may be configured to continue subroutine A as shown in Figure 5B. Before moving on to a description of the exemplary operation of the system controller 24 or camera control unit, it is important to note that the timing of reporting and monitoring of device data 42 associated with each of the surgical devices 12 may vary. More clearly, the frequency and associated timing of each instance of reporting and monitoring, as described later with reference to steps 76, 78, and step 88, may be configured differently for each device. Such operations can help facilitate the independent control of each surgical device 12 without requiring time-coordinated control of the surgical device 12 using a central controller.

[0030] Simultaneously with steps 74-78 discussed with reference to the surgical device 12, steps 80-88 may be configured to be processed by the system controller 24. During operation, the method 70 associated with the video console 12a may be configured to be initiated in response to the acquisition and / or processing of image data 26 (80). Generally, the system controller 24 (e.g., CCU) may be configured to process the image data to identify surgical conditions or various features associated with the surgical site 28 (84). Throughout the acquisition of image data and the identification of the visual-based conditions of the surgical site 28, the system controller 24 may be configured to further access the device data 42 reported by the surgical device 12 and generate aggregated data 44 containing the device data 42 and one or more visual-based conditions via the device network 14 (86). Once the aggregated data 44 is generated, the data 44 may be configured to be reported or broadcast via the device network 14 (88). In some cases, the characteristics or visual condition of the surgical site 28 may be additionally reported to the aggregated data 44 if such information is available and processed by the CCU or system controller. Following step 88, the system controller 24 may also be configured to continue with subroutine A, as shown in Figure 5B.

[0031] Referring here to Figure 5B, a method step associated with any of the surgical devices 12, including the video console 12a and the associated system controller 24, proceeds through subroutine A and returns to the method shown in Figure 5A via reference path B. Controllers 16 and 24 of the surgical devices 12 may be configured to receive or selectively access device data 42 and / or aggregate data 44, respectively, in step 96. Based on the device data 42 and / or aggregate data 44, each of the controllers 16 and 24 may be configured to independently identify a proposed or automated operating state based on the corresponding programming associated with each of the surgical devices 12 (98). As previously stated, the proposed or automated operating state of the surgical device 12 may be identified based on the active profile 34 of the user profile 32. Therefore, the proposed state described in step 100 may be configured to be identified in response to previously identified user preferences related to a combination of device data 42, and / or surgical states communicated via the device network 14 by the surgical device 12, including the video console 12a.

[0032] Based on the active profile 34 and the corresponding programming or control instructions, each of the controllers 16, 24 may identify a proposed operating state and determine in step 100 whether the proposed operating state associated with the device data 42 and / or aggregated data 44 is an automatically controlled state, a pre-approved state, or a manual state. If the proposed state is automatically accepted and updated by the controllers 16, 24 of the surgical device 12, the proposed operating state may be configured to be applied in step 102. If the proposed state requires confirmation, each of the corresponding controllers 16, 24 associated with the surgical device 12 may be configured to respond according to their associated programming or control instructions and proceed to step 104 to request manual approval or approval. If manual approval is required, method 70 may proceed to step 104 to communicate the proposed operating state to a user interface (e.g., a display 18, a computer, or a tablet 20) and present a prompt requesting permission for the proposed state. If the proposed state is approved in step 106, method 70 may proceed to step 108, in which case the approval of the proposed state may be communicated via the device network 14, which allows the corresponding controllers 16, 24 to access the approval as a message. Once the approval is communicated via the device network 14 via a message, the relevant controllers 16, 24 may be configured to apply the operating state proposed in step 102, as described above. Following step 102, method 70 may be configured to continue operating according to the reference path B, in accordance with steps 74-78 and 80-88, as described above.

[0033] In addition to communicating the approval of the proposed operating state in step 108 to the relevant controllers 16, 24, method 70 may also be configured to communicate the updates to the control scheme 30 of the active profile 34 described above with reference to Figure 1 (110). Over time, such updates to the control scheme 30 may be implemented by the system 10 to provide a customized user experience associated with each of the user profiles 32. As a result, the control settings and configurations for each of the surgical devices 12 may be proposed by the system 10 and used to input the corresponding settings associated with the user profile 32. Furthermore, the system may be configured to not only automatically detect states related to device data and aggregate data, but also to provide advanced learning behavior that, as described in step 106, further generates custom programming of settings and the operation of the surgical devices 12 in response to the control settings recorded for each user. In this way, the system 10 may be configured to automatically update the control scheme 30 associated with the operation of the surgical devices 12 for each of the user profiles 32.

[0034] Referring here to Figure 6, an exemplary operation of the pump controller 12c of the surgical pump 120 is described with reference to a bleeding event detected by the visual module 60 of the system controller 24. The bleeding event may be the result of the operation of an excision tool 122 (e.g., a shaver, rasp, drill, burr, etc.) applied to a surgical site 28 representing a cavity in the patient. In the illustrated embodiment, the pump 120 may be configured to control the inflow of surgical fluid by monitoring the supply pressure or local pressure at the surgical site 28. For example, in some implementations, the surgical pump 120 may include an inflow pump, which may be configured to operate in combination with an outflow pump to control the flow rate and / or pressure. The inflow may be configured to be supplied via a cannula of a camera device 22, and the fluid outflow may be configured to be collected from the surgical site 28 via the lumen of the cutting or excision tool 122 and the outflow port 124 of the cannula 126. If only an inflow pump is included in the surgical pump 120, the fluid exchange or flow through the surgical site 28 or cavity may be a result of the pressure difference between the surgical site 28 and the outflow port 124. In such cases, the flow rate may be indirectly controlled. In other cases, the surgical pump 120 may be configured to include both inflow and outflow control pumps or devices. Thus, the pump controller 12c may be configured to control the inflow and / or outflow to and from the surgical site 28 to maintain hemostasis of the patient cavity, ensuring that the corresponding anatomical structure is effectively displayed in the field of view 130 of the camera device 22.

[0035] Throughout a surgical procedure similar to that exemplified, various actions may result in reduced visibility of the field of view 130, as represented by exemplary images 130a–130d. For example, the operation of a cutting tool 122 or excision device may introduce particulate matter (e.g., tissue, bone fragments, etc.) into the patient's cavity. Furthermore, the removal of tissue, perforation, incision, or various surgical tasks may result in bleeding or massive bleeding, affecting visibility. As shown in the first image 130a, the surgical site 28 is represented in a low-visibility state that may correspond to a bleeding event resulting from an excision operation. As illustrated, the first image 130a may also include reduced visibility resulting from particulate matter and the introduction of blood into the surgical fluid within the patient's cavity. In response to the detection of such a low-visibility state by module 60, the system controller 24 may be configured to activate a washing or rinsing technique to adjust the pressure and / or fluid exchange rate to remove particulate matter, blood, and / or tissue from the patient's cavity.

[0036] In the illustrated embodiment, the visual module 60 determines the low visibility state using the visibility threshold T of the visibility index. vThe system may be configured to identify a blood-positive detection that has visibility below a certain level and affects the clarity of the image data 26. In such embodiments, the visual module 60 may be configured to process the image data 26 to identify that bleeding is likely occurring at the surgical site 28. In response to the detection of a potential bleeding event, the system controller 24 may be configured to prompt the user of the system 10 to activate or to activate a lavage routine. As will be discussed in more detail later, the prompt may be configured to receive a confirmation input to accept the proposed activation, with a conditional mapping of one or more user inputs on the user interface of the camera device 22 or various connected devices 12. In either case, in response to an automatic or user-activated lavage routine, the pump controller 12c may be configured to activate an increase in the surgical fluid supplied to the inflow via the pump 120 to increase the pressure and apply a tampon insertion technique to the bleeding at the surgical site 28. Furthermore, in some implementations, the outflow transmitted through the outflow port 124 of the cannula 126 may be configured to control the inflow pressure and simultaneously wash away blood and / or particulate matter from the surgical site 28. As shown by the sequence of images 130a-130d, the washing routine may be configured to control bleeding by increasing the fluid pressure while rinsing the patient's cavity. In this way, the system 10 can ensure that the field of view 130 of the camera device 22 is free of blood and debris, allowing the surgical procedure to be completed effectively.

[0037] As described herein, the image data 26 may be configured to correspond to still images, a series of image frames in an image feed, and / or any form of video or visual representation of the field of view 130 of the camera device 22. For example, the image data may be configured to correspond to a video feed captured at a frame rate that shows changes in the field of view 130 over time. Alternatively, the image data may be configured to correspond to images sampled periodically, or images sampled in response to one or more events or actions associated with the system 10 that can be communicated via the device network 14. Thus, the system 10 and the corresponding methods can be implemented in various ways to suit the desired application.

[0038] In addition to prompting or automatically initiating a washing procedure, the visual module 60 of the system controller 24 may be configured to additionally monitor the effectiveness of the washing or rinsing procedure by monitoring the temporal changes in the corresponding image data 26 represented by images 130a to 130d. In response to improved visibility, the visual module 60 may be configured to update the visual state to indicate that the bleeding event is under control. During operation, the visual module 60 may be configured to identify the cessation of the bleeding event as a result of improved visibility of the image data, as shown in the fourth image 130d, or as a result of a corresponding improvement in visibility, as shown in the third image 130c. As a result of the improved visibility detected by the visual module 60, the pump controller 12c may be notified of the updated operating state of the control system 10 and prompted to stop the washing routine and return to the baseline pressure. In this way, the control system 10 may be configured to provide improved patient outcomes, actively evaluate the quality and / or characteristics of image data 26 acquired by the camera device 22, and limit the adverse effects of extravascular migration by limiting the duration of increased pressure applied to the surgical site 28.

[0039] Referring here to Figure 7, image data 26 captured by the camera device 22 is shown in various states of visibility or clarity to demonstrate exemplary visual processing methods that may be applied by the visual module 60. Starting with a representative clear image 132, the visual module 60 may be configured to process the image data 26 in multiple segments 134 or sectors that can be processed independently to distinguish the relative locations of corresponding visibility states within the image data 26. In some implementations, one or more of the segments 134 (e.g., a central segment or manually identified segments) may be configured to be prioritized as particularly important to the overall visibility and operation of the camera device 22. Based on the prioritization of segments 134, the visual module 60 may be configured to increase the monitoring frequency of one or more preferred segments 134 and / or to compare the corresponding visibility of the preferred segments with a visibility threshold that requires a higher relative level of visibility compared to the unpreferred and / or centrally located segments 134. In this way, the visual module 60 may provide customized monitoring settings to suit various applications or preferences.

[0040] As further shown in Figure 7, the visibility associated with the image data 26 may be configured to be classified based on a variety of visually identifiable or optically detectable characteristics, which can be classified based on one or more indices or classifications indicating a change from a high level of clarity to a low level of clarity. Exemplary images of the clarity range associated with the operation of the visual module 60 are shown as images 138a and 138b corresponding to the content shown in the image data 26. In an exemplary embodiment, the visual module 60 may be configured to classify the image data independently based on the blood classification of the blood classification unit 140 and the visibility index 142. As shown in the first set of images 138a, the images show a decrease in the level of visibility from top to bottom, representing a change from a high level of clarity to a low level of clarity as determined by the visibility index 142. A predetermined or user-specific visibility threshold T vDepending on the circumstances, the system 10 may prompt or otherwise control the operation of the pump controller 12c to improve visibility at the surgical site 28.

[0041] The visibility index 142 may be configured to provide the system controller 24 and pump controller 12c with general insight into how to control inflow and, in some cases, outflow of the surgical pump 120, and the blood classification unit 140 may be configured to identify specific conditions at the surgical site 28 and activate or suggest different control strategies for the pump based on the presence of blood. For example, low visibility associated with the visibility index 142 may be communicated via the device network 14 to notify the pump controller 12c to increase fluid exchange or rinsing at the surgical site 28. Furthermore, a blood-positive classification from the blood classification unit 140 may indicate that a decrease in visibility of the visibility index 142 is associated with bleeding at the surgical site 28. This condition information informs the pump controller 12c that increased fluid exchange should also be accompanied by an increase in fluid pressure provided through the inflow in order to effectively perform a tampon insertion procedure on the bleeding and to clarify the image data 26. In this way, the blood classification unit 140 and the visibility index 142 may be configured to be independently monitored by the visual module 60 so that the surgical pump can change control settings associated with different events detected in the image data 26.

[0042] During operation, the blood classification unit 140 of the visual module 60 may be configured to detect or classify image data as blood-positive or blood-negative based on the color content or color response associated with the illumination of the surgical site 28 across a range of light wavelengths. The visibility index 142 may be configured to be measured by the visual module 60 as a relative level of sharpness or contrast associated with the image data 26. In exemplary embodiments, the visibility index 142 may be configured to be identified based on a visibility regression model compared to ground truth data based on a measure of image quality. For example, a neural image evaluation (NIMA) model may be used to identify a visibility index 142 that is trained to prioritize the visibility of the central segment 134 on those segments 134 extending around the periphery of the field of view 130. In this way, the visual module 60 may be configured to process the image data 26 of each segment 134 to determine a relative level of clarity quantified by the distribution of the visibility index 142. The data used to train the NIMA model to measure visibility index 142 may include, but not be limited to, sharpness, local standard deviation, local entropy, mean standard contrast normalization (MSCN), blind / reference image spatial quality rating (BRISQUE), natural image quality rating (NIQE), manually annotated scores, and / or various similar image processing methods, and may include one or more combinations of image quality or color content indices. The NIMA model is included as an example, but it will be understood that other models or regression models may be similarly applicable.

[0043] In addition to the blood classification unit 140 and the visibility index 142, the visual module 60 may be configured to further process the image data 26 to identify various additional identifiable characteristics. For example, in some implementations, a particulate or particulate content index 144 may be identified within the image data 26 to provide further insights for triggering or prompting one or more control settings of the surgical apparatus 12. For clarity, the visibility index 142 may be configured to quantify various aspects related to visibility in the image data 26, which may include characteristics also detected by the particulate content index 144. However, in some cases, it may be beneficial to track the presence of debris or discrete particle content that may alternatively affect visibility in relation to particle movement and / or intermittent obstructions, which would otherwise not be detected within segment 134 and could not be associated with the visibility index 142. As shown in the second exemplary image 138b, the increase in the detected particulate content level 168 is shown independently of underlying turbidity that can consistently cloud areas corresponding to one or more of the segments 134. During operation, the particle content index 144 may be configured to be identified by the visual module 60 by detecting the movement of one or more discrete objects identified in the image data 26. Identification of discrete particles or moving particles associated with the particle content index 144 may be configured to be detected based on a variety of procedures by the visual module 60, including but not limited to background subtraction, frame difference, time difference, optical flow, and various similar image processing techniques. In this way, the visual module 60 may be configured to provide additional information to the state module 62 so that the state associated with the image data 26 of the surgical site 28 can be reported to the surgical apparatus 12. In particular, the indication of an increase in particle content may be configured to trigger the pump controller 12c to initiate or suggest the initiation of inflow and / or outflow fluid adjustments (e.g., increasing inflow pressure, increasing fluid exchange rate, adjusting suction, etc.).As shown below in Figure 10, in addition to the visibility index 142 and the particle content index 144, additional image processing techniques may be used to identify various aspects related to the operation of the camera device 22, such as, for example, but not limited to, identifying the focus of the camera device 22 or identifying the location of the camera device 22 (e.g., deployed inside or outside a patient's cavity).

[0044] Referring here to Figure 8, an example of a visual processing method controlled by the visual module 60 is illustrated with reference to an exemplary image 150 of a surgical site 28. In the illustrated embodiment, the image data 26 includes a blood-positive identifier 152 identified by the blood classification unit 140 of the visual module 60. The blood-positive identifier 152 may be configured to be identified by the visual module 60 in response to the red content of one or more image data 26 segments 134 that exceed the minimum blood classification threshold 154, as previously described. In some cases, features of the image data 26, including red content, uniform swirls, or other features, may be mixed or accumulated beyond the maximum blood classification threshold 156. Image data 26 exceeding the maximum blood classification threshold 156 may contain excessive color content or hue without sufficient brightness to easily classify the conditions related to the field of view 130 as corresponding to a blood-positive condition or identifier 152 or other conditions related to the operation of the camera device 22. For example, an unknown blood classification identified by the blood classification unit 140 that exceeds the maximum blood classification threshold 156 may correspond to an out-of-focus image, an occluded or obscured image sensor, or other visually indeterminate features presented in the image data 26. Detection of such unknown conditions associated with the blood classification unit 140 can be identified by the system controller 24, which may output one or more commands and / or prompts to the user of the control system 10 to indicate that the camera device 22 may be out of focus or otherwise blocked. Therefore, even conditions exceeding the maximum blood classification threshold 156 may be identified by the visual module 60, which may identify and update the identification of the surgical site 28's condition reported by the condition module 62. Furthermore, while the operation of the blood classification unit 140 is primarily described with reference to the color content or hue of the image data 26, it will be understood that due to limitations related to patent disclosure, colors cannot be readily presented. Therefore, the presence of color must be described, not visually indicated.

[0045] When the visual module 60 determines the blood classification as a blood positive identification 152, the detected visibility 158 of the exemplary image 150 may be configured to be further determined based on the visibility index 142. In some cases, the detected visibility 158 may be pre-configured or associated with a visibility threshold T, which may be associated with user preferences. v The configuration may be compared to the above. For such user preferences, as mentioned above, the configuration may include loading an active profile 34 corresponding to the user profile 32. The detected visibility 158 is the visibility threshold T v In response to a value smaller than , the visual module 60 may be configured to communicate the state of the surgical site 28 to a state module 62 indicating limited visibility and blood-positive identification 152. The visual state may then be output from the state module 62 and reported to aggregated data 44 via the device network 14 so that each of the surgical devices 12 can access the reported state and the corresponding device controller 16 can act in response to the aggregated data.

[0046] As mentioned above, the visibility threshold T in combination with blood positive identification 152 v In response to the identification of a detected visibility 158 below a certain threshold, the pump controller 12c may be configured to prompt the user of the system 10 to manually activate the lavage setting of the surgical pump 120, or otherwise to activate it automatically. The lavage setting or routine may be configured to increase the flow into the patient cavity and, as a result, increase the pressure at the surgical site 28 to slow and control any associated bleeding events. In this way, the system controller 24 can provide prompts or automatic operation of the surgical pump 120 to ensure that the image data 26 remains clear throughout the various procedures.

[0047] As described in various embodiments, the activation of one or more proposed settings for controlling the operation of the surgical pump 120 via the pump controller 12c may be configured to require manual activation by the user of the system 10. In various implementations, the activation or confirmation of one or more proposed control states of the surgical device 12 may be presented on the display 18 to prompt the user for confirmation. Furthermore, one or more of the device controllers 16 or system controllers 24 may be configured to selectively map inputs to one of the corresponding user interfaces of the surgical device 12 to receive prompts or confirmations of proposed control settings. For example, in response to proposed lavage settings identified by the pump controller 12c and / or system controller 24, as reported via the device network 14, the system controller 24 may be configured to selectively map user inputs of the camera device 22 to temporarily receive confirmations of proposed control settings for the surgical pump 120. A practical example of such operation may include a prompt on the display 18 requesting the user of the system 10 to "press camera input 1 to activate lavage." Next, the user may engage with the corresponding camera input of the camera device 22 to initiate the proposed or prompted cleaning process. In this way, the control system 10 may be configured to provide coordinated operation of the surgical device 12 by communicating the corresponding states and proposed operating settings via the device network 14, prompting the user of the system 10 to confirm and initiate various settings or operations.

[0048] While the operation of the control system 10 has been described as semi-automated or a proposed configuration, the operation of the pump controller 12c and the system controller 24 may also be configured with one or more setting preferences associated with the user profile 32 of the active profile 34. As previously mentioned, the user's user profile 32 may be configured to be loaded as an active profile 34 for each procedure associated with the operation of the control system 10. The user profile 32 may be configured to include various preferences or operating characteristics that are set by or trained based on past operations associated with each corresponding one. For example, the user profile 32 may include, as will be described later with reference to Figure 9, the initiation of hemostasis, washing or rinsing routines, as well as the minimum blood classification threshold 154, visibility threshold T v and / or particle threshold T p To maintain the preferences associated with the surgical pump, the system may be configured to include operating settings for the pump controller 12c for controlling the surgical pump by referencing conventional operation. The control system 10 may be configured to prompt each user for one or more preferences and / or learn preferences associated with the operation of one or more surgical devices 12 from past use. Thus, the system 10 may provide customized operation associated with each user profile 32 so that the proposed operation of the surgical device is tailored to the individual needs of a particular user.

[0049] The operation of the surgical pump 120 may vary considerably depending on the type of procedure and user preference, but may be approximately 50 mmHg, for example, as an example of a conventional operating pressure required to maintain hemostasis of the patient's cavity at the surgical site 28. Furthermore, the pressure level increase associated with the proposed irrigation settings may be configured to correspond to a pressure increase in the range of approximately 10% to 50% of the typical operating pressure or baseline pressure. For example, the irrigation routine may be handled by the pump controller 12c to control the surgical pump 120 to increase the inflow pressure from 50 mmHg to 70 mmHg. As previously mentioned with reference to Figure 6, the improved clarity of the image data 26 may be monitored by the visual module 60 to determine whether the blood classification unit 140, visibility index 142, and / or particle content index 144 have improved to below the corresponding thresholds, each of which may be customized or adjusted by the user and stored in the user profile 32. In response to the improved clarity detected in the image data 26, the visual module 60 may be configured to update the visual state and communicate the updated state information to the state module 62. The updated fluid state at the surgical site 28 may be further reported via the device network 14. In response to the updated state associated with clarity at the surgical site 28, the pump controller 12c may be configured to reduce the pressure of the surgical pump 120 and return its operation to a baseline of approximately 50 mmHg.

[0050] The typical fluid flow rate of the inflow supplied by the surgical pump 120 may be approximately 50–600 ml / min, and may generally vary between 100–400 ml / min. In response to the inflow and outflow of fluid, the associated pressure measured at or estimated relative to the surgical site 28 may vary from approximately 30 mmHg and 80 mmHg throughout the operation of the pump 120. The baseline pressure associated with each surgery or surgical procedure may be stored in each user profile 32 of the associated user (e.g., surgeon, physician, etc.). In this way, the system 10 may be configured to identify the different operating characteristics of the pump 120, as well as the remaining device controllers of the surgical apparatus 12 of the system 10, for each user profile 32. Similarly, the operating pressure and characteristics of System 10 may be updated and stored in the active profile 34 of the user profile 32 according to each procedure, and may be configured to adjust and / or update the operating characteristics of System 10 (e.g., flow rate, baseline pressure, etc.) based on the proposed state 210 and the corresponding confirmation input 212, as described herein. In this way, System 10 may actively store and update operating preferences, including flow rate, baseline pressure, blood content level, and / or visibility thresholds, associated with the data of the active profile 34 for each operation or procedure, in the corresponding user profile 32. As previously stated, the user profile 32 may be configured to be stored in one or more servers 50 or databases for access and use in subsequent operations of System 10 or systems compatible with the system discussed herein.

[0051] While specifically discussed with reference to the camera device 22 and the cutting or excision tool 122, the conditional mapping of the corresponding user interface inputs of the surgical device 12 may be configured to selectively map to various inputs in order to verify or activate the proposed or semi-automatic control settings of the surgical pump 120. For example, the control system may be configured to selectively map various controls or inputs of surgical devices 12 or peripheral devices that communicate with the control system 10 via the device network 14. In some implementations, the system controller 24 may be configured to selectively map user inputs necessary to verify the proposed or semi-automated control settings to one or more surgical devices 12 that are in active operation. For example, throughout the operation of the control system 10, the system controller 24 may be configured to monitor the device status associated with device data 42 reported via the device network 14 to determine which of the surgical devices 12 are in active use or have been recently activated. If the surgical device 12 is used within a predetermined time (e.g., 10 seconds, 30 seconds, 60 seconds, etc.), the system controller 24 or the corresponding device controller 16 may be configured to conditionally map one or more inputs of the device 12 (e.g., resection wand, resection tool, camera device 22, etc.) to provide confirmation of proposed or semi-automatic control of the surgical pump 120. Accordingly, the control system 10 may be configured to provide flexible operation that can be coordinated among various surgical devices 12 to improve accessibility to one or more operations identified by the visual module 60 and the state module 62, as described in this disclosure.

[0052] The operation of the visual module 60 will now be described with reference to Figures 9A and 9B, with reference to an exemplary image 160 showing a surgical site 28. In some implementations, the visual module 60 may be configured to process individual frames of image data 26 over time and classify the frames based on identified image content into a first class 162 and a second class 164. Generally, this process can be achieved through a machine learning process that implements a multitask classifier. During operation, the multitask classifier may be configured to implement a class-launched mapping algorithm to classify the content of image frames into one or more containing classes. Classes 162 and 164 may include a first containing class 162 that identifies the size and distribution of image data 26 within the blood-containing classification 162a, and a second image class that represents the blood-absent classification 162b. After calculation, the configuration may specify a blood content ratio 168 for the first containing class 162 as identifying the relationship between the blood-containing classification 162a and the blood-absent classification 162b. By tracking the presence or absence of blood in the image data, the first content class 162 can improve the accuracy and reliability of blood content associated with each frame of the image data 26.

[0053] In addition to the first class 162, frames of image data 26 may be classified into a second content class 164, which includes valid class 164a and unknown class 164b. The second class 164 may be configured to indicate whether the content of the image data 26 can be classified by procedure 166 shown in Figure 9B, or whether it is valid for classification by procedure 166. The second class 164 or validity classification of the image data may be useful in providing a positive determination or classification of image frames that may not be reliable in providing automatic control via procedure 166. Therefore, a multitask classifier applied by the classification procedure may be implemented to reliably identify the blood content in a video stream captured by the camera device 22 and to provide automatic or computer-assisted control of the system 10. As shown in the following embodiments, the multitask classifier may also be implemented via class-launch mapping, which can be achieved by applying a trained image classification model. In the specific example considered, a gradient-weighted class-launch mapping procedure (e.g., GradCAM) is applied to generate classes 162, 164.

[0054] The operation of the visual module 60 used to identify classes 162 and 164 will be described, primarily with reference to Figure 9A, with reference to the range of the exemplary scope image 170. The scope image 170 shows the range of blood content, from a clear image 170a without blood at the top of the spectrum to a blood-saturated image 170b at the bottom of the spectrum. Although not clearly visible due to the monochromatic color scale, each image between the clear image 170a and the blood-saturated image 170b has an increasing level of blood content and a corresponding increase in red coloration. The red coloration is generally indicated by an increase in cloudiness as it moves from the top to the bottom of the spectrum. The class launch maps 172 resulting from each of the exemplary scope images 170 will be described with reference to the image classification procedure of the following embodiment.

[0055] During operation, the visual module 60 may be configured to apply a classification procedure 166 to generate a class launch map 172 that identifies blood-present classification 162a and blood-absent classification 162b for each processed image frame. As shown in Figure 9A, each of the corresponding exemplary class launch maps 172 is shown for blood-present classification 162a and blood-absent classification 162b aligned with the corresponding scope image 170. As illustrated, a scope image 170 with a low blood content, close to a clear image 170a, has a high class content for blood-absent classification 162b. In contrast, a scope image 170 with a high blood content, close to a blood-saturated image 170b, has a higher content in the class launch map 172 for blood-present classification 162a. Based on the contrast between blood-absent classification 162b and blood-present classification 162a, the blood abundance ratio 168 can clearly distinguish between content in image data corresponding to blood and content that does not contain blood. In response to the ratio 168 of the class activation map 172 for blood classifications 162a and 162b, each of the device controllers 16 may be configured to respond to one or more recommended or automatic operating states.

[0056] Referring particularly to the operation of the arthroscopy pump controller 12c, the identification of the blood abundance ratio 168 may be configured to control the pressure of the pump 12c over a user-defined pressure range. Further details regarding user-defined or patient-based control of the surgical pump 120 are described later with reference to Figure 14. Further referring to Figure 9A, the operation of the pump controller 12c is described in response to the blood content shown in exemplary image 160. In the illustrated embodiment, the blood content associated with exemplary image 160 may be configured to correspond to identified blood content classifications 174 and identified blood absence classifications 176. In the illustrated embodiment, a representative class activation map 172 is identified for reference. As illustrated, the identified classifications 174, 176 may be configured to correspond to intermediate levels of blood content between clear image 170a and blood-saturated image 170b. Based on the identified blood classifications 174, 176, the classification procedure 166 may be configured to identify a calculated blood abundance ratio 168. As illustrated, the calculated blood abundance ratio 168 may be configured to correspond to a moderate level of blood content similarly identified in classifications 174 and 176. In response to the calculated blood abundance ratio 168, the pump controller 12c may be configured to automatically suggest a corresponding pressure level within a user-specified range of pressure limits for a particular procedure or operation. Thus, the classification procedure 166 may be configured to detect the blood abundance ratio 168 in order to accurately detect the blood content in the patient's cavity and to responsively control the pump pressure to correct or control the presence of blood in the image data 26.

[0057] Referring further to Figure 9A, the classification procedure 166 may be configured to further determine whether each frame of the image data 26 is valid 164a or unknown 164b through the identification of a second class 164. During operation, an indication that the image data is valid 164a may be reported via the device network to notify the device controller 16 of the surgical apparatus 12 that the image data 26 is executable to support automatic or semi-automatic operation. Alternatively, if the image data 26 is classified as unknown 164b for one or more image frames in a series, the resulting device data 42 or aggregated data 44 may be configured to notify the device controller 16 of the surgical apparatus 12 to pause or cancel semi-automatic or visually assisted operation. In this way, the classification procedure 166 may be configured to ensure that precise control is maintained for the various situations presented in the image data 26.

[0058] Referring here to Figure 9B, a process diagram illustrating the classification procedure 166 in more detail is shown. During operation, the classification procedure 166 may be configured to receive image data 26 as a video feed containing multiple image frames 166a. Each of the image frames 166a may be provided to a neural network or trained model containing multiple layers, indicated as CONV blocks and ID blocks. Each of the layers 166b may be configured to correspond to detection and processing techniques that can highlight different objects and / or features within the image frames 166a. After processing via the convolutional layers 166b, the heatmap of the image data 26 may be combined and / or averaged via the layer combination module 166c. Following these combinations, the class launch map 172 from the image frames 166a may be processed as the outputs 166d of a first class 162 and a second class 164. The blood abundance ratio 168 may be calculated using the first class 162 or blood / no blood classification 162a, 162b. Furthermore, the determination of valid / unknown classifications 164a, 164b may be configured to output as a compatibility indicator 166e indicating whether the image data 26 is valid to support the automatic or semi-automatic operation described herein. Based on this process, the classification procedure 166 may be implemented to provide a status indicator that identifies the blood content to each of the surgical devices 12 on the device network 14.

[0059] Referring to Figure 10, a flowchart is shown illustrating a method 180 for monitoring image data and controlling the operation of the surgical pump 120. Method 180 may be configured to start in step 182 in response to the startup of the system 10. Once started, the system controller 24 may be configured to start the camera device 22 to capture and process image data 26 (184). Furthermore, the system controller 24 may be configured to receive and communicate various device settings and control states in device data 42 and aggregated data 44 via the device network 14 (186). As previously stated, the visual module 60 may be configured to include a scope position detection routine that can determine whether the camera device 22 is inserted into the surgical site 28 (188). For example, in step 188, the visual module 60 may be configured to process the image data 26 to determine whether the features and illumination associated with the image data 26 are related to the surgical site 28 or the environment outside the surgical site (e.g., the operating room or surgical area). One or more characteristics, such as the range of light wavelengths, the features of the image data, and the focal length of the camera device 22, may vary considerably when the image data is imaged within the surgical site 28 relative to the environment outside the surgical site 28 in the operating room or surgical operating room. Such behavior may also be called scope-in ​​versus scope-out detection and may indicate whether the camera device 22 and the corresponding image data 26 are imaged depicting the surgical site 28 within the patient's cavity. In response to a negative or scope-out determination of the image data 26 representing the patient's cavity in the surgical site 28, the method 180 may be configured to maintain the user interface or system interface and the corresponding controls presented on the display 18 in preoperative interface mode (190) and return to step 186. If the image data 26 indicates an inserted or scope-in ​​state in step 188, the system controller 24 may be configured to update the controller interface to intraoperative interface mode (192).

[0060] In response to the intraoperative interface mode applied in step 192, system controller 24 may be configured to control vision module 60 to start detecting the visual state of surgical site 28 (194). As described above, the visual state of surgical site 28 may be detected by referring to blood classifier 140, visibility indicator 142, and particle content indicator 144. As shown, the method incorporates the determination of the visual state in successive steps 196, 198, and 200. In step 196, the image data 26 is determined to be blood positive or blood negative. In step 198, the detected particle content 168 is compared with a particle content threshold T p . Further, in step 200, the detected visibility 158 is compared with a visibility threshold T v . In response to each combination of the states identified in steps 196, 198, and 200, vision module 60 may be configured to classify the visual state of surgical site 28 based on representative image data 26 of step 202. The visual state is reported to state module 62 by vision module 60 and may be reported via device network 14. In this way, each of surgical devices 12 communicating via device network 14 may be configured to update or adapt its operating settings to correspond to the detected visual state 60. Although steps 196, 198, and 200 are described in sequence, the detection of the clarity of image data 26 may be rearranged based on priority or, in some cases, may be processed simultaneously by vision module 60. The processing may also be performed frame by frame or periodically at a sampling frequency, depending on the sophistication of the system hardware.

[0061] Following the identification and reporting of the visual state in step 202, method 180 may proceed to step 204, where one or more of the device controllers 16 of the surgical apparatus 12 and / or system controller 24 may be configured to identify the proposed control state. The control state may be configured to identify based on the blood classification unit 140, the visibility index 142, and / or the particle content index 144, as discussed with reference to the pump controller 12c and the surgical pump 120. If the proposed operation of the surgical pump 120 requires confirmation, method 180 may proceed to step 206, where it may be configured to determine whether a prompt is confirmed via one or more of the various user interface inputs of the surgical apparatus 12 (e.g., conditionally mapped to an input, or otherwise presented on the user interface of the computer or tablet 20). If no confirmation is received in step 206, method 180 may be configured to return to step 188. If confirmation is received, in this embodiment, the proposed control state of the pump controller 12c may be configured to update the operation of the surgical pump 120 to conform to the control state proposed in step 208. In this way, the control system 10 may be configured to provide semi-automated or accelerated control operation of the surgical pump 120 in response to a visual state identified by the visual module 60.

[0062] Referring further to Figure 10, the proposed or active control state of the surgical pump 120 may be configured to be updated by repeating or cycling steps 194-208, as described above. For example, when method 180 proceeds to step 204, the visual state identified in steps 196-200 may be configured to update the image data 26 to indicate that it no longer contains positive blood classification, excessive particle content, and / or reduced visibility states. In response to the updated visual state, the proposed control state identified in step 204 may indicate that the surgical pump 120 should return to baseline pressure and typical fluid exchange rate. Thus, the continued operation of method 180 may be configured to provide prompts related to the activation of a control state in order to improve clarity or visibility of the image data 26, or it may be configured to provide actions to limit the pressure associated with the patient's cavity for the surgical procedure in order to improve the outcome by limiting extravascular migration.

[0063] In some implementations, System 10 may provide one or more operational states and corresponding detection routines that may be step or stage specific in relation to one or more surgical procedure actions. As previously stated, the system may be configured to provide tracking of each stage or step of a pre-configured medical procedure based on the detection of the use of various surgical instruments 12, the operational characteristics of the tools reported via the instrument network 14, and / or corresponding states or features detected in the image data 26 using the procedure tracking module 64. In response to the detection of a procedure stage or step, the visual module 60 may be configured to identify one or more states, treatments, or pending actions of System 10 that may result in a change in the clarity or visibility of the image data 26. While some conditions may not be readily identifiable based solely on the processing of the image data 26 by the image processor(s) 48, System 10 may be configured to estimate one or more states of a surgical site based on a combination of changes detected in the image data 26 (e.g., discoloration, cloudiness, etc.) and corresponding identification of a procedure stage or step. For example, in some steps of a procedure, the visibility of image data 26 may be reduced by adding one or more treatments or biologics to the fluid within the patient's cavity at the surgical site 28. In response to such processing, the clarity or color of image data 26 may change, but rinsing the cavity may limit the benefits of the processing. Therefore, in response to the detection of a procedure step that may include the administration of a treatment or biologic that may affect the visibility of image data 26, the system may be configured to infer that the corresponding change in the visibility of the image data is a result of the procedure step (e.g., administration of a biologic) within the patient's cavity.

[0064] In response to such detections, the controller 24 may be configured to omit or suppress corrective pump actions (e.g., rinsing or flushing cavities) despite apparent changes in the image data 26. For example, the visual module 60 may, in response to the treatment tracking module 64 indicating that this step of the procedure generally involves the administration of a biologic or additive treatment that may affect the visibility of the image data 26, adjust the visibility threshold T v The system may be configured to temporarily adjust or suppress the corresponding corrective action. Similarly, the controller 24 may be configured to output a message or prompt the user of the system 10 to identify or confirm whether the change in image quality is the result of the administration of a biologic to the surgical fluid or an additive treatment. In response to a positive confirmation, the corrective action of the pump controller 12c may be configured to be suppressed for a predetermined period of time or until the treatment tracking module 64 indicates that the visibility of the image data 26 is no longer controlled based on the biologic or additive treatment. Examples of biologic treatments that can be performed and inferred or detected by the vision module 60 include, but are not limited to, bone marrow aspirate (BMA) concentrate, cultured and proliferated mesenchymal stem cells (MSCs) and stromal cells, autologous blood products (including modified plasma [ACP], platelet-rich plasma [PRP], etc.), growth factors, hyaluronic acid, and autologous chondrocyte transplantation (ACI), autologous matrix-induced chondrogenesis (AMIC), etc.

[0065] Referring here to Figure 11, a plot is shown showing the fluid pressure applied by the surgical pump 120, as well as the corresponding fluid consumption tracked over time throughout the surgical procedure. As previously mentioned, in some implementations, the control system 10 (e.g., pump controller 12c) may be configured to track the operation of one or more of the surgical devices 12, including the surgical pump 120, to determine each user's preferences and update the preferences in the corresponding user profile 32. The data shown in Figure 11 may be configured to correspond to sample pump data that can be stored to identify the preferences of the relevant user, so that suggestions and associated responses can be stored in the user profile 32. In this way, the system 10 can learn the preferences associated with each of the user profiles 32 to improve the proposed operating state, including baseline pressure, fluid inflow and outflow rates, thresholds associated with detection in the blood sorting unit 140, visibility index 142, and particle content index 144, as well as other operating settings of the device 12 of the system 10.

[0066] In some implementations, the threshold associated with the visibility of the surgical site 28 in the image data 26 may be identified as part of the user's preferences associated with each of the user profiles 32. For example, the visibility threshold T associated with the blood classification unit 140. v , particle content threshold T p The system may be configured to track the visibility threshold T associated with the blood classification unit 140 as one or more visibility tolerances or sensitivity settings. For example, some users may want to flush or lavage the surgical site in response to a limited change in visibility. In such cases, the system 10 tracks the visibility threshold T associated with the blood classification unit 140. v , particle content threshold T pThe system may be configured to reduce one or more of the blood thresholds or to increase the corresponding sensitivity to the corresponding user profile 32. In this configuration, only slight changes in visibility may trigger an automatic response or prompt. As an alternative configuration, some users may prefer to flush or lavage the surgical site in response to a significant change in visibility. In such cases, the system 10 may adjust the visibility threshold T associated with the blood classification unit 140. v , particle content threshold T p The configuration may increase one or more of the blood thresholds or increase the corresponding sensitivity associated with the user profile 32. In this way, slight changes in visibility may not trigger an auto-response or prompt, and such proposed actions may require a significant decrease in visibility before being activated. Thus, these user profiles 32 can be described as having high tolerances for changes in visibility associated with one or more of the blood classification unit 140, the visibility index 142, and the particle content index 144.

[0067] As shown in Figure 11, the pressure readings related to the operation of the surgical pump 120 include the display of the proposed control state updates 210a, 210b, 210c, 210d, and 210e, as well as the acknowledgment input 212 received by one or more of the user interfaces of the surgical apparatus 12. For example, in the first proposed state 210a, the visual module 60 has a visibility threshold T associated with the visibility index 142. vThe system may be configured to identify a blood-positive identification 152 from the blood classification unit 140 in combination with a detected visibility 158 of less than 158. Based on aggregated data 44, including the reported states identified by the visual module 60 and the state module 62, the system controller 24 may be configured to output a prompt on the display 18 requesting confirmation of the updated control state for the washing routine. The corresponding confirmation input 212 following the first proposed state 210a may be received by the device network 14 and configured to initiate increased pressure and fluid exchange associated with the washing routine. Throughout the washing routine, the visual module 60 monitors image data 26 to determine visibility and blood content when the minimum visibility T v The configuration may determine whether the blood content has improved to exceed a certain level or no longer shows a significant blood content. Based on this determination, the visual state may be updated by the visual module 60 and output via the state module 62 in the aggregated data 44. As shown, the second proposed state 210b is confirmed by the confirmation input 212. As shown, the pressure associated with the washing routine may be configured to return to a baseline pressure of approximately 50 mmHg. As previously stated, each of the confirmation inputs 212 may be configured to map to the user interface of the surgical apparatus 12, in particular to one or more inputs or user inputs associated with the surgical apparatus 12 that have been identified as active for a given time for the proposed state.

[0068] Later in the procedure, a third proposed state 210c may include a configuration in which baseline pressure can be reduced at the surgical site 28. Such a proposal may be identified based on a report from the visual module 60 that the image data 26 is free of debris and blood over a period of time that may vary based on a specific procedure, patient, or user associated with the active profile 34. For example, if the image data 26 is free of debris (e.g., visibility threshold T v and / or particle content threshold T pAfter a predetermined time or procedure-specific period (e.g., 5-15 minutes), when it is determined that the blood does not contain a blood negative with visibility exceeding a certain threshold, the controller 24 may be configured to suggest or adjust an updated baseline pressure for the operation of the surgical pump 120. In response to the suggested condition, the pump controller 12c may control the pressure of the inflow to the surgical site 28 to gradually decrease over a longer period (e.g., 5-15 minutes) to the updated baseline pressure (e.g., decrease from 52 mmHg to 44 mmHg). In this way, the system 10 may be configured to automatically suggest changes in baseline pressure to improve the patient's recovery. The subsequently suggested conditions are a fourth suggested condition 210d and a fifth suggested condition 210e resulting from the blood positivity identification 152 by the blood classification unit 140, and a visibility threshold T, respectively, identified by the visibility index 142. v This includes a reduction in visibility below a certain level. Each of the proposed control states may be identified in response to an analysis of image data 26 processed throughout the operation, thereby allowing the user to operate the system 10 through a series of intuitive prompts that do not distract from or cause delays in the surgical procedure.

[0069] Throughout the operation of the control system 10, the system may be configured to monitor and record cumulative pressure and fluid consumption associated with the operation of the surgical pump 120 for each procedure associated with the active profile 34 in order to track preferences related to baseline pressure, fluid exchange rate, and increased pressure associated with rinse and / or washing routines. In this way, the control system 10 can track the preferences of each user corresponding to the user profile 32 and improve the suggestions made for the operation of the surgical pump 120. Furthermore, the system may be configured to record data related to the operation of the surgical pump 120 in a comprehensive user library, which can monitor the benefits associated with various pressure settings, fluid consumption rates, and various other settings associated with the operation of the pump in order to compare and document the data in relation to patient outcomes and pain scores in order to improve patient outcomes related to the corresponding procedures.

[0070] Referring here to Figure 12, a typical diagram of the user interface is shown on the display 18. In addition to updating the control state of the surgical pump 120, the operation of the visual module 60 and the state module 62 may be configured to cause the system controller 24 to update one or more color schemes or interface menus associated with the operation of the surgical pump 120. For example, during the baseline operation of the surgical pump 120, the system controller 24 may be configured to display the operation information of the control system 10 in a first color scheme (e.g., black and white). In response to the detection of an updated visual state by the visual module 60, the system controller 24 may be configured to update the color scheme to a second color scheme (e.g., green on a black background) to highlight the identification of the proposed control state of the surgical pump 120. Furthermore, in response to confirmation of the activation of the proposed control state, the system controller 24 may update the graphic user interface and the corresponding color scheme to maintain the second color scheme but focus on the updated operation information of the surgical pump 120. For example, the system controller 24 may be configured to display a cleaning indicator 220, as well as related pressure indicators 222 and / or elapsed time indicators 224 associated with the cleaning routine. In this way, relevant information related to the proposed operating state activated for the surgical pump 120 may be presented and highlighted to the user of the control system 10 on the display 18.

[0071] While the active cleaning settings for the corresponding color schemes have been discussed with reference, the system controller 24 may also be configured to activate or modify the corresponding data or information displayed on the color scheme and graphic user interface to highlight any of the proposed, confirmed, or automatically activated processes identified in response to the visual state of the surgical site 28 determined by the visual module 60, as discussed herein.

[0072] Referring here to Figure 13, the operation of various surgical devices 12 that communicate with and / or form the control system 10 will be described with reference to the control of the surgical pump 120. As provided in the following embodiments, the operation of the surgical pump 120 may be configured to be controlled in response to the operation of various surgical devices 12 that communicate via the device network 14. For example, the control system 10 may be configured to communicate with one or more patient monitors 12g via the device network 14. In this configuration, patient information, including vital signs such as heart rate and blood pressure, may be reported by the patient monitors 12g via the device network 14. In some implementations, a visual module 60 and / or a state module 62 may be configured to report data related to vital signs and report corresponding information in conjunction with the state of the surgical device 12 in aggregated data 44 on the device network 14. Thus, the device controller 16 of the surgical device 12 may be configured to update or prompt the user to take into account updated operating settings or operating states to match the information reported by the patient monitors 12g.

[0073] In certain embodiments, the display of the patient's blood pressure may be monitored by the state module 62 to identify examples of elevated blood pressure that could serve as lead indicators of potential bleeding at the surgical site 28. In response to the detection of elevated blood pressure by the state module 62, the pump controller 12c may be configured to prompt the user via the display 18 to increase the fluid pressure at the surgical site 28 using the surgical pump 120. As described in various embodiments, the user may then confirm the suggestion via input to one or more of the user interfaces provided by the surgical apparatus 12. Furthermore, in some cases, the operation of the visual module 60 may be updated based on the elevation of blood pressure to increase the sensitivity of the blood classification unit 140 and / or visibility index 142 or adjust the motion detection settings in response to the detection of an elevation in blood pressure (e.g., an increase in diastolic pressure, systolic pressure, mean arterial pressure, etc.). For example, the system may be configured to prioritize the operation of the blood classification unit 140 in response to the display of elevated blood pressure, or it may be configured to decrease the bleeding detection threshold 154 associated with the detection of blood identified in the image data 26 at the surgical site 28 in response to the increased likelihood of blood detection due to the elevated blood pressure state and the possibility of bleeding. In this way, the visual module 60 may be configured to adjust detections associated with the blood classification unit 140 and / or visibility index 142 so that detections associated with conditions related to elevated blood pressure are expected, detected with increased sensitivity, and prompt the user to apply washing, rinsing, or various pump settings in response to the condition. Examples of elevated blood pressure may include a systolic pressure above 120 mmHg and a diastolic pressure remaining below 80 mmHg, or a case where the systolic pressure rises above 120 mmHg and the diastolic pressure rises above 80 mmHg.

[0074] In some implementations, the operation of system 10 may be configured to update similarly, or it may be configured to prompt the user to update the operation based on the active operation of surgical equipment 12, such as a resection tool 122, an ablation or electrosurgical device, or various electromechanical surgical instruments. As shown in Figure 13, the operation of the resection tool 122 will be described with reference to the inflow and outflow provided to the surgical site 28. As illustrated, the outflow may be configured to be provided through both the resection tool 122 (e.g., a shaver) and the outflow port 124 of the cannula 126. In response to the activation of the resection tool 122, the pump controller 12c may be configured to stop the corresponding outflow from the resection tool 122 and increase the outflow through the outflow port 124. Such operation can ensure that the fluid pressure does not increase in the patient cavity represented by the surgical site 28 resulting from the reduction in outflow. In this way, the control system 10 can maintain homeostasis at the surgical site while also limiting the possibility of blockage of the outflow associated with the resection tool 122.

[0075] In some implementations, the visual module 60 and the status module 62 may be configured to update settings associated with the blood classification unit 140, visibility index 142, and / or particle content index 144 based on the type of the implemented excision tool 122 and / or the blade or cutting attachment style associated with the excision tool 122. For example, in some cases, based on the blade or cutting attachment associated with the excision tool 122, the user may prefer to see a certain level of bleeding or blood presented in the image data 26 at the surgical site 28 to ensure that the procedure is effective. In such circumstances, the cutting tool and / or cutting instrument style associated with the excision tool 122 may be configured to be reported via the device network 14 through the excision tool 122 and / or to be manually entered into the system controller 24 during procedure setup for operation. In response to the display of the cutting attachment, blade style, cutting instrument, or similarly interchangeable components of the excision tool 122, the visual module 60 may be configured to update corresponding thresholds for one or more of the blood classification unit 140, visibility index 142, and / or particle content index 144. For example, in response to the activation or connection of a file (e.g., a power file), pick, power pick, or similar device detected or identified via the device network 14, the visual module 60 may be configured to increase the minimum blood classification threshold 154 without triggering an automatic or facilitated adjustment of the pressure or fluid flow supplied by the surgical pump 120, thereby ensuring that blood is visible in the image data 26.

[0076] As described in the embodiments above, the type of the excision tool, cutting accessory, blade type, cutting ratio, invasiveness, etc., may be reported by the corresponding device controller 16 via the device network 14 as a model, serial number, tool ID, accessory ID, etc. This information may be configured to be automatically identified by the corresponding device controller 16, for example, by one or more identifying circuits (e.g., radio frequency identifying circuits or conventional computerized memory devices such as electrically erasable programmable read-only memory or similar devices) of the corresponding surgical device 12 and accessory. Furthermore, the information identifying the surgical device 12 and accessory may be configured to be manually entered preoperatively. Once loaded, each of the controllers 16, 24 may identify the corresponding use on the network 14, enabling the system 10 to estimate the clarity of the image data 26 and the impact of the corresponding adjustments on the operation of the visual module 60, as discussed herein.

[0077] In yet another embodiment, the activation of the excision tool 122 may result in a significant decrease in visibility or an increase in turbidity, depending on the type of cutting instrument or accessory associated with the excision tool 122. Therefore, based on the style and / or type of accessory associated with the excision tool 122 (e.g., blade, burr, pick, rasp, drill, etc.), the system controller 24 and / or pump controller 12c may be configured to predict various levels of turbidity and / or associated bleeding. In such cases, the control system 10 may be configured to adjust the associated pressure and suction corresponding to the inflow and outflow associated with the operation of the surgical pump 120, based on the style, model, type, ratio, or other variables that may vary in relation to the specific excision tool 122 implemented. For example, in some cases, the pressure adjustment associated with the inflow of the surgical fluid may be configured to be increased by about 5% to 50% or more from the baseline pressure, with respect to the detection of a decrease in field of view by the visual module 60, depending on the style, model, type, etc. of the accessory associated with the excision tool 122 and the possible resulting impact on visibility at the surgical site 28. Similarly, the system controller 24 and / or pump controller 12c may be configured to adjust one or more rinsing settings that adjust the fluid exchange rate supplied by the inflow and returned via the outflow to clear visibility related to the relevant operation of the excision tool 122. For example, the pump controller 12c may be configured to increase the fluid exchange rate of the surgical fluid supplied to the surgical site 28 by about 5% to 50% or more, depending on the expected resulting turbidity or particle content related to the operation of a particular model or style of the implemented excision tool 122.By tracking the model and type of the implemented excision tool 122, the control system 10 may be configured to flexibly respond to the expected levels of turbidity, particulate matter, and / or the likely blood content at the surgical site 28, and to automatically initiate or suggest customized pressure or fluid exchange settings related to the operation of the surgical pump 120 in combination with the specific excision tool 122 to suppress unwanted pressure at the surgical site 28, and to customize or adjust the response of the surgical pump 120 to specific conditions expected for the surgical apparatus 12 being performed.

[0078] In some cases, the control system 10 may be configured to additionally adjust the operation of the surgical pump 120 to help maintain optimal performance for one or more of the surgical devices 12. For example, in response to the operation of an electrosurgical or ablation probe, the electrosurgical console 12b may be configured to communicate the corresponding operating configuration, model, style, etc., of the electrosurgical device via the device network 14. In response to the display of an electrosurgical probe (e.g., an RF ablation probe), the pump controller 12c may be configured to respond by reducing the fluid exchange rate of the surgical fluid to the surgical site 28 associated with inflow and outflow (e.g., reducing fluid permeation or exchange by 5% to 25% or more). By limiting the supply and / or suction rates associated with the operation of the surgical pump 120, the pump controller 12c may be configured to ensure that excessive suction does not affect the generation of plasma produced by the ablation probe.

[0079] In some implementations, the temperature of the surgical site 28 may be detected by an ablation probe, a camera device 22, or other surgical device 12 and reported via the device network 14. In response to an increase in temperature level, suction or inflow and outflow controlled by the surgical pump 120 may increase the fluid exchange rate at the surgical site 28 in response to the temperature increase that may result from the operation of the ablation probe. In this way, the control system 10 may be configured to decrease or adjust the baseline pressure and associated fluid exchange rate in response to the use of the ablation probe controlled by the electrosurgical console 12b, and to selectively increase fluid exchange at the surgical site 28 as needed, thereby avoiding a temperature rise that exceeds one or more predetermined temperature thresholds in the patient's cavity. Such operation can limit the temperature rise in the patient's cavity while ensuring that the performance of the ablation probe is maintained.

[0080] As provided in various aforementioned embodiments, the control system 10 may offer various benefits in relation to facilitating the recommended control of surgical devices 12, particularly the surgical pump 120, and / or the automatic activation of various control settings, in order to improve the effective operation of the system 10. By combining the distributed control provided on the device network with the operation of the visual module 60 and the state module 62, the control system 10 may monitor the operation of each of the relevant surgical devices 12 in relation to activities and / or characteristics that may be associated with or identified within the patient cavity at the surgical site 28, and ensure that each of the corresponding elements of the control system 10 or each of the devices 12 can work together effectively. While specific embodiments of combined operation are provided throughout this application, it should be understood that each of the control routines and coordinated operation of the surgical devices 12 may be applied individually or in combination without departing from the concepts of the invention disclosed herein.

[0081] Referring here to Figure 14, a flowchart is shown illustrating a method 230 implemented for setting or adjusting one or more operating parameters of a surgical device 12 that communicates with the control system 10. Generally, method 230 may be implemented to set user-specific pressure limits or guide rails associated with the operation of the surgical pump 120, which may be associated with a user profile 32. In addition to user-specified limits, the pressure settings of the surgical pump 120 may be configured to be adjusted based on one or more existing conditions or patient health characteristics, which may be entered by staff and / or accessed by a patient database communicating with the system 10. In some cases, the pressure settings associated with the operation of the pump 120 may be configured to be further controlled in response to patient information, including vital signs such as heart rate and blood pressure, which may be reported by a patient monitor 12g during the procedure. Thus, method 230 may be flexibly implemented to adjust a maximum pressure setting that can be automatically applied to control bleeding or massive bleeding, and a minimum pressure setting that can be maintained between bleeding events as a baseline pressure for operating the surgical pump 120.

[0082] As shown in Figure 14, method 230 may be configured to be initiated by starting a setup procedure for the control system 10 (232). The setup procedure may be configured to include one or more configuration steps 234a, 234b, 234c, which may include accessing or receiving data from a user input device, a database, and / or a server 236. As previously stated, the configuration data may be configured to be received in the form of patient data 234a, procedure data 234b, and / or user / surgeon preference data 234c, which can be accessed via the user profile 32. The patient data 234a may be configured to correspond to one or more existing conditions, e.g., irregular blood pressure (e.g., hypertension, hypotension), heart disease, diabetes, age, atrophy, or other patient health characteristics. Furthermore, the patient data 234a in step 234 may be configured to correspond to intraoperative patient data reported by the patient monitor 12g via the device network 14. The pressure setting of the surgical pump 120 may be configured to be additionally set based on a particular type of procedure, as shown in step 234b. Furthermore, as shown in step 234c, the maximum and minimum pressure settings of the surgical pump 120 may be configured to be adjusted based on the user's or surgeon's preference. Although each of the data inputs related to the operation of the system 10 is described separately in steps 234a, 234b, and 234c, it will be understood that user or surgeon preference data may incorporate maximum and minimum pressure settings for specific procedures outlined in step 234b, and based on various existing health conditions related to each patient, as described in step 234a. Thus, the settings of the system 10 for the semi-automatic operation of the surgical pump 120 may be configured to be customized to suit the specific needs of the patient, procedure, and / or user / surgeon.

[0083] When the data associated with steps 234a, 234b, and 234c is accessed by the device controller 16 of the surgical pump 120, the controller 16 may be configured to determine control settings based on one or more of the relevant parameters identified in steps 234a, 234b, and 234c, and to adjust the control settings for operating the system 10, in particular the surgical pump 120 (238). Based on each of the input parameters from steps 234a, 234b, and 234c, the device controller 16 may set maximum and minimum pressure settings, during which the pressure of the surgical pump 120 may be configured to be controlled automatically or by a proposed setting update. In this way, the operation of the surgical pump 120 or other device 12 may be configured to be adjusted in response to the visual conditions described with reference to Figures 6-9 (240). With patient, user, and / or procedure-based pressure settings identified in step 238 and recorded in step 240, the automatic control procedure for the surgical pump 120 may be configured to be initiated in step 240.

[0084] The automatic or assisted device control routine 240 may operate as generally outlined with reference to Method 180. Alternatively, the pressure control of the surgical pump 120 between the maximum and minimum pressure settings may be configured to be progressively adjusted in response to the blood abundance ratio 168 or other indicators of blood presence in the image data 26. For example, instead of adjusting the pressure of the surgical pump 120 in response to particulate matter content or visibility falling below a predetermined threshold, the device controller 16 of the surgical pump 120 may be configured to gradually adjust (increase, decrease) the pump pressure in response to an increase or decrease in the blood abundance ratio 168. To effectively provide recommended or automated updates to the pressure settings of the surgical pump 120, the control routine 240 may continue by monitoring patient data reported by the patient monitor 12g and may be configured to update the pressure settings and / or limits defined in step 238 (242). In step 242, routine 240 may also be configured to process and / or classify the image data 26 to identify the visual state recorded by the camera device 22. As previously mentioned, the device data 42 and image data 26 may be reported via the device network 14 as individual device data and / or aggregated data 44, and the device data 42 and image data 26 may be compiled by the system controller 24 or the camera control unit (CCU).

[0085] Using patient data, device data 42, and / or aggregated data 44, the device controller 16 may be configured to automatically adjust the pressure of the surgical pump 120 in response to each of the data inputs reported in step 242 (244). If there is an automatic or assisted pump pressure control setting identified in step 244, routine 240 may be configured to continue by displaying the control setting to the user on one or more displays 18 (246). When displaying the automatic and / or recommended control setting for the pump 120, the device controller 16 may be configured to additionally identify the basis or reason for the pressure setting, which may be associated with patient data or procedure data related to the patient and device data monitored in steps 234a, 234b, and / or step 242. In this way, method 230 may provide a useful basis for a proposed or automated pressure setting update that may be detected based on the state of the image data even before the user has identified that a bleeding event has occurred.

[0086] Throughout the operation of System 10, the control routine 240 may be configured to monitor patient data and corresponding conditions presented in the image data to determine whether the automatic or suggested pressure setting is within the maximum and minimum pressure settings set in step 240 (248). For example, during a bleeding event, if the patient begins bleeding and the image data reports a high blood abundance ratio 168, the device controller 16 may be configured to adjust to the maximum pressure setting associated with the user's preference. However, in such circumstances, the system may be configured to prompt the user to initiate manual control, which may extend to a maximum or minimum pressure setting that exceeds the user-specified maximum and minimum pressure settings and may be associated with limits set by the device manufacturer (250). In such circumstances, the automatic or assisted device control routine 240 may be disabled, at least temporarily, to allow the user or surgeon to adjust the pressure setting within the maximum and minimum limits set by the manufacturer of the surgical pump 120. In this way, the control routine 240 may be configured to adjust the pressure setting of the pump 120 to identify conditions that may require manual intervention to provide the best possible care to the patient. Alternatively, in step 248, if the automatic or assisted pump pressure setting remains within the user-specified maximum and minimum pressure settings, routine 240 may return to step 242 to monitor and maintain the visibility presented in image data 26.

[0087] Although Method 230 is described as primarily providing control of the pressure of the pump 120 in response to image data 26, the system may also be configured to adjust or control the pressure and / or inflow and outflow of the surgical pump 120 in response to detected operation of one or more surgical devices 12 reported via the device network 14, as described in Step 242. For example, the activation of an excision or electrosurgical device reported in device data 42 and / or aggregated data 44 may be configured to prompt the device controller 16 of the surgical pump 120 to increase the inflow and outflow through the surgical site to improve visibility of the image data 26. Such indications of the operation of the devices 12 may be applied in combination with various detection and classification routines described with reference to Figures 6-9 to optimize the system 10's response to various control scenarios. By automating or suggesting control settings based on the image data 26 and device data reported via the device network 14, the system 10 can improve the operation of the surgical pump 120 in cooperation with each of the devices communicating via the device network 14. Furthermore, by adjusting the control or pressure settings of the pump 120 based on patient data, procedure data, and / or user / surgeon preferences, the disclosure may provide a customized user experience that can improve the operation of the surgical pump 120 to improve surgical outcomes.

[0088] According to some aspects of the present disclosure, a surgical control system for a plurality of surgical devices comprises at least one device controller communicating with each of the plurality of surgical devices via a communication bus. The at least one device controller is configured to communicate device data including a message indicating at least one of a control state and a detected state associated with the operation of each of the plurality of surgical devices. A camera device is configured to communicate with a plurality of medical devices via a communication bus and comprises at least one controller, the controller being configured to capture image data indicating a surgical site, process the image data, identify the surgical state of the surgical site in response to the image data, generate aggregated data including the surgical state and the device data in a packet communicated via the communication bus, and communicate the aggregated data via the communication bus. Each of the plurality of device controllers independently controls the operation of the plurality of surgical devices in response to at least one of the device data and the aggregated data.

[0089] Depending on the circumstances, this disclosure may implement one or more of the following structures or configurations in various combinations: The surgical status is identified in response to a combination of the device data and the image data, and includes an indication of at least one of the following: the patient's condition, the procedure step, the surgical device status, and the surgical site status; The device data includes a message that communicates at least one of the control states and detected states that are associated with the operation of each of the plurality of medical devices and are monitored by one or more of the plurality of medical devices; The detected state is detected by at least one of the surgical devices, and the state of the surgical site is identified; The detected state includes at least one of the following: temperature data, pressure data (pressure of the fluid in the cavity), flow rate data, and current data (feedback); The detected state is the operating state of one or more of the multiple surgical devices; The detected state is the patient's condition (e.g., patient vital signs, health statistics) monitored by the patient monitors of the multiple surgical devices; At least one device controller comprises multiple device controllers associated with each of the multiple surgical devices; Each of the device controllers independently reports and monitors the device data communicated as messages from each of the plurality of surgical devices; The plurality of surgical devices respond independently and selectively to the aggregated data packets or the device data according to the programming control structure of the corresponding device controller of the plurality of device controllers; At least one controller processes the device data as a factor in determining the surgical state; The factors associated with the device data instruct the at least one controller to identify the surgical state within a subset of multiple state categories; At least one controller estimates the current surgical status in response to the device data; At least one controller further infers the current surgical state in response to a previously identified surgical state, and / or, the surgical state is identified in response to a combination of the device data and a previously identified surgical state. At least one controller is configured to identify a proposed control configuration for at least one of the surgical devices based on the aggregated data.

[0090] According to another aspect of the present disclosure, a method for controlling a surgical control system that communicates with a plurality of surgical devices includes broadcasting device data via a communication bus that communicates with a plurality of device controllers of the plurality of surgical devices, including a message indicating at least one of a control state and a detected state related to the operation of each of the plurality of surgical devices; capturing image data indicating a surgical site using a camera device; identifying the surgical state of the surgical site in response to the image data; generating aggregated data in a packet including the surgical state and the device data; and communicating the aggregated data via the communication bus, wherein each of the plurality of device controllers communicates to independently control the operation of the plurality of surgical devices in response to at least one of the device data and the aggregated data.

[0091] Depending on the circumstances, this disclosure may implement one or more of the following structures or configurations in various combinations: The device data includes a message communicating at least one of the control states and detected states reported by each of the plurality of medical devices and monitored by one or more of the plurality of medical devices; The detected state is detected by at least one of the surgical devices, and the state of the surgical site is identified; Based on the aggregated data, identify a proposed control configuration for at least one of the surgical devices; Display the proposed control configuration as a prompt requesting confirmation of the proposed control configuration; Mapping the inputs of the user interface of the active medical device, which is a plurality of medical devices or a camera device, such that the inputs of the user interface are mapped to receive confirmation of the proposed control configuration in response to the activation of the inputs; In response to the input to the user interface associated with the confirmation, the user interface communicates the confirmation of the proposed control configuration to the communication bus.

[0092] According to yet another aspect of the present disclosure, a surgical control system for a plurality of surgical devices includes a plurality of device controllers communicating with each of the plurality of surgical devices via a communication bus, each of which independently reports the device data from one of the plurality of surgical devices, monitors the device data communicated as a message from each of the plurality of surgical devices, and the device data indicates at least one of a control state and a detected state associated with the operation of each of the plurality of surgical devices; and a camera device communicating with the plurality of medical devices via the communication bus, comprising at least one controller, the at least one controller configured to capture image data indicating a surgical site, and to identify a surgical state of the surgical site in response to the image data, the surgical state being reported by the camera device via the communication bus, wherein each of the plurality of device controllers independently controls the operation of the plurality of surgical devices in response to at least one of the device data and the surgical state.

[0093] Naturally, any described process or step within a described process may be combined with other disclosed processes or steps to form a structure within the scope of this apparatus. The exemplary structures and processes disclosed herein are for illustrative purposes only and should not be construed as limitations.

[0094] Furthermore, it goes without saying that the aforementioned structure and method can be modified and altered without deviating from the concept of this device. And naturally, unless otherwise explicitly stated in the language of those claims, these concepts are intended to be covered by the following claims.

[0095] The above description is to be considered only for the exemplary embodiments. Those skilled in the art, or those who make or use the apparatus, will conceive of modifications to the apparatus. Therefore, the drawings and the embodiments described above are for illustrative purposes only and are not intended to limit the scope of the apparatus, which is understood to be defined by the following claims, which shall be interpreted in accordance with the principles of patent law, including the doctrine of equivalents.

Claims

1. A surgical control system for multiple surgical devices, At least one device controller that communicates with each of the plurality of surgical devices via a communication bus, and is configured to communicate device data including a message indicating at least one of the control state and detected state associated with the operation of each of the plurality of surgical devices, A camera device that communicates with the plurality of medical devices via the aforementioned communication bus, comprising at least one controller, the at least controller is To acquire image data showing the surgical site, Processing the aforementioned image data, In response to the aforementioned image data, the surgical status of the surgical site is identified, To generate aggregated data including the surgical procedure status and the device data in packets communicated via the communication bus, A camera device is configured to communicate the aggregated data via the aforementioned communication bus, Includes, A surgical control system in which each of the plurality of device controllers independently controls the operation of the plurality of surgical devices in response to at least one of the device data and the aggregated data.

2. The surgical control system according to claim 1, wherein the surgical state is identified in response to a combination of the device data and the image data, and includes a display of at least one of the patient's condition, a procedure step, the surgical device state, and the surgical site state.

3. The surgical control system according to any one of claims 1 to 2, wherein the device data includes a message that communicates at least one of a control state and a detected state, which is associated with the operation of each of the plurality of medical devices and is monitored by one or more of the plurality of medical devices.

4. The surgical control system according to claim 3, wherein the detected state is detected by at least one of the surgical devices, and the state of the surgical site is identified.

5. The surgical control system according to claim 4, wherein the detected state includes at least one of temperature data, pressure data, flow rate data, and current data.

6. The surgical control system according to claim 3, wherein the detected state is an operating state of one or more of the plurality of surgical devices.

7. The surgical control system according to claim 3, wherein the detected state is the state of a patient monitored by the patient monitors of the plurality of surgical devices.

8. The surgical control system according to any one of claims 1 to 7, wherein the at least one device controller comprises a plurality of device controllers associated with each of the plurality of surgical devices.

9. The surgical control system according to claim 8, wherein each of the device controllers independently reports and monitors the device data communicated as messages from each of the plurality of surgical devices.

10. The surgical control system according to any one of claims 1 to 9, wherein the plurality of surgical devices respond independently and selectively to the aggregated data packets or the device data according to the programming control structure of the corresponding device controller of the plurality of device controllers.

11. The surgical control system according to any one of claims 1 to 10, wherein the at least one controller processes the device data as a factor in determining the surgical state.

12. The surgical control system according to claim 11, wherein the factors associated with the device data instruct the at least one controller to identify the surgical state within a subset of a plurality of state categories.

13. The surgical control system according to any one of claims 1 to 12, wherein at least one controller estimates the current surgical state in response to the device data.

14. The surgical control system according to claim 13, wherein at least one controller further infers the current surgical state in response to a previously identified surgical state, such that the surgical state is identified in response to a combination of the device data and the previously identified surgical state.

15. The surgical control system according to any one of claims 1 to 14, wherein the at least one controller is configured to identify a proposed control configuration for at least one of the surgical devices based on the aggregated data.

16. A method for controlling a surgical control system that communicates with multiple surgical devices, Broadcasting device data including a message indicating at least one of the control state and detected state associated with the operation of each of the multiple surgical devices via a communication bus that communicates with multiple device controllers of the multiple surgical devices, This involves using a camera device to capture image data indicating the surgical site, In response to the aforementioned image data, the surgical status of the surgical site is identified, To generate aggregated data in a packet, including the surgical procedure status and the device data, Communicating the aggregated data via the communication bus, wherein each of the plurality of device controllers independently controls the operation of the plurality of surgical devices in response to at least one of the device data and the aggregated data. Methods that include...

17. The method according to claim 16, wherein the device data includes a message communicating at least one of a control state and a detected state, which is reported by each of the plurality of medical devices and monitored by one or more of the plurality of medical devices.

18. The surgical control system according to claim 17, wherein the detected state is detected by at least one of the surgical devices, and the state of the surgical site is identified.

19. A method according to any one of claims 16 to 18, further, Based on the aggregated data, identify the proposed control configuration for at least one of the surgical devices, A method comprising displaying the proposed control configuration as a prompt requesting confirmation of the proposed control configuration.

20. The method according to claim 19, further, A method comprising mapping the inputs of a user interface of an active medical device, which is a plurality of medical devices or a camera device, such that the inputs of the user interface are mapped to receive confirmation of the proposed control configuration in response to the activation of the inputs.

21. The method according to claim 19, wherein, in response to the input to the user interface associated with the confirmation, the user interface communicates the confirmation of the proposed control configuration to the communication bus.

22. A surgical control system for multiple surgical devices, A plurality of device controllers communicating with each of the plurality of surgical devices via a communication bus, each of the device controllers independently reports the device data from one of the plurality of surgical devices, monitors the device data communicated as a message from each of the plurality of surgical devices, and the device data indicates at least one of the control state and detected state associated with the operation of each of the plurality of surgical devices, A camera device that communicates with the plurality of medical devices via the communication bus, comprising at least one controller, the at least one controller configured to capture image data indicating a surgical site, to identify the surgical state of the surgical site in response to the image data, and the surgical state being reported by the camera device via the communication bus, Includes, A surgical control system in which each of the plurality of device controllers independently controls the operation of the plurality of surgical devices in response to at least one of the device data and the surgical state.

Images