System and method for audio signal arrangement and projection

Abstract

Systems and methods for projecting audio signals to an operator of a teleoperated surgical system to convey a spatial orientation associated with the audio signals to the operator are disclosed. Characteristics of the audio signals, such as direction and volume, can be selected to give the teleoperator the impression of being locally positioned next to a patient. Characteristics of the audio signals can also be modified to provide a spatial transition between an actual location of an audio source within a physical audio environment and a simulated location of the audio source within a virtual audio environment.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application 62 / 889,086, filed August 20, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure relates to systems for arranging and / or projecting audio signals, and more specifically to systems having intelligent audio arrangement and / or projection. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during invasive medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, clinicians can insert medical instruments to reach target tissue locations. Minimally invasive medical instruments include devices such as therapeutic instruments, diagnostic instruments, and surgical instruments. They can also include imaging instruments, such as endoscopic instruments. Imaging instruments provide the user with a field of view within the patient's anatomy. Some minimally invasive medical instruments and imaging instruments can be remotely operated or computer-assisted. Examples of medical teleoperations include the da... Surgical systems and da S TM Surgical systems. Each of these systems includes a surgeon's console, a manipulator assembly, a high-performance three-dimensional (3-D) vision system, and one or more medical devices coupled to the manipulator assembly.

[0005] During medical procedures, communication may occur between the operator of a remote operating system and someone near the patient. For example, the operator may wish to send instructions to a healthcare professional near the patient, and / or a healthcare professional near the patient may wish to send instructions to the operator. Therefore, robust communication capabilities can lead to safer, more efficient, and overall more successful clinical outcomes with remote operating systems.

[0006] Therefore, it would be advantageous to provide a system that offers improved communication during surgical procedures. Summary of the Invention

[0007] The embodiments of the present invention are best summarized by the claims appended to the specification.

[0008] In one aspect of the invention, an audio system includes one or more audio sensors, a tracking system, an audio reproduction system, and an audio placement controller coupled to the one or more audio sensors, the tracking system, and the audio reproduction system. The audio placement controller can be configured to perform operations including: receiving an audio signal at the one or more audio sensors; receiving tracking data from the tracking system associated with the actual location of the source of the audio signal in a physical audio environment; determining a simulated location of the audio signal in a virtual audio environment; and reproducing the audio signal via the audio reproduction system. At least one characteristic (e.g., volume or direction) of the audio signal reproduced in the virtual audio environment can be determined based on the relationship between the actual location and the simulated location.

[0009] In one embodiment, one or more audio sensors (e.g., microphones) may be configured to detect audio signals near the patient in the medical system. These audio signals may correspond to verbal communication between the patient or persons in the vicinity. Alternatively, the audio signal may correspond to the audio output of a medical monitoring device (e.g., a heart rate monitor, blood pressure monitor, blood oxygen sensor, etc.). Furthermore, the audio signal may correspond to the audio output of the medical system.

[0010] In some embodiments, the tracking data may correspond to a tracking device worn by the audio signal source, or it may be derived from the audio signal. For example, audio sensors may be used to determine the location of the source based on the volume of the audio received at each respective sensor, or based on the time difference of the audio arriving at each audio sensor via triangulation.

[0011] In some embodiments, the simulated location of the audio signal may represent the actual location of the source. For example, if the actual location of the source is 5 feet directly to the left of the endoscope used as a reference point, the simulated location might be 5 feet directly to the left of the operator. In other embodiments, the simulated location of the audio signal may differ from the actual location based on tracking data. The simulated location can be determined based on one or more properties of the audio signal. For example, the simulated location can be determined based on determining that the audio signal is associated with an emergency message or is being conveyed to a listener of an audio system. This determination can be made at least in part based on the name or keywords stated in the audio or the volume of the audio received at the audio sensor. In some cases, artificial neural networks can be used to determine the simulated location.

[0012] Audio reproduction systems may include stereo systems, surround sound systems, or headphones worn by the listener of the audio system.

[0013] The audio system may also include a synthesized audio source. The audio placement controller may be further configured to receive synthesized audio signals from the synthesized audio source, determine a second analog position of the synthesized audio signal within the virtual audio environment, and emit synthesized audio signals with characteristics based on this second analog position within the virtual audio environment via an audio reproduction system. The synthesized audio signals may correspond to an audio representation of a patient's physiological processes. This allows an operator to auditorily monitor the physiological processes.

[0014] On the other hand, the audio system may include memory and one or more processors coupled to the memory. The one or more processors may be configured to read instructions from the memory and perform operations including: receiving an audio signal detected near a patient during a medical procedure; receiving tracking information associated with the source of the audio signal; determining the simulated location of the audio signal in a virtual audio environment; and reproducing the audio signal to a remote listener via an audio reproduction system. The audio reproduction system may provide characteristics (e.g., volume, pitch, direction, playback speed, etc.) for the audio signal based on the spatial relationship between the remote listener and the simulated location in the virtual audio environment.

[0015] In another aspect, a method may include receiving an audio signal detected near a patient during a medical procedure performed using a medical system, receiving tracking data associated with the source of the audio signal, determining the simulated location of the audio signal in a virtual audio environment, and reproducing the audio signal to an operator of the medical system via an audio reproduction system. At least one characteristic of the reproduced audio signal may be based on its simulated location in the virtual audio environment.

[0016] In another aspect, a method includes receiving an audio signal from an operator of at least one sensor of a medical system, identifying a target associated with the audio signal located near a patient associated with the medical system, receiving tracking data associated with the target, and reproducing the audio signal near the patient via an audio reproduction system. The audio reproduction system may focus the reproduced audio signal toward the target based on the tracking data.

[0017] In another aspect of this disclosure, an audio system includes at least one speaker, a synthesized audio source, and an audio placement controller coupled to the synthesized audio source. The audio placement controller can be configured to perform operations including: receiving a synthesized audio signal from the synthesized audio source; receiving position data associated with the synthesized audio signal; determining a simulated position of the synthesized audio signal in a virtual audio environment based on the position data; and emitting the synthesized audio signal via at least one speaker. At least one characteristic of the audio signal emitted in the virtual audio environment can be determined based on the position information.

[0018] In some embodiments, the synthesized audio signal may correspond to a patient's physiological processes. These processes may include at least one of respiration, heartbeat, blood flow, or any other suitable process. In other embodiments, the synthesized audio signal may correspond to a region of interest within the patient's anatomy. The simulated position may correspond to the spatial relationship between the field of view displayed to the operator and the actual location of the region of interest.

[0019] In some embodiments, the audio system may also include an imaging system. A region of interest can be detected by the imaging system. The imaging system may include a camera positioned within the patient's anatomical structures. Alternatively, the imaging system may include an ultrasound probe, a CT scanner, an X-ray machine, or an MRI machine. A tracking system can register the spatial relationship between the field of view and a reference point of the imaging system. The tracking system can provide positional data to the audio placement controller.

[0020] In some embodiments, at least one speaker may be a directional speaker. At least one characteristic may include selection of the direction from which the synthesized audio is emitted from the directional speaker. At least one speaker may also include at least two speakers. At least one characteristic may include selection of at least one of the at least two speakers that emits the synthesized audio. This selection may be based on a first direction associated with a spatial relationship and a second direction associated with a second spatial relationship between the operator and at least one speaker. In some embodiments, it may be desirable that these spatial relationships are the same or similar.

[0021] The region of interest may include at least one of the following: the desired location of the instrument within the anatomical structure, the undesired location of the instrument within the anatomical structure, the identification of a foreign body, or bleeding.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the disclosure as described in the following detailed description will be apparent to those skilled in the art. Attached Figure Description

[0023] Figure 1 This is a simplified diagram of a remotely operated medical system according to an embodiment of the present disclosure.

[0024] Figure 2 This is a simplified diagram of an audio system with intelligent audio signal arrangement according to some embodiments.

[0025] Figure 3A This is a simplified diagram illustrating the transition between a physical audio environment and a virtual audio environment according to some embodiments.

[0026] Figure 3B Based on some embodiments at different time points Figure 3A A simplified diagram of the transition between physical and virtual audio environments.

[0027] Figure 4 This is a simplified diagram of an audio system with intelligent audio projection according to some embodiments.

[0028] Figure 5 This is a simplified diagram of a method for arranging audio signals according to some embodiments.

[0029] Figure 6 This is a simplified diagram of a method for projecting audio signals according to some embodiments.

[0030] Figure 7 This is a simplified diagram of an audio system with intelligent audio signal arrangement according to some embodiments.

[0031] Figure 8 This is a simplified diagram of a method for generating an audio signal according to some embodiments. Detailed Implementation

[0032] In the following description, specific details of some embodiments consistent with this disclosure are set forth. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize other elements within the scope and spirit of this disclosure, although not specifically described herein. Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0033] In some cases, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring various aspects of the embodiments.

[0034] The following examples will describe various instruments and parts thereof according to their state in three-dimensional space. As used herein, the term “position” refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom, which can be described using variations in Cartesian X, Y, and Z coordinates, such as along the Cartesian X, Y, and Z axes). As used herein, the term “orientation” refers to the rotational arrangement of an object or part of an object (three rotational degrees of freedom—e.g., described using roll, pitch, and yaw). As used herein, the term “attitude” refers to the position of an object or part of an object in at least one translational degree of freedom, and the orientation of the object or part of the object in at least one rotational degree of freedom. For an asymmetric rigid body in three-dimensional space, a complete attitude can be described using a total of six degrees of freedom.

[0035] Furthermore, while some examples described herein relate to surgical procedures or tools, or medical procedures and tools, the disclosed techniques are applicable to non-medical procedures and tools. For example, the tools, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general robotic applications, and sensing or manipulating non-tissue workpieces. Other example applications relate to appearance enhancement, imaging of human or animal anatomy, collecting data from human or animal anatomy, setting up or dismantling systems, and training medical or non-medical personnel. Additional example applications include procedures performed on tissue removed from human or animal anatomy (without returning the human or animal anatomy), and applications for performing procedures on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical, medical treatment, or diagnostic procedures.

[0036] Figure 1 This is a simplified diagram of a remotely operated medical system 100 according to some embodiments. In some embodiments, the remotely operated medical system 100 may be suitable for, for example, surgical, diagnostic, therapeutic, or biopsy procedures. Figure 1 As shown, medical system 100 typically includes a manipulator component 102 for operating medical device 104 to perform various procedures on patient P. The manipulator component may be remotely operated, or may include both remotely operated sub-components and non-remotely operated sub-components for manual, robotic, and / or remote operation control of medical device 104. Manipulator component 102 is mounted to or near operating table T. An operator input system (e.g., master control component 106) allows operator O (e.g., as...) Figure 1 The surgeon, clinician, or physician shown examines the intervention site and controls the manipulator assembly 102.

[0037] The master control component 106 may be located at the operator console, which is typically located in the same room as the operating table T, for example, to one side of the surgical table where the patient P is located. However, it should be understood that the operator O may be located in a different room or in a completely different building than the patient P. The master control component 106 typically includes one or more control devices for controlling the manipulator component 102. The control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual operation controllers, voice recognition devices, body motion or presence sensors, etc. To provide the operator O with a strong sense of direct control of the instrument 104, the control devices may be provided with the same degrees of freedom as the associated medical instrument 104. In this way, the control devices provide the operator O with a sense of remote presentation or integration of the control devices with the medical instrument 104.

[0038] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104 and still provide remote presentation to the operator O. In some embodiments, the control device may optionally be a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating the device (e.g., for closing a gripper, applying a potential to an electrode, delivering medication, etc.).

[0039] Manipulator assembly 102 supports medical device 104 and may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, commonly referred to as a setup structure) and a remotely operated manipulator. Manipulator assembly 102, or more specifically, the remotely operated manipulator, may optionally include multiple actuators or motors that drive inputs on medical device 104 in response to commands from a control system (e.g., control system 112). Actuators may optionally include drive systems that, when coupled to medical device 104, can advance medical device 104 into an anatomical opening created naturally or surgically. Other drive systems can move the distal end of medical device 104 in multiple degrees of freedom, including three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian axes). Furthermore, actuators may be used to actuate articulated end effectors of medical device 104 for grasping tissue in the jaws of biopsy equipment, etc. Actuator positioning sensors (such as resolvers, encoders, potentiometers, and other mechanisms) can provide medical system 100 with sensor data describing the rotation and orientation of the motor shaft. This positioning sensor data can be used to determine the motion of an object manipulated by the actuator.

[0040] The remotely operated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instrument of the manipulator assembly 102. Such subsystems may include: a positioning / position sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining positioning, orientation, velocity, rate, attitude, and / or shape along a distal end and / or one or more segments that may constitute a flexible body of the medical device 104; and / or a visualization system for capturing images from the distal end of the medical device 104.

[0041] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical device 104 generated by a subsystem of the sensor system 108. The display system 110 and the master control component 106 can be oriented so that an operator O can control the medical device 104 and the master control component 106 through remotely presented perception.

[0042] In some embodiments, medical device 104 may have a visualization system that may include a viewing endoscope assembly that records concurrent or real-time images of the surgical site and provides the images to an operator or operator O via one or more displays of medical system 100 (e.g., one or more displays of display system 110). The concurrent images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with medical device 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include a processor of control system 112. The processor of control system 112 may execute instructions corresponding to the methods and operators described herein.

[0043] Display system 110 can also display images of surgical sites and medical instruments captured by a visualization system. In some examples, remote-operated medical system 100 can configure controls for medical instrument 104 and master control component 106 such that the relative positioning of the medical instrument is analogous to the relative positioning of the operator O's eyes and hands. In this way, operator O can manipulate medical instrument 104 with hand controls as if viewing a physically present workspace. Physical presence means that the image presentation is a realistic perspective image simulating a viewpoint, as if the operator were physically manipulating medical instrument 104 with their hands rather than through remote-operated medical system 100.

[0044] In some examples, the display system 110 can present images of the surgical site recorded preoperatively or intraoperatively using image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The preoperative or intraoperative image data can be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or rate-based information) and / or as images from models created based on the preoperative or intraoperative image dataset.

[0045] In some embodiments, typically for the purpose of image-guided surgical procedures, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered (i.e., dynamically referenced) with preoperative or concurrent images / models. This can be done to present a virtual image of the internal surgical site to the operator O from the viewpoint of the medical device 104. In some examples, this viewpoint may be from the tip of the medical device 104. Images and / or other graphic or alphanumeric indicators of the tip of the medical device 104 may be overlaid on the virtual image to assist the operator O in controlling the medical device 104. In some examples, the medical device 104 may not be visible in the virtual image.

[0046] In some embodiments, display system 110 may display a virtual navigation image in which the actual position of medical device 104 is registered with preoperative or concurrent images to present a virtual image of medical device 104 within the surgical site to operator O from an external viewpoint. An image of a portion of medical device 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist operator O in controlling medical device 104. As described herein, a visual representation of data points may be rendered onto display system 110. For example, measured data points, moving data points, registered data points, and other data points described herein may be displayed visually on display system 110. Data points may be visually represented in the user interface as multiple points or spots on display system 110 or as a rendering model (e.g., a grid or line model created based on the data point set). In some examples, data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed in display system 110 after each processing operation to change the data points has been performed. In some embodiments, a virtual navigation image may be presented on the display 110, which depicts a model of the anatomical pathway from the perspective of an instrument inserted along or through the corresponding actual anatomical pathway.

[0047] The remote-operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control component 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., these instructions stored on a non-transitory machine-readable medium) for implementing some or all of the methods described according to the aspects disclosed herein, including instructions for providing information to the display system 110. While the control system 112... Figure 1 The simplified schematic is shown as a single block, but the system may include two or more data processing circuits, wherein a portion of the processing may optionally be executed on or near manipulator component 102, another portion of the processing may be executed at master component 106, and so on. The processor of control system 112 may execute instructions that correspond to the processes disclosed herein and described in more detail below. Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programming instructions may be implemented as multiple independent programs or subroutines, or they may be integrated into multiple other aspects of the remote operating system described herein. In one embodiment, control system 112 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0048] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to this feedback, the control system 112 may transmit a signal to the master control component 106. In some examples, the control system 112 may transmit a signal instructing one or more actuators of the manipulator component 102 to move the medical device 104. The medical device 104 may extend through an opening in the patient P's body to an internal surgical site within the patient P's body. Any suitable conventional and / or specialized actuators may be used. In some examples, one or more actuators may be detached from or integrated with the manipulator component 102. In some embodiments, one or more actuators and the manipulator component 102 are provided as part of a remotely operated trolley positioned adjacent to the patient P and the operating table T.

[0049] The control system 112 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of the acquired anatomical pathway. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. Software, which can be used in combination with manual input, is used to convert the recorded images into a segmented two-dimensional or three-dimensional composite representation of a part or the entire anatomical organ or region. The image dataset is associated with this composite representation. The composite representation and the image dataset describe the various locations and shapes of the pathway and their connectivity. Images used to generate the composite representation may be recorded preoperatively or intraoperatively during the clinical procedure. In some embodiments, the virtual visualization system may use a standard representation (i.e., not patient-specific) or a mixture of a standard representation and patient-specific data. Composite representations and any virtual images generated from composite representations can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0050] During the virtual navigation procedure, sensor system 108 can be used to calculate the approximate position of medical device 104 relative to the anatomy of patient P. This position can be used to generate macroscopic (external) tracking images of the anatomy of patient P and virtual internal images of the anatomy of patient P. The system may implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display surgical images recorded during the medical procedure and preoperatively, such as those from a virtual visualization system. For example, PCT Publication WO 2016 / 191298 (published December 1, 2016) (disclosing “Systems and Methods of Registration for Image Guided Surgery”) discloses such a system, which is incorporated herein by reference in its entirety. The remotely operated medical system 100 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems, and / or suction systems. In some embodiments, the remotely operated medical system 100 may include more than one manipulator component and / or more than one master control component. The exact number of manipulator components will depend on other factors such as the surgical procedure and space constraints within the operating room. The master control components 106 can be placed side-by-side or they can be positioned separately. Multiple master control components allow more than one operator to control one or more manipulator components in various combinations.

[0051] Audio communication is typically a useful mechanism for transmitting information between a patient (P) and an operator (O) during surgical procedures. Therefore, a medical system (e.g., a remote-operated medical system 100) can detect audio signals from the vicinity of the patient (P) and reproduce these signals for the operator (O). However, multiple audio sources may be present near the patient (P) during a surgical procedure, making it difficult for the operator (O) to distinguish the audio sources and / or understand the context of the audio signals. For example, when multiple medical personnel are near the patient (P), the operator (O) may have difficulty determining who is speaking, who they are talking to, etc. Similarly, the operator (O) may be visually immersed in the operator console and may lack visual cues to accompany the audio signals provided through speakers at the console.

[0052] Therefore, when reproducing audio signals for operator O, it is generally desirable to detect and transmit the spatial relationships between various audio sources. Accurately and / or realistically reconstructing spatial relationships (i.e., recreating each audio signal from its physical location directly corresponding to the audio source) is one possible approach, but further enhancements are possible. For example, once the locations of the various audio sources are determined, each audio source can be mapped to an analog location that may or may not match the physical location of the audio source. In this way, it is possible to artificially alter the perceived location of high-priority audio signals that are close to the listener (e.g., emergency messages and / or audio sent directly to operator O), and similarly, to position low-priority audio signals (e.g., conversations by medical personnel) further away from the listener.

[0053] Figure 2 This is a simplified diagram of an audio system 200 with intelligent audio signal arrangement according to some embodiments. In some embodiments, the audio system 200 may be associated with and / or incorporated into a medical system 210 (e.g., a remotely operated medical system 100). Consistent with such embodiments, the medical system 210 may include a patient P, an operating table T, a manipulator assembly 211, and one or more medical devices 212 and / or be associated with the patient P, operating table T, manipulator assembly 211, and one or more medical devices 212, which typically correspond to... Figure 1 Features similar to those marked in the text. For example... Figure 2 As depicted, the patient P, operating table T, manipulator assembly 211, and medical device 212 are located near the patient within the patient environment or patient reference frame 222. In illustrative embodiments, the patient environment 222 may correspond to an operating room. In some embodiments, various personnel 213a-n (e.g., physicians, surgeons, nurses, etc.) may be located within the patient environment 222 when performing surgical procedures.

[0054] In some embodiments, the medical system 210 may further include an operator O, a main control component 214, and a display system 215 and / or be associated with the operator O, the main control component 214, and the display system 215, which typically correspond to... Figure 1 Features similar to those marked in the text. One of the medical devices 212 may have a visualization system that may include a viewing endoscope assembly (e.g., an endoscope) that records concurrent or real-time images of the surgical field of view and provides the image of the field of view to the operator or operator O via a display 215. The image of the field of view on the display 215 helps the operator O to have a spatial understanding of the patient environment 222.

[0055] like Figure 2 As depicted, the operator 214, the main control component 214, and the display system 215 are located in the operator environment or operator reference frame 224. In the illustrative embodiment, the operator environment 224 may correspond to a room physically separate from the patient environment 222. However, it should be understood that the operator environment 224 and the patient environment 222 may be located in the same room and / or may overlap with each other.

[0056] In some embodiments, the audio system 200 may include one or more audio sensors 230 located in and / or near the patient environment 222. Typically, the audio sensors 230 detect various audio signals emitted from and / or audible within the patient environment 222. Exemplary audio signals include speech between persons 213a-n, speech between persons 213a-n and patient P, sounds associated with monitoring equipment (e.g., the audio output of a heart rate monitor), ambient noise, etc. In some embodiments, the audio sensors 230 may have a fixed location (e.g., a microphone mounted on a wall, operating table T, manipulator assembly 211, etc.) and / or a dynamic location (e.g., a microphone clipped onto persons 213a-n and / or patient P) within the patient environment 222.

[0057] The audio system 200 may also include a tracking system 240 located in and / or near the patient environment 222. In some embodiments, the tracking system 240 may determine the location of various audio sources within the patient environment 222. For example, the tracking system 240 may collect location information associated with persons 213a-n, patient P, monitoring equipment, and / or any other source of audio picked up by the audio sensors 230. In some embodiments, the tracking system 240 may use RFID-based tracking, wherein RFID tags (and / or any other suitable tracking device) are used to track persons 213a-n. In some embodiments, the tracking system 240 may include one or more cameras and / or image processors to use image processing techniques to locate audio sources. In some embodiments, the tracking system 240 may perform audio signal processing to locate audio sources based on audio signals detected by the audio sensors 230 (e.g., using triangulation techniques based on the time of arrival or relative intensity of audio signals detected by multiple audio sensors 230).

[0058] The audio system 200 may further include an audio placement controller 250, which receives audio signal data from an audio sensor 230 and / or tracking data from a tracking system 240. In some embodiments, the audio placement controller 250 may include one or more processors to process the received audio signals and tracking data. According to some embodiments, the audio placement controller 250 may map the received audio signals to a virtual audio environment based on the tracking data. Typically, the virtual audio environment is an audio environment provided to an operator located in an operator environment 224, in which the perceived location and / or directionality of emitting audio signals can be arranged to correspond to the operator's field of view in the patient environment 222. More specifically, the audio placement controller 250 may place each audio source at a simulated location within the virtual audio environment based on the actual and / or desired spatial positioning of various audio sources relative to a listener (e.g., an operator).

[0059] In some embodiments, the audio placement controller 250 may perform a direct mapping between the physical location of an audio source in the patient environment 222 (as indicated by tracking data) and the simulated location of the audio source in the virtual audio environment. That is, the physical location and the simulated location may remain constant and correspond to the same location. However, in some embodiments, the simulated location may differ from the physical location. In some embodiments, the audio placement controller 250 may change the placement of the audio source in the virtual audio environment (compared to the physical location) based on one or more properties of the audio signal. For example, the audio placement controller 250 may determine that one of the persons 213a-n is speaking with a particular urgency (e.g., based on the speaker's loudness, pitch, tone, etc.). Therefore, the audio placement controller 250 may place the speaker in a simulated location in the virtual audio environment close to the listener to ensure that urgent messages receive appropriate attention.

[0060] In addition to arranging the audio source within a virtual audio environment, the audio arrangement controller 250 can set and / or adjust other properties of the audio signal. For example, the audio arrangement controller 250 can adjust the frequency and / or playback speed of the audio signal based on tracking data and / or other properties of the audio signal. In an illustrative embodiment, when tracking data indicates that the audio source is in motion, the audio arrangement controller 250 can adjust the frequency of the corresponding audio signal to enhance (and / or reduce) the Doppler effect caused by motion.

[0061] In some embodiments, the audio arrangement controller 250 may receive synthetic audio signals from one or more synthetic audio sources 260. In some embodiments, the synthetic audio signals may not correspond to the actual audio signals picked up by the audio sensor 230, but may be artificially generated. For example, the synthetic audio signals may correspond to audio representations of physiological processes (e.g., heartbeat, breathing, etc.), alarms, notifications, etc.

[0062] In some embodiments, in addition to the audio sensor 230 and / or the synthesized audio source 260, the audio placement controller 250 may also receive audio signals from various other sources. For example, the audio placement controller 250 may receive audio signals from remote personnel ( Figure 2 (Not shown) It receives verbal communications and can place remote personnel in a virtual audio environment. Therefore, the audio placement controller 250 can combine audio signals from multiple sources and / or channels and map them to desired locations within the virtual audio environment.

[0063] The audio system 200 also includes an audio reproduction system 270 located in and / or near the operator environment 224. The audio reproduction system 270 receives output signals from the audio placement controller 250 and generates audio outputs having desired spatial and / or directional characteristics corresponding to the virtual audio environment. In some embodiments, the audio reproduction system 270 may include a stereo system, a surround sound system, headphones, etc. In this way, the audio reproduction system 270 can provide the operator O with the impression that each audio signal in the virtual audio environment originates from a simulated location within the virtual audio environment.

[0064] Figure 3A This is a simplified diagram illustrating the transition between a physical audio environment 310 (e.g., patient environment 222) and a virtual audio environment 320 according to some embodiments. Figure 2 In some consistent examples, the conversion can be performed using a controller that receives audio signal data and tracking data (e.g., audio layout controller 250). Although the physical audio environment 310 and the virtual audio environment 320 are represented in two dimensions in Figure 3, it should be understood that the location of the audio source may additionally or alternatively be represented in one and / or three dimensions.

[0065] like Figure 3A As described, the physical audio environment 310 includes three audio sources 331-333 located at different positions relative to the central position 315. Typically, the central position 315 can correspond to any point, but it can also correspond to a defined location (e.g., the center of the operating room, the patient's position, the position of medical instruments, the position of an imaging system establishing the operator's field of view, etc.). Audio sources 331-333 can actually include any audio source that may be present in a surgical setting, such as verbal communication between medical personnel (e.g., personnel 213a-n), patient monitoring equipment, ambient noise, etc.

[0066] The virtual audio environment 320 includes three audio sources 331'-333' corresponding to audio sources 331-333. In some embodiments, the virtual audio environment 320 may include one or more synthesized audio sources 340 that do not correspond to any of the audio sources 331-333. As previously referenced Figure 2 The synthesized audio source 340 discussed here can correspond to physiological processes (e.g., heartbeat, breathing, etc.), alarms, etc. Display systems (e.g., display system 215) can be used to transmit large amounts of statistical data, status, configurations, etc., to the operator, potentially making the display cluttered with information. Such information can be difficult for the operator to process while maintaining focus on operating the equipment. In this regard, at least to reduce the excessive distracting information provided by display system 215, it may be advantageous to provide some information to the operator auditorily rather than visually. The synthesized audio source 340 can provide such auditory information.

[0067] Audio sources 331'-333' and 340 are arranged at simulated positions relative to listener position 325. In some embodiments, listener position 325 may correspond to the position of the operator's field of view. Since listener position 325 can be positioned at the center of virtual audio environment 320, a listener located at listener position 325 (e.g., operator O) will perceive that the audio signals from audio sources 331'-333' and 340 originate from their respective positions in virtual audio environment 320.

[0068] like Figure 3A As depicted, the positions of audio sources 331'-333' may or may not match the positions of audio sources 331-333 in the physical audio environment 310. For example, audio source 331' may be positioned in the virtual audio environment 320 at approximately the same location as in the physical audio environment 310. As another example, audio source 332' may be significantly more centered in the virtual audio environment 320 than its corresponding counterpart in the physical audio environment 310, while audio source 333' may be farther from the center in the virtual audio environment 320 than in the physical audio environment 310. These virtual arrangements can be accomplished, for example, by increasing the volume of the audio from audio source 332, reducing noise in the signal from audio source 332, altering the pitch, velocity, or other audio characteristics of the audio from audio source 332, and / or decreasing the volume of other audio sources. There may be a variety of reasons for artificially moving audio source 332' closer to the listener in the virtual audio environment 320. For example, when audio source 332 might correspond to a member of a medical team speaking with unusual urgency. In another example, a member of the medical team may state keywords and / or key phrases (e.g., the operator's name) indicating that he or she wishes to speak directly to operator O. Therefore, positioning the audio source 332' in a centered position close to the listener may help convey to operator O the impression that a team member is speaking directly to operator O.

[0069] Furthermore, it may be desirable to transform the dynamic physical location of audio sources 331-333 into the static virtual location of audio sources 331'-333'. In other words, personnel or equipment may move around the physical audio environment 310 during surgical procedures (e.g., at a specific time point). Figure 3A and at different times Figure 3B (The differences between them are shown), but the operator O may not be aware of their movement because the operator's attention is focused on the display system. In order to maintain the understanding of who is speaking based on directional cues in the virtual audio environment 320, the virtual positions of the audio sources 331'-333' in the virtual audio environment 320 can remain stationary, while the corresponding actual audio sources 331-333 move around the room, as indicated by their positions in the virtual audio environment 320. Figure 3Aand Figure 3B The position changes as shown. That is, even when the audio source 332 (which could be, for example, a nurse) moves from the left to the right of the center position 315 (e.g., the manipulator component), the sound from the audio source can be directed from a common angle to the operator (located at the center position 325 of the virtual audio environment) (e.g., from a speaker only to the operator's left) to create the impression that the audio source 332 is stationary to the operator's left. This allows the operator to distinguish the audio feedback from that particular audio source 332 from the audio feedback from other audio sources 331 and 333.

[0070] A variety of techniques can be used to generate virtual audio environments by mimicking and / or modifying certain properties of the physical audio environment. For example, to simulate a physical audio environment in a way that gives the operator the impression that they are at the center of the physical audio environment, the direction and volume of various sounds (alarms, voices, tools, etc.) in the physical audio environment can be detected and their properties similar to those of the sounds occurring at the operator's simulated location (e.g., the center of the physical audio environment) can be recreated in the virtual audio environment. By replicating the properties of the various sounds in a similar or identical manner to those sounds occurring at the simulated location, the operator can receive audio information as if they were actually present at the simulated location, thus creating the impression that the operator is present at that simulated location (e.g., standing next to a patient or at the location of an imaging system that generates the operator's field of view).

[0071] Additionally, the nature of the sound can be modified within a virtual audio environment to give the operator a different impression than that actually occurring at the simulated location. In other words, it is possible to use... Figure 2 The tracking system 240, audio placement controller 250, and synthesized audio source 260 modify the sound provided to the operator to enhance or supplement the real-world audio. Contemporaneous techniques for modifying sound in the virtual audio environment include, for example, increasing volume and / or changing the direction of one or more sounds as described above. In this regard, for example, the virtual location of a nurse could be brought closer to the operator by increasing the volume of the nurse's voice, whereas in a physical audio environment, the nurse might be far away from the operator, making the nurse difficult to hear. In contrast, the volume of other sounds might be reduced, such as conversations between medical students or other observers in a corridor. In this regard, the audio system could determine that observers are not active participants in the operation based on their location in the room, or that their comments are unnecessary based on the volume of their speech. Therefore, the audio system could completely filter out their voices from the virtual audio environment to avoid distracting the operator. As another anticipated technique, the volume of ambient noise (e.g., the operation of tools and equipment, such as ventilators) could be reduced.

[0072] Another anticipated technique for modifying sound in a virtual audio environment includes reducing echo or reverberation. Operating rooms are typically characterized by smooth, rigid surfaces such as tiled floors and stainless steel tables. These surfaces can reflect sound waves, creating background noise that can distract the operator and potentially interfere with other, more important sounds. Therefore, when generating an audio signal for the virtual audio environment 320, the audio system 200 can reduce the volume of reflected sound waves from the physical audio environment 310 or eliminate reflected sound waves by filtering out repetitive waveforms associated with reflections and reverberation.

[0073] Another anticipated technique involves altering the content of the speech. In other words, certain phrases or comments that were stated by a person in the physical audio environment 310 and determined to be unimportant can be omitted from the audio generated in the virtual audio environment 320. For example, when a member of a medical team makes a statement addressed to another team member rather than the operator (e.g., by instructing by beginning the statement with the team member's name), that portion of the audio may not be generated in the virtual audio environment 320. Instead, the portion of the team member's statement beginning with the operator's name can be sent to the virtual audio environment 320. Of course, various combinations of these and other anticipated techniques can be used to achieve the desired virtual audio environment 320.

[0074] Figure 4 This is a simplified diagram of an audio system 400 with intelligent audio projection according to some embodiments. Similar to audio system 200, audio system 400 addresses the need for improved audio communication between the patient's vicinity (P) and the operator (O). Unlike audio system 200, in which the operator (O) is the listener, audio system 400 is configured for scenarios where the operator (O) is the speaker. Although audio system 200 and audio system 400 are described as separate systems for clarity, it should be understood that audio system 200 and audio system 400 can be combined to provide bidirectional audio communication between the patient's vicinity (P) and the operator (O).

[0075] Similar to audio system 200, audio system 400 may be associated with and / or incorporated into medical system 410 (e.g., remotely operated medical system 100). Consistent with such embodiments, medical system 410 may include a patient P, operating table T, manipulator assembly 411, and medical device 412 and / or be associated with them, the patient P, operating table T, manipulator assembly 411, and medical device 412 typically corresponding to Figure 1 Features similar to those marked in the text. For example... Figure 4As depicted, the patient P, operating table T, manipulator assembly 411, and medical device 412 are located in the patient's periphery or environment 422. In illustrative embodiments, the patient environment 422 may correspond to an operating room. In some embodiments, various personnel 413a-n (e.g., physicians, surgeons, nurses, etc.) may be located within the patient environment 422 when performing surgical procedures.

[0076] In some embodiments, the medical system 410 may further include an operator O, a main control component 414, and a display system 415 and / or associated therewith, wherein the operator O, the main control component 414, and the display system 415 typically correspond to Figure 1 Features similar to those marked in the text. For example... Figure 4 As depicted, the operator 414, main control component 414, and display system 415 are located in the operator environment 424. In the illustrative embodiment, the operator environment 424 may correspond to a room physically separate from the patient environment 422. However, it should be understood that the operator environment 424 and the patient environment 422 may be located in the same room and / or may overlap with each other.

[0077] In some embodiments, the audio system 400 may include an audio sensor 430 located in and / or near the operator's periphery 420. Typically, the audio sensor 430 is configured to detect verbal communication from the operator O. For example, the audio sensor 430 may include a microphone clipped to the operator O, mounted and / or built into the main control component 414 and / or the display system 415, etc.

[0078] In some embodiments, operator O may wish to speak to one or more targets in patient environment 422. For example, operator O may wish to speak to a specific person among persons 413a-n and / or may wish to speak to patient P. Therefore, audio system 400 may include target recognizer 440 that identifies one or more targets that operator O wishes to converse with. In some examples, one or more targets may be manually identified by operator O (e.g., by making a selection via master component 414). In some examples, one or more targets may be identified based on one or more keywords and / or key phrases spoken by operator O (e.g., by stating the target's name). In some examples, one or more targets may be automatically identified (e.g., by determining possible targets based on the content and / or topic of the speech). In some embodiments, one or more audio sensors 430 may be used to determine targets based on the direction in which operator O projects spoken audio. For example, volume detected at multiple audio sensors 430 may be used to identify targets based on the location of persons 413a-n within the virtual audio environment.

[0079] The audio system 400 may also include a tracking system 450 located in and / or near the patient environment 422. In some embodiments, the tracking system 450 may determine the location of potential targets within the patient environment 422. For example, the tracking system 450 may collect location information associated with persons 413a-n and / or patient P. In some embodiments, the tracking system 450 may use RFID-based tracking, wherein RFID tags (and / or any other suitable tracking device) are used to track persons 413a-n. In some embodiments, the tracking system 450 may include one or more cameras and / or image processors to use image processing techniques to locate potential targets.

[0080] The audio system 400 may further include an audio projection controller 460 that receives audio signal data from an audio sensor 430, target data from a target recognizer 440, and / or target tracking data from a tracking system 450. In some embodiments, the audio projection controller 460 may include one or more processors to process the received audio signal data, target data, and target tracking data. According to some embodiments, the audio projection controller 460 may determine a projection profile of the audio signal based on the target data and / or target tracking data. In particular, the projection profile may identify spatial variations in the volume, frequency, and / or other attributes of the audio signal so that the audio signal reaches a desired target.

[0081] The audio system 400 also includes an audio reproduction system 470 located in and / or near the patient environment 422. The audio reproduction system 470 receives an output signal from the audio projection controller 460 and reproduces an audio output having desired spatial and / or directional characteristics corresponding to the projection profile. In some embodiments, the directional audio reproduction system 470 may include a directional speaker system, headphones worn by personnel 413a-n, a set of speakers arranged at different locations within the patient environment 422, etc. In this way, the audio reproduction system 470 allows the operator O to communicate with a specific target within the patient environment 222 without causing auditory interference to non-targets.

[0082] Figure 5 This is a simplified diagram of a method 500 for arranging audio signals according to some embodiments. According to... Figure 2 In some consistent embodiments, method 500 may be performed by a controller of an audio system (e.g., audio placement controller 250) during a medical procedure. In some embodiments, method 500 may allow an operator of the medical system used in the medical procedure to listen to audio signals emanating from the patient's surroundings during the medical procedure.

[0083] At process 510, audio signals detected in the vicinity of the patient are received. In some embodiments, the audio signals may correspond to verbal communication from the patient during the medical procedure, verbal communication from persons in the vicinity of the patient, audio output from patient monitoring equipment (e.g., a heart rate monitor), audio output from the medical system, ambient noise, etc. In some embodiments, the audio signals may be received from microphones (and / or other suitable audio transducers) located in and / or near the patient (e.g., in the operating room with the patient).

[0084] At process 520, tracking data associated with an audio signal source is received. As previously described, the audio signal source may correspond to a patient, personnel located in the vicinity of the patient, patient monitoring equipment, medical systems, etc. In some embodiments, the tracking data may identify the relative and / or absolute location of these sources in the vicinity of the patient. In some embodiments, the tracking data may correspond to RFID tracking data based on RFID tags (and / or any other suitable tracking devices) worn by personnel in the vicinity of the patient. In some examples, this tracking data may be derived from the received audio signal, for example, by triangulation of the source's location based on the intensity of the audio signal.

[0085] At process 530, a simulated position of the audio signal in the virtual audio environment is determined. The simulated position of the audio signal may be relative to the listener's position in the virtual audio environment. In some embodiments, the simulated position may match the physical position of the audio signal indicated by tracking data. In other words, a direct mapping occurs between the physical position of the audio signal in the patient environment 222 (as indicated by the tracking data) and the simulated position of the audio source in the virtual audio environment. In some embodiments, the simulated position may not match the physical position of the audio signal indicated by the tracking data. In other words, a mapping of the tracking data is performed with respect to the physical position of the audio signal in the patient environment 222. The modified mapping creates a simulated position of the audio source in the virtual audio environment that differs from the physical position of the audio signal in the patient environment 222. In some embodiments, the simulated position of the audio source in the virtual audio environment is based on its spatial relationship to the listener's position. For example, the simulated position may be determined based on one or more attributes of the audio signal (e.g., pitch, content of spoken communication, keywords and / or key phrases included in spoken communication, etc.). In this way, high-priority audio signals (e.g., emergency messages, messages sent directly to the listener, etc.) can be artificially positioned closer to the listener, while low-priority audio signals (e.g., ambient noise, conversations not involving the listener, etc.) can be artificially positioned further away from the listener. In some examples, machine learning techniques can be used to determine the simulated position. For example, artificial neural networks can be developed and trained to predict the desired simulated position of the audio signal based on tracking data, the properties of the audio signal, and / or various other factors that may affect the desired simulated position. In addition to determining the simulated position of the audio signal, other properties of the audio signal can be selected at process 530. For example, frequency, playback speed, and / or other properties can be adjusted. More generally, various audio properties and / or spatial properties of the audio signal can be manipulated as needed to create a desired audio impression on the listener.

[0086] At process 540, an audio signal is reproduced via an audio reproduction system to provide a listener (e.g., a remote listener located outside the patient's vicinity, such as an operator of a medical system) with the impression that the reproduced audio signal originates from a simulated location in the virtual audio environment determined at process 530. In some embodiments, the audio reproduction system may correspond to a stereo system, a surround sound system, headphones, and / or any other type of audio reproduction system capable of transmitting the spatial characteristics of the audio signal.

[0087] Figure 6 This is a simplified diagram of a method 600 for projecting audio signals according to some embodiments. According to... Figure 4In some consistent embodiments, method 600 may be performed by a controller of an audio system (e.g., audio projection controller 460) during a medical procedure. In some embodiments, method 600 may allow an operator of the medical system used in the medical procedure to transmit audio signals to target individuals and / or locations within the vicinity of the patient during the medical procedure.

[0088] At process 610, an audio signal is received from the operator of the medical system. In some examples, the audio signal may correspond to verbal communication received via a microphone located around and / or clipped to the operator. In some examples, the audio signal may be synthesized and / or may correspond to pre-recorded audio received in response to a selection made by the operator via an input interface (e.g., pressing a button to initiate playback of the audio signal).

[0089] At process 620, a target associated with the audio signal is identified. Typically, the target corresponds to one or more entities located in the vicinity of the patient. For example, the target may include the patient, people in the vicinity of the patient, etc. In some examples, the target may be determined based on manual input from the operator (e.g., selections made via an input interface), the content of the audio signal, keywords and / or key phrases included in the audio signal, etc.

[0090] At process 630, tracking data associated with the target is received. As previously described, the target of the audio signal may correspond to a patient, a person located in the vicinity of the patient, etc. In some embodiments, the tracking data may identify the relative and / or absolute position of the target within the vicinity of the patient. In some embodiments, the tracking data may correspond to RFID tracking data based on RFID tags (and / or any other suitable tracking devices) worn by persons in the vicinity of the patient. In some examples, this tracking data may be derived from sensor data collected in the vicinity of the patient (e.g., image and / or audio sensor data).

[0091] At process 640, an audio signal is reproduced around the patient via an audio reproduction system. The audio reproduction system directs or focuses the reproduced audio signal onto the target based on tracking data received at process 630. In some embodiments, the audio reproduction system may include a directional sound system, multiple speakers distributed around the patient, and / or any other audio reproduction system capable of reproducing the audio signal in a localized manner. In some embodiments, the audio signal may be reproduced at or near the target location at maximum volume, while the volume elsewhere may be reduced to mitigate noise pollution and / or non-target interference around the patient.

[0092] Go to Figure 7The diagram illustrates an audio system 700. An imaging system 790 (which may be a supplementary imaging system) can be used to capture images of the internal components of a patient's anatomy. For example, the imaging system 790 may be an endoscope, camera, ultrasound probe, CT scanner, MRI machine, and X-ray machine, or any other suitable device. While the imaging system 790 is capturing images, the operator O may be operating a manipulator assembly 711 (e.g., manipulator assembly 102) and more specifically, one or more instruments 712 (which may include, for example, imaging instruments, such as endoscopes or endoscope / tool ​​assemblies) via a master control assembly 714 and a display system 715 located in the operator environment 224. The imaging system 790 can be configured to identify areas of interest within the anatomy of the patient P within the patient environment 722. For example, excessive bleeding, identification of foreign bodies (e.g., surgical screws), or the proximity of instruments to specific anatomical features may be areas that the operator needs to be aware of. However, the endoscopic field of view from instrument 712 presented to operator O on display system 715 may not record or capture the region of interest because the event or feature is outside the endoscopic field of view. Therefore, imaging system 790 may be operable to detect the region of interest independently of operator O. The position and orientation of instrument field of view 791 relative to instrument system field of view 792 may be known. For example, tracking system 740 may determine, register, and track the spatial relationship between instrument 712 (and more specifically, in some cases, the field of view 791 of instrument 712) and imaging system 790. In some embodiments, tracking system 740 may be similar to tracking system 240. In some embodiments, the position of instrument field of view 791 relative to imaging system field of view 792 may be based on the fact that the kinematic relationship between instrument 712 and imaging system 790 is known. After identifying the region of interest, the tracking system 740 can determine the spatial relationship between the field of view 791 of the instrument 712 or the starting point of the field of view 791 (e.g., the far end of the imaging instrument) and the specific location of the region of interest within the field of view 792 of the imaging system.

[0093] Based on the spatial relationship between the region of interest (ROI) and the field of view 791, the audio placement controller 750 can determine the analog position of the synthesized audio signal from the synthesized audio source 760. The analog position can represent the ROI outside and relative to the field of view 791 (e.g., a patient bleeding area, a risky area). The synthesized audio source 760 can be an alarm generator or other components / software configured to generate an audio alarm in response to the identification of the ROI. In some embodiments, the synthesized audio source 760 can be part of the imaging system 790, the audio placement controller 750, and / or the tracking system 740. The audio placement controller 750 can use the analog position of the synthesized audio signal to select one or more speakers 770a, 770b of the audio generation system through which the synthesized audio signal is emitted. The selection of one or more speakers can include determining the volume at which the synthesized audio signal is played from the one or more speakers. For example, in a virtual audio environment (which may correspond to operator environment 724), an analog position directly to the left of the operator might cause the audio placement controller 750 to select only the 80dB left speaker 770a, while an analog position directly in front of the operator might cause the audio placement controller 750 to select both speakers 770a and 770b, both at 70dB. Analog positions between these two directions could result in the left speaker 770a playing the synthesized audio signal at 75dB, while the right speaker 770b plays the synthesized audio signal at 65dB.

[0094] exist Figure 7 In the illustrated embodiment, the region of interest detected in the field of view 792 of the imaging system 790 is primarily to the left of the field of view 791 of the instrument 712 (e.g., an endoscope). Therefore, the audio arrangement controller 750 can determine that a synthetic audio signal containing a warning to the operator should be emitted primarily from the left speaker 770a to warn or enhance the operator's awareness of the region of interest to the left of the field of view 791, which is being viewed by the operator O on the display system 715. This can prompt the operator O to manipulate the instrument 712 to look to the left and view the event or feature of interest.

[0095] It should be understood that the imaging system 790 may not include an endoscope, but may be a CT scanner, ultrasound probe, etc., and its images can be spatially registered relative to the instrument 712 via the tracking system 740. Furthermore, when the operator O manipulates the endoscope of the instrument 712 to overlap the field of view 791 with the field of view 792 of the imaging system 790, the display system 715 can overlay the anatomical images from the imaging system 790 with the camera view of the instrument 712.

[0096] Figure 8 This is a simplified diagram of a method 800 for generating audio signals according to some embodiments. According to... Figure 7In some consistent embodiments, method 800 may be performed by a controller of an audio system (e.g., audio placement controller 750) during a medical procedure. In some embodiments, method 800 may allow an operator of a medical system used in a medical procedure to experience a virtual audio environment comprising synthesized audio signals that provide directional indication of regions of interest outside the operator's field of view.

[0097] At process 810, a synthesized audio signal can be received from a synthesized audio source (e.g., synthesized audio source 760). Process 810 can be performed at any time during method 800, not necessarily before other processes. In some embodiments, synthesized audio source 760 can be an alarm generator or other components / software configured to generate audio indications associated with a region of interest.

[0098] At process 820, location data of a region of interest within the patient environment can be received. In some embodiments, the region of interest may be a region within the patient's anatomy. In some embodiments, the region of interest may be a region within the patient body outside the field of view (e.g., field of view 791) of the operator's primary imaging system (e.g., an endoscope). In some embodiments, the region of interest may be detectable in the field of view (e.g., field of view 792) of a supplementary imaging system via image processing or other detection techniques. The fields of view of the two imaging systems can be registered such that the orientation of the location data of the region of interest outside the operator's field of view is known relative to the operator's field of view.

[0099] At process 830, the simulated position of the synthesized audio signal is determined in the virtual audio environment based on position data. In some embodiments, the virtual audio environment is an audio environment provided to an operator located in operator environment 724, wherein the perceived position and / or directionality from which the synthesized audio signal originates can be arranged to correspond to the operator's field of view in patient environment 722.

[0100] At process 840, a synthesized audio signal is generated at a simulated location in the virtual audio environment via an audio generation system (e.g., speakers 770a, 770b). In some embodiments, the virtual audio environment is the audio environment experienced by operator O in operator environment 724.

[0101] Some examples of processors (such as the processors of audio placement controller 250 and / or audio projection controller 460) may include non-transient tangible machine-readable media comprising executable code that, when run by one or more processors (e.g., the processors of audio placement controller 250 and / or audio projection controller 460), can cause one or more processors to perform the processes of methods 500, 600, and / or 800. Some common forms of machine-readable media that may include processes of methods 500 and / or 600 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media suitable for reading by a processor or computer.

[0102] Although some illustrative embodiments have been shown and described, extensive modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of these embodiments may be employed without correspondingly using other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be limited only by the appended claims, and it is appropriate to interpret the claims broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. An audio system comprising: One or more audio sensors; Tracking system; Audio reproduction system as well as An audio placement controller, coupled to the one or more audio sensors, the tracking system, and the audio reproduction system, wherein the audio placement controller is configured to perform operations including the following: Receive audio signals at one or more audio sensors; The tracking system receives tracking data, which is associated with the actual location of the source of the audio signal in the physical audio environment. Determine the simulated position of the audio signal in the virtual audio environment; as well as The audio signal is reproduced at the simulated location in the virtual audio environment via the audio reproduction system.

2. The audio system of claim 1, wherein the one or more audio sensors are configured to detect the audio signal in a patient environment.

3. The audio system of claim 2, wherein the audio signal corresponds to verbal communication between a patient or medical personnel in the patient environment.

4. The audio system of claim 2, wherein the audio signal corresponds to the audio output of a patient monitoring device in the patient environment.

5. The audio system of claim 2, wherein the audio signal corresponds to the audio output of a medical system in the patient environment.

6. The audio system of claim 1, wherein the tracking data corresponds to a tracking device attached to the source of the audio signal.

7. The audio system of claim 1, wherein the tracking data is derived from the audio signal.

8. The audio system of claim 1, wherein the analog position of the audio signal is different from the actual position based on the tracking data.

9. The audio system of claim 1, wherein the simulated position is determined based on one or more properties of the audio signal.

10. The audio system of claim 1, wherein the simulated location is determined based on determining that the audio signal is associated with an emergency message or that the audio signal is being communicated to a listener of the audio system.

11. The audio system of claim 1, wherein the simulated position is determined using an artificial neural network.

12. The audio system of claim 1, wherein the audio reproduction system comprises a stereo system, a surround sound system, or headphones worn by a listener of the audio system.

13. The audio system of claim 1, further comprising a synthesized audio source, wherein the operation further comprises: Receive synthesized audio signals from the synthesized audio source; Determine the second analog position of the synthesized audio signal in the virtual audio environment; and The synthesized audio signal, having characteristics based on the second analog position in the virtual audio environment, is emitted via the audio reproduction system.

14. The audio system of claim 13, wherein the synthesized audio signal corresponds to an audio representation of a patient’s physiological processes.

15. A system comprising: Memory; and One or more processors coupled to the memory, the one or more processors being configured to read instructions from the memory and perform operations including: Receive audio signals detected in the patient environment; Receive tracking information associated with the location of the source of the audio signal in the patient environment; Determine the simulated position of the audio signal in the virtual audio environment; and The audio signal is reproduced in an operator environment via an audio reproduction system, wherein the audio reproduction system provides the audio signal having characteristics based on the spatial relationship between the listener's location and the simulated location in the virtual audio environment.

16. The system of claim 15, wherein the audio signal corresponds to verbal communication by a person in the patient environment.

17. The system of claim 15, wherein the audio signal corresponds to the audio output of a medical device in the patient environment.

18. The system of claim 15, wherein the tracking information is received from a tracking device attached to the source of the audio signal.

19. The system of claim 15, wherein determining the simulated position of the audio signal in the virtual audio environment includes changing the position of the source of the audio signal from the patient environment.

20. The system of claim 15, wherein determining the simulated location of the audio signal in the virtual audio environment includes the location of the source of the audio signal matched from the patient environment.

21. A method comprising: Receive audio signals detected in the patient's environment during medical procedures performed using the medical system; Receive tracking data associated with the source of the audio signal; Determine the simulated position of the audio signal in the virtual audio environment; and The audio signal is reproduced in the operator's environment via an audio reproduction system, wherein at least one characteristic of the reproduced audio signal is based on the simulated position in the virtual audio environment.

22. The method of claim 21, wherein the audio signal corresponds to verbal communication by a person in the patient environment.

23. The method of claim 21, wherein the audio signal corresponds to the audio output of a medical device in the patient environment.

24. The method of claim 21, wherein the tracking data is received from a tracking device attached to the source of the audio signal.

25. The method of claim 21, wherein determining the simulated position of the audio signal in the virtual audio environment comprises changing the position of the source of the audio signal from the patient environment.

26. The method of claim 21, wherein determining the simulated location of the audio signal in the virtual audio environment includes the location of the source of the audio signal matched from the patient environment.

27. An audio system comprising: Audio generation system; Synthesized audio sources; and An audio placement controller, coupled to the synthesized audio source, wherein the audio placement controller is configured to perform operations including the following: Receive synthesized audio signals from the synthesized audio source; Receive location data of the region of interest in the patient environment; Based on the location data, the simulated position of the synthesized audio signal in the virtual audio environment is determined; and The synthesized audio signal is generated at the simulated location in the virtual audio environment via the audio generation system.

28. The audio system of claim 27, wherein the synthesized audio signal corresponds to the physiological processes of a patient in the region of interest.

29. The audio system of claim 28, wherein the physiological process includes at least one of breathing, heartbeat, or blood flow.

30. The audio system of claim 27, wherein the synthesized audio signal includes an alarm associated with a condition detected in the region of interest.

31. The audio system of claim 27, wherein the region of interest is within a patient anatomy within the patient environment, and wherein the region of interest is outside the field of view of a first imaging system located within the patient anatomy.

32. The audio system of claim 31, wherein determining the simulated position of the synthesized audio signal in the virtual audio environment comprises determining the spatial relationship between the position data of the region of interest and the field of view of the first imaging system.

33. The audio system of claim 31, wherein the region of interest is detectable in the field of view of the second imaging system within the patient environment.

Images