Apparatus and system for improving troubleshooting in medical systems

Abstract

To improve troubleshooting in medical systems.SOLUTION: The method comprises: obtaining an audio stream comprising a speech by a user of a medical system during an operation; obtaining data streams from equipment associated with the medical system; detecting within the audio stream an indicator to a problem, the indicator occurring during a first time window; selecting an audio segment within the audio stream and a data segment in each of the data streams, the data segment corresponding in time to the audio segment, the audio segment and the data segments comprising audio and data, respectively, captured during the time window; and outputting the audio segment and one data segment.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 522,249, entitled “APPARATUS AND SYSTEM FOR IMPROVING TROUBLESHOOTING IN MEDICAL SYSTEMS,” filed June 21, 2023, which is incorporated herein by reference.

[0002] FIELD OF THEINVENTION FIELD OF THE DISCLOSURE The present disclosure relates generally to medical systems, and more particularly to systems and methods for improving troubleshooting in medical systems. [Background technology]

[0003] In addition to their primary function, many advanced systems, especially medical systems, provide continuous, real-time recording of user and system activity.

[0004] For example, user actions may be logged, including actions associated with a user interface device such as a keyboard or pointing device, actions associated with the system and its associated parameters, such as the application of energy to tissue during electrophysiological (EP) mapping and ablation operations, etc. Additionally, specific events bookmarked by the user may also be logged.

[0005] Additionally, raw data acquired by the system may be collected and logged, such as various digital samples of measurements taken by electrodes or other measurement devices, a record of the location of devices used during operation, such as one or more catheters, etc.

[0006] Further types of recorded data may include data processing results such as visualization of device pathways or morphological characteristics of an electrocardiogram (ECG), which may or may not include intermediate computational results.

[0007] Additionally, logs may be obtained from the underlying software, including, for example, recording of code execution traces programmed by the software developer, for purposes such as debugging or performance monitoring, recording of system events, network messages, etc.

[0008] Due to the data types mentioned above and the significant amounts of additional data types that are recorded per unit time, and especially per operation, searching for specific events within this wealth of data, e.g. for troubleshooting purposes, is a challenging task.

[0009] Traditionally, data is reviewed and such events retrieved when something fails to operate, particularly when needed to evaluate physician actions and compliance with required medical standards. [Brief description of the drawings]

[0010] A more complete understanding of the present disclosure will be obtained from the following detailed description of the embodiments of the present disclosure when read in conjunction with the drawings. [Figure 1] FIG. 1 is a schematic, depiction of an exemplary medical environment including a catheter-based electrophysiological (EP) mapping and ablation system, according to some exemplary embodiments of the present disclosure. [Diagram 2] FIG. 2 is a depiction of various data sources and data extracted and recorded therefrom, according to certain exemplary embodiments of the present disclosure. [Diagram 3] FIG. 1 is a flow diagram of steps in a method for troubleshooting a medical system, according to some exemplary embodiments of the present disclosure. [Figure 4]FIG. 1 is a schematic block diagram of a computing platform for troubleshooting a medical system, according to some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known circuits, control logic, and details of computer program instructions for conventional algorithms and processes have not been shown in detail in order to avoid unnecessarily obscuring the present invention.

[0012] Software programming code embodying aspects of the invention is typically maintained in permanent storage, such as a computer readable medium. In a client-server environment, such software programming code may be stored on a client or a server. The software programming code may be embodied in any of a variety of known media for use with data processing systems, including, but not limited to, magnetic and optical storage devices, such as disk drives, magnetic tapes, compact discs (CDs), digital video discs (DVDs), and computer instruction signals embodied in a transmission medium, with or without a carrier wave upon which the signals are modulated. For example, the transmission medium may include a communications network, such as the Internet. Additionally, while the invention may be embodied in computer software, the functionality necessary to carry out the invention may alternatively be embodied, in part or in whole, using hardware components, such as application specific integrated circuits or other hardware, or some combination of hardware components and software.

[0013] Overview In the following disclosure, the term "word" should be interpreted broadly to cover any word or phrase including one or more words, syllables, etc. A word may be in any language or dialect and may include single syllables or combinations, or voice gestures such as "uh-uh", "oh no", etc.

[0014] In the following disclosure, the term "stream" should be interpreted broadly to cover any sequence of digital data received from a source, at any frequency or time resolution and in any format, whether associated with an audio stream or another data stream. The data may be processed or otherwise used online as it is being received, or may be stored on a storage device and processed at a later time, for example as a file containing all or part of the received data.

[0015] In many medical procedures, users, such as physicians, employ sophisticated equipment. Two such examples include electrophysiology mapping and ablation systems, and phacoemulsification systems.

[0016] In the following disclosure, the term "problem" should be interpreted broadly to cover any technical, software, or other problem with the systems and equipment that may be used by a physician or other staff member and investigated by a developer. The term "problem" may be used interchangeably with the term "troubleshooting event."

[0017] A vast amount of data may be collected during operation, including measurements, user actions, system actions, events of the underlying computing system, intermediate and final processing results, etc. Typically, the information is used to monitor or evaluate user actions, for example to ensure that a physician has made the right decisions and performed the right actions at the right time.

[0018] Because medical systems are generally complex, they are naturally prone to problems, including bugs, misuse issues, etc., especially during the development or deployment of a new system or a new version thereof. Thus, it may be necessary to troubleshoot such problems.

[0019] However, as detailed above, the sheer amount of collected data makes troubleshooting difficult, since it is generally unclear if and when a problem occurred or when it was noticed by the user.

[0020] According to some embodiments of the present disclosure, in addition to recording all data streams from the system, as is done currently, microphones, audio recorders, and user action recorders may be used to capture and record audio and other data entered by users, including physicians and possibly additional staff members such as clinical assistants (CAS). Audio may be recorded throughout the entire operation or during one or more portions thereof.

[0021] The recorded audio may be analyzed by one of the personnel members to detect troubleshooting indicators.

[0022] The indicators may be words that indicate the presence of any problem, such as "Ugh", "Problem", "Disabled", "Not working", "Screen frozen", "Can't move mouse", "System is very slow", "System unresponsive", "Can't click that button", "Can't find that option / button", "Button is unavailable", "Button / option disappeared", "Clicking does nothing", "Window / dialog closed / disappeared", "Can't move / minimize / maximize window / pane", "Error / warning means nothing", "Poor performance", "How can I clear / remove / resolve that error / warning", "System not working, no errors displayed", "Restart system", etc.

[0023] One or more words may also be domain specific and may indicate a problem with a particular system. For example, the following words or phrases may relate to an electrophysiological mapping and ablation system or its operation: "Ablation time / temperature / power seems wrong", "Force indication is wrong", "Catheter is stuttering", "No connection to piu / ultrasound machine / recording system", "Maybe we should turn off piu", "Annotations are wrong", "ECG monitor froze / stopped", "Missing points", "Missing contours", "Catheter visualization does not work", "Can't save / archive / restore study", etc.

[0024] Further instructions may include identifying problematic emotional states within the audio, such as stress, anger, anxiety, etc.

[0025] In some embodiments, inappropriate words may also be searched for, as such words may indicate frustration with the system.

[0026] In some embodiments, words may be recognized by fully transcribing the captured audio and searching the transcription for collections of words that indicate a problem, whether general or specific.

[0027] In a further embodiment, words may be recognized using small vocabulary speech recognition techniques where only specific words are searched for.

[0028] Words may be searched in an ambiguous manner such that similar words or phrases may also be detected, e.g., using "we" instead of "I", a different order of words within a phrase, etc. For example, each detected word may be associated with a confidence, and only those words identified with a confidence above a threshold may be output for further processing.

[0029] Once an indicator is detected, a time slot including a first time window in which the word or emotional state was detected, a second time window preceding the first time window, and optionally a third time window following the first time window may be marked for investigation. In some embodiments, each of the first, second, and third time windows may be of a predetermined (and possibly different) length, e.g., 1 second, 10 seconds, 1 minute, 10 minutes, 15 minutes, etc. In some embodiments, the second time window may go back from the first time window to a predetermined event, such as insertion of the catheter into the vein, arrival of the catheter in the heart, application of the ablation, start of the operation, etc. In some embodiments, the third time window may extend from the first time window to a predetermined event, such as the catheter being withdrawn from the heart, end of the operation, etc. The timing of the predetermined event may be obtained from a log of the operation, from another recording, from one or more words detected in the audio recording, etc. In some embodiments, the preceding event may relate to an underlying executed program, such as spawning a thread, launching a module, etc. The third time slot may continue through a.

[0030] Once the time slots, including the first time window, the second time window, and the third time window, are determined, they may be applied to other captured streams, including, for example, ongoing measurements, user actions, system actions, system events, network events, etc., such that only the segments corresponding to the time slots are further processed.

[0031] In some embodiments, any identifying details of the subject of the action may be concealed from the audio and data streams within a time slot. For example, the patient's or physician's name, if mentioned in the audio segment, may be concealed and removed from any data segments or screen captures, etc., to protect their anonymity and privacy. In some embodiments, useful details such as the subject's gender or age (e.g., references to the patient as "he" or "she") may be maintained or concealed according to regulations, sensitivities, etc.

[0032] The portions of the audio and data streams corresponding to the time slots may then be provided, for example, immediately or at any later time, to another system or person responsible for troubleshooting the problem, e.g., determining whether there was in fact a problem, what led to the problem, how the situation should be prevented, etc. In some embodiments, the system may be repaired and updated to prevent such situations. In other embodiments, the user may receive guidance as to how the situation should be prevented, etc.

[0033] The present disclosure thus provides for automatic detection of troubleshooting situations during operation, which may include problems with the equipment being used, and provides relevant audio and data information to the person or system responsible for troubleshooting. The present disclosure thus enables troubleshooting of situations so that the problematic situation can be eliminated or corrected, either by repairing the system or by educating the user.

[0034] The present disclosure also provides for effective and efficient troubleshooting since it is not necessary to review the entirety of data collected during operation: the troubleshooter or system need only review the relevant time period when the user noticed the problem, as well as the preceding and / or subsequent time windows.

[0035] System Description 1, which illustrates an exemplary catheter-based electrophysiology mapping and ablation system 10. However, it should be understood that the present disclosure is not limited to such systems and may be used with any other type of medical system, such as a phacoemulsification system, etc.

[0036] The exemplary system 10 may include one or more catheters that may be percutaneously inserted by a physician 24 through the vascular system of a patient 23 and into a chamber or vasculature of the heart 12. Typically, a delivery sheath catheter is inserted into the left or right atrium near a desired location of the heart 12. One or more catheters may then be inserted into the delivery sheath catheter to reach a desired location within the heart 12. The multiple catheters, including, for example, catheter 14, may include catheters dedicated to sensing Intracardiac Electrogram (IEGM) signals, catheters dedicated to ablation, and / or catheters for both sensing and ablation. The physician 24 may place a distal tip 28 of catheter 14 in contact with the heart wall to ablate a target site in the heart 12.

[0037] Catheter 14 may be an exemplary catheter for use with an irreversible electroporation (IRE) / pulsed field ablation (PFA) system. A distal tip portion 28 of catheter 14 may include one or more electrodes connected by wires that extend through catheter 14 to a console 30.

[0038] Once the distal end of the catheter 14 reaches the target location within the heart 12, the physician 24 may further manipulate the catheter 14 to place one or more electrodes disposed on the distal tip 28 in contact with the target location, such as the ostium of a pulmonary vein.

[0039] The proximal end of the catheter 14 receives energy from an IRE ablation energy generator 50 via the control console 30. The IRE power source 50 can generate PFA energy including, but not limited to, monopolar, bipolar, or combinations of high voltage DC pulses, such as may be used to achieve irreversible electroporation (IRE). The bipolar pulses may be applied between one or more electrode pairs.

[0040] The distal tip 28 portion may include one or more sensors, such as a force sensor, to sense the force applied by the electrodes to the heart wall. The force sensor may sense and report the total force applied by the electrodes of the distal tip 28 on the heart wall, which may be calculated based on the force applied by each electrode contacting the heart wall on the heart wall.

[0041] The distal tip 28 portion may include a position sensor for tracking the position and orientation of the distal or other sensors. The position sensor may be a magnetic-based position sensor including three magnetic coils for sensing three-dimensional (3D) position and orientation. The magnetic-based position sensor may operate in conjunction with a location pad 25 including multiple magnetic coils 32 configured to generate a magnetic field within a predefined working volume. The real-time position of either the distal tip 28 or the electrodes of the catheter 14 may be tracked based on the magnetic fields generated using the location pad 25 and sensed by the magnetic-based position sensor. Further details of magnetic based position sensing technology are described in U.S. Patent Nos. 5,5391,199; 5,443,489; 5,558,091; 6,172,499; 6,239,724; 6,332,089; 6,484,118; 6,618,612; 6,690,963; 6,788,967; and 6,892,091.

[0042] Readings from a force sensor, position sensor, or other sensor may be reported continuously, for example as a digital stream.

[0043] The system 10 may include one or more electrode patches 38 positioned for skin contact on the patient 23 to establish a positional reference for the location pads 25 and impedance-based tracking of the electrodes. For impedance-based tracking, a current is directed to the electrodes and sensed at the electrode skin patch 38 such that the position of each electrode may be triangulated via the electrode patch 38. Details of impedance-based position tracking technology are described in U.S. Patent Nos. 7,536,218, 7,756,576, 7,848,787, 7,869,865, and 8,456,182.

[0044] Recorder 11 may record and display electrograms 21 captured using surface ECG electrodes 18, as well as intracardiac electrograms (IEGMs) captured using corresponding catheters. Recorder 11 may include pacing capability for pacing the cardiac rhythm and / or may be electrically connected to a stand-alone pacer.

[0045] The patient interface unit (PIU) 30 is an interface configured to establish electrical communication between the catheters, other electrophysiological equipment, a power source, and a workstation 55 for controlling the operation of the system 10. The electrophysiological equipment of the system 10 may include, for example, multiple catheters, location pads 25, body surface ECG electrodes 18, electrode patches 38, an ablation energy generator 50, and a recorder 11. Optionally and preferably, the PIU 30 additionally includes processing capabilities for implementing real-time calculations of catheter positions and performing ECG calculations.

[0046] The workstation 55 may include a processor unit having a memory, a memory or storage device having suitable operating software stored therein, and a user interface functionality. The workstation 55 may optionally provide multiple functions including: (1) modeling the endocardial anatomical structures in three dimensions (3D) and rendering a model or anatomical map 20 for display on the display device 27; (2) displaying activation sequences (or other data) compiled from recorded electrograms 21 in a representative visual display or image superimposed on the rendered anatomical map 20 on the display device 27; (3) displaying real-time positions and orientations of multiple catheters within the cardiac chambers; (4) displaying sites of interest, such as where ablation energy has been applied, on the display device 27; and (5) displaying a current PFA index 51 calculated based on the force and position of the catheter 14 on the display device 27. It is understood that the above-mentioned functions are merely exemplary and that a different set of functions may be provided to the user. One commercially available product embodying elements of system 10 is available as the CARTO™ 3 System, available from Biosense Webster, Inc., 31A Technology Drive, Irvine, CA 92618.

[0047] The system 10 may include one or more microphones 56 for capturing speech by the physician 24. The microphone 56 may be head-mounted on the head of the physician 24, embedded within the workstation 55, fixed in the room, or located elsewhere in the system or its environment. The microphone 56 may be wired or wirelessly connected to a device, such as the workstation 55 or another device adapted to record audio captured by the microphone 56.

[0048] Referring now to FIG. 2, a partial exemplary depiction of data collected during operation is shown, according to some exemplary embodiments of the present disclosure.

[0049] 2 shows a microphone 56, which may or may not be connected to a headset 204, and which may or may not be embedded within the workstation 55. The microphone 56 may output a digital signal 208.

[0050] One or more electrodes 212 of the catheter may each output one or more digital streams 216 indicative, for example, of the ongoing pressure exerted by the electrode on the tissue wall, ongoing position measurements, and the like.

[0051] The workstation 55 or another computer platform associated therewith, hardware or software, may output a digital stream 224 that includes system events, interrupts, and the like.

[0052] Each of the keyboard 228 and pointing device 236, such as a mouse, touch screen, joystick, etc., associated with the workstation 55 may also output digital streams 232 and / or 240, respectively.

[0053] It will be appreciated that the audio streams shown are merely exemplary and that fewer, additional or different data streams may be provided and used.

[0054] The problem may then be searched for either in the audio stream or in the data stream. For example, an audio analysis module, which may be embedded in the workstation 55 or in another remote or colocated computing device, may receive and analyze the audio stream 208 to search for words indicative of the problem.

[0055] The audio analysis module may apply a full transcription to the audio stream 208 and search the transcription for a predefined list of words. Additionally or alternatively, the audio analysis module may perform small vocabulary speech recognition, specifically searching for words from a predefined list.

[0056] Additionally or alternatively, the audio analysis module may perform emotional state analysis of the audio stream 208 to recognize points when the speaker's emotional state is likely to be stressful, anxious, angry, etc.

[0057] In the example of FIG. 2, the word "problem" is detected by the audio analysis module between time T1 and time T2.

[0058] In response to detecting the word (or emotional state), a time point at least earlier than T1, for example T0, may be determined.

[0059] In some embodiments, T0 may be arbitrarily selected, for example, a predetermined period of time before T1.

[0060] In further embodiments, T0 may be selected upon a recognizable event preceding T1, such as the insertion of a catheter, a computerized event reported by workstation 55, such as a network problem, a user interface event, or the like.

[0061] The preceding event may be searched for within the audio stream, within any of the data streams, such as data streams 224, 232, or 240, or may be received from an external source.

[0062] Similarly, a time point T3 following T2 may be selected. T3 may be selected arbitrarily, for example 1 minute, 10 minutes, 15 minutes after T2, etc. Alternatively, T3 is identified when an event occurs, at the end of an action, etc., as detailed above.

[0063] Optionally, identifying details of the patient, doctor, or another entity may be concealed in the audio and in the data stream between T0 and T3. For example, the patient's name being spoken may be concealed in the audio, the patient's printed name may be concealed in either of the data streams, etc. It is understood that the names of the patient and doctor, or any other identifying details of any of the participants in the operation, may be entered at the start of the operation and may also be available in the organization's or operation's computerized systems. Thus, details may be retrieved, searched, and concealed in the audio, in the text recorded as input through a keyboard, in the recorded screen, or in any other recorded information. Concealment may help ensure privacy and anonymity of the doctor, the patient, and any other parties involved.

[0064] The segments of the audio and data streams T0 through T3 may then be provided to a user, a system, or the like.

[0065] It will be understood that the devices and streams shown in Figure 2 are merely exemplary and that an actual system may output different streams. Some exemplary data streams may include one or more of the following: Raw electrocardiogram (ECG) digital sample recordings collected from ECG electrodes 18 attached to the patient's body surface (e.g., at the limbs, chest or back) and from intracardiac electrode catheters; ECG processed data, such as automatically detected ECG morphological features, called "annotations", that relate to specific times in the ECG for normal or abnormal heart rhythms; Intracardiac catheter navigation recording, e.g. of the three-dimensional positions visited by an intracardiac catheter, Visualization of intracardiac catheter shafts (e.g., as three-dimensional geometric curves), Catheter localization and visualization algorithm computation results, whether intermediate or final, for algorithm optimization, debugging, and research; Localization based on electromagnetic signals resulting from artificial electromagnetic fields, for example generated by a low-frequency signal generator placed under the patient table; Localization based on synthetic microcurrents artificially generated between a body surface patch and a catheter electrode, - therapeutic radio frequency ablation and pulsed field ablation applications, including time, duration, and repetition information recording; Recording of applied forces, e.g. measurements of the mechanical forces applied by the physician while pushing a catheter against the inner wall of the heart to measure an ECG or to perform an ablation; Recording of physician-initiated pacing events, i.e. cardiac stimulation by application of electrical pulses to induce potential arrhythmias or to verify ablation efficacy to eradicate arrhythmias; A user action logger, which contains a record of actions performed by a user using the system user interface; Recording of user ultrasound frames or anatomical markers, e.g., short text bookmarks typed into the system by the user; Software internal loggers, including recording of code execution traces programmed by the software developer for debugging or system computing performance monitoring purposes; Recording the occurrence and disappearance of predefined system errors and warnings displayed to a user, such as a physician or clinical assistant; Operating system event logs, e.g. related to computing resource consumption; Operating system application or kernel crash event recording; In systems where the catheter is equipped with an ultrasound transducer, recording of ultrasound frames and anatomical markers (contours) applied manually by the user or automatically by the software.

[0066] Reference is now made to FIG. 3, which is a flow diagram of steps in a method for troubleshooting a medical system, according to some exemplary embodiments of the present disclosure.

[0067] At step 304, an audio stream including audio representing speech by one or more users, such as a physician or clinical assistant, during an action being performed on a subject may be received by a computing device. The audio stream may be received in real time from an audio capture device, such as a microphone, or after the stream has been stored, for example in the form of a file, on a storage device. The audio stream may be captured by a microphone, whether in the form of a headset, ear pod, located somewhere in the room, or embedded in a workstation associated with the medical system.

[0068] At step 308, one or more data streams may be received from one or more sources associated with the operation. As detailed above, the data streams may be received from one or more sensors, from the computing platform including results of processing received data, from events associated with the computing platform or its peripherals and environment, etc.

[0069] In step 312, the audio stream may be analyzed. In some exemplary embodiments, the analysis may include some pre-processing, such as noise reduction, silence or tone removal, audio separation into speech by different speakers, etc. Following pre-processing, one or more troubleshooting indicators may be detected in the audio stream occurring during, for example, the first time window. Some indicators may include the occurrence of words from a predefined list, for example, a list of words that indicate a problem, particularly a problem with equipment of the medical system. The words may be detected by fully transcribing the audio stream and searching the list of words in the transcription, or by applying small vocabulary speech recognition to the list of words. The words may be detected at a start time and at a end during the first time window. Other indicators may include stressful, angry, or similar emotional states identified by an emotional state analysis of the audio stream, which may also indicate a problem. In some examples, the data stream may also be searched for such words, and the time at which the words are found may be indicated.

[0070] In step 316, an audio segment may be selected from the audio stream and one or more data segments corresponding in time to the audio segment may be selected from one or more of the data streams. The audio segment and the data segment may include audio and data captured over a first time window. The audio segment and the data segment may further include audio and data captured over a second time window preceding the first time window. The second time window may immediately precede the first time window. The second time window may start at the start of an operation or at a predefined event, as detailed above.

[0071] The audio and data segments may also include audio and data captured over a third time window following the first time window. The third time window may immediately follow the first time window. The third time window may end at the end of an operation or at a predefined event, as detailed above. In some embodiments, a user may indicate one or more of the boundaries of the first, second, or third time windows to indicate where the user expects to find information for troubleshooting.

[0072] In step 320, the audio and data segments may be processed to conceal, where applicable, any identifying details of the subject of the action, such as the name or identity of the subject mentioned in the audio or coded either in the data stream. In some embodiments, further details such as gender or age may also be concealed. The spoken name of the patient may be concealed in the audio segment, and text or binary identifying details such as printed name, gender, age, etc. may be concealed from the data segment.

[0073] The audio segment and at least one data segment may be output in step 324. The segments may be stored as one or more files, provided as a stream over a communications channel, or transmitted over the communications channel to a user, such as a developer.

[0074] In some embodiments, the customized software may help the developer analyze the situation by playing segments of an audio stream; reproducing user actions such as keyboard, mouse, touch screen or other events; visualizing data streams, e.g., showing events or displaying text data, displaying graphs, etc.

[0075] Replaying the flow and events, including user actions, that took place before and after a problem was indicated by a user may increase a developer's ability to reproduce bugs that might otherwise be deemed unreproducible. The developer may then be able to better analyze the situation and provide a solution.

[0076] Referring now to FIG. 4, a block diagram of a computing platform 400 for troubleshooting a medical system is shown, in accordance with some example embodiments of the present disclosure.

[0077] It will be understood that computing platform 400 may be embedded within workstation 55 or another computing platform of the system, but may also be a stand-alone computing platform or may be embedded elsewhere, either on-premise or remotely, and may be in operative communication with workstation 55.

[0078] Computing platform 400 may be implemented as one or more computing platforms that may be operatively connected to one another. For example, one or more remote computing platforms may be implemented, for example, on a cloud computer. Other computing platforms may be part of an associated organizational computer network. In other embodiments, all functionality may be provided by one or more computing platforms that are all part of the organizational network.

[0079] Computing platform 400 may comprise one or more processors 404, which may be one or more central processing units (CPUs), graphics processing units (GPUs), electronic circuits, integrated circuits (ICs), etc., located or not on the same computing platform. Processor 404 may be configured to provide the required functionality, for example, by loading into memory and activating software modules stored in storage device 412, which will be described in more detail below.

[0080] The computing platform 400 may include input / output (I / O) devices 406, such as a microphone, a display, a keyboard, a touch screen, a mouse or other pointing device, speakers, etc. The I / O devices 406 may be used to receive input from and provide output to a user, such as a system developer, a system administrator, a physician, a clinical assistant, etc., or to receive words to be detected provided as text or audio, display data streams, play audio streams, etc.

[0081] The computing platform 400 may include a communication device 408 for communicating with other devices or other computing platforms, e.g., obtaining information from one or more sensors, obtaining operational information from a controller of a medical system, storing data on a remote storage device, etc. The communication module 408 may be adapted to interface with any communication channel, such as a Local Area Network (LAN), a Wide Area Network (WAN), a cellular network, etc., and to use any associated communication protocol.

[0082] The computing platform 400 may include a storage device 412, such as a hard disk drive, a flash disk, a random access memory (RAM), a memory chip, etc. In some exemplary embodiments, the storage device 412 may hold program code operable to cause the processor 404 to perform operations associated with any of the modules listed below or steps of the method of FIG. 3 above. The program code may include one or more executable units, such as functions, libraries, stand-alone programs, etc., adapted to execute instructions as detailed below.

[0083] Alternatively or additionally, the instructions provided may be stored on a non-transitory, tangible computer readable medium, such as a magnetic, optical, or electronic memory.

[0084] The storage device 412 may comprise a communication module 416 for transmitting and receiving data to and from other systems or components, such as a control system, one or more sensors, an external storage device, a display system, etc., through the communication device 408. In particular, the communication module 416 may be operable in receiving measurements from sensors associated with the electrodes and / or position information of the respective electrodes, processing results thereof, etc.

[0085] The storage device 412 may include an audio and data receiving module 420 for receiving an audio stream containing the user's voice and one or more data streams containing data associated with the medical system or computing system, as detailed in steps 304 and 308 of FIG. 3 above.

[0086] The storage device 412 may include one or more audio analysis modules 424 for detecting one or more words from a list of predefined words, as described in more detail above in connection with step 312 of Figure 3. The audio analysis module 424 may also include an emotional state analysis module for detecting stressful or other emotional states that may indicate problems in the audio stream.

[0087] The storage device 412 may include one or more time window determination modules 428 for determining a time window preceding a segment in which a word, emotional state, or another indicator was detected. The time window may go back to the start of the action or another predefined event. The time window determination module 428 may also be operable in determining a time window resulting from a segment in which a word, emotional state, or another indicator was detected. The time window may go forward to the end of the action or another predefined event.

[0088] The storage device 412 may include a patient detail concealment module 432 for concealing identifying details of subjects within selected segments, as described in relation to step 320 of FIG. 3 above.

[0089] The storage device 412 may include a segment output module 436 for outputting the determined audio and data segments including audio and data of the first time window, the second time window, and the third time window, as described in connection with step 324 of FIG. 3 above.

[0090] The storage device 412 may include a data and control flow management module 440 for invoking the above modules in the correct order and with the required inputs, for example, for determining the time window after which an indicator is found following receipt of the audio and data streams.

[0091] The storage device 412 may include a user interface 444 for receiving input from and providing output to a user, such as a system developer, a system administrator, a physician, a clinical assistant, etc. The user interface 444 may, for example, receive from the user words to be detected, which may be provided as text or audio, display data streams, play audio streams, etc.

[0092] The storage device 412 may further comprise or be in operative communication with a storage space for storing the received audio and data streams, portions of the audio and data streams including the first time window, the second time window, and the third time window, detected problems, etc.

[0093] The present invention may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0094] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in a groove on which instructions are recorded, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as being a signal that is itself ephemeral, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted through a current line.

[0095] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0096] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, such as Java, C, C++, Python, etc. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to individualize the electronic circuitry to carry out aspects of the present invention.

[0097] Aspects of the present invention are described herein with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer readable program instructions.

[0098] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to generate a machine such that the instructions are executed by the processor of the computer or other programmable data processing device to create means for performing the functions / acts specified in the blocks of the flow diagrams and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium that can cause a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer readable storage medium on which the instructions are stored includes an article of manufacture that includes instructions that implement aspects of the functions / acts specified in the blocks of the flow diagrams and / or block diagrams.

[0099] The computer-readable program instructions may be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts defined in the flow diagrams and / or block diagram blocks.

[0100] The flow diagrams and block diagrams in the figures illustrate the structure, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flow diagrams or block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing the illustrated logical function. In some alternative implementations, the functions shown in the blocks may occur in an order other than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented by a dedicated hardware-based system that performs a particular function or operation, or may operate or execute a combination of dedicated hardware and computer instructions. EXAMPLES

[0101] Example 1 1. A method comprising: capturing, during an operation involving the medical system, an audio stream including speech by a user using the medical system to perform an operation, the operation being performed on an object; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within an audio stream, the indicator occurring during at least a time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least a time window; outputting the audio segment and the at least one data segment.

[0102] Example 2 2. The method of example 1, wherein the indicator for the problem is detected by recognizing in the audio stream at least one word selected from a list of predefined words.

[0103] Example 3 2. The method of example 1, wherein the indicator for the problem is detected by recognizing an emotional state indicative of the problem within the audio stream.

[0104] Example 4 2. The method of example 1, wherein the audio segment and the at least one data segment include audio and data, respectively, captured during a second time window preceding the time window.

[0105] Example 5 The method of example 4, wherein the second time window begins at a predetermined event.

[0106] Example 6 The method of example 4, wherein the second time window begins at a start time of the action.

[0107] Example 7 2. The method of embodiment 1, wherein the audio segment and the at least one data segment include audio and data, respectively, captured during a third time window contiguous to the time window.

[0108] Example 8 8. The method of example 7, wherein the third time window ends at a predetermined event or at an end time of the action.

[0109] Example 9 2. The method of example 1, further comprising concealing identifying details of a subject or user in the audio segment and at least one data segment.

[0110] Example 10 2. The method of example 1, wherein at least one data stream includes data related to actions performed by a user.

[0111] Example 11 11. The method of example 10, wherein the action performed by the user includes a user interface initiating action.

[0112] Example 12 2. The method of example 1, wherein at least one data stream includes data related to actions taken by the system.

[0113] (Example 13) 13. The method of example 12, wherein the action is taken by the system in response to a user action.

[0114] Example 14 14. The method of example 13, wherein at least one data stream includes data related to measurements taken by the system.

[0115] Example 15 The method of example 1, wherein the system is an electrophysiological (EP) mapping and ablation system.

[0116] (Example 16) The method of example 1, wherein the at least one data stream includes at least one type of data selected from the group consisting of raw electrocardiogram (ECG) digital sample recordings collected from ECG electrodes, ECG processed data, intracardiac catheter navigation recordings, intracardiac catheter shaft visualization, catheter localization and visualization algorithm calculation results, electromagnetic signal based localization, current based localization, therapeutic radio frequency ablation or pulsed field ablation application including duration or repetition information recording, recording of applied force measurements, a user action log including recording of actions performed by a user using a user interface module, keyboard, mouse, or touch screen actions, and ultrasound frames or anatomical markers.

[0117] (Example 17) The method of example 1, wherein the system is an electrophysiological (EP) mapping and ablation system.

[0118] (Example 18) The method of example 1, wherein the operation is a phacoemulsification operation.

[0119] (Example 19) 1. A computerized apparatus having a processor coupled to a memory unit, the processor being adapted to perform the following steps: acquiring an audio stream including speech by a user of the medical system during an operation involving the medical system, the operation being performed on a subject; acquiring at least one data stream from equipment associated with the medical system; detecting an indicator for a problem in the audio stream, the indicator occurring during at least a first time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data captured respectively during at least the time window; and outputting the audio segment and the at least one data segment.

[0120] (Example 20) 1. A computer program product comprising a non-transitory computer-readable medium carrying program instructions that, when read by a processor, cause the processor to: acquire an audio stream including speech by a user of the medical system during an operation involving the medical system, the operation being performed on the subject; acquire at least one data stream from equipment associated with the medical system; detect an indicator for a problem within the audio stream, the indicator occurring during at least a first time window; select an audio segment within the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data captured during at least the time window, respectively; and output the audio segment and the at least one data segment.

[0121] Although the embodiments described herein primarily address cardiac diagnostic applications, the methods and systems described herein may also be used in other medical applications.

[0122] It will be understood that the embodiments described above are given by way of example, and that the present disclosure is not limited to what is particularly shown and described herein above. Rather, the scope of the present disclosure includes both combinations and subcombinations of the various features described herein above, as well as variations and modifications thereof not disclosed in the prior art that would occur to one skilled in the art upon reading the foregoing description.

[0123] [Embodiment] (1) A method comprising: acquiring, during an operation involving a medical system, an audio stream including speech by a user using the medical system to perform the operation, the operation being performed on an object; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within the audio stream, the indicator occurring during at least a time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; outputting the audio segment and the at least one data segment. (2) The method of embodiment 1, wherein the indicator of the problem is detected by recognizing in the audio stream at least one word selected from a list of predetermined words. (3) The method of claim 1, wherein the indicator of the problem is detected by recognizing an emotional state indicative of a problem within the audio stream. (4) The method of embodiment 1, wherein the audio segment and the at least one data segment comprise audio and data, respectively, captured during a second time window preceding the time window. (5) The method of embodiment 4, wherein the second time window begins at a predetermined event.

[0124] (6) The method of embodiment 4, wherein the second time window begins at a start time of the action. (7) The method of embodiment 1, wherein the audio segment and the at least one data segment comprise audio and data, respectively, captured during a third time window contiguous to the time window. (8) The method of embodiment 7, wherein the third time window ends at a predetermined event or at an end time of the action. (9) The method of embodiment 1, further comprising concealing identifying details of the subject or user in the audio segment and the at least one data segment. (10) The method of embodiment 1, wherein the at least one data stream includes data related to actions performed by the user.

[0125] (11) The method of embodiment 10, wherein the action performed by the user includes a user interface initiating action. (12) The method of embodiment 1, wherein the at least one data stream includes data related to actions taken by the system. (13) The method of claim 12, wherein the action is taken by the system in response to a user action. (14) The method of claim 13, wherein the at least one data stream includes data related to measurements taken by the system. (15) The method of embodiment 1, wherein the system is an electrophysiological (EP) mapping and ablation system.

[0126] (16) The method of embodiment 1, wherein the at least one data stream includes at least one type of data selected from the group consisting of raw electrocardiogram (ECG) digital sample recordings collected from ECG electrodes, ECG processed data, intracardiac catheter navigation recordings, intracardiac catheter shaft visualization, catheter localization and visualization algorithm calculation results, electromagnetic signal based localization, current based localization, therapeutic radio frequency ablation or pulsed field ablation application including duration or repetition information recording, recording of applied force measurements, a user action log including a record of actions performed by the user using a user interface module, keyboard, mouse, or touch screen actions, and ultrasound frames or anatomical markers. (17) The method of embodiment 1, wherein the system is an electrophysiological (EP) mapping and ablation system. (18) The method of claim 1, wherein the operation is a phacoemulsification operation. (19) A computerized apparatus having a processor coupled to a memory unit, the processor comprising: acquiring an audio stream comprising speech by a user of a medical system during an operation involving the medical system, the operation being performed on an object; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within the audio stream, the indicator occurring during at least a first time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; and outputting said audio segment and said at least one data segment. (20) A computer program product comprising a non-transitory computer-readable medium bearing program instructions, the instructions, when read by a processor, causing the processor to: capturing an audio stream including speech by a user of a medical system during an operation involving the medical system, the operation being performed on a subject; acquiring at least one data stream from a device associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator occurring during at least a first time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; and outputting said audio segment and said at least one data segment.

Claims

1. 1. A computerized apparatus having a processor coupled to a memory unit, the processor comprising: acquiring an audio stream comprising speech by a user of a medical system during an operation involving the medical system, the operation being performed on an object; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within the audio stream, the indicator occurring during at least a first time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; and outputting said audio segment and said at least one data segment.

2. 1. A computer program product comprising a non-transitory computer-readable medium bearing program instructions, the instructions, when read by a processor, causing the processor to: capturing an audio stream including speech by a user of a medical system during an operation involving the medical system, the operation being performed on a subject; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within the audio stream, the indicator occurring during at least a first time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; and outputting said audio segment and said at least one data segment.

3. 1. A method comprising: acquiring, during an operation involving a medical system, an audio stream including speech by a user using the medical system to perform the operation, the operation being performed on an object; acquiring at least one data stream from a device associated with the medical system; detecting an indicator for a problem within the audio stream, the indicator occurring during at least a time window; selecting an audio segment in the audio stream and at least one data segment in each data stream of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data respectively captured during at least the time window; outputting the audio segment and the at least one data segment.

4. The method of claim 3 , wherein the indicator for the problem is detected by recognizing in the audio stream at least one word selected from a predetermined list of words.

5. The method of claim 3 , wherein the indicator for the problem is detected by recognizing an emotional state indicative of a problem within the audio stream.

6. The method of claim 3 , wherein the audio segment and the at least one data segment comprise audio and data, respectively, captured during a second time window preceding the time window.

7. The method of claim 6 , wherein the second window of time begins at a predetermined event.

8. The method of claim 6 , wherein the second window of time begins at a start time of the action.

9. The method of claim 3 , wherein the audio segment and the at least one data segment comprise audio and data, respectively, captured during a third time window contiguous to the time window.

10. The method of claim 9 , wherein the third time window ends at a predetermined event or at an end time of the action.

11. The method of claim 3 , further comprising concealing identifying details of the subject or user in the audio segment and the at least one data segment.

12. The method of claim 3 , wherein the at least one data stream includes data related to actions performed by the user.

13. The method of claim 12 , wherein the action performed by the user comprises a user interface initiating action.

14. The method of claim 3 , wherein the at least one data stream includes data related to actions taken by the system.

15. The method of claim 14 , wherein the action is taken by the system in response to a user action.

16. The method of claim 15 , wherein the at least one data stream includes data related to measurements taken by the system.

17. The method of claim 3 , wherein the system is an electrophysiological (EP) mapping and ablation system.

18. 4. The method of claim 3, wherein the at least one data stream comprises at least one type of data selected from the group consisting of: raw electrocardiogram (ECG) digital sample recordings collected from ECG electrodes, ECG processed data, intracardiac catheter navigation recordings, intracardiac catheter shaft visualization, catheter localization and visualization algorithm calculation results, electromagnetic signal based localization, current based localization, therapeutic radio frequency ablation or pulsed field ablation application including duration or repetition information recording, applied force measurement recording, user action log including recording of actions performed by the user with a user interface module, keyboard, mouse, or touch screen actions, and ultrasound frames or anatomical markers.

19. The method of claim 3 , wherein the system is an electrophysiological (EP) mapping and ablation system.

20. The method of claim 3 , wherein the operation is a phacoemulsification operation.

Images