Apparatus and system for improving fault diagnosis in medical systems

Abstract

A method, apparatus and computer program product are disclosed, the method comprising: obtaining an audio stream during an operation involving a medical system, the audio stream comprising speech of a user of the medical system, where the operation is performed on an object; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator occurring at least during a first time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; and outputting the audio segment and the at least one data segment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 522,249, filed on June 21, 2023, entitled “APPARATUS AND SYSTEM FORIMPROVING TROUBLESHOOTING IN MEDICAL SYSTEMS,” which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to medical systems, and particularly to systems and methods for improving fault diagnostics in medical systems. Background Art

[0004] Many advanced systems, especially medical systems, in addition to their primary function, continuously and in real time record user and system activities.

[0005] For example, the user's actions can be recorded in the log, including actions associated with a user interface device (such as a keyboard or pointing device), actions associated with the system and their related parameters, such as the application of energy to tissue during electrophysiological (EP) mapping and ablation procedures, etc. In addition, certain events tagged by the user can also be recorded in the log.

[0006] Additionally, raw data obtained by the system may be collected and logged, such as various digital samples of measurements made by electrodes or other measuring devices, location records of devices used during operation (such as one or more catheters), and the like.

[0007] Another type of recorded data may include data processing results, such as visualizations of device pathways or electrocardiogram (ECG) morphological features, which may or may not include intermediate calculation results.

[0008] Additionally, logs may be obtained from the underlying software, including, for example, logging of code execution traces programmed by software developers or similar items for purposes such as debugging or performance monitoring, logging system events, network messages, and the like.

[0009] Since the total amount of the above and additional data types recorded per time unit, in particular per operation, is very large, searching for specific events (eg, for the purpose of troubleshooting) within this large amount of data is a challenging task.

[0010] Traditionally, when problems with operation occur, particularly when physician actions and compliance with required medical standards need to be assessed, this data is reviewed and searched for such events. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present disclosure will be more fully understood through the following detailed description of examples of the present disclosure in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a schematic illustration of an exemplary medical environment including a catheter-based electrophysiological (EP) mapping and ablation system according to some exemplary embodiments of the present disclosure;

[0013] Figure 2 is an illustration of various data sources and data extracted and recorded therefrom according to some exemplary embodiments of the present disclosure;

[0014] Figure 3 is a flow chart of steps in a method for troubleshooting a medical system according to some exemplary embodiments of the present disclosure; and

[0015] Figure 4 is a schematic block diagram of a computing platform for fault diagnosis of a medical system according to some exemplary embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] In the following description, numerous specific details are set forth in order to provide a comprehensive understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced without these specific details. In other cases, well-known circuits, control logic components, and details of computer program instructions for conventional algorithms and processes are not shown in detail to avoid unnecessarily obscuring the present invention.

[0017] Software programming code embodying aspects of the present invention is typically stored in a permanent storage device such as a computer-readable medium. In a client-server environment, such software programming code may be stored on a client or a server. The software programming code may be contained in any of a variety of known media used with a data processing system. This includes, but is not limited to, magnetic and optical storage devices such as disk drives, tapes, compact disks (CDs), digital video disks (DVDs), and computer instruction signals contained in a transmission medium with or without a carrier wave modulating the signal. For example, the transmission medium may include a communication network such as the Internet. In addition, although the present invention may be embodied in computer software, the functionality required to implement the present invention may alternatively be embodied in part or in whole using hardware components, such as application specific integrated circuits or other hardware, or some combination of hardware components and software.

[0018] Overview

[0019] In the following disclosure, the term "word" is broadly interpreted to cover any word or phrase comprising one or more words, syllables, etc. The word may be in any language or dialect, and may include a single syllable or a combination, or a vocal expression, such as "uh-uh", "oh no", etc.

[0020] In the following disclosure, the term "stream", whether associated with an audio stream or another data stream, is to be broadly interpreted as covering any sequence of digital data received from a source at any frequency or time resolution and in any format. The data may be processed as it is received or otherwise used online, or stored in a storage device and processed later, for example as a file comprising all or part of the received data.

[0021] In many medical procedures, users such as physicians use complex equipment. Two such examples include electrophysiological mapping and ablation systems and phacoemulsification systems.

[0022] In the following disclosure, the term "problem" is interpreted broadly to cover any technical problem, software problem, or other problem with systems and equipment used by physicians or other staff and that developers can investigate. The term "problem" can be used interchangeably with the term "troubleshooting time".

[0023] A large amount of data can be collected during operation, including measurements, user actions, system actions, events of the underlying computing system, intermediate and final processing results, and other data. Often, this information is used to monitor or evaluate the user's actions, for example, to ensure that the physician makes the right decisions and performs the right actions at the right time.

[0024] Because medical systems are often complex, they are naturally prone to problems, including vulnerabilities, misuse issues, or other problems, and are particularly prone to problems during the development or deployment of new systems or new versions thereof. Therefore, troubleshooting of these problems may be required.

[0025] However, as detailed above, the large amount of collected data makes troubleshooting difficult because it is often unknown if and when a problem occurred or was noticed by a user.

[0026] According to some embodiments of the present disclosure, in addition to recording all data streams from the system as is currently done, microphones, audio recorders, and user motion recorders may be used to capture and record audio and other data input by the user, including the physician and possibly additional staff, such as a clinical assistant (CAS). The audio may be recorded during the entire operation, or one or more portions of the audio 30 may be recorded.

[0027] One of the recorded persons may analyze the recorded audio to detect fault diagnostic indicators.

[0028] The indicator may be a word indicating the presence of any problem, such as "uh huh", "problem", "disabled", "not working", "screen is frozen", "I can't move the mouse", "system is very slow", "system is not responding", "I can't click that button"; "I can't find that option / button", "button is disabled", "button / option has disappeared", "I clicked it and nothing happened", "window / dialog has closed / disappeared", "I can't move / minimize / maximize window / pane", "error / warning makes no sense to me", "poor performance", "how do I erase / remove / solve that error / warning", "system is not working and doesn't show errors", "reboot system", etc. 10 One or more words may also be domain specific and may indicate a problem with a particular system. For example, the following words or phrases may be related to an electrophysiological mapping and ablation system or its operation: "ablation time / temperature / power seems wrong", "force indication is wrong", "catheter is rattling", "no connection to piu / ultrasound machine / recording system", "maybe we should shut down piu", "annotation is incorrect", "ECG monitor has frozen / stopped", "missing a point", "missing contour", "catheter visualization does not work", "I can't save / archive / restore study", etc.

[0029] Additional indications may include identification of problematic emotional states within the audio, such as stress, anger, worry, etc.

[0030] In some embodiments, inappropriate words may also be searched, as such words may indicate frustration with the system. 20 In some embodiments, words may be identified by fully transcribing the captured audio and searching the transcription for a set of words that indicate a problem, whether general or specific.

[0031] In other embodiments, small vocabulary speech recognition techniques (where only specific words are searched) may be used to recognize words.

[0032] The words may be searched in a fuzzy manner so that similar words or phrases may also be detected, such as using "we" instead of "I", a different order of words in a phrase, etc. For example, each detected word may be associated with a degree of certainty, and only words identified with a degree of certainty exceeding a threshold may be output for further processing.

[0033] Once an indicator has been detected, a time slot including the following items may be marked for investigation: a first time window during which the word or emotional state was detected, a second time window before the first time window, and optionally a third time window after the first time window. In some embodiments, each of the first time window, the second time window, and the third time window may have a predetermined (and possibly different) length, such as one second, ten seconds, one minute, ten minutes, fifteen minutes, etc. In some embodiments, the second time window may be backed up from the first time window until a predetermined event, such as the insertion of a catheter into a vein, the catheter reaching the heart, the application of ablation, the start of the operation, etc. In some embodiments, the third time window may extend from the first time window until a predetermined event, such as the catheter being pulled out of the heart, the end of the operation, etc. The timing of the predetermined event may be obtained from a log of the operation, from another record, from one or more words detected in an audio recording, etc. In some embodiments, the previous event may be related to an underlying executed program (such as spawning a thread, activating a module, etc.). The third time slot may continue until

[0034] Once a time slot including a first time window, a second time window, and a third time window is determined, 10 the time slot can be applied to other captured streams, including, for example, ongoing measurements, user actions, system actions, system events, network events, etc., so that only fragments corresponding to the time slots of these other captured streams are further processed.

[0035] In some embodiments, any identifying details of the subject of the operation can be hidden from the audio stream and the data stream during this time slot. For example, if the patient or physician's name is mentioned in the audio clip, the patient's or physician's name can be hidden, the patient's or physician's name can be removed from any data clips or from screenshots, etc., to protect their anonymity and privacy. In some embodiments, information details such as the subject's gender or age (e.g., referring to the patient as "he" or "she") can be maintained or hidden depending on regulations, sensitivity, etc.

[0036] The portion of the audio stream and data stream corresponding to the time slot can then be provided, for example, immediately or any time later, to another system or personnel responsible for troubleshooting the problem (e.g., determining whether there is indeed a problem, what caused the problem, how to prevent this situation, etc.). In some embodiments, the system can be repaired and updated to prevent such situations. In other embodiments, the user can receive guidance on how to prevent this situation, etc.

[0037] Thus, the present disclosure provides for automatic detection of troubleshooting situations during operation, where these situations may involve problems with the equipment being used and providing relevant audio and data information to the person or system responsible for the troubleshooting. Thus, the present disclosure enables troubleshooting of problematic situations so that the situation can be eliminated or corrected by repairing the system or by educating the user.

[0038] The present disclosure also provides effective and efficient fault diagnosis, because there is no need to examine the entire data collected during operation. The person or system responsible for fault diagnosis only needs to examine the relevant time period when the user noticed the problem 30 and the time window before and / or after.

[0039] System Description

[0040] refer to Figure 1 , which shows an example catheter-based electrophysiological mapping and ablation system 10. However, it should be understood that the present disclosure is not limited to such a system and may also be used with any other type of medical system, such as a phacoemulsification system or other systems.

[0041] The exemplary system 10 may include one or more catheters that can be inserted percutaneously through the vascular system of the patient 23 by a physician 524 into a chamber or vascular structure of the heart 12. Typically, a delivery sheath catheter is inserted into the left atrium or right atrium near a desired location in the heart 12. One or more catheters may then be inserted into the delivery sheath catheter to reach the desired location in the heart 12. Multiple catheters including, for example, catheter 14 may include a catheter dedicated to sensing intracardiac electrogram (IEGM) signals, a catheter dedicated to ablation, and / or a catheter for both sensing and ablation. The physician 24 may place the distal tip 28 of the catheter 14 in contact with the heart wall for ablation of a target site in the heart 12.

[0042] The catheter 14 may be an exemplary catheter to be used with an irreversible electroporation (IRE) / pulsed field ablation (PFA) system. The distal tip section 28 of the catheter 14 may include one or more electrodes connected by wires extending through the catheter 14 to a console 30.

[0043] Once the distal end of catheter 14 has reached the target location within heart 12, physician 24 may further manipulate catheter 14 to place one or more electrodes disposed on distal tip 28 into contact with the target location, such as the ostium of a pulmonary vein.

[0044] The proximal end of the catheter 14 receives energy from an IRE ablation energy generator 50 through a console 30. The IRE power source 50 can generate, but is not limited to, PFA energy, including high-voltage DC pulses of monopolar, bipolar, or a combination thereof, which can be used to achieve irreversible electroporation (IRE). The bipolar pulse can be applied between one or more pairs of electrodes.

[0045] The distal tip 28 segment may include one or more sensors, such as force sensors, for sensing the force applied by the electrodes on the heart wall. The force sensor may sense and report the total force applied by the electrodes of the distal tip 28 on the heart wall, which may be calculated based on the force applied by each electrode touching the heart wall 25.

[0046] The distal tip 28 segment may include a position sensor or other sensor for tracking the position and orientation of the distal end. The position sensor may be a magnetic-based position sensor that includes three magnetic coils for sensing three-dimensional (3D) position and orientation. The magnetic-based position sensor may operate in conjunction with a positioning pad 25 that includes a plurality of magnetic coils 32 configured to generate a magnetic field in a predetermined working space 30. The real-time position of the distal tip 28 of the catheter 14 or any electrode may be tracked based on the magnetic field generated by the positioning pad 25 and sensed by the magnetic-based position sensor. Details of magnetic-based position sensing technology are described in U.S. Patents 5,5391,199, 5,443,489,

[0047] No. 5,558,091, No. 6,172,499, No. 6,239,724, No. 6,332,089, No. 6,484,118,

[0048] No. 6,618,612, No. 6,690,963, No. 6,788,967 and No. 6,892,091.

[0049] Readings from force sensors, position sensors or other sensors may be reported continuously, for example as a digital stream 5 .

[0050] The system 10 may include one or more electrode patches 38 positioned in contact with the skin of the patient 23 to establish a position reference for the positioning pad 25 and impedance-based tracking of the electrodes. For impedance-based tracking, current is directed to the electrodes and sensed at the electrode skin patches 38 so that the position of each electrode can be triangulated via the electrode patches 38. Details of the impedance-based position tracking techniques are described in U.S. Patents 7,536,218, 7,756,576, 7,848,787, 7,869,865, and 8,456,182.

[0051] The recorder 11 can record and display the electrogram 21 captured with the surface ECG electrodes 18 and the intracardiac electrogram (IEGM) captured with the corresponding catheter. The recorder 11 may include pacing capabilities for pacing the cardiac rhythm and / or may be electrically connected to an independent pacemaker. 15 The patient interface unit (PIU) 30 is an interface configured to establish electrical communication between the catheter, other electrophysiological equipment, a power supply, and a workstation 55 for controlling the operation of the system 10. The electrophysiological equipment of the system 10 may include, for example, multiple catheters, a positioning pad 25, a surface ECG electrode 18, an electrode patch 38, an ablation energy generator 50, and a recorder 11. Optionally and preferably, the PIU 30 further includes processing capabilities for achieving real-time calculation of the position of the catheter and for performing ECG calculations. 20 The workstation 55 may include a memory, a processor unit with a memory or storage device having appropriate operating software stored therein, and user interface capabilities. The workstation 55 may provide a plurality of functions, which may optionally include: (1) three-dimensional (3D) modeling of the endocardial anatomy and rendering the model or anatomical map 20 for display on the display device 27, (2) displaying on the display device 27 an activation sequence (or other data) compiled from recorded electrograms 21 as representative visual markers or images superimposed on the rendered anatomical map 20, (3) displaying the real-time position and orientation of multiple catheters within the heart chamber, (4) displaying on the display device 27 sites of interest, such as locations where ablation energy has been applied, and (5) displaying on the display device 27 a current PFA index 51 calculated based on the force and position of the catheter 14. It should be understood that the above-described functions are merely exemplary and that a different set of functions may be provided to the user. A commercial product 30 embodying elements of the system 10 is the CARTO 100, available from Biosense Webster, Inc. of 31A Technology Drive, Irvine, CA 92618. TM 3 systems.

[0052] The system 10 may include one or more microphones 56 for capturing the speech of the physician 24. The microphone 56 may be worn on the head of the physician 24, embedded in the workstation 55, fixed in the room, or located anywhere else within the system or its environment. The microphone 56 may be connected to a device such as the workstation 55 or another device suitable for recording the audio captured by the microphone 56, either wired or wirelessly.

[0053] Reference now Figure 2 , which shows a partial exemplary illustration of data collected during operation according to some exemplary embodiments of the present disclosure.

[0054] Figure 2A microphone 56 is shown, which may or may not be connected to the headset 204 , embedded within the workstation 55 , etc. The microphone 56 may output a digital signal 208 .

[0055] The one or more electrodes 212 of the catheter may each output one or more digital streams 216 indicating, for example, the continuous pressure exerted by the electrode on the tissue wall, continuous position measurements, and the like.

[0056] The hardware or software of the workstation 55 or another computer platform associated therewith may also output a digital stream 224 including system events, interrupts, and the like.

[0057] A keyboard 228 and a pointing device 236 (such as a mouse, touch screen, joystick, etc.) associated with the workstation 55 may each also output digital streams 232 and / or 240, respectively.

[0058] It should be understood that the audio streams shown are exemplary only, and that fewer, more, or different data streams may be provided and used.

[0059] The question may then be searched within the audio stream or within any data stream.For example, an audio analysis module, which may be embedded within the workstation 55 or in another remote or collocated computing device, may receive and analyze the audio stream 208 to search for words indicative of a question.

[0060] The audio analysis module may apply a full transcription to the audio stream 208 and search the transcription for a list of predetermined words. Additionally or alternatively, the audio analysis module may perform small vocabulary speech recognition and search specifically for words from the predetermined list.

[0061] Additionally or alternatively, the audio analysis module may perform an emotional state analysis of the audio stream 208 to identify points in time when the speaker's emotional state appears to be nervous, worried, angry, etc.

[0062] exist Figure 2 In the example of , between time T1 and time T2 , the audio analysis module has detected the word “problem”.

[0063] In response to detecting the word (or emotional state), at least one time point earlier than T1 may be determined, such as T0.

[0064] In some embodiments, T0 may be selected arbitrarily, such as a predetermined period of time before T1.

[0065] In other embodiments, T0 may be selected based on an identifiable event preceding T1, such as the insertion of a catheter, a computerized event reported by workstation 55 (such as a network problem), a user interface event, etc. The prior event may be searched within the audio stream, within any of the data streams (such as data streams 224, 232, or 240), or retrieved from an external source.

[0066] Similarly, a time point T3 after T2 can be selected. T3 can be arbitrarily selected, for example, 1 minute, 10 minutes, 15 minutes, etc. after T2. Alternatively, T3 is identified when the event described in detail above occurs, when the operation ends, etc. Alternatively, the identification details of the patient, doctor, or another entity in the data stream between T0 and T3 can be hidden in the audio. For example, the patient's name spoken in the audio can be hidden, the printed name of the patient in any of the data streams in the data stream can be hidden, etc. It should be understood that any other identification details of the patient and doctor's name or any of the operation participants are input at the beginning of the operation, and can also be obtained in the computerized system of the organization or operation. Therefore, these details can be retrieved, searched, and hidden in the audio, the recorded text entered by the keyboard, the recorded screen, or any other recorded information. This hiding can help ensure the privacy and anonymity of doctors, patients, and any other involved personnel.

[0067] The audio segment and data stream from T0 to T3 may then be provided to a user, system, or the like.

[0068] It should be understood that Figure 2 The devices and streams shown are exemplary only, and an actual system may output 20 different multiple streams. Some exemplary data streams may include one or more of the following:

[0069] From an electrocardiogram attached to the patient's body (e.g., at a limb, chest, or back)

[0070] (ECG) electrodes 18 and raw ECG digital sample records collected from intracardiac electrode catheters;

[0071] ECG processed data, such as automatically detected ECG morphology features, which are called

[0072] “Annotations”, which relate to specific times in the ECG for normal or abnormal heart rhythms; 25

[0073] Intracardiac catheter navigation records, e.g., three-dimensional locations accessed by a catheter within the heart;

[0074] Intracardiac catheter axis visualization (e.g., as a 3D geometric curve);

[0075] Catheter positioning and visualization of algorithm calculation results for algorithm optimization, debugging and research,

[0076] And whether it is intermediate or final;

[0077] Positioning based on electromagnetic signals (generated, for example, by an artificial electromagnetic field generated by a low-frequency signal generator placed under the patient table);

[0078] Based on electrical current (e.g., synthetic current artificially generated between the body surface patch and the catheter electrode)

[0079] Microcurrent) positioning;

[0080] Therapeutic radiofrequency ablation or pulsed field ablation application, including timing, duration and recurrence

[0081] Re-information records;

[0082] The force applied (e.g., by a physician when pushing a catheter against the inner wall of the heart to measure 5

[0083] ECG or mechanical force applied when performing ablation);

[0084] Physician-initiated pacing events (i.e., cardiac stimulation by applying electrical pulses)

[0085] Generates potential arrhythmias or is used to verify the effectiveness of ablation to eradicate arrhythmias

[0086] abnormal effectiveness);

[0087] User action logger, which includes a record of actions performed by the user using the system user interface;

[0088] User ultrasound frames or anatomical labels, e.g. short text typed into the system by the user

[0089] Records of tags;

[0090] · Software internal logger, which includes support for debugging functions programmed by software developers

[0091] or code execution tracking records for system computing performance monitoring purposes;15

[0092] · The occurrence and disappearance of predefined system errors and the notification to the user (such as a physician or clinical assistant)

[0093] Records of warnings displayed by the

[0094] Operating system event logs, which are related to, for example, computing resource consumption;

[0095] Operating system application or kernel crash event records;

[0096] • In systems where the catheter is equipped with an ultrasound transducer, the recording of ultrasound frames and anatomical markers (contours) applied manually by the user or automatically by software.

[0097] Reference now Figure 3 , which shows a flowchart of steps in a method for fault diagnosis of a medical system according to some exemplary embodiments of the present disclosure.

[0098] At step 304, an audio stream including audio representing the speech of one or more users (such as a physician or a clinical assistant) during the operation performed on the object may be received by the computing device. The audio stream may be received in real time from an audio capture device such as a microphone, or after the stream has been stored on a storage device, such as in the form of a file. The audio stream may be captured by a microphone, whether the microphone is in the form of a headset, earphones, located anywhere in the room, or embedded in a workstation associated with the medical system.

[0099] At step 308, one or more data 30 streams may be received from one or more sources associated with the operation. As detailed above, the data streams may be received from one or more sensors, from a computing platform including results of processing received data, events associated with the computing platform or its peripherals and environment, and the like.

[0100] In step 312, the audio stream can be analyzed. In some exemplary embodiments, the analysis may include some pre-processing, such as noise reduction, mute or tone removal, separation of speech into different speakers' speech, and the like. After pre-processing, one or more fault diagnosis indicators may be detected in the audio stream, which may occur, for example, during the first time window. Some indicators may include the occurrence of words from a predetermined list (e.g., a list of words indicating a problem, particularly a problem with the equipment of a medical system). Words may be detected by completely transcribing the audio stream and searching the word list in the transcription, or applying small vocabulary speech recognition to the word list. These words may be detected at the start time and continue to be detected during the first time window. Other indicators may include nervous, angry, or similar emotional states identified by emotional state analysis of the audio stream, which may also indicate problems. In some examples, such words may also be searched in the data stream, and the time at which these words are found may be indicated.

[0101] In step 316, an audio segment may be selected from the audio stream, and one or more data segments corresponding to the audio segment in time may be selected from one or more of the data streams. The audio segment and the data segment may include audio and data captured in a first time window, respectively. The audio segment and the data segment may also include audio and data captured in a second time window before the first time window. The second time window may be immediately before the first time window. The second time window may start at the start of the operation or at a predetermined event, as described in detail above.

[0102] The audio segment and the data segment may also include the audio and data captured on the third time window after the first time window, respectively. The third time window may be immediately after the first time window. The third time window may end at the end of the operation or at a predetermined event, as described in detail above. In some embodiments, the user may indicate one or more boundaries of the boundaries of the first time window, the second time window, or the third time window, so that the user indicates where to expect to find information for troubleshooting.

[0103] At step 320, where applicable, the audio clips and data clips may be processed to hide all identifying details of the object of the operation, such as the name or ID of the object mentioned in the audio or encoded in any data stream, etc. In some embodiments, additional details such as gender or age may also be hidden. The patient's spoken name in the audio clip may be hidden, and text or binary identifying details such as printed name, gender, age, etc. may be hidden from the data clip.

[0104] The audio segment and at least one data segment may be output at step 324. The segments may be stored as one or more files, provided as a stream over a communication channel, sent to a user such as a developer via a communication channel, and so on.

[0105] In some embodiments, custom software can help developers analyze situations by playing clips of audio streams, reproducing user actions (such as keyboard, mouse, touch screen, or other events), visualizing data streams (e.g., showing 5 events or displaying text data, displaying graphics), etc.

[0106] The playback of the processes and events (including user actions) that occurred before and after the user has indicated a problem can increase the developer's ability to reproduce a bug that might otherwise be considered unreproducible. The developer can then better analyze the situation and can provide a solution.

[0107] Reference now Figure 4 , which shows a block diagram of a computing platform 400 for fault diagnosis of a medical system according to some exemplary embodiments of the present disclosure.

[0108] It should be understood that computing platform 400 may be embedded within workstation 55 or another computing platform of the system, but may also be a standalone computing platform or embedded elsewhere, either locally or remotely, and in operable communication with workstation 55 .

[0109] Computing platform 400 may be implemented as one or more computing platforms operatively connected to one another. For example, one or more remote computing platforms may be implemented, for example, on a cloud computer. Other computing platforms may be part of an associated organization's computer network. In other embodiments, all functionality may be provided by one or more computing platforms that are all part of an organization's network.

[0110] The computing platform 400 may include one or more processors 404 located or not located on the same computing platform, which may be one or more central processing units (CPUs), GPUs, microprocessors, electronic circuits, integrated circuits (ICs), etc. The processor 404 may be configured to provide the desired functionality, for example, by loading into memory and activating software modules stored on the storage device 412 described in detail below.

[0111] The computing platform 400 may include input / output (I / O) devices 406, such as a microphone, a display, a keyboard, a touch screen, a mouse or another pointing device, speakers, etc. The I / O devices 406 may be used to receive input from a user (such as a system developer, a system administrator, a physician, a clinical assistant) and provide output to the user, or, for example, receive words to be detected as text or as audio, display a data stream and play an audio stream, etc.

[0112] The computing platform 400 may include a communication device 408 for communicating with other devices or other computing platforms, such as obtaining information from one or more sensors, obtaining operational information from the controller 30 of the medical system, storing data on a remote storage device, etc. The communication module 408 may be adapted to interface with any communication channel (such as a local area network (LAN), a wide area network (WAN), a cellular network, etc.), and use any relevant communication protocol.

[0113] The computing platform 400 may include a storage device 412, such as a hard drive, a flash drive, a random access memory (RAM), a memory chip, etc. In some exemplary embodiments, the storage device 412 may retain program code for causing the processor 404 to execute any of the modules listed below or above. Figure 3 The program code may include one or more executable units, such as functions, libraries, independent programs, etc., adapted to perform instructions as described in detail below.

[0114] Alternatively or additionally, the provided instructions may be stored on a non-transitory tangible computer-readable medium, such as a magnetic, optical or electronic memory.

[0115] The storage device 412 may include a communication module 416, which is used to transmit data to other systems or components (such as a control system, one or more sensors, an external storage device, a display system, etc.) through the communication device 408 and receive data from other systems or components. Specifically, the communication module 416 can be used to receive measurement results from sensors associated with electrodes and / or position information of corresponding electrodes, processing results thereof, etc.

[0116] The storage device 412 may include an audio and data receiving module 420 for receiving an audio stream including a user's voice and one or more data streams including data associated with a medical system or a computing system, as described above in Figure 3 Detailed description of steps 304 and 308.

[0117] The storage device 412 may include one or more audio analysis modules 424, the one or more audio analysis modules for detecting one or more words from a predetermined list of words, as described above with reference to FIG. Figure 3 The audio analysis module 424 may also include an emotional state analysis module for detecting stress or other emotional states within the audio stream that may indicate a problem.

[0118] The storage device 412 may include one or more time window determination modules 428 for determining a time window before a segment of a word, an emotional state, or another indicator is detected. The time window may be rolled back to the start of the operation, or rolled back to another predetermined event. The time window determination module 428 may also be used to determine a time window after a segment of a word, an emotional state, or another indicator is detected. The time window may be advanced until the end of the operation, or advanced to another predetermined event.

[0119] The storage device 412 may include a patient detail hiding module 432 for hiding the identifying details of the objects within the selected segment, as described above with reference to FIG. Figure 3 Step 320 is described in association with .

[0120] The storage device 412 may include a segment output module 436, which is used to output the determined audio segments and data segments, which include the audio and data of the first time window, the second time window, and the third time window, as described above with reference to FIG. Figure 3 Step 324 is described in association with FIG.

[0121] The storage device 412 may include a data and control flow management module 440, which is used to activate the above modules in the correct order and with the required inputs, such as determining a time window after finding an indicator, the determining operation after receiving an audio stream and a data stream, etc.

[0122] The storage device 412 may include a user interface 444 for receiving input from a user (such as a system developer, system administrator, physician, clinical assistant, etc.) and providing output to the user. The user interface 444 may, for example, receive a word to be detected from the user, which may be provided as text or as audio, display a data stream, play an audio stream, etc. 15

[0123] The storage device 412 may also include or be in operative communication with a storage space that stores, for example, received audio and data streams, portions of the audio and data streams comprising the first time window, the second time window, and the third time window, detected problems, and the like.

[0124] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, which are used to cause a processor to perform various aspects of the present invention.

[0125] A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: 25 portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the foregoing. 30 As used herein, a computer readable storage medium should not be understood as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted by wire.

[0126] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, transmission optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0127] The computer-readable program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages ​​(such as programming languages ​​such as Java, C, C++, Python, etc.). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or 15 completely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., by using the Internet of an Internet service provider). In some embodiments, in order to perform various aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) 20 may execute computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0128] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each block in the flowchart illustrations and / or block diagrams and the combination of blocks in the flowchart illustrations and / or block diagrams can be implemented by computer-readable program instructions.

[0129] These computer-readable program instructions may be provided to a general-purpose computer, a processor of a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create instructions for implementing the flowcharts and / or block diagrams. Figure 1These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide a computer, a programmable data processing device and / or other device to work in a specific manner, so that the computer-readable storage medium with instructions stored therein includes a computer-readable storage medium with instructions for implementing the flowchart and / or the block diagram. Figure 1 An article of manufacture that specifies various aspects of the functions / actions in one or more blocks.

[0130] The computer readable program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus to cause a series of operation steps to be executed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer, other programmable device or other apparatus implement the flowchart and / or block diagram. Figure 1 Functions / actions specified in one or more boxes.

[0131] The flow charts and block diagrams in the accompanying drawings illustrate the possible specific implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram may represent a module, a fragment or a part of an instruction, and the instruction includes one or more executable instructions for implementing a specified logical function. In some alternative specific implementations, the function indicated in the box may not occur in the order indicated in the accompanying drawings. For example, depending on the function involved, the two boxes shown in succession may actually be executed substantially simultaneously, or the box may sometimes be executed in the opposite order. It should also be noted that each box in the block diagram and / or the flow chart diagram and the combination of the boxes in the block diagram and / or the flow chart diagram can be implemented by a system based on special-purpose hardware that performs a specified function or action or by a combination of special-purpose hardware and computer instructions.

[0132] Example

[0133] Example 1

[0134] A method comprising:

[0135] obtaining an audio stream during a procedure involving a medical system, the audio stream comprising speech of a user using the medical system to perform the procedure, wherein the procedure is performed on a subject;

[0136] obtaining at least one data stream from equipment associated with the medical system;

[0137] detecting an indicator of a problem within the audio stream, the indicator appearing at least during a time window;

[0138] selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; and

[0139] The audio segment and the at least one data segment are output.

[0140] Example 2

[0141] The method of embodiment 1, wherein the indicator of the problem is detected by identifying at least one word selected from a predetermined list of words within the audio stream.

[0142] Example 3

[0143] The method of embodiment 1 wherein the indicator of the question is detected by identifying an emotional state within the audio stream that is indicative of a question.

[0144] Example 4

[0145] A method as in embodiment 1, wherein the audio segment and the at least one data segment include audio and data, respectively, captured during a second time window prior to the time window.

[0146] Example 5

[0147] The method according to embodiment 4, wherein the second time window starts at a predetermined event.

[0148] Example 6

[0149] The method according to embodiment 4, wherein the second time window starts when the operation starts.

[0150] Example 7

[0151] The method as in embodiment 1, wherein the audio segment and the at least one data segment include audio and data, respectively, captured during a third time window that is continuous with the time window.

[0152] Example 8

[0153] The method according to embodiment 7, wherein the third time window ends at a predetermined event or when the operation ends.

[0154] Example 9

[0155] The method of embodiment 1 further comprises concealing identifying details of the object or the user within the audio segment and the at least one data segment.

[0156] Example 10

[0157] The method of embodiment 1, wherein the at least one data stream includes data related to an action performed by the user.

[0158] Embodiment 11

[0159] A method as described in embodiment 10, wherein the action performed by the user includes a user interface activation action.

[0160] Example 12

[0161] The method of embodiment 1, wherein the at least one data stream includes data related to actions taken by the system.

[0162] Embodiment 13

[0163] The method of embodiment 12, wherein the action is taken by the system in response to a user action.

[0164] Embodiment 14

[0165] The method of embodiment 13, wherein the at least one data stream includes data related to measurements made by the system.

[0166] Embodiment 15

[0167] The method of embodiment 1, wherein the system is an electrophysiological (EP) mapping and ablation system.

[0168] Example 16

[0169] A method according to embodiment 1, wherein the at least one data stream includes at least one type of data selected from the group consisting of: raw ECG digital sample records collected from electrocardiogram (ECG) electrodes; ECG processed data; intracardiac catheter navigation records; intracardiac catheter axis visualization; catheter positioning and visualization algorithm calculation results; electromagnetic signal-based positioning; current-based positioning; therapeutic radiofrequency ablation or pulsed field ablation application including duration or repetition information records; records of applied force measurements; user action logs, including records of actions performed by the user using a user interface module; keyboard, mouse or touch screen actions; and ultrasound frames or anatomical markers.

[0170] Embodiment 17

[0171] The method of embodiment 1, wherein the system is an electrophysiological (EP) mapping and ablation system.

[0172] Embodiment 18

[0173] A method according to Example 1, wherein the operation is a phacoemulsification operation.

[0174] Embodiment 19

[0175] A computerized device having a processor coupled to a memory unit, the processor adapted to perform the following steps: obtaining an audio stream during an operation involving a medical system, the audio stream comprising speech of a user of the medical system, wherein the operation is performed on a subject; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator occurring during at least a first time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment comprising audio and data, respectively, captured during at least the time window; and outputting the audio segment and the at least one data segment.

[0176] Embodiment 20

[0177] A computer program product comprises a non-transitory computer-readable medium storing program instructions, which instructions, when read by a processor, cause the processor to perform: obtaining an audio stream during an operation involving a medical system, the audio stream comprising speech of a user of the medical system, wherein the operation is performed on an object; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator appearing at least during a first time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; and outputting the audio segment and the at least one data segment.

[0178] Although the embodiments described herein are primarily directed to cardiac diagnostic applications, the methods and systems described herein may also be used in other medical applications.

[0179] It should be understood that the above embodiments are cited by way of example, and the present disclosure is not limited to what is specifically shown and described above. On the contrary, the scope of the present disclosure includes the combination and sub-combination of the aforementioned various features and their variations and modifications, which should occur to those skilled in the art when reading the above description and which are not disclosed in the prior art.

Claims

1. A method comprising: obtaining an audio stream during a procedure involving a medical system, the audio stream comprising speech of a user using the medical system to perform the procedure, wherein the procedure is performed on a subject; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator appearing at least during a time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; and The audio segment and the at least one data segment are output.

2. The method according to claim 1, wherein: The indicator of the question is detected by identifying at least one word selected from a predetermined list of words within the audio stream. Test.

3. The method according to claim 1, wherein: The indicator of the problem is indicated by The problem is detected by identifying an emotional state indicative of the problem within the audio stream.

4. The method according to claim 1, wherein: The audio segment and the at least one data segment respectively include audio captured during a second time window prior to the time window. Frequency and data.

5. The method according to claim 4, wherein: The second time window is predetermined Starts when the item is ready.

6. The method according to claim 4, wherein: The second time window starts when the operation starts.

7. The method according to claim 1, wherein: The audio clip and the at least one digital The data segments respectively include the captured data during a third time window continuous with the time window. Audio and data.

8. The method according to claim 7, wherein: The third time window ends at a predetermined event or when the operation ends.

9. The method of claim 1, further comprising concealing identifying details of the object or the user within the audio segment and the at least one data segment.

10. The method according to claim 1, wherein: The at least one data stream includes data related to actions performed by the user.

11. The method according to claim 10, wherein: The action performed by the user comprises a user interface activation action.

12. The method according to claim 1, wherein: The at least one data stream includes data related to actions taken by the system.

13. The method according to claim 12, wherein: The action is taken by the system in response to a user action.

14. The method according to claim 13, wherein: The at least one data stream comprises data related to measurements made by the system.

15. The method according to claim 1, wherein: The system is an electrophysiological (EP) mapping and ablation system.

16. The method according to claim 1, wherein: The at least one data stream includes at least one type of data selected from the group consisting of: raw ECG digital sample records collected from electrocardiogram (ECG) electrodes; ECG processed data; intracardiac catheter navigation records; intracardiac catheter axis visualization; catheter positioning and visualization algorithm calculation results; electromagnetic signal-based positioning; current-based positioning; Therapeutic radiofrequency ablation or pulsed field ablation applications including recording of duration or repetition information; recording of measurements of applied force; user action log, including recording of actions performed by the user utilizing a user interface module; keyboard, mouse or touch screen actions; and ultrasound frames or anatomical markers.

17. The method according to claim 1, wherein: The system is an electrophysiological (EP) mapping and ablation system.

18. The method according to claim 1, wherein: The procedure is a phacoemulsification procedure.

19. A computerized device having a processor coupled to a memory unit, the processor being adapted to perform the following steps: obtaining an audio stream, the audio stream including speech of a user of the medical system, during an operation involving the medical system, wherein the operation is performed on a subject; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator appearing at least during a first time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; and The audio segment and the at least one data segment are output.

20. A computer program product comprising a non-transitory computer readable medium storing program instructions, which when read by a processor cause the processor to execute: obtaining an audio stream, the audio stream including speech of a user of the medical system, during an operation involving the medical system, wherein the operation is performed on a subject; obtaining at least one data stream from equipment associated with the medical system; detecting an indicator of a problem within the audio stream, the indicator appearing at least during a first time window; selecting an audio segment within the audio stream and at least one data segment in each of the at least one data stream, the at least one data segment corresponding in time to the audio segment, the audio segment and the at least one data segment respectively comprising audio and data captured during at least the time window; as well as The audio segment and the at least one data segment are output.

Images