Synchronizing audio video data and medical data

Abstract

The invention relates to a computer-implemented method of synchronizing audio-video data with medical data, the method comprising: receiving the audio-video data (40) recorded by a first device (20), the audio-video data comprising an audio channel and a video channel simultaneously capturing a medical procedure performed in a medical environment; receiving medical data (30) recorded by a second device, the medical data capturing physiological parameters of the patient during the medical procedure; classifying one or more sounds from the audio channel of the audio video data as generated by an instrument in the medical environment using one or more machine learning algorithms; and synchronizing the audio and video data with the medical data based on the occurrence time of the one or more sounds.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for synchronizing audiovisual data and medical data, a computer-implemented method for training a machine learning algorithm, and a transient or non-transient computer-readable medium. More specifically, the medical data may capture physiological parameters of a patient during a medical procedure, and the audiovisual data may capture the medical procedure. Background Art

[0002] Medical data (such as medical images) can be captured for various reasons, for example, to record a patient's condition or physiological parameters during a medical procedure. For example, during image-guided interventions, medical images such as X-ray images can be used to monitor the progress of the procedure and view the inside of the patient, thereby providing the ability to observe devices (such as guidewires, catheters, and stents) inside the patient in real time. Sometimes, medical professionals will also record a video of the medical procedure on a separate device (e.g., a mobile device such as a cell phone). Sometimes it is necessary to synchronize the medical data and the recorded video.

[0003] It is an object of the present invention to improve the prior art. Summary of the Invention

[0004] According to a first aspect of the present invention, there is provided a computer-implemented method for synchronizing audio-video data and medical data, the method comprising: receiving audio-video data recorded by a first device, the audio-video data comprising an audio channel and a video channel simultaneously capturing a medical procedure performed in a medical environment; receiving medical data recorded by a second device; classifying one or more sounds from an audio channel of the audio-video data as being produced by an instrument in the medical environment using one or more machine learning algorithms; and synchronizing the audio-video data with the medical data based on a time of occurrence of the one or more sounds that have been classified as being produced by the instrument.

[0005] In this way, it is easy to synchronize the audiovisual data with the medical data, for example for the purpose of generating a composite video, because the method does not rely on acoustic features (humans speaking specific terms), which can be distracting to medical personnel. Instead, the method provides a passive means for synchronizing the corresponding data modalities.

[0006] In an embodiment, the apparatus may be a medical imaging system comprising the second device, and the medical data may include a medical image, such as an X-ray image.

[0007] In another example, the method further includes recording one or more events associated with the medical instrument in an event log, and wherein synchronization can include temporally matching one or more sounds with corresponding one or more events from the event log. This can provide advantages by enabling passive synchronization of features present in medical data at different stages with the instruments used during those stages. In particular, events can be recorded while the second device is capturing medical images. For example, when the second device is a C-arm X-ray imaging device used during certain medical interventions in a catheterization lab, temporal matching can involve recorded catheterization lab events.

[0008] In some examples, one or more sounds produced by the instrument include a sound characteristic produced by a speaker system controlled by the second device, the sound characteristic indicating a system clock time of the second device, wherein synchronizing the audio-visual data includes temporally matching the one or more sounds with one or more timestamps indicating the system clock time.

[0009] In this way, accurate synchronization is provided by actively projecting the time signature from the second device to the first device.

[0010] In an embodiment, generating the sound signature may include generating a first sound pattern at a first frequency and generating a second sound pattern at a second frequency, wherein the first frequency and the second frequency may be above a human audible frequency range, and a difference between the first frequency and the second frequency may be within a human audible frequency range. In this manner, the first frequency and the second frequency are not detectable by humans, thereby preventing medical personnel from being distracted during a medical procedure, while the difference frequency is within a recording frequency range of a microphone.

[0011] In an embodiment, the computer-implemented method may further comprise: detecting a display of the second device in the video channel of the audio-video data using one or more machine learning algorithms; identifying one or more display features on the display of the second device using the one or more machine learning algorithms; and wherein synchronizing the audio-video data with the medical data further comprises matching the time of occurrence of the one or more display features in the audio-video data with the time of occurrence of one or more features displayed on the display of the second device. In this manner, the accuracy of synchronization is improved by matching features displayed on the second device with features recorded by the first device. In this manner, image features may be used to fine-tune the sound of the instrument used for detection.

[0012] In an embodiment, the one or more display features may include a system clock of the second device. The system clock may be accurate to the granularity of the time measurement of the displayed clock. In the case where the time is provided in seconds, this can help fine-tune synchronization.

[0013] In an embodiment, the one or more display features may include medical data.When a display feature changes, its timing can be detected almost instantaneously, again fine-tuning the synchronization.

[0014] According to another aspect of the present invention, there is provided a computer-implemented method for generating a composite video. The method comprises: synchronizing audio-visual data and medical data as described above; detecting one or more video features from the medical procedure in the video channel of the audio-visual data; and generating the composite video by identifying a portion of the synchronized audio-visual data and medical data for display based on the time of occurrence of the or each video feature.

[0015] In an embodiment, the one or more video features may include a first video feature and a second video feature, and wherein generating the composite video may include: identifying a first occurrence time of the first video feature in the video channel; identifying a second occurrence time of the second video feature in the video channel; and identifying the portion of the composite video as between the first occurrence time and the second occurrence time.

[0016] In an embodiment, the computer-implemented method may further comprise displaying the portion of the composite video on a display of the first device.

[0017] In an embodiment, the first video feature may include a recording start, and the second video feature may include a recording end. Each recording must have a start and an end, so this is a reliable way to provide the first display feature and the second display feature. Furthermore, this approach is passive, as no other features need to be actively detected.

[0018] In an embodiment, generating the composite video data may include enhancing the video channel with the medical data.

[0019] In an embodiment, the one or more display features may include the medical data, wherein enhancing the video channel with the medical data may include replacing medical data identified in the video channel as one or more display features on a display of the second device with the medical data recorded by the second device. In this manner, the medical data captured by the second device may be at a higher resolution than the medical data displayed on the display of the second device and recorded by the first device.

[0020] In an embodiment, the medical data may be a medical image.

[0021] According to another aspect of the present invention, there is provided a computer-implemented method for training one or more machine learning algorithms for classifying one or more sounds from an audio channel of audio-visual data as being generated by an instrument in a medical environment, the method comprising: providing an audio channel from the audio-visual data and a label associating one or more sounds in the audio channel with the instrument; and, training the one or more machine learning algorithms using supervised learning based on the label to perform the classification of the one or more sounds in the audio channel. Preferably, the one or more machine learning algorithms trained according to the present invention are used in the synchronization method further described and claimed herein.

[0022] According to another aspect of the present invention, there is provided a transitory or non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the method of any preceding claim.

[0023] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The embodiments of the invention described herein may be best understood with reference to the accompanying drawings, in which:

[0025] Figure 1 A perspective view of a medical environment including medical instruments and medical imaging equipment is shown;

[0026] Figure 2 shows the capture of Figure 1 a stereoscopic image of a mobile device for audio-video recording of a medical procedure performed in a medical environment;

[0027] Figure 3 A block diagram of a document management system for managing medical data and audio / video data according to an embodiment of the present invention is shown;

[0028] Figure 4 A schematic diagram illustrating a method for synchronizing medical data and audio / video data according to an embodiment of the present invention is shown;

[0029] Figure 5 The method for recognizing sound from the audio channel of audio video data according to an embodiment of the present invention is shown. Figure 4 Schematic diagram of the synchronization method;

[0030] Figure 6shows before and after stereoscopic views of a mobile device displaying a video and a composite video including medical data and audio-video data, respectively, according to an embodiment of the present invention;

[0031] Figure 7 shows before and after stereoscopic views of a mobile device displaying a video and a composite video including medical data and audio-video data, respectively, according to an embodiment of the present invention;

[0032] Figure 8 A flowchart illustrating a method for synchronizing audio and video data with medical data according to an embodiment of the present invention; and

[0033] Figure 9 A flowchart of a method for training one or more machine learning algorithms according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] The methods described herein may be computer-implemented methods. In particular, the methods may include at least a computer-implemented method for generating a composite video and a computer-implemented method for training one or more machine learning algorithms. A computer implementing the methods may include a storage device (e.g., a memory) and a processor. The computer may be a hardware computer, and thus the storage device and the processor may be a hardware storage device and a hardware processor, respectively.

[0035] The computer-implemented method may be provided as instructions stored on a transient or non-transient computer-readable medium. The computer-readable medium may be stored in a storage device of a computer. When the instructions are executed by a processor, the processor performs any method steps described herein.

[0036] refer to Figure 1 , a medical procedure is performed in a medical environment 10 (e.g., an operating room). The medical environment 10 includes an instrument 12. The instrument may include medical instruments for performing the medical procedure, such as a workbench, a drip chamber, a catheter, a scalpel, etc. In addition, the instrument may include a medical data capture device (also referred to herein as a second device) 12. Figure 1In the embodiment, the medical data capture device 12 is a C-arm X-ray imaging system. In other cases, a different medical data capture device may be used, where the medical procedure requires it. The medical data capture device 12 may include a sensor 14 and a display 16. The sensor 14 may be configured to capture physiological parameters of the patient during the medical procedure. In the case of a C-arm, an emitter or source 18 may also be provided. The emitter 18 may emit X-ray signals and the sensor 14 may detect X-rays. Thus, the medical data generated by the C-arm may be a medical image. The medical image may be an X-ray. In other embodiments, the medical image may be, for example, a computed tomography (CT) scan image, an ultrasound scan image, a magnetic resonance imaging scan, etc. The display 16 may display the medical image in real time during the medical procedure to provide guidance to medical personnel.

[0037] refer to Figure 2 , an audio and video recorder 20 for recording audio and video data is also within the medical environment 10. The audio and video recorder 20 can be a mobile device and can be further referred to as a first device herein. The mobile device can be a smartphone or a tablet. The audio and video recorder 20 can also be part of another system, such as an augmented reality (AR) headset or a head-mounted camera.

[0038] During a medical procedure, a user (e.g., a medical professional) may wish to record the stages of the medical procedure. For example, the recording may be used for training. The user may use a first device 20 to record the stages of a medical procedure 28. The first device 20 may include a camera 22, a microphone 24, and a display 26. The display 26 may be in the form of a touch screen.

[0039] The camera 22 can be configured to capture video data of its field of view. In this context, the field of view includes the medical environment 10. The microphone 24 can be configured to capture audio data from near the first device 20. The internal clock of the first device 20 can be used to synchronize the video data and the audio data. The synchronized audio data and visual data can be referred to as audio-video data. In this way, the audio-video data can include an audio channel representing the audio data and a video channel representing the video data. The audio channel and the video channel have simultaneously captured the medical procedure or at least one stage of the medical procedure in the medical environment. As will be described in more detail below, the display 16 of the second device 12 can also be recorded in the video channel of the audio-video data. The display 16 can display the medical data (e.g., medical image) 30 captured by the second device 12.

[0040] refer to Figure 3 , a database system can be provided to manage and store medical data captured by one or more second devices 12. The database system can also manage and store audio and video data captured by one or more first devices 20.

[0041] The database system may include a server 34. The first device 20 and the second device 12 may be communicatively linked to the server 34. The communication link may be provided as a wireless link, such as Bluetooth or Wi-Fi, or may be a physical connection, such as via a cable or wire. In some embodiments, the first device 20 and the second device 12 may be directly connected together, that is, in addition to being indirectly connected via the server 34. The direct connection may be provided wirelessly, such as Bluetooth or Wi-Fi, or may be a physical connection, such as via a cable or wire.

[0042] One or more user interfaces 36 may also be provided and communicatively linked to the server 34 to access, view, and edit, for example, any data stored by the server 34. In this manner, the server 34 may be connected to a storage medium storing any audio-visual data 40 and medical data 30 received from the respective first and second devices 20, 12.

[0043] The database system may be a Picture Archiving and Communication System (PACS). The medical data may conform to a standard, such as Digital Imaging and Communications in Medicine (DICOM).

[0044] According to one or more embodiments of the present invention, a method for synchronizing audiovisual data and medical data is provided. The method may be performed by a document management system, by the first device 20, by the second device 12, or by another device (eg, the server 34).

[0045] The method begins by receiving audio-visual data 40 recorded by a first device 20 and receiving medical data 30 recorded by a second device 12. Next, the method includes classifying one or more sounds from an audio channel of the audio-visual data 40 as being produced by an instrument in the medical environment 10 using a machine learning algorithm.

[0046] refer to Figure 4 , the first device 20 captures audio-visual data 40 of a stage of a medical procedure. The audio-visual data 40 includes an audio channel 42 and a video channel 44. Although the audio-visual data 40 is shown on the display 26 of the first device 20, this is shown schematically and for illustrative purposes only. In practice, the display 26 does not visually display the audio-visual data in terms of the trace / waveform data and image frames.

[0047] refer to Figure 5, the audio channel 42 is input to a machine learning algorithm 46. The machine learning algorithm 46 may be a supervised machine learning algorithm. The supervised machine learning algorithm may be trained by providing the audio channel from the audio video data and labels identifying one or more sounds in the audio channel of an instrument used in a medical environment. The machine learning algorithm may be trained using supervised learning based on the labels to classify one or more sounds in the audio channel as being associated with the instrument.

[0048] The machine learning algorithm may be a neural network. In this embodiment, the neural network may be a recurrent neural network. In other embodiments, the neural network may be a convolutional neural network or a transformer network.

[0049] Training can include forward propagation and backward propagation. In the forward propagation, samples 52 of the audio channel from the training data are input to the neural network. Samples 52 can be acquired periodically. Samples 52 can have substantially equal durations. Each sample 52 passes through the neural network, which outputs a value using an output layer. The output layer can include a softmax layer. The softmax layer can include multiple nodes, each node representing the probability that the sample of audio data was generated by a specific event. The specific event can be a sound generated by a medical instrument in a medical environment.

[0050] The neural network generates an output vector 48 comprising a plurality of values. Each value corresponds to a classification of the sample 52. In other words, when the neural network determines that the sample corresponds to a particular source or has the highest probability of corresponding to a particular source, a value for that event is provided. For example, a value of 1 may correspond to the first source.

[0051] For example, output 0 may correspond to background noise, output 1 may correspond to a piece of medical equipment being moved, eg, a table being lowered, output 2 may correspond to a piece of medical equipment being moved, eg, the second device 12 being moved, and so on.

[0052] A loss function is used to calculate the loss or error between the output vector 48 and the ground truth vector 50. Backpropagation is used to optimize the hyperparameters of the neural network, such as the weights within the layers, based on the loss function. The loss function can be a least absolute deviation (L1) loss function or a least square error (L2) loss function.

[0053] As described above, the task of the neural network is to detect and classify characteristic audio events (one or more sounds) in the presence of other background noise while taking into account acoustic variations in the recorded audio due to the position relative to the C-arm and the acoustic characteristics of the recording device, the room, etc. One method for generating training data for such a network is to collect audio data; and, label the collected audio data using a system event log, wherein the audio data is synchronized with the system event log and associated with the medical data at various locations and associated with various recording devices in the presence of various background noises. Alternatively, a series of events known in the medical data can be triggered and recordings obtained. These recordings can be made using various recording systems, various background sounds, from various locations in the medical environment relative to the medical data capture instrument, etc. The neural network is trained to identify relevant sounds from the instrument from these recordings that correspond to the recorded events in the system log.

[0054] Further references Figure 4 In this embodiment, the neural network recognizes four sounds. The present invention is not limited to the specific existence of four sounds. These four sounds are Figure 4 In the example, the video events are indicated as VE1, VE2, VE3 and VE4. VE may be an acronym for video event. The term "video event" is intended to mean an audio-visual event, since the actual event is the sound in the audio channel.

[0055] The first video event VE1 or first sound may be the sound of a C-arm starting to move. The second video event VE2 or second sound may occur after the first video event VE1. The second video event VE2 may be the sound of a C-arm machine's pedal being pressed. The third video event VE3 or third sound may be the sound of the C-arm stopping movement or engaging the C-arm's brake. The fourth video event VE4 or fourth sound may be the sound of the pedal being released. In this manner, the method includes classifying sounds as being generated by medical instruments in a medical environment during a stage of a medical procedure.

[0056] When sequences of events like these are detected, the timing between these events creates very specific patterns.

[0057] These sound generating events can occur in an intubation lab or catheterization lab equipped with the first and second devices such that their clocks are synchronized. In this way, any pattern of sound generating events detected in the audio stream of the first device 20 can be correlated with a matching event pattern in the event log of the second device 12, thereby allowing synchronization of the two clocks.

[0058] For example, the symbols CE1, CE2, ..., CEn can be used to describe catheterization laboratory events. Figure 4In the illustrated embodiment, there are nine catheterization lab events CE1 to CE9. The first catheterization lab event CE1 and the second catheterization lab event CE2 may be due to the patient table being moved upward. The third catheterization lab event CE3 and the fourth catheterization lab event CE4 may be due to the table being translated toward the C-arm. The fifth to eighth catheterization lab events CE5 to CE8 may correspond to the first to fourth video events VE1 to VE4, respectively. The ninth catheterization lab event CE9 may be any event, such as adjusting the table or pressing the C-arm pedal to acquire a subsequent image.

[0059] It should be noted that the first through fourth video events VE1 through VE4 and the fifth through eighth catheterization lab events CE5 through CE8 occur during a phase of a medical procedure. The phase of the medical procedure may be a particularly important phase. For example, it may be assumed that a physician or other catheterization lab staff member will record the phase of the medical procedure on the first device 20 because it is a particularly important or significant phase.

[0060] In this way, the method includes synchronizing audio-visual data with medical data based on the time of occurrence of sounds. Specifically, synchronization is achieved by temporally matching the occurrence of sounds from audio channels VE1 to VE4 with matching catheterization lab events (CE5 to CE8).

[0061] It should be understood that the instrument can be a medical imaging system including a second device, and the medical data can include a medical image. The method can also include recording one or more events performed by the medical imaging system in an event log when the medical image is captured by the second device. Synchronization can include temporally matching the one or more sounds with corresponding one or more events from the event log.

[0062] Briefly return to Figure 1 In another embodiment, the second device 12 may further include a speaker 50. The speaker 50 is synchronized with the medical data recorded by the second device 12 because it is connected to the same system clock. In other words, the speaker 50 is controlled by the second device 12. The speaker 50 is configured to generate sound. The sound may be in the form of a sound characteristic. The sound characteristic may indicate the system clock of the second device. For example, the time pattern of the pulses indicates the time of the system clock (e.g., a short pulse indicates a 0 and a longer pulse indicates a 1, and the emission of a binary sequence encodes the clock time).

[0063] The microphone 24 of the first device 20 can detect the sound signature. Audio and video data recorded simultaneously with the sound signature is recorded and can be marked with a timestamp. In this manner, synchronizing the audio and video data includes aligning the timestamps in the audio and video data with matching timestamps in the medical imaging system. If the time shift between the clocks is a fraction of the clock resolution, some interpolation may be required, as will be understood by those skilled in the art.

[0064] In the same or another embodiment, the sound signature may include a first sound pattern and a second sound pattern. Speaker 50 may be configured to generate a first sound pattern at a first frequency and a second sound pattern at a second frequency. The first frequency may be 40 kHz, and the second frequency may be 50 kHz. Those specific frequency values ​​may be different, although the first and second frequencies are preferably above the audible limit of human hearing. Furthermore, the first and second frequencies are preferably different from each other. In this way, nonlinearities in the sound recording hardware associated with microphone 24 can be exploited. This is because nonlinearities in the mechanical membrane of microphone 24 and the initial amplifier stage will generate sum and difference frequency components. For the example using frequencies of 40 kHz and 50 kHz, the difference frequency is 10 kHz. 10 kHz is within the recording frequency range of microphones specifically configured to record human-audible sound, such as the microphones of mobile devices such as smartphones. Therefore, even if the first and second frequencies are above the audible range of the human ear, they will result in an audible 10 kHz signal recorded by first device 20. In this way, the sound signature can be emitted without distracting medical personnel within medical environment 10. Alternatively, the speaker 50 may include a first speaker 52 and a second speaker 54. The first speaker 52 may be configured to generate a first sound pattern at a first frequency. The second speaker 54 may be configured to generate a second sound pattern at a second frequency.

[0065] In addition to the synchronization methods described above, further embodiments may include additional synchronization methods.

[0066] For example, the method may further include detecting, by the first device 20, the display 16 of the second device 12 in the video channel of the audio-video data using a machine learning algorithm. Furthermore, the machine learning algorithm may be used to identify one or more display features on the display 16. The machine learning algorithm may include a neural network. The neural network may be a convolutional neural network. The convolutional neural network may be trained to classify the display as a display and further trained to classify one or more display features. Classifying the display feature may include classifying the display feature as the system clock of the second device 12. Additionally or alternatively, the machine learning algorithm may be trained to classify the display feature as medical data 30 being displayed on the display 16.

[0067] In this manner, synchronizing the audio-visual data with the medical data may also include matching a time of occurrence of one or more displayed features with a time of occurrence of display of the one or more features on a display by the second device.

[0068] Reference Figure 3 Regardless of the particular synchronization method used, the synchronized audio-visual data and medical data may be stored in a storage medium.

[0069] Additionally, the method may comprise generating a composite video by identifying a portion of the synchronized audio-video data and medical data for display based on the time of occurrence of the or each video feature.

[0070] For example, a video feature may include a first video feature and a second video feature. A first occurrence time of the first video feature may be identified in the video channel. A second occurrence time of the second video feature may be identified in the video channel. The portion of synchronized audio, video data and medical data may be identified as being between the first occurrence time and the second occurrence time. In this manner, a composite video may be generated that includes the synchronized audio, video data and medical data between the first occurrence time and the second occurrence time. The composite video may be displayed on a display.

[0071] refer to Figure 1 , the display may be the display 26 of the first device 20, the display 16 of the second device 12, or another display.

[0072] In one or more embodiments, the first video feature may be recording start. In other words, when the user of first device 20 begins recording a stage of a medical procedure, the time when recording begins is identified. The second video feature may be recording end. In other words, when the user of first device 20 stops recording a stage of a medical procedure, the time when recording ends is identified. It can be assumed that the user is recording a significant stage of the medical procedure. Using recording start and end does not require identifying further video features. In other words, using recording start and end is a passive way to identify significant stages of a medical procedure.

[0073] Additionally, in one or more embodiments, any data (e.g., medical data) that is not identified as being between the first video feature and the second video feature may be discarded or deleted. In this way, storage space may be freed up ( Figure 3 ).

[0074] The data captured during the recorded time period can also be used as training data to train the various machine learning algorithms described herein.

[0075] refer to Figure 6In one or more embodiments, a composite video may be constructed by enhancing the video channel of the audio-video data with the medical image. For example, Figure 6 The first device 20 on the left shows the original video channel being displayed on the display 26 . Figure 6 The first device 20 on the right shows a video channel of audio-visual data 40 enhanced with a medical image 30 .

[0076] refer to Figure 7 In one or more embodiments, a composite video may be constructed by replacing the medical data in the video channel of the audio-video data with the medical images recorded directly by the second device 12. For example, Figure 7 The first device 20 on the left shows a video channel of audio-visual data 40 displayed on the display 26. The video channel includes a medical image ( Figure 1 ). Figure 6 The first device 20 on the right shows a medical image in the video channel of the audio-video data replaced with a medical image 30 recorded by the second device 12. Replacing the medical image with the medical image captured by the second device 12 is advantageous because it has a higher resolution than the video captured by the first device 20, which is also subject to various image artifacts (moire effects, aliasing, mirror reflections of the screen, etc.) due to capturing the data shown on the screen.

[0077] In one or more embodiments, it may be desirable to tag a composite video based on speech recognition from an audio channel of the audio video data.

[0078] In such an embodiment, keywords can be looked for to identify the stage of a process or event of interest. These tags can be attached to the stored medical data for more efficient viewing when generating training materials or placed as subtitles in video content. Some example phrases to listen for include:

[0079] “Prepping the groin” means we are close to starting the invasive portion of the procedure.

[0080] "Close" indicates the end of the invasive portion of the procedure.

[0081] "Start Run" may indicate that contrast media is about to be injected or that fluoroscopic imaging is about to be acquired.

[0082] "Hold your breath" may indicate that important medical data is about to be collected in the next few seconds (eg, a fluoroscopy sequence with breath hold).

[0083] "You can breathe" indication deadline now completes collecting data

[0084] "Administering heparin" is important for documentation of the medication.

[0085] If medical data is currently labeled manually, a deep learning system can be trained over time to associate labels with phrases from speech recognition on audio tracks and begin suggesting labels to gradually automate the process.

[0086] In another embodiment, personally identifiable information such as patient names and other details can be automatically removed when medical data is included in a document. This can be achieved by removing the identifiable information from the DICOM header. If this information is present as an overlay in the medical image data or in the audio-visual data, this can be achieved by using any method that can detect personally identifiable information in the medical image data (e.g., a patient's name, national health number, date of birth, etc. overlaid on the medical image data that may be blurred) or in the audio-visual data (e.g., a patient's face that may be blurred, or a catheterization lab worker mentioning the patient's age in a video channel that may be cut out of the audio channel).

[0087] refer to Figure 8 Although the foregoing description describes various embodiments in detail, some embodiments can be summarized as involving a computer-implemented method for synchronizing audio-video data and medical data. The method includes various steps. At step S100, the method includes receiving audio-video data recorded by a first device, the audio-video data including an audio channel and a video channel that simultaneously capture a medical procedure performed in a medical environment. At step S102, the method includes receiving medical data recorded by a second device. At step S104, the method includes using one or more machine learning algorithms to classify one or more sounds from an audio channel of the audio-video data as being produced by an instrument in the medical environment. At step S106, the method includes synchronizing the audio-video data with the medical data based on a time of occurrence of the one or more sounds.

[0088] refer to Figure 9 Although the foregoing description describes various embodiments in detail, some embodiments can be summarized as a computer-implemented method involving training one or more machine learning algorithms using various steps. At step S200, the method includes providing an audio channel from audio-video data and labels identifying one or more sounds in the audio channel of an instrument used in a medical environment. At step S202, the method includes training the one or more machine learning algorithms using supervised learning based on the labels to classify the one or more sounds in the audio channel as being associated with the instrument.

[0089] Although the present invention has been described and illustrated in detail in the drawings and the foregoing description, such illustration and description are to be considered illustrative or exemplary rather than restrictive; the invention is not limited to the disclosed embodiments. For example, it is possible to operate the present invention in which more than two capture devices are used and more than two modalities are used (e.g., more than audio-visual data and medical images).

[0090] If there are many information sources, you may want to show different subsets of sources at different times in the process, depending on what is going on. When generating training material, a user interface can be used: First, a video is created with all the information panes and a checkbox on each pane. The user can then check / uncheck the desired panes in time. Now the system generates a new video (e.g., a composite video) showing only the desired information panes at different times. AI methods can be used to predict what the desired pane will look like depending on time (e.g., a process card).

[0091] From a study of the drawings, the present disclosure and the appended claims, other variations of the disclosed embodiments may be understood and implemented by those skilled in the art in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may perform the functions of several items recited in the claims. The fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium supplied with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method for synchronizing audio-visual data with medical data during a medical procedure, the method comprising: receiving (S100) the audio-video data (40) recorded by the first device (20), the audio-video data comprising an audio channel and a video channel simultaneously capturing the medical procedure performed in the medical environment (10), Receiving (S102) the medical data (30) recorded by the second device (12); using one or more machine learning algorithms to classify (S104) one or more sounds from the audio channel of the audio-video data as being produced by an instrument in the medical environment; as well as The audio-visual data is synchronized with the medical data based on the occurrence time of the one or more sounds that have been classified as being generated by the instrument (S106).

2. The computer-implemented method of claim 1, in, The apparatus is a medical imaging system comprising the second device, and the medical data comprises a medical image.

3. The computer-implemented method according to claim 1 or 2, in, The method further includes recording one or more events associated with the medical instrument in an event log, and The synchronizing (S106) includes temporally matching the one or more sounds with corresponding one or more events from the event log.

4. A computer-implemented method according to any preceding claim, in, the one or more sounds produced by the instrument include a sound characteristic produced by a speaker system (50) controlled by the second device (12), the sound characteristic being indicative of a system clock time of the second device (12), Wherein synchronizing the audio-visual data (40) comprises matching the one or more sounds in time with one or more time stamps indicating the system clock time.

5. The computer-implemented method of any preceding claim, further comprising: Using one or more machine learning algorithms, detecting a display (16) of the second device (12) in the video channel of the audio-video data (40); identifying one or more display features on the display (16) of the second device using the one or more machine learning algorithms; and Wherein, the synchronization of the audio-video data (40) with the medical data also includes matching the occurrence time of the one or more display features in the audio-video data with the occurrence time of the one or more features being displayed on the display by the second device (12).

6. The computer-implemented method of claim 5, wherein: The one or more display characteristics include a system clock of the second device (12).

7. A computer-implemented method according to claim 5 or claim 6, wherein: The one or more display features include the medical data (30).

8. A computer-implemented method of generating a composite video, the method comprising: Synchronising audio-visual data with medical data as claimed in any preceding claim; detecting one or more video features from the medical procedure in the video channel of the audio video data; as well as The composite video is generated by identifying portions of the synchronized audio-video data and medical data for display based on the time of occurrence of the or each video feature.

9. The computer-implemented method of claim 8, wherein: The one or more video features include a first video feature and a second video feature, and wherein generating the composite video includes: identifying a first occurrence time of the first video feature in the video channel; identifying a second occurrence time of the second video feature in the video channel; and The portion of the composite video is identified as being between the first occurrence time and the second occurrence time.

10. The computer-implemented method of claim 9, wherein: The first video feature comprises a start of recording and the second video feature comprises an end of recording.

11. A computer-implemented method according to any one of claims 8 to 10, wherein: Generating the composite video data includes enhancing the video channel using the medical data (30).

12. The computer-implemented method of claim 11, wherein: The one or more display features include the medical data (30), wherein enhancing the video channel with the medical data (30) includes replacing the medical data in the video channel identified as the one or more display features on the display of the second device (12) with the medical data (30) recorded by the second device (12).

13. A computer-implemented method of training one or more machine learning algorithms for classifying one or more sounds from an audio channel of audio-video data as being produced by an instrument in a medical environment, the method comprising: providing (S200) an audio channel from audio video data (40) and a tag associating one or more sounds in the audio channel with an instrument used in a medical environment (10); as well as The one or more machine learning algorithms are trained ( S202 ) using supervised learning based on the labels to perform the classification of the one or more sounds.

14. A transient or non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the method of any preceding claim.

Images