SYNCHRONIZING AUDIOVISUAL AND MEDICAL DATA

Abstract

The present invention relates to a computer-implemented method for synchronizing audiovisual data and medical data, the method comprising: receiving the audiovisual data (40) recorded by a first device (20), wherein the audiovisual data includes an audio channel and a video channel simultaneously capturing a medical procedure performed in a medical environment; receiving the medical data (30) recorded by a second device, wherein the medical data captures physiological parameters of a patient during the medical procedure; classifying, using one or more machine learning algorithms, one or more sounds from the audio channel of the audiovisual data as being generated by equipment in the medical environment; and synchronizing the audiovisual data with the medical data based on a time of occurrence of one or more sounds.

Description

AREA

[0001] The present invention relates to a computer-implemented method for synchronizing audiovisual data and medical data, a computer-implemented method for training a machine learning algorithm, and a transitory or non-transitory computer-readable medium. More specifically, the medical data may capture physiological parameters of a patient during a medical procedure, and the audiovisual data may capture the medical procedure. STATE OF THE ART

[0002] Medical data, such as medical images, can be captured for a variety of reasons, for example, to record a patient's condition or physiological parameters during a medical procedure. During an image-guided procedure, for example, medical images, such as X-rays, can be used to monitor the progress of the procedure and view the patient's internal organs, providing the ability to observe devices such as guidewires, catheters, and stents inside the patient in real time. Sometimes, medical professionals also record a video of the medical procedure on a separate device, such as a mobile device like a cell phone. Sometimes synchronization of the medical data and the recorded video is desired.

[0003] An object of the present invention is to improve the state of the art. SUMMARY

[0004] According to a first aspect of the present invention, there is provided a computer-implemented method for synchronizing audiovisual data and medical data, the method comprising: receiving the audiovisual data recorded by a first device, the audiovisual data including an audio channel and a video channel concurrently capturing the medical procedure performed in a medical environment; receiving the medical data recorded by a second device; classifying, using one or more machine learning algorithms, one or more sounds from the audio channel of the audiovisual data as being generated by equipment in the medical environment; and synchronizing the audiovisual data with the medical data based on a time of occurrence of the one or more sounds classified as being generated by the equipment.

[0005] In this way, the audiovisual data and the medical data can be easily synchronized, for example, to create a composite video, since this method does not rely on sound features such as human pronunciation of certain terms, which could distract medical personnel. Instead, this method provides a passive means of synchronizing the respective data modalities.

[0006] In one embodiment, the equipment may be a medical imaging system comprising the second device, and the medical data may include medical images, such as x-ray images.

[0007] In further examples, the method further comprises logging, in an event log, one or more events associated with the medical equipment, and wherein synchronizing may comprise synchronizing the one or more tones in time with the respective one or more events from the event log. This may provide an advantage because it enables passive synchronization of the features present in the medical data at different phases and the equipment used during those phases. In particular, events may be logged while medical images are being acquired by the second device. For example, if the second device is a C-arm x-ray imaging device used in certain medical procedures in a cath lab, the synchronization in time may include logged cath lab events.

[0008] In certain examples, the one or more tones generated by the equipment include a sound signature generated by a speaker system controlled by the second device, the sound signature indicative of a system clock time of the second device, wherein synchronizing the audiovisual data comprises timing the one or more tones with one or more timestamps indicative of the system clock time.

[0009] In this way, accurate synchronization is provided by actively projecting the time signature from the second device to the first device.

[0010] In one embodiment, generating the sound signature may comprise: generating a first sound pattern having a first frequency; generating a second sound pattern having a second frequency, wherein the first frequency and the second frequency may be above a human-audible frequency range, and a difference between the first frequency and the second frequency may be within the human-audible frequency range. In this way, the first and second frequencies are imperceptible to humans, and medical personnel are thus not distracted during the medical procedure, even though the difference frequency is within a recording frequency range of a microphone.

[0011] In one embodiment, the computer-implemented method may further comprise: detecting, using one or more machine learning algorithms, a display of the second device in the video channel of the audiovisual data; identifying, using the one or more machine learning algorithms, one or more display features on the display of the second device; and wherein synchronizing the audiovisual data with the medical data further comprises matching a time of occurrence of the one or more display features in the audiovisual data with a time of occurrence of the one or more features on the display by the second device. In this way, the accuracy of the synchronization is improved by matching the features displayed on the second device with the features recorded by the first device.In this way, the tones that the equipment in use detects can be fine-tuned using image feature recognition.

[0012] In one embodiment, the one or more display features may include a system clock of the second device. The accuracy of system clocks may correspond to the level of detail of the time measurements of the displayed clock. Providing the time in seconds may aid in fine-tuning synchronization.

[0013] In one embodiment, the one or more display features may include the medical data. When display features change, their timing can be detected almost instantaneously, allowing for fine-tuning of the synchronization.

[0014] According to another aspect of the present invention, a computer-implemented method for generating a composite video is provided. The method comprises: synchronizing audiovisual data and medical data as described above; detecting one or more video features of the medical procedure in the video channel of the audiovisual data; and generating the composite video by identifying a portion of the synchronized audiovisual data and medical data for display based on a time of occurrence of the or each video feature.

[0015] In one embodiment, the one or more video features may comprise first and second video features, and wherein generating the composite video may comprise: identifying a first time of occurrence of the first video feature in the video channel; identifying a second time of occurrence of the second video feature in the video channel; and identifying the portion of the composite video as being between the first time of occurrence and the second time of occurrence.

[0016] In one embodiment, the computer-implemented method may further comprise displaying the portion of the composite video on a display of the first device.

[0017] In one embodiment, the first video feature may include the recording start, and the second video feature may include the recording end. Each recording must have a start and end point, and therefore this is a reliable way to provide the first and second display features. This approach is also passive, as no additional features need to be actively detected.

[0018] In one embodiment, generating the composite video data may include augmenting the video channel with the medical data.

[0019] In one embodiment, the one or more display features may include the medical data, wherein augmenting the video channel with the medical data may include replacing medical data in the video channel identified as the one or more display features on the display of the second device, wherein the medical data is recorded by the second device. In this way, the resolution of the medical data acquired by the second device may be higher than the medical data displayed on the display of the second device and recorded by the first device.

[0020] In one embodiment, the medical data may be a medical image.

[0021] According to another aspect of the invention, there is provided a computer-implemented method for training one or more machine learning algorithms for the task of classifying one or more sounds from an audio channel of audiovisual data as being generated by equipment in a medical environment, the method comprising: providing the audio channel of audiovisual data and a label associating one or more sounds in the audio channel with the equipment; and training the one or more machine learning algorithms using supervised learning based on the label to perform the classification of the one or more sounds in the audio channel. Preferably, the one or more machine learning algorithms trained hereby are employed in a synchronization method as further described and claimed herein.

[0022] According to a further aspect of the invention, a transitory or non-transitory computer-readable medium is provided having instructions stored thereon which, when executed by a processor, cause the processor to perform the method according to any one of the preceding claims.

[0023] These and other aspects of the invention will become apparent from and be explained with reference to the embodiment(s) described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The embodiments of the inventions described herein are best understood by reference to the accompanying drawings in which: Fig. 1 shows a perspective view of a medical environment including medical equipment and a medical imaging device; Fig. Figure 2 shows a perspective view of a mobile device used to capture an audiovisual recording of a medical procedure performed in the medical environment of Fig. 1 is executed; Fig. 3 shows a block diagram of a document management system for managing the medical data and the audiovisual data according to an embodiment of the invention; Fig. 4 shows a schematic diagram illustrating a method for synchronizing medical data and audiovisual data according to an embodiment of the invention; Fig. 5 shows a schematic diagram illustrating a method for identifying sounds from an audio channel of the audiovisual data according to an embodiment of the invention for use in the method for synchronizing Fig. 4 illustrates; Fig. 6 are perspective views of a mobile device (before and after) displaying a video and a composite video including the medical data and the audiovisual data, respectively, according to an embodiment of the invention; Fig. 7 shows perspective views of a mobile device (before and after) displaying a video and a composite video with the medical data and the audiovisual data, respectively, according to an embodiment of the invention; Fig. 8 shows a flowchart of a method for synchronizing audiovisual data and medical data according to an embodiment of the invention and Fig. 9 shows a flowchart of a method for training one or more machine learning algorithms according to an embodiment of the invention. DESCRIPTION OF EMBODIMENTS

[0025] The methods described herein may be computer-implemented methods. In particular, the methods may include at least one computer-implemented method for generating a composite video and one computer-implemented method for training one or more machine learning algorithms. The computer implementing the methods may include storage, e.g., memory, and a processor. The computer may be a hardware computer, and thus, the storage and processor may each be hardware storage and a hardware processor.

[0026] The computer-implemented methods may be provided as instructions stored on a transitory or non-transitory computer-readable medium. The computer-readable medium may be stored in the memory of the computer. When executed by the processor, the instructions cause the processor to perform any of the method steps described herein.

[0027] With reference to Fig. 1, a medical procedure is performed in a medical environment 10, e.g., an operating room. The medical environment 10 includes equipment 12. The equipment may include medical equipment for performing the medical procedure, e.g., a table, a drip, a catheter, a scalpel, etc. Additionally, the equipment may include a device for acquiring medical data (hereinafter also referred to as a second device) 12. In Fig. 1, the medical data acquisition device 12 is a C-arm x-ray imaging system. In other cases, other medical data acquisition devices may be used if the medical procedure requires it. The medical data acquisition device 12 may include a sensor 14 and a display 16. The sensor 14 may be configured to acquire physiological parameters of a patient during the medical procedure. In the case of a C-arm, an emitter or source 18 may also be provided. The emitter 18 may emit x-ray signals, and the sensor 14 may detect x-rays. The medical data generated by the C-arm may thus be a medical image. The medical image may be an x-ray.In other embodiments, the medical image may be, for example, a computed tomography (CT) scan image, an ultrasound scan image, a magnetic resonance imaging scan, etc. The display 16 may display the medical image in real time during the medical procedure to provide guidance to medical personnel.

[0028] With reference to Fig. 2, the medical environment 10 also contains an audiovisual recorder 20 for recording audiovisual data. The audiovisual recorder 20 may be a mobile device and is otherwise referred to herein as a first device. The mobile device may be a smartphone or tablet. The audiovisual recorder 20 may also be part of another system, for example, an augmented reality headset (AR headset) or a head-mounted camera.

[0029] During the medical procedure, a user, e.g., a medical professional, may want to record a phase of the medical procedure. The recording may be used, for example, for training. The user may record the phase of the medical procedure 28 using the first device 20. The first device 20 may include a camera 22, a microphone 24, and a display 26. The display 26 may be in the form of a touchscreen.

[0030] The camera 22 may be configured to capture video data of its field of view. The field of view, in this context, includes the medical environment 10. The microphone 24 may be configured to capture audio data from the vicinity of the first device 20. The video data and the audio data may be synchronized using the internal clock of the first device 20. The synchronized audio and video data may be referred to as audiovisual data. In this way, the audiovisual data may include an audio channel representing the audio data and a video channel representing the video data. The audio channel and the video channel may capture the medical procedure or at least one phase of the medical procedure simultaneously in the medical environment. As described in more detail below, the display 16 of the second device 12 may also be recorded in the video channel of the audiovisual data.The display 16 may display medical data (e.g., a medical image) 30 acquired by the second device 12.

[0031] With reference to Fig. 3, a database system may be provided to manage and store medical data acquired by one or more second devices 12. The database system may also manage and store audiovisual data acquired by one or more first devices 20.

[0032] The database system may include a server 34. The first device 20 and the second device 12 may be communicatively linked to the server 34. The communication connection may be provided as a wireless connection, e.g., Bluetooth or Wi-Fi, or may be a physical connection, e.g., via a cable or wire. In some embodiments, the first and second devices 20, 12 may be directly connected to each other, i.e., in addition to the indirect connection via the server 34. The direct connection may be provided wirelessly, e.g., via Bluetooth or Wi-Fi, or may be a physical connection, e.g., via a cable or wire.

[0033] Additionally, one or more user interfaces 36 may be provided and communicatively linked to the server 34, for example, to access, display, and edit all data stored by the server 34. In this way, the server 34 may be connected to a storage medium that stores all audiovisual data 40 and medical data 30 received from the respective first 20 and second 12 devices.

[0034] The database system can be a Picture Archiving and Communication System (PACS). The medical data can conform to a standard. The standard can be Digital Imaging and Communication in Medicine (DICOM).

[0035] According to one or more embodiments of the invention, a method for synchronizing the audiovisual data and the medical data is provided. The method can be performed by the document management system, the first device 20, the second device 12, or another device, e.g., the server 34.

[0036] The method starts with receiving the audiovisual data 40 recorded by the first device 20 and receiving the medical data 30 recorded by the second device 12. Thereafter, the method includes classifying, using a machine learning algorithm, one or more sounds from the audio channel of the audiovisual data 40 as being generated by equipment in the medical environment 10.

[0037] With reference to Fig. 4, the first device 20 acquires the audiovisual data 40 of the phase of the medical procedure. The audiovisual data 40 includes the audio channel 42 and the video channel 44. Although the audiovisual data 40 is shown on the display 26 of the first device 20, this is purely schematic and serves only as an illustration. In practice, the display 26 does not visibly display the audiovisual data in the form of track / waveform data and image frames.

[0038] With reference to Fig. 5, the audio channel 42 is input to a machine learning algorithm 46. The machine learning algorithm 46 may be a supervised machine learning algorithm. The supervised machine learning algorithm may be trained by providing an audio channel of audiovisual data and a label that identifies one or more sounds in the audio channel of equipment used in a medical setting. The machine learning algorithm may be trained using supervised learning based on the label to classify the one or more sounds in the audio channel as associated with the equipment.

[0039] The machine learning algorithm may be a neural network. In this embodiment, the neural network may be a recurrent neural network. In other embodiments, the neural network may be a convolutional neural network or a transformation network.

[0040] Training may include prepropagation and backpropagation. During prepropagation, samples 52 from the audio channel of the training data are input to the neural network. The samples 52 may be taken periodically. The samples 52 may be of substantially equal duration. Each sample 52 is passed through the neural network, which outputs a value using an output layer. The output layer may include a softmax layer. The softmax layer may include a plurality of nodes, each of which represents the probability that the audio data sample was generated by a particular event. The particular event may be a sound generated by medical equipment in the medical environment.

[0041] The neural network generates an output vector 48 that includes a plurality of values. Each value corresponds to a classification of a sample 52. In other words, if the neural network decides that the sample corresponds to a particular source or corresponds to it with the highest probability, a value is provided for this event. For example, a value of one may correspond to a first source.

[0042] For example, output zero may correspond to background noise, output one may correspond to a piece of moving medical equipment, e.g., a table being lowered, output two may correspond to a piece of moving medical equipment, e.g., the second device 12 being moved, etc.

[0043] Using a loss function, a loss or error is calculated between the output vector 48 and a ground-truth vector 50. Backpropagation is used to optimize the hyperparameters, such as the weights within the layers, of the neural network based on the loss function. The loss function can be a least absolute deviation (L1) loss function or a least squared error (L2) loss function.

[0044] As explained above, the task of the neural network is to detect and classify the characteristic audio events (one or more tones) in the presence of other background noise, taking into account acoustic changes in the recorded audio due to the position relative to the C-arm and the acoustic properties of the recording device, the room, etc. One method for generating training data for such a network is to collect audio data synchronized with a system event log and linked to the medical data at a plurality of positions and with different recording devices in the presence of a plurality of background noises, and to use the system event log to label the collected audio data. Alternatively, a known series of events in the medical data could be triggered, and a recording obtained.These recordings can be made with different recording systems, with different background noises, from different positions in the medical environment relative to the medical data acquisition equipment, etc. The neural network is trained to identify relevant equipment sounds from these recordings that correspond to recorded events in the system logs.

[0045] With further reference to Fig. 4, in this embodiment, four tones are identified by the neural network. The present invention is not limited to the presence of specific four tones. These four tones are Fig. 4 are referred to as VE1, VE2, VE3, and VE4. VE can be an acronym for video event. The term "video event" refers to an audiovisual event, since the actual event is sound in the audio channel.

[0046] The first video event (VE1) or sound may be the sound of the C-arm starting to move. The second video event, VE2, or sound may occur after the first video event, VE1. The second video event, VE2, may be the sound of a pedal of the C-arm device being pressed. The third video event, VE3, or sound may be the sound of the C-arm stopping moving or the C-arm brake engaging. The fourth video event, VE4, or sound may be the sound of the pedal being released. In this way, the method includes classifying the sounds as being generated by medical equipment in the medical environment during a phase of the medical procedure.

[0047] When a sequence of events like these is detected, the timing between these events creates a very specific pattern.

[0048] These tone-generating events may occur in a catheterization lab or cath lab equipped with a first and a second device such that their clocks are synchronized. In this way, each pattern of tone-generating events detected in the audio data stream of the first device 20 can be associated with a matching event pattern in the event log of the second device 12, enabling synchronization of the two clocks.

[0049] For example, the cath lab events can be described using the labels CE1, CE2, ..., CEn. In the Fig. In the embodiment shown in Figure 4, there are nine cath lab events, CE1 through CE9. A first cath lab event, CE1, and a second cath lab event, CE2, may result from a patient table being moved upward. The third and fourth cath lab events, CE3, CE4, may result from the table being moved toward the C-arm. The fifth through eighth cath lab events, CE5 through CE8, may correspond to the first through fourth video events, VE1 through VE4, respectively. The ninth cath lab event, CE9, may be any event, such as adjusting the table or pressing the C-arm pedal to acquire a subsequent image.

[0050] Note that the first through fourth video events VE1 through VE4 and the fifth through eighth cath lab events CE5 through CE8 occur during a phase of the medical procedure. The phase of the medical procedure may be a particularly important phase. For example, it may be assumed that a physician or other cath lab personnel would record a phase of the medical procedure on the first device 20 because it is a particularly important or significant phase.

[0051] In this way, the method includes synchronizing the audiovisual data with the medical data based on the time of occurrence of the tones. Specifically, synchronization is enabled by aligning the occurrence of the tones from the audio channel VE1 to VE4 with the corresponding catheter lab events CE5 to CE8.

[0052] It should be noted that the equipment may be the medical imaging system comprising the second device, and the medical data may include medical images. The method may further comprise logging one or more events performed by the medical imaging system when the medical images are acquired by the second device in an event log. Synchronizing may comprise timing the one or more tones with the respective one or more events from the event log.

[0053] To briefly Fig. Returning to Figure 1, in another embodiment, the second device 12 may further include a speaker 50. The speaker 50 is synchronized with the medical data recorded by the second device 12 because it is connected to the same system clock. In other words, the speaker 50 is controlled by the second device 12. The speaker 50 is configured to generate a tone. The tone may be in the form of a tone signature. The tone signature may be indicative of the system clock of the second device. For example, the temporal pattern of the pulses is indicative of the system clock time (for example, a short pulse indicates a 0 and a longer pulse indicates a 1, and a binary sequence encoding the time is transmitted).

[0054] The microphone 24 of the first device 20 can detect the audio signature. The audiovisual data recorded simultaneously with the audio signature is recorded and can be time-stamped. Thus, synchronizing the audiovisual data involves aligning the time stamps in the audiovisual data with corresponding time stamps in the medical imaging system. If the time offset between the clocks is a fraction of the clock resolution, some interpolation may be required, which will be known to those skilled in the art.

[0055] In the same or a different embodiment, the sound signature may include a first sound pattern and a second sound pattern. The loudspeaker 50 may be configured to generate the first sound pattern at a first frequency and the second sound pattern at a second frequency. The first frequency may be 40 kHz and the second frequency may be 50 kHz. These specific frequency values ​​may be different, although it is preferable that the first and second frequencies be above the audible threshold for human hearing. Additionally, it is preferable that the first and second frequencies be different from each other. In this way, the nonlinearity of the sound recording hardware associated with the microphone 24 can be exploited. This is because the nonlinearities in a mechanical diaphragm of the microphone 24 and the initial amplifier stage produce sum and difference frequency components.In the example of using frequencies of 40 kHz and 50 kHz, the difference frequency is 10 kHz. 10 kHz is within the microphone recording frequency range of most devices specifically configured to record human-audible sounds, e.g., microphones of mobile devices such as smartphones. Thus, although the first and second frequencies are above the audible range of the human ear, they still result in an audible 10 kHz signal being recorded by the first device 20. In this way, the sound signature can be transmitted without distracting medical personnel in the medical environment 10. Alternatively, the speaker 50 may include a first speaker 52 and a second speaker 54. The first speaker 52 may be configured to generate the first sound pattern at a first frequency.The second speaker 54 may be configured to generate the second tone pattern at a second frequency.

[0056] In addition to the above-mentioned synchronization methods, further embodiments may include additional synchronization methods.

[0057] For example, the method may further include recognizing the display 16 of the second device 12 in the video channel of the audiovisual data by the first device 20 using a machine learning algorithm. Furthermore, the machine learning algorithm may be used to identify one or more display features on the display 16. The machine learning algorithm may include a neural network. The neural network may be a convolutional neural network. The convolutional neural network may be trained to classify the display as a display and may also be trained to classify the one or more display features. The classifications of the display features may include classifying the display features as the system clock of the second device 12.Additionally or alternatively, the machine learning algorithm may be trained to classify the display features as the medical data 30 displayed on the display 16.

[0058] In this way, synchronizing the audiovisual data with the medical data may further comprise aligning a time of occurrence of the one or more display features with a time of occurrence of the display of the one or more features on the display by the second device.

[0059] With reference to Fig. 3, the synchronised audiovisual data and medical data may be stored in the storage medium regardless of the specific synchronisation method used.

[0060] Additionally, the method may include generating a composite video by identifying a portion of the synchronized audiovisual data and medical data for display based on a time of occurrence of the or each video feature.

[0061] For example, the video features may include a first video feature and a second video feature. A first time of occurrence of the first video feature may be identified in the video channel. A second time of occurrence of the second video feature may be identified in the video channel. The portion of the synchronized audiovisual and medical data may be identified as being between the first and second times of occurrence. In this way, the composite video may be generated to include the synchronized audiovisual and medical data between the first and second times of occurrence. The composite video may be displayed on a display.

[0062] With reference to Fig. 1, the display may be the display 26 of the first device 20, the display 16 of the second device 12, or another display.

[0063] In one or more embodiments, the first video feature may be the start of a recording. In other words, when a user of the first device 20 starts recording the phase of the medical procedure, the time of the recording start is identified. The second video feature may be the end of the recording. In other words, when a user of the first device 20 stops recording the phase of the medical procedure, the time of the recording end is identified. It is assumed that the user is recording an important phase of the medical procedure. By using the recording start and end, it is no longer necessary to identify further video features. In other words, the use of the recording start and end is a passive way to identify an important phase of the medical procedure.

[0064] Additionally, in one or more embodiments, any data (e.g., medical data) that is not identified as being between the first and second video features may be discarded or deleted. In this way, storage space ( Fig. 3) be released.

[0065] The data collected during the recording period may also be used as training data to train the various machine learning algorithms described herein.

[0066] With reference to Fig. 6, in one or more embodiments, the composite video may be created by enhancing the video channel of the audiovisual data with the medical image. For example, the first device 20 on the left in Fig. 6 the raw video channel shown on the display 26. The first device 20 on the right in Fig. 6 shows the video channel of the audiovisual data 40, which is extended by the medical image 30.

[0067] With reference to Fig. 7, in one or more embodiments, the composite video may be created by replacing medical data in the video channel of the audiovisual data with the medical image recorded directly by the second device 12. For example, the first device 20 shows, on the left in Fig. 7 shows the video channel of the audiovisual data 40 displayed on the display 26. The video channel includes the medical image that the first device 20 has acquired by focusing on the display of the second device 16 ( Fig. 1). The first device 20 on the right in Fig. 6 shows the medical image in the video channel of the audiovisual data replaced by the medical image 30 recorded by the second device 12. Replacing the medical image with the medical image acquired by the second device 12 is advantageous because it has a higher resolution than the video acquired by the first device 20, which also suffers from various image artifacts caused by capturing data displayed on a screen (moiré effect, aliasing, specular reflections from the screen, etc.).

[0068] In one or more embodiments, it may be desirable to label the composite video based on speech recognition from the audio channel of the audiovisual data.

[0069] In such an embodiment, one could search for keywords to determine the stage of the procedure or relevant events. These labels could be appended to the stored medical data for more efficient review or inserted into the video content as subtitles when generating training material. Some phrases to look out for include: “Prepare groin” indicates that the invasive part of the procedure is about to begin “Close” indicates the end of the invasive part of the procedure “We are doing a run” may indicate that contrast medium will be injected or fluoroscopy imaging will be performed shortly “Hold your breath” to ensure that important medical data is collected in the next few seconds (e.g., fluoroscopy sequence with breath held) “You can breathe”, data collection is completed for now “Give heparin”, important for documenting the administration of medication

[0070] If medical data is currently labeled manually, a deep learning system could be trained over time to associate labels with sentences from speech recognition on the audio track and begin suggesting labels to gradually automate the process.

[0071] In further embodiments, when including medical data in the documentation, personal information such as the patient's name and other details could be automatically removed. This can be achieved by removing personal information from DICOM headers. If this information is present as an overlay in medical image data or in audio-video data, this can be achieved by using a method that can detect personal information in the medical image data (e.g., patient name, national health number, date of birth, etc., which are overlaid into the medical image data and may be blurred) or in the audio-video data (e.g., the patient's face, which may be blurred, or the cath lab staff indicating the patient's age in the video channel, which may be cropped from the audio channel).

[0072] With reference to Fig. 8, while the foregoing description describes the various embodiments in detail, some embodiments can be summarized as relating to a computer-implemented method for synchronizing audiovisual data and medical data. The method comprises various steps. In step S100, the method comprises receiving the audiovisual data recorded by a first device, wherein the audiovisual data includes an audio channel and a video channel concurrently capturing the medical procedure performed in a medical environment. In step S102, the method comprises receiving medical data recorded by a second device.In step S104, the method includes classifying, using one or more machine learning algorithms, one or more tones from the audio channel of the audiovisual data as being generated by equipment in the medical environment. In step S106, the method includes synchronizing the audiovisual data with the medical data based on a time of occurrence of the one or more tones.

[0073] With reference to Fig.9, while various embodiments are described in detail in the foregoing description, some embodiments can be summarized as relating to a computer-implemented method for training one or more machine learning algorithms using various steps. In step S200, the method includes providing an audio channel of audiovisual data and a label identifying one or more sounds in the audio channel of equipment used in a medical environment. In step S202, the method includes training the one or more machine learning algorithms using supervised learning based on the label to classify the one or more sounds in the audio channel as associated with the equipment.

[0074] While the invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered as illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. For example, it is possible to practice the invention in an embodiment in which more than two recording devices are used and more than two modalities are used (e.g., more than audiovisual data and medical images).

[0075] If many information sources are available, it may be necessary to display a different subset of the sources at different times in the procedure, depending on the situation. A user interface could be used to generate training material: First, a video is created with all information fields and a checkbox for each one. The user can then check / uncheck the desired fields at a given time. The system then generates a new video (e.g., a composite video) that shows only the desired information fields at different times. AI techniques could be used to predict what the desired fields are over time (e.g., procedure cards).

[0076] Further variations of the disclosed embodiments may be understood and implemented by those skilled in the art practicing the claimed invention from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a," "an," or "another" does not exclude a plurality. A single processor or other unit may perform the functions of several of the elements recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not mean that a combination of those measures cannot be used to advantage. A computer program may be embodied on a suitable medium, such as, for example, a hard disk, a memory card, a removable disk, or a removable media device.The invention may be stored / distributed on an optical storage medium or a semiconductor medium supplied with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. Any reference signs in the claims are not to be construed as limitations on the scope of protection.

Claims

[1] A computer-implemented method for synchronizing audiovisual data and medical data in a medical procedure, the method comprising: Receiving (S100) the audiovisual data (40) recorded by a first device (20), wherein the audiovisual data includes an audio channel and a video channel that simultaneously capture the medical procedure performed in a medical environment (10), Receiving (S102) the medical data (30) recorded by a second device (12); Classifying (S 104), using one or more machine learning algorithms, one or more sounds from the audio channel of the audiovisual data as being generated by equipment in the medical environment and Synchronizing (S106) the audiovisual data with the medical data based on a time of occurrence of one or more sounds classified as being generated by the equipment. [2] The computer-implemented method of claim 1, wherein the equipment is a medical imaging system comprising the second device, and the medical data includes medical images. [3] Computer-implemented method according to claim 1 or 2, the method further comprising logging one or more events associated with the medical equipment in an event log and wherein synchronizing (S106) comprises synchronizing the one or more tones with the respective one or more events from the event log. [4] The computer-implemented method of any preceding claim, wherein the one or more tones generated by the equipment include a sound signature generated by a speaker system (50) controlled by the second device (12), the sound signature indicative of a system clock time of the second device (12), wherein synchronizing the audiovisual data (40) comprises timing the one or more tones with one or more timestamps indicative of the system clock time. [5] A computer-implemented method according to any one of the preceding claims, further comprising: Detecting, using one or more machine learning algorithms, a display (16) of the second device (12) in the video channel of the audiovisual data (40); Identifying, using one or more machine learning algorithms, one or more display features on the display (16) of the second device and wherein synchronizing the audiovisual data (40) with the medical data further comprises matching a time of occurrence of one or more display features in the audiovisual data with a time of occurrence of the one or more features on the display by the second device (12). [6] The computer-implemented method of claim 5, wherein the one or more display features comprise a system clock of the second device (12). [7] A computer-implemented method according to claim 5 or claim 6, wherein the one or more display features comprise the medical data (30). [8] A computer-implemented method for generating a composite video, the method comprising: Synchronizing audiovisual data and medical data according to any one of the preceding claims; Detecting one or more video features of the medical procedure in the video channel of the audiovisual data and Generating the composite video by identifying a portion of the synchronized audiovisual data and medical data for display based on a time of occurrence of the or each video feature. [9] The computer-implemented method of claim 8, wherein the one or more video features comprise first and second video features, and wherein generating the composite video comprises: identifying a first time of occurrence of the first video feature in the video channel; Identifying a second time of occurrence of the second video feature in the video channel and Identifying the portion of the composite video as occurring between the first time of occurrence and the second time of occurrence. [10] The computer-implemented method of claim 9, wherein the first video feature comprises the recording start and the second video feature comprises the recording end. [11] The computer-implemented method of any one of claims 8 to 10, wherein generating the composite video data comprises augmenting the video channel with the medical data (30). [12] The computer-implemented method of claim 11, wherein the one or more display features comprise the medical data (30), wherein augmenting the video channel with the medical data (30) comprises replacing medical data in the video channel identified as the one or more display features on the display of the second device (12) with the medical data (30) recorded by the second device (12). [13] A computer-implemented method for training one or more machine learning algorithms for the task of classifying one or more sounds from an audio channel of audiovisual data as being generated by equipment in a medical environment, the method comprising: Providing (S200) an audio channel from audiovisual data (40) and an identifier linking one or more tones in the audio channel to the equipment used in a medical environment (10); and Training (S202) the one or more machine learning algorithms using supervised learning based on the label to perform the classification of the one or more sounds. [14] A transitory or non-transitory computer-readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the method according to any one of the preceding claims.

Images