Cardiac ablation operation video intelligent subtitle generation method, system and device and medium

By constructing a knowledge graph of cardiac ablation surgery and a dual-stream Transformer architecture, the problem of insufficient professional term recognition and context understanding in cardiac ablation surgery videos is solved, and high accuracy and efficient smart subtitle generation is achieved, suitable for a variety of medical education and clinical training Scene.

CN119091354AInactive Publication Date: 2024-12-06QINGPU BRANCH OF ZHONGSHAN HOSPITAL AFFILIATED TO FUDAN UNIV (SHANGHAI QINGPU DISTRICT CENT HOSPITAL)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411209794.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When the existing automatic video subtitle generation technology handles professional medical videos of cardiac ablation surgery, the professional term recognition is inaccurate and the context is insufficient, making it difficult to meet the needs of medical education and clinical training.

Method used

Construct a knowledge graph of cardiac ablation surgery, combines the dual-stream Transformer architecture, utilizes external knowledge streams and internal video streams for cross-attention mechanisms, generates subtitle text, and post-processes to ensure professionalism and fluency.

Benefits of technology

It improves the professionalism and accuracy of subtitles, enhances contextual understanding, improves subtitle generation efficiency and is highly adaptable, and is suitable for a variety of scenarios, such as automatic subtitles generation of surgical teaching videos and real-time subtitles generation of surgical live broadcasts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091354A_ABST
    Figure CN119091354A_ABST
Patent Text Reader

Abstract

The invention discloses a cardiac ablation operation video intelligent subtitle generation method, system and device and a medium. The method comprises the following steps that a cardiac ablation operation knowledge graph is constructed, wherein the knowledge graph comprises a cardiac anatomical structure, instruments designed for the cardiac ablation operation and operation steps of the cardiac ablation operation; the cardiac ablation operation video is preprocessed to extract video feature data, and the video feature data comprises key frames, surgical instruments, anatomical structures, audio and text information; and based on the cardiac ablation operation knowledge graph, processing the video feature data to generate a subtitle text. According to the method, by introducing the professional knowledge graph of the cardiac ablation operation and the double-flow Transform architecture, external knowledge and internal video information can be utilized at the same time, the overall understanding of the operation process is improved, and the professionalism and accuracy of cardiac ablation operation video intelligent subtitle generation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and artificial intelligence technology, and in particular to a method, system, device and computer-readable storage medium for generating intelligent subtitles for cardiac ablation surgery videos based on knowledge graph enhancement. Background Art

[0002] Cardiac ablation surgery is an important method for treating arrhythmias. It restores normal heart rhythm by locating and ablating abnormal electrical signals inside the heart. In order to improve the teaching and training effects of the surgery, it is usually necessary to add professional and accurate subtitles to the cardiac ablation surgery videos.

[0003] In traditional technology, subtitles are produced manually, which is not only time-consuming and labor-intensive, but also prone to problems such as lack of professionalism and inaccurate descriptions.

[0004] Later, automatic subtitle generation technology emerged. Existing automatic subtitle generation technology mainly relies on speech recognition and natural language processing technology. Although these technologies have made significant progress in daily conversations and general videos, they still face great challenges when faced with complex medical terms and professional terms. For example, professional terms involved in cardiac ablation surgery, such as "atrioventricular node" and "coronary sinus", are often difficult to accurately identify and translate. In addition, contextual information during the operation is crucial to understanding and describing the surgical steps, but existing technologies are still insufficient in processing contextual information.

[0005] In summary, the existing automatic video subtitle generation technology often has defects such as inaccurate recognition of professional terms and insufficient context understanding when processing professional medical videos such as cardiac ablation surgery, which makes it difficult to meet the needs of medical education and clinical training. Summary of the invention

[0006] The embodiments of the present application provide a method, system, device and computer-readable storage medium for generating intelligent subtitles for cardiac ablation surgery videos, thereby solving the technical problems of inaccurate recognition of professional terms and insufficient understanding of context in the field of cardiac ablation surgery videos in the automatic video subtitle generation method in the prior art, and improving the professionalism and accuracy of intelligent subtitle generation for cardiac ablation surgery videos.

[0007] In a first aspect, an embodiment of the present application provides a method for generating intelligent subtitles for a cardiac ablation surgery video, comprising the following steps:

[0008] Constructing a knowledge graph of cardiac ablation surgery, wherein the knowledge graph includes cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operation steps of cardiac ablation surgery;

[0009] Preprocessing the cardiac ablation surgery video to extract video feature data, wherein the video feature data includes key frames, surgical instruments, anatomical structures, audio and text information;

[0010] Based on the cardiac ablation surgery knowledge graph, the video feature data is processed to generate subtitle text.

[0011] Preferably, the construction of a cardiac ablation surgery knowledge graph specifically includes:

[0012] Collect professional literature, teaching materials and surgical guidelines related to cardiac ablation surgery;

[0013] Extract key concepts related to cardiac ablation procedures;

[0014] Establish relationships between the key concepts;

[0015] A graph database is used to store the key concepts and the relationships between the key concepts to form a knowledge graph of cardiac ablation surgery.

[0016] Preferably, the preprocessing of the cardiac ablation surgery video to extract video feature data specifically includes:

[0017] Decode cardiac ablation surgery videos and extract key frames;

[0018] Detect surgical instruments and anatomical structures in every frame of a cardiac ablation procedure video;

[0019] Converting audio from a cardiac ablation procedure video to text;

[0020] Extract existing text information from cardiac ablation surgery videos.

[0021] Preferably, the processing of the video feature data based on the cardiac ablation surgery knowledge graph to generate subtitle text specifically includes:

[0022] Using a two-stream Transformer architecture, one data stream is the external knowledge stream, which encodes the cardiac ablation surgery knowledge graph; the other data stream is the internal video stream, which encodes the video feature data;

[0023] The two-stream Transformer architecture applies a cross-attention mechanism between the external knowledge stream and the internal video stream to achieve information interaction;

[0024] The output of the external knowledge stream and the internal video stream are fused to generate subtitle text.

[0025] Furthermore, the cross-attention mechanism is specifically as follows: first, the attention weight between the external knowledge stream output and the internal video stream output is calculated, and then the representation of the external knowledge stream and the internal video stream is updated according to the attention weight.

[0026] Preferably, the method further comprises: post-processing the generated subtitle text to ensure professionalism and fluency;

[0027] The post-processing specifically comprises the following steps:

[0028] Proofread the generated subtitles using professional medical dictionaries to ensure the accuracy of professional terminology;

[0029] Apply rule-based grammar checking to ensure sentence fluency;

[0030] Align the generated subtitles to the video timeline.

[0031] In a second aspect, an embodiment of the present application provides a system for generating intelligent subtitles for a cardiac ablation surgery video, the system comprising:

[0032] A cardiac ablation surgery knowledge graph construction module, which is used to construct a cardiac ablation surgery knowledge graph, wherein the knowledge graph includes cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operation steps of cardiac ablation surgery;

[0033] A cardiac ablation surgery video preprocessing module, used for preprocessing the cardiac ablation surgery video to extract video feature data, wherein the video feature data includes key frames, surgical instruments, anatomical structures, audio and text information;

[0034] A subtitle text generation module is used to process the video feature data based on the cardiac ablation surgery knowledge graph to generate subtitle text.

[0035] Preferably, the system further comprises:

[0036] The post-processing module is used to post-process the generated subtitle text to ensure professionalism and fluency; the post-processing includes: using a professional medical dictionary to proofread the generated subtitles to ensure the accuracy of professional terminology; applying rule-based grammar checking to ensure the fluency of sentences; aligning the generated subtitles with the video timeline.

[0037] In a third aspect, an embodiment of the present application provides a device for generating intelligent subtitles for cardiac ablation surgery videos, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for generating intelligent subtitles for cardiac ablation surgery videos when executing the program.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for generating intelligent subtitles for cardiac ablation surgery videos.

[0039] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0040] 1. Improved professionalism and accuracy of subtitles: The present invention introduces a professional knowledge map of cardiac ablation surgery, which can accurately identify and describe the professional terms, anatomical structures and operation steps in the surgery;

[0041] 2. Enhanced contextual understanding: By adopting a dual-stream Transformer architecture, the present invention can simultaneously utilize external knowledge and internal video information to improve the overall understanding of the surgical process;

[0042] 3. Improved subtitle generation efficiency: The present invention can generate subtitles intelligently and automatically, greatly reducing the workload of manual subtitle production;

[0043] 4. Strong adaptability: The method of the present invention can be optimized for different types of cardiac ablation surgeries and has good scalability;

[0044] 5. Multi-scenario application: The method of the present invention can be used in various scenarios such as automatic generation of subtitles for surgical teaching videos, intelligent assistance for surgical records, and real-time subtitle generation for live surgical broadcasts. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0046] Figure 1 This is a flow chart of the method for generating intelligent subtitles for a cardiac ablation surgery video provided in Example 1 of the present application;

[0047] Figure 2 This is the example scenario of the method for generating intelligent subtitles for cardiac ablation surgery videos in embodiment 1 of the present application in cardiac valve catheterization;

[0048] Figure 3 This is a structural block diagram of the intelligent subtitle generation system for cardiac ablation surgery video provided in Example 2 of the present application;

[0049] Figure 4 This is a structural block diagram of the device for generating intelligent subtitles for cardiac ablation surgery videos provided in Example 3 of the present application.

[0050] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0051] The embodiments of the present application provide a method, system and computer-readable storage medium for generating intelligent subtitles for cardiac ablation surgery videos, thereby solving the technical problems of inaccurate recognition of professional terms and insufficient understanding of context in the field of cardiac ablation surgery videos in the automatic video subtitle generation method in the prior art, and improving the professionalism and accuracy of intelligent subtitle generation for cardiac ablation surgery videos.

[0052] In order to better understand the above technical solution, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices consistent with some aspects of the present application as detailed in the attached claims.

[0053] Embodiment 1

[0054] Figure 1 Flow chart of the method for generating intelligent subtitles for a cardiac ablation surgery video provided in this embodiment. The method for generating intelligent subtitles for a cardiac ablation surgery video specifically includes the following steps:

[0055] Step S1: Construct a knowledge graph of cardiac ablation surgery, wherein the knowledge graph includes professional knowledge in the field of cardiac ablation surgery, such as cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operating steps of cardiac ablation surgery.

[0056] Specifically, step S1 includes the following steps:

[0057] Step S11: Collect professional literature, teaching materials and surgical guidelines related to the field of cardiac ablation surgery;

[0058] Step S12: extracting key concepts related to cardiac ablation surgery, such as: cardiac anatomical structures (such as atria, ventricles, pulmonary veins, etc.), instruments involved in cardiac ablation surgery (such as ablation catheters, three-dimensional mapping systems, etc.), and operating steps of cardiac ablation surgery (such as puncture, mapping, ablation, etc.);

[0059] Step S13: establishing the relationship between the key concepts, such as "position relationship", "operation sequence", "cause and effect relationship", etc.;

[0060] Step S14: Use a graph database (such as Neo4j) to store the key concepts and the relationships between the key concepts to form a knowledge graph of cardiac ablation surgery.

[0061] Step S2: preprocessing the cardiac ablation surgery video to extract video feature data, where the video feature data includes key frames, surgical instruments, anatomical structures, audio, and text information;

[0062] Specifically, step S2 includes the following steps:

[0063] Step S21: using the OpenCV library to decode the cardiac ablation surgery video and extract key frames;

[0064] OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library that provides a large number of algorithms for processing and analyzing images and videos. In this embodiment, the cardiac ablation surgery video is decoded based on the OpenCV library to extract key frames in the video;

[0065] Step S22: using the pre-trained YOLOv5 model to detect surgical instruments and anatomical structures in each frame of the cardiac ablation surgery video;

[0066] YOLOv5 is an advanced object detection algorithm, which is the fifth version of the YOLO (You Only Look Once) series of algorithms. The core idea of ​​the YOLOv5 algorithm is to transform the object detection problem into a regression problem by extracting features from the input image through the backbone network and then converting it into a set of detection boxes and corresponding category probability scores.

[0067] In this embodiment, the YOLOv5 model is first trained using a large number of cardiac ablation surgery video samples. The pre-trained YOLOv5 model has the ability to detect surgical instruments and anatomical structures in each frame of the cardiac ablation surgery video.

[0068] Step S23: using the pre-trained DeepSpeech model to convert the audio of the cardiac ablation surgery video into text;

[0069] DeepSpeech is a speech-to-text conversion technology based on deep learning. It uses machine learning technology, especially deep learning algorithms, to achieve parsing of speech input and text output. DeepSpeech uses convolutional neural networks (CNN) for feature extraction, long short-term memory networks (LSTM) to capture temporal patterns, and generates text predictions through a fully connected layer connected to the LSTM output.

[0070] In this embodiment, the DeepSpeech model is first trained using a large number of cardiac ablation surgery video samples. The pre-trained DeepSpeech model has the ability to convert the audio of the cardiac ablation surgery video into text.

[0071] Step S24: extracting existing text information in the cardiac ablation surgery video, such as surgery time, patient information, etc.

[0072] Extracting the existing text information in the cardiac ablation surgery video can be achieved through the following technologies: speech recognition technology (such as Google Speech-to-Text, IBM Watson Speech to Text, etc.), video subtitle extraction tools (such as Subtitle Edit, Aegisub, etc.), video OCR technology (such as ABBYY FineReader, Tesseract OCR, etc.). Through these technologies, the existing text information can be effectively extracted from the cardiac ablation surgery video.

[0073] Step S3: Based on the cardiac ablation surgery knowledge graph, the video feature data is processed to generate subtitle text.

[0074] In this embodiment, a dual-stream Transformer architecture is used for information processing. The dual-stream Transformer architecture is a model structure used in multimodal tasks, which can process two different types of data streams at the same time. In this architecture, each modality has its own independent Transformer processing flow, that is, two independent encoders (Encoders), which process the data of each modality respectively, and realize the interaction and fusion of information through the cross-attention mechanism.

[0075] Specifically, in this embodiment, a dual-stream Transformer architecture is used, one data stream is an external knowledge stream, that is, the cardiac ablation surgery knowledge graph is encoded and input; the other data stream is an internal video stream, that is, the video feature data is encoded and input. The dual-stream Transformer architecture applies a cross-attention mechanism between the external knowledge stream and the internal video stream to achieve information interaction; and fuses the output of the external knowledge stream and the internal video stream to generate subtitle text.

[0076] Among them, the cardiac ablation surgery knowledge graph is encoded and input, specifically including:

[0077] Convert the node and edge information in the cardiac ablation surgery knowledge graph into vector representation;

[0078] Use Graph Attention Network (GAT) to encode the converted cardiac ablation surgery knowledge graph;

[0079] Input the encoded cardiac ablation surgery knowledge graph into the Transformer encoder.

[0080] Encode the video feature data and input it, including:

[0081] Use the pre-trained ResNet-50 model to extract visual features of key frames;

[0082] Use the BERT model to encode text features, where the text features include text converted from audio and text information already in the video;

[0083] Combine visual features and text features and input them into the Transformer encoder;

[0084] Among them, the PyTorch Geometric library is used to implement the Graph Attention Network (GAT), and PyTorch is used to implement the Transformer encoder. PyTorch is an open source machine learning library that is widely used in applications such as computer vision and natural language processing.

[0085] The ResNet-50 model is a deep residual network (ResNet) that contains 50 convolutional layers. Its design solves the gradient vanishing and gradient exploding problems that occur as the network depth increases by introducing a residual learning framework, allowing for successful training of deeper network structures. In this implementation, the ResNet-50 model is trained with a large number of key frames, and the pre-trained ResNet-50 model has the ability to extract visual features of key frames.

[0086] The BERT model comes with Transformer, which uses the self-attention mechanism to process serialized data.

[0087] Furthermore, the cross attention mechanism is specifically as follows: first, the attention weight between the output of the external knowledge stream and the output of the internal video stream is calculated, and then the representation of the two streams is updated according to the attention weight. The cross attention mechanism is implemented using PyTorch.

[0088] Furthermore, when generating subtitle text, the external knowledge stream after interaction is first combined with the output of the internal video stream, and then the subtitle text is generated using the Transformer decoder. The Transformer decoder is implemented using PyTorch.

[0089] In a preferred embodiment, the method for generating intelligent subtitles for a cardiac ablation surgery video further includes:

[0090] Step S4: Post-process the generated subtitle text to ensure professionalism and fluency.

[0091] Specifically, step S4 includes the following steps:

[0092] Step S41: Proofread the generated subtitles using a professional medical dictionary to ensure the accuracy of professional terminology.

[0093] Step S42: Apply rule-based grammar checking to ensure the fluency of the sentence.

[0094] Step S43: Align the generated subtitles with the video timeline.

[0095] Among them, grammar checking can be implemented using the NLTK library.

[0096] The method for generating intelligent subtitles for cardiac ablation surgery videos provided in the above embodiment first integrates a large number of professional terms and definitions related to cardiac ablation surgery to build a comprehensive medical knowledge base for assisting speech recognition and subtitle generation. Then, through deep learning and natural language processing technology, the contextual information in the cardiac ablation surgery video is analyzed to accurately understand and describe the various steps in the cardiac ablation surgery process. Finally, by combining the medical knowledge base, contextual information and speech recognition technology, it is possible to accurately identify and describe the professional terms and operating steps in the cardiac ablation surgery process, and automatically generate accurate and professional surgical subtitles.

[0097] The method provided in this embodiment can significantly improve the efficiency and accuracy of generating subtitles for cardiac ablation surgery videos, and provide strong support for medical education and clinical training.

[0098] The following is an explanation with reference to clinical examples.

[0099] Preparation: A 10-minute video of a cardiac ablation procedure

[0100] Step 1: Knowledge graph construction

[0101] -Added knowledge specific to cardiac ablation, such as concepts such as "pulmonary vein isolation" and "circular ablation";

[0102] -Establish the sequential relationship between surgical steps

[0103] Step 2: Video Preprocessing

[0104] -Key instruments for testing: annular ablation catheter, three-dimensional mapping system

[0105] -Identify key anatomical structures: left atrium, pulmonary vein ostia

[0106] - Extract the doctor's voice instructions

[0107] Step 3: Dual stream processing and subtitle generation

[0108] - External knowledge flow focusing on expertise in cardiac ablation

[0109] -Internal video stream captures the movement of surgical instruments and the doctor's operation

[0110] Step 4: Post-processing

[0111] - Ensure accurate use of professional terms such as "pulmonary vein isolation" and "circular ablation"

[0112] - Adjust the subtitle display time to synchronize with the surgical operation

[0113] Output example:

[0114] 00:01:15-00:01:30: "The doctor inserted the circular ablation catheter into the left atrium"

[0115] 00:02:45-00:03:00: "Begin circumferential ablation of the left superior pulmonary vein"

[0116] 00:05:30-00:05:45: "Complete electrical isolation of the left pulmonary vein."

[0117] Figure 2 The example scenario of the intelligent subtitle generation method for cardiac ablation surgery videos in cardiac valve catheterization is demonstrated. Figure 2 A video frame of a heart valve catheterization is shown in Figure 1. The model is able to identify the scene in the picture and generate corresponding surgical captions. Figure 2 In the video, the model identified the catheter and valve in the image and generated the caption “ablation catheter enters the ventricle.”

[0118] This example scenario demonstrates the ability of the method in this embodiment to identify the key steps of heart valve catheterization and generate corresponding subtitles. By combining knowledge graphs and deep learning technology, the method in this embodiment can well understand the scenes and actions in the surgical video and generate accurate and clear surgical subtitles.

[0119] The intelligent subtitle generation method provided in this embodiment can provide real-time subtitle support in cardiac ablation surgery videos, helping doctors and nurses better understand the surgical process and improve the safety and efficiency of the surgery. At the same time, this technology can also be used for surgical training and education to help novice doctors better understand surgical steps and techniques.

[0120] Embodiment 2

[0121] Based on the same inventive concept as the method for generating intelligent subtitles for cardiac ablation surgery videos in the aforementioned embodiment 1, this embodiment further provides a system for generating intelligent subtitles for cardiac ablation surgery videos, such as Figure 3 As shown, the system comprises:

[0122] The cardiac ablation surgery knowledge graph construction module 1 is used to construct the cardiac ablation surgery knowledge graph, which includes professional knowledge in the field of cardiac ablation surgery such as the cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operating steps of cardiac ablation surgery.

[0123] A cardiac ablation surgery video preprocessing module 2 is used to preprocess the cardiac ablation surgery video to extract video feature data, wherein the video feature data includes key frames, surgical instruments, anatomical structures, audio and text information;

[0124] The subtitle text generation module 3 is used to process the video feature data based on the cardiac ablation surgery knowledge graph to generate subtitle text.

[0125] In a preferred embodiment, the cardiac ablation surgery video intelligent subtitle generation system further includes:

[0126] The post-processing module is used to post-process the generated subtitle text to ensure professionalism and fluency. The post-processing includes: proofreading the generated subtitles using a professional medical dictionary to ensure the accuracy of professional terms; applying rule-based grammar checking to ensure the fluency of sentences; and aligning the generated subtitles with the video timeline.

[0127] The various specific processes and specific examples of the method for generating intelligent subtitles for cardiac ablation surgery videos in the aforementioned embodiment 1 are also applicable to the system for generating intelligent subtitles for cardiac ablation surgery videos in the present embodiment. Through the detailed description of embodiment 1, those skilled in the art can clearly know the implementation method of the system for generating intelligent subtitles for cardiac ablation surgery videos in the present embodiment. Therefore, for the sake of brevity of the specification, they will not be described in detail here.

[0128] Embodiment 3

[0129] Based on the same inventive concept as the method for generating intelligent subtitles for a cardiac ablation surgery video in the aforementioned embodiment 1, this embodiment further provides a device for generating intelligent subtitles for a cardiac ablation surgery video, such as Figure 4 As shown, a computer program is stored thereon, and when the program is executed by a processor, the steps of the method for generating intelligent subtitles for a cardiac ablation surgery video described in Embodiment 1 are implemented.

[0130] Among them, Figure 4In the embodiment of the present invention, a bus architecture (represented by bus 300) is shown, which may include any number of interconnected buses and bridges, and bus 300 links various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are not further described herein. Bus interface 306 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, namely a transceiver, which provides a unit for communicating with various other devices over a transmission medium.

[0131] The processor 302 is responsible for managing the bus 300 and general processing, while the memory 304 may be used to store data used by the processor 302 when performing operations.

[0132] The cardiac ablation surgery knowledge graph storage can provide a RESTful API interface for other modules to call.

[0133] Embodiment 4

[0134] Based on the same inventive concept as the method for generating intelligent subtitles for cardiac ablation surgery videos in the aforementioned embodiment 1, this embodiment also provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the method for generating intelligent subtitles for cardiac ablation surgery videos described in embodiment 1.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0140] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating intelligent subtitles for cardiac ablation surgery videos, characterized in that: The steps include: Constructing a knowledge graph of cardiac ablation surgery, wherein the knowledge graph includes cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operation steps of cardiac ablation surgery; Preprocessing the cardiac ablation surgery video to extract video feature data, wherein the video feature data includes key frames, surgical instruments, anatomical structures, audio and text information; Based on the cardiac ablation surgery knowledge graph, the video feature data is processed to generate subtitle text.

2. The method for generating intelligent subtitles for cardiac ablation surgery video according to claim 1, characterized in that: The construction of the cardiac ablation surgery knowledge graph specifically includes: Collect professional literature, teaching materials and surgical guidelines related to cardiac ablation surgery; Extract key concepts related to cardiac ablation procedures; Establish relationships between the key concepts; A graph database is used to store the key concepts and the relationships between the key concepts to form a knowledge graph of cardiac ablation surgery.

3. The method for generating intelligent subtitles for cardiac ablation surgery video according to claim 1, characterized in that: The preprocessing of the cardiac ablation surgery video to extract video feature data specifically includes: Decode cardiac ablation surgery videos and extract key frames; Detect surgical instruments and anatomical structures in every frame of a cardiac ablation procedure video; Converting audio from a cardiac ablation procedure video to text; Extract existing text information from cardiac ablation surgery videos.

4. The method for generating intelligent subtitles for a cardiac ablation surgery video according to claim 1, characterized in that: The processing of the video feature data based on the cardiac ablation surgery knowledge graph to generate subtitle text specifically includes: Using a two-stream Transformer architecture, one data stream is the external knowledge stream, which encodes the cardiac ablation surgery knowledge graph; the other data stream is the internal video stream, which encodes the video feature data; The two-stream Transformer architecture applies a cross-attention mechanism between the external knowledge stream and the internal video stream to achieve information interaction; The output of the external knowledge stream and the internal video stream are fused to generate subtitle text.

5. The method for generating intelligent subtitles for a cardiac ablation surgery video according to claim 4, characterized in that: The cross-attention mechanism is specifically as follows: first, the attention weight between the external knowledge stream output and the internal video stream output is calculated, and then the representation of the external knowledge stream and the internal video stream is updated according to the attention weight.

6. The method for generating intelligent subtitles for a cardiac ablation surgery video according to claim 1, characterized in that: The method further comprises: post-processing the generated subtitle text to ensure professionalism and fluency; The post-processing specifically comprises the following steps: Proofread the generated subtitles using professional medical dictionaries to ensure the accuracy of professional terminology; Apply rule-based grammar checking to ensure sentence fluency; Align the generated subtitles to the video timeline.

7. A cardiac ablation surgery video intelligent subtitle generation system, characterized in that: The system comprises: A cardiac ablation surgery knowledge graph construction module, which is used to construct a cardiac ablation surgery knowledge graph, wherein the knowledge graph includes cardiac anatomical structure, instruments designed for cardiac ablation surgery, and operation steps of cardiac ablation surgery; A cardiac ablation surgery video preprocessing module, used for preprocessing the cardiac ablation surgery video to extract video feature data, wherein the video feature data includes key frames, surgical instruments, anatomical structures, audio and text information; A subtitle text generation module is used to process the video feature data based on the cardiac ablation surgery knowledge graph to generate subtitle text.

8. The intelligent subtitle generation system for cardiac ablation surgery video according to claim 7, characterized in that: The system further comprises: The post-processing module is used to post-process the generated subtitle text to ensure professionalism and fluency; the post-processing includes: using a professional medical dictionary to proofread the generated subtitles to ensure the accuracy of professional terminology; applying rule-based grammar checking to ensure the fluency of sentences; aligning the generated subtitles with the video timeline.

9. A device for generating intelligent subtitles for a cardiac ablation surgery video, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for generating intelligent subtitles for cardiac ablation surgery videos as described in claim 1 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for generating intelligent subtitles for cardiac ablation surgery videos as described in claim 1 are implemented.