Method and system for automatic generation of surgical record based on dense video description

Abstract

The application discloses a kind of based on dense video description Surgical Record Automatic Generation Method and system, mainly solve the problem that existing method is poorly adapted to change surgical scene, generate redundant caption and unstable generation process.Its implementation scheme is: obtaining surgical video, it is divided into training set, verification set and test set;Build including feature extraction module, matching enhancement coding and decoding module, the surgical record automatic generation network of redundant output module, and the total loss function of the network is combined into the existing three loss functions;Using training set and verification set to the surgical record automatic generation network is iteratively trained and verified, the best parameter on verification set is selected as the trained surgical record automatic generation network: test set video is input to the trained network and obtains the caption and time stamp corresponding to surgical video.The application can reduce the redundancy of generated caption, improve the matching stability in caption generation process and the accuracy of generation result, and can be used for dense video caption generation task in complex scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology, and in particular relates to an automatic method for generating surgical records, which can be used for dense video subtitle generation tasks in complex scenarios. Background Technology

[0002] In today's medical field, surgical records occupy a crucial position. As a core component of surgical case documentation, they span multiple important areas, including medical practice, legal disputes, medical education, and case discussions. From a medical practice perspective, surgical records provide detailed evidence for the formulation and adjustment of subsequent treatment plans. Doctors can use this information to accurately grasp the surgical outcome and the patient's recovery status, thereby making scientific and reasonable medical decisions. In legal dispute situations, surgical records serve as impartial adjudicators. Their accurate presentation of the entire surgical process ensures that the legitimate rights and interests of both doctors and patients are balanced on the scales of law, becoming the core evidentiary support for adjudicating medical disputes. At the case discussion level, surgical records provide direction for medical team members, enabling them to accurately analyze the gains and losses of the surgery, deeply explore areas for optimization, and promote the continuous improvement and advancement of medical technology.

[0003] However, the traditional method of manually writing surgical records has many drawbacks. On the one hand, it is extremely inefficient. After completing complex surgeries with high intensity, doctors are already physically and mentally exhausted, yet they still need to spend a great deal of time and energy meticulously recording surgical details. This undoubtedly greatly increases the workload of doctors, leading to a significant reduction in work efficiency. On the other hand, manual recording is heavily influenced by subjective factors and is prone to omissions and errors. The surgical process is rapidly changing and highly complex. When doctors are fully focused on the surgical procedure, they often find it difficult to simultaneously perform comprehensive and accurate recording work. This results in frequent omissions of key information or errors in recording, posing potential risks to medical quality and safety.

[0004] With the rapid development of computer vision and natural language processing technologies, multimodal technologies have emerged and shone brightly in numerous fields, bringing unprecedented hope to the automated generation of surgical records. Video captioning technology, as an emerging force within this field, has attracted considerable attention. Its core task lies in skillfully transforming video content into natural language descriptions, playing a crucial role in the automated generation of surgical records. Based on this, dense video captioning technology has emerged and risen rapidly. It focuses on accurately locating multiple key events in unedited videos and endowing them with vivid descriptions in natural language, opening up new avenues for the automated generation of surgical records. However, its application faces numerous challenges in the unique context of surgical scenarios.

[0005] Regarding training stability, the Hungarian matching algorithm, upon which dense video captioning technology relies, exhibits serious deficiencies in surgical scenarios. The complexity and precision of surgical procedures require the model to accurately process massive amounts of complex information. However, the Hungarian matching algorithm is highly sensitive to input changes; even minor perturbations can cause significant differences in label assignment, leading to drastic fluctuations and a lack of stability in the matching results. This not only interferes with the model training process but also significantly reduces the accuracy and reliability of the generated surgical records, failing to meet the stringent requirements of clinical practice.

[0006] Regarding redundancy, the number of predefined events often far exceeds the number of actual events, leading to rampant redundancy in predicted time intervals. This drawback directly impacts surgical records, manifesting as a large amount of repetitive and trivial information, severely diminishing the readability and usability of the reports. Doctors must expend considerable time and effort sifting through these lengthy and redundant surgical records to extract key information, significantly affecting medical efficiency and decision-making accuracy.

[0007] To overcome these challenges, scholars in the field have conducted tireless research and proposed numerous solutions:

[0008] Inspired by open-domain neural machine translation and imaging captioning tasks, Zhang et al. divided the task into two aspects: understanding the surgical activity in the current view and modeling the relationship between visual information and textual description. They introduced a transformer backbone encoder-decoder network with self-critical reinforcement learning to generate surgical records from surgical images and evaluated the effectiveness of their method on the DAISI dataset.

[0009] Patent document CN118942612A discloses a method for generating surgical reports based on three-dimensional surgical feature maps. This method converts multiple three-dimensional surgical feature maps from a fully connected layer surgical video into three-dimensional surgical feature vectors. A classifier then uses these three-dimensional surgical feature vectors to identify surgical operations in the surgical video, obtaining the probability that each surgical operation in the video corresponds to a preset surgical operation. Based on the probability of each surgical operation, a surgical report is generated from the surgical video.

[0010] The aforementioned methods for generating surgical video subtitles are limited by fixed patterns or specific scene data, resulting in poor adaptability to changing surgical scenarios and a large amount of redundancy in the generated subtitles. They are also prone to information loss and mismatch due to the difficulty in aligning different types of data during multimodal information fusion, making it impossible to accurately describe the details of the surgical situation and meet the needs of automated generation of surgical records. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of the existing technology by proposing an automatic method for generating surgical records based on dense video description, so as to reduce subtitle redundancy in surgical scenes and improve the matching stability and accuracy of the generation results during the subtitle generation process.

[0012] To achieve the above objectives, the technical solution of the present invention includes:

[0013] 1. A method for automatically generating surgical records based on dense video description, characterized in that it includes:

[0014] Obtain surgical videos and randomly divide them into training, validation, and test sets in a 6:2:2 ratio;

[0015] A surgical record automatic generation network based on dense video description is constructed, which includes: a surgical instrument-guided feature extraction module, a matching enhancement encoding and decoding module, a redundancy removal output module, and the existing Giou loss function, cross-entropy loss function and FocalLoss classification loss function are combined into the total loss function L of the network.

[0016] The network for automatically generating surgical records based on dense video descriptions was iteratively trained and validated using training and validation sets:

[0017] The test set videos are input into a pre-trained surgical record automatic generation network based on dense video description to obtain subtitles and timestamps corresponding to the surgical videos.

[0018] Furthermore, the surgical instrument-guided feature extraction module includes:

[0019] CLIP image encoder, used to extract basic visual features from input video. v ;

[0020] CLIP text encoder, used to extract basic text features from input video. t The two encoders are connected in parallel for output;

[0021] The feature fusion module is used to process visual features f v and text features f t Feature fusion is performed to obtain enhanced global visual features f.

[0022] Furthermore, the matching enhancement encoding / decoding module includes:

[0023] An event encoder, consisting of a deformable Transformer network, is used to encode global video features f to extract segment-level features f. e ;

[0024] The query initialization module, composed of a fully connected neural network, is used to evaluate fragment-level features f.e Does the event feature exist? An initial event query (t) is generated by selecting the top N features with the highest confidence. c ,t l ,E);

[0025] The denoising and reconstruction module is used to reconstruct real event time tags. Inject random noise to transform it into noisy event queries.

[0026] Event decoder, used to initialize event query (t) c ,t l E) and noisy event queries Each with segment-level feature f e Interact to decode and generate event feature set D and denoised event feature set D2. dn .

[0027] Furthermore, the redundancy removal output module includes:

[0028] A counter, consisting of a fully connected network, is used to determine the number of clusters K;

[0029] The event redundancy removal module is used to aggregate similar events in the key event feature set D to reduce the number of events and obtain the aggregated event feature set F;

[0030] The output module, consisting of a locator and a subtitle generator, is used to determine the aggregated event feature set F and the denoised event feature set D. dn Generate the final output (t) s ,t e ,s), where t s It is the start time of the event, t e "S" represents the event end time, and "s" represents the corresponding subtitle description.

[0031] 2. A surgical record automatic generation system based on dense video description, characterized in that it includes:

[0032] The data preprocessing module is used to randomly divide the surgical videos into training, validation, and test sets.

[0033] The feature extraction module is used to fuse video features and text features to obtain global video features;

[0034] The encoding / decoding module is used to convert global video features into event features;

[0035] The output module is used to aggregate event features and generate output for each event feature;

[0036] The model building module is used to connect the feature extraction module, the encoding / decoding module and the output module to form an automatic surgical record generation network based on dense video description.

[0037] The model training module is used to iteratively train the constructed surgical record automatic generation network using the training set and loss function L, and obtain the model parameters that perform best on the validation set to obtain the trained surgical record automatic generation network.

[0038] The model testing module is used to test the trained surgical record automatic generation network using test set videos, and obtain the subtitles and timestamps corresponding to the surgical videos.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] Firstly, this invention constructs a feature extraction module guided by surgical instruments, which fuses the visual and textual features of the input video. By leveraging the complementary advantages of visual and textual modalities, the feature extraction process can better focus on the movements and interactions of the instruments in the surgical video, providing a higher quality feature foundation for the subsequent decoding process, thereby improving the accuracy of the generated subtitles.

[0041] Secondly, by constructing a matching enhancement encoding and decoding module, the present invention injects random noise into the real event time stamps, and then decodes the noisy data to restore the original data, which can make the matching results more accurate and reliable, and improve the matching stability in the subtitle generation process.

[0042] Thirdly, by constructing a redundancy removal output module, using a counter to predict the number of clusters, and using a clustering algorithm to aggregate event features, this invention not only reduces the interference of repeated events on the generated results, but also reduces the repeated generation of redundant information, making the generated subtitles more concise and easier to read. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the automatic surgical record generation method provided in Embodiment 1 of the present invention.

[0044] Figure 2 This is a schematic diagram of the network structure for automatically generating surgical records in the method of this invention;

[0045] Figure 3 This is a block diagram of the surgical record automatic generation system provided in Embodiment 2 of the present invention;

[0046] Figure 4 This is an example of caption results generated using the present invention on a tumor resection surgery dataset.

[0047] Detailed Implementation Methods

[0048] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative effort should all fall within the protection scope of the present invention.

[0049] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

[0050] Example 1: A method for automatically generating surgical records based on dense video description.

[0051] Reference Figure 1 The implementation steps of this example include the following:

[0052] Step 1: Dataset preprocessing.

[0053] Obtain surgical videos and randomly divide them into training, validation, and test sets in a 6:2:2 ratio.

[0054] Step 2: Establish a feature extraction module for surgical instrument guidance.

[0055] (2.1) Extract the frame sequence V from the surgical videos in the training set in chronological order, and use the image encoder of the CLIP model to extract visual features f for each frame. v Simultaneously, surgical instrument keywords W from the labeled training set are collected, and their text features f are extracted using the text encoder of the CLIP model. t ;

[0056] (2.2) Visual features f are processed through the feature fusion module. v and text features f t Feature fusion is performed to obtain enhanced global visual features f:

[0057] (2.2.1) Calculate visual features f using dot product operation v and text features f t Similarity between v ,f t The similarity results are then Softmax normalized to generate a normalized similarity matrix M. vt :

[0058] Similarity(f v ,f t )=f v ·f tT

[0059] M vt =Softmax(Similarity(f v ,f t ))

[0060] Where Similarity(·,·) represents the similarity dot product operation, and Softmax(·) represents the normalization operation;

[0061] Normalized similarity matrix M vt This represents the probability distribution of matching between each visual feature and all text features;

[0062] (2.2.2) Based on the similarity matrix M vt Select the k text features f that are most relevant to the visual features of each frame from all text features. t ′:

[0063] f t =Select(f) t M vt ,k)

[0064] Where k is a hyperparameter, representing the pre-defined number of selected text features;

[0065] (2.2.3) The filtered text features f t ′ and visual features f v The features are concatenated to form a fused multimodal feature representation, and sinusoidal positional coding is used to add temporal information to the features, resulting in the global feature representation f of the video:

[0066] f = Concat(f) v ,f t ′)+PE

[0067] Wherein, Concat(·) represents the feature concatenation operation.

[0068] PE stands for sinusoidal position code, which is used to preserve the time information of the frame sequence.

[0069] Step 3: Establish a matching enhancement encoding / decoding module.

[0070] (3.1) Establish an event encoder composed of a deformable Transformer network:

[0071] The Multi-Scale Deformable Attention Module (MSDA) of a Deformable Transformer network is used as the encoder to encode global features f and extract fragment-level features f. e :

[0072]

[0073] in, These are three learnable weight matrices in the encoder.

[0074] (3.2) Establish a query initialization module composed of a fully connected neural network to evaluate the fragment-level features f. e Does the event feature exist? An initial event query (t) is generated by selecting the top N features with the highest confidence. c ,t l E):

[0075] The fully connected neural network includes linear layers, multilayer perceptrons, and embedding layers, wherein:

[0076] Linear layers form a binary classification head for evaluating fragment-level features f e Does it include key events? Select the top N features with the highest confidence.

[0077] In the formula, FNN represents a binary classification head composed of linear layers, and N is a pre-defined number of events;

[0078] Multilayer perceptrons form a regression head used to determine relative position (t) c ,t l ):

[0079] In the formula, MLP represents a regression head composed of multilayer perceptrons, and t c It is the center of time, t l It refers to the length of time;

[0080] The embedding layer will select Mapped to query feature E:

[0081] In the formula, Embedding(·) represents the embedding layer;

[0082] Select N relative positions (t) c ,t l The initialization event query (t) is formed by combining the initialization event query (t) with the corresponding query feature E. c ,t l E).

[0083] (3.3) Establish a noise reduction and reconstruction module for real event time labels. Inject random noise to transform it into noisy event queries.

[0084] Two hyperparameters, λ and ψ, are set to control the labeling of real events. Injected random noise yields noisy event time labels. Where λ represents the number of queries that generate noise, and ψ represents the threshold for noise offset in the time dimension;

[0085] Noisy event time tags Make a judgment: if the noise offset and If none of them exceed ψ, they are marked as positive samples; otherwise, they are marked as negative samples.

[0086] Noise-added event time tags The noisy event features E are obtained after the embedding layer. dn This is then combined with time tags to create a noisy event query.

[0087] (3.4) Construct an event decoder consisting of a linear layer and a deformable Transformer network, using a multi-scale deformable attention module (MSDA), to process the initial query (t). c ,t l E) and noisy event queries Decode the event to generate event feature set D and denoised event feature set D'. dn :

[0088] The linear layer will initialize the query (t) c ,t l E) and noisy event queries The data is concatenated and mapped to the query feature Q;

[0089] The Multi-Scale Deformable Attention Module (MSDA) is used to combine query features with fragment-level features f e Interact and decode to generate event-level features f' e :

[0090]

[0091] in, These are three learnable weight matrices for the multi-scale deformable attention module in the decoder;

[0092] Based on different query types, the event-level feature f' e Divided into event feature set D and denoised event feature set D dn .

[0093] Step 4: Establish a redundancy removal output module.

[0094] (4.1) Construct a counter consisting of linear layers to predict the number of clusters K:

[0095] The event feature set D is input into the linear layer to obtain the predicted value K′ of the number of events contained in the video;

[0096] To ensure that the number of predicted events is no less than the number of actual events, a constant c is added to correct the predicted values, resulting in the number of clusters K:

[0097] K = K′ + c

[0098] (4.2) Establish a redundancy removal module based on the K-means clustering algorithm to group events with similar characteristics into the same class, and perform max pooling on events within the class to synthesize information and obtain the aggregated event feature set F:

[0099] (4.2.1) Calculate the event feature set D = {x1, x2, ..., x...} N The mean u of};

[0100] (4.2.2) Select the feature furthest from u as the first cluster center u1;

[0101] (4.2.3) Based on the number of clusters K, gradually select the point u closest to the current cluster center. i All u will be obtained k As the initial cluster centers, U = {u1, u2, ..., u...} i ,...,u K};

[0102] (4.2.4) Calculate the cosine similarity between each data point in the event feature set and the cluster center U, assign each data point to the cluster center cluster with the maximum cosine similarity, and select the data point closest to the centroid of the cluster in each cluster center cluster as the new cluster center of the cluster.

[0103] (4.2.5) Repeat step (4.2.4) until the clustering classification results are stable;

[0104] (4.2.6) Perform max pooling on the similar event features within the same cluster in the clustering results to obtain aggregated event features. Aggregate event features for all K types The features are merged to obtain the aggregated event feature set F.

[0105] (4.3) Construct an output module consisting of a locator and a subtitle generator, concatenate it to the redundancy removal module, and then combine the aggregated event feature set F and the denoised event feature set D. dn The input is fed into the output module to obtain the final generated result (t). s ,t e ,s), where t sIt is the start time of the event, t e "S" represents the event end time, and "s" represents the corresponding subtitle description.

[0106] Step 5: Construct a network for automatically generating surgical records based on dense video descriptions.

[0107] (5.1) The surgical instrument-guided enhanced feature extraction module, matching enhanced decomposition module, and redundancy removal output module described above are cascaded to form an automatic surgical record generation network based on dense video description, such as... Figure 2 As shown;

[0108] (5.2) Constructing the loss function L for the automatic surgical record generation network:

[0109] (5.2.1) Constructing the reconstruction loss L dn It consists of three parts: denoising reconstruction localization loss, denoising reconstruction classification loss, and denoising reconstruction subtitle loss.

[0110]

[0111] in, It is the denoising reconstruction localization loss, which is composed of the Giou loss function;

[0112] It is the denoising and reconstruction classification loss, which is composed of the Focal Loss classification loss function;

[0113] It is the loss for denoising and reconstructing subtitles, which is composed of the cross-entropy loss function;

[0114] β giou β cls and β cap Respectively represent and and The corresponding weights;

[0115] (5.2.2) Construction result generation loss L c It consists of four parts: localization loss, classification loss, event count loss, and caption loss.

[0116] L c =β1L giou +β2L cls +β3L ec +β4L cap

[0117] Among them, L giou The localization loss is composed of the Giou loss function;

[0118] L cls The classification loss is composed of the Focal Loss classification loss function;

[0119] L ec The event count loss is composed of the cross-entropy loss function.

[0120] L cap The subtitle loss is composed of the cross-entropy loss function;

[0121] β1, β2, β3 and β4 represent the relationships with L, respectively. giou L cls L ec and L cap The corresponding weights;

[0122] (5.2.3) Construct the query initialization loss L s It consists of query initialization location loss and query initialization classification loss:

[0123]

[0124] in, It is the query initialization location loss, which is composed of the Giou loss function;

[0125] It is the query initialization classification loss, which is composed of the Focal Loss classification loss function;

[0126] α giou and α cls Respectively represent and and The corresponding weights;

[0127] (5.2.4) Combining the reconstruction loss, result generation loss, and query initialization loss described above, we obtain the total loss function L of the network:

[0128] L = γ dn L dn +γ c L c +γsLs

[0129] Among them, L dn It is a reconstruction loss, L c It is the result generation loss, L s It is the query initialization loss, γ dn γ c γ s These are the weights of these three types of losses.

[0130] Step 6: Iteratively train the network for automatically generating surgical records based on dense video descriptions:

[0131] (6.1) Set the maximum number of iterations (Epoch) for network training and the initial learning rate (L).r ;

[0132] (6.2) Extract frame sequences from the surgical videos in the training set in chronological order and input them into the constructed surgical record automatic generation network. First, obtain the global feature representation f through the surgical instrument-guided enhanced feature extraction module, and then obtain the event feature set D and the denoised event feature set D through the matching enhanced encoder-decoder module. dn Finally, the prediction result is obtained through the redundancy removal output module;

[0133] (6.3) Calculate the training loss L based on the prediction results, optimize the network parameters using gradient descent, and validate the network on the validation set;

[0134] (6.4) Repeat (6.2) to (6.3) until the maximum number of iterations Epoch, select the parameters that perform best on the validation set, and obtain the trained surgical record automatic generation network based on dense video description.

[0135] Step 7: Input the test set videos into the trained surgical record automatic generation network to obtain the subtitle descriptions and corresponding timestamps of the surgical videos in the test set.

[0136] Example 2: Automatic generation system for surgical records based on dense video description.

[0137] Reference Figure 3 This example includes a data preprocessing module 1, a feature extraction module 2, an encoding / decoding module 3, an output module 4, a model building module 5, a model training module 6, and a model testing module 7. Its working principle is as follows:

[0138] Data preprocessing module 1 is used to acquire surgical videos, uniformly crop the videos to ensure that the duration does not exceed 5 minutes, sample each video at a sampling rate of 1 frame per second, and randomly divide them into training set, validation set and test set;

[0139] The feature extraction module 2 is used to extract features from the preprocessed video and input them to the encoding / decoding module 3;

[0140] The encoding / decoding module 3 converts the video features output by the feature extraction module into event features and inputs them into the output module 4;

[0141] The output module 4 aggregates the event features output by the encoding and decoding module and generates a prediction result for each event feature;

[0142] The model building module 5 is used to connect the feature extraction module 2, the encoding / decoding module 3 and the output module 4 in series to form an automatic surgical record generation network based on dense video description.

[0143] Model training module 6 is used to iteratively train the network built in model building module 5, obtain the model parameters that perform best on the validation set, and obtain the trained surgical record automatic generation network.

[0144] Model testing module 7 is used to test the trained surgical record automatic generation network using test set videos to obtain subtitles and timestamps corresponding to the surgical videos.

[0145] The effects of this invention can be further illustrated through simulation experiments:

[0146] I. Simulation Conditions

[0147] Experimental environment: Development was conducted using the PyCharm platform on Ubuntu 18.04 operating system, with Python 3.7 as the programming language and PyTorch 1.5 deep learning framework. Hardware configuration included an Intel i7 9700K processor and a GeForce RTX 2080 graphics card.

[0148] Experimental data: Tumor resection surgery dataset and CholecT50 dataset, where:

[0149] The surgical videos in the tumor resection surgery dataset were collected from 30 patients with soft tissue tumors who underwent surgery, including various soft tissue tumors such as neurofibromas, lipomas, fibromas, and vascular malformations. The video lengths in this dataset range from 47 seconds to 46 minutes and 27 seconds. The annotations contain 454 sentences of description and corresponding timestamps, involving 140 unique text elements and covering 7 common surgical instruments.

[0150] The CholecT50 dataset includes 50 endoscopic videos of laparoscopic cholecystectomy, extracted from the surgical procedure at a rate of 1 frame per second and presented in triplet format.<instrument,verb,target> The data was labeled with information about surgical activities. These labels contained 100 different triplets, covering 6 surgical instruments, 10 actions, and 15 targets.

[0151] In this simulation experiment, timestamps are obtained by skipping non-action video frames and merging adjacent video frames with the same triplet annotations. The corresponding sentence descriptions are obtained by connecting the elements of the triplet annotations with conjunctions.

[0152] II. Simulation Content

[0153] Simulation Experiment 1: Under the above simulation conditions, the present invention and four existing video captioning methods, TAD-CG, MT, PDVC and GVL, were tested on the test set of the tumor resection surgery dataset to generate captions and timestamps corresponding to the surgical videos. The generation results of each method were evaluated using multiple evaluation metrics, and the scores of each metric are shown in Table 1.

[0154] Table 1. Experimental evaluation results of the tumor resection surgery dataset.

[0155]

[0156] Wherein, BLEU-4: represents the accuracy score of matching four consecutive words between the generated sentence and the labeled sentence;

[0157] Rouge: Recall score calculated based on the difference between the word frequencies in the generated sentence and those in the labeled sentence;

[0158] Meteor: Considers different forms of words and synonyms in a sentence, and uses the F1 score as the evaluation metric;

[0159] CIDEr: Based on the semantic similarity between the generated sentence and the annotated sentence, it calculates the weighted frequency of words appearing in the corpus, and combines TF-IDF weights to evaluate the consistency score between the generated text and the reference text.

[0160] The names of the four existing video captioning methods are as follows:

[0161] TAD-CG is an abbreviation for Bidirectional attentive fusion with context gating for dense video captioning.

[0162] MT is an abbreviation for End-to-end dense video captioning with masked transformer.

[0163] PDVC stands for End-to-end dense video captioning with parallel decoding.

[0164] GVL stands for Learning Grounded Vision-Language Representation for Versatile Understanding in Untrimmed Videos.

[0165] As shown in Table 1, compared with the latest existing method GVL, the present invention significantly improves the BLEU-4, Rouge and CIDEr indices by 34.4%, 15.4% and 24.19%, respectively. In addition, the present invention also achieves suboptimal performance in the Meteor index.

[0166] The captions generated by this invention and two other methods, PDVC and GVL, from test videos in a tumor resection surgery dataset were visualized, and the results are shown in Figure 4.

[0167] from Figure 4 It can be seen that this invention, along with PDVC and GVL methods, can accurately generate descriptions of surgical instruments, such as "electrosurgical knife," and actions, such as "cutting." However, only this invention can accurately describe the tissue being acted upon, such as "tumor," at each stage. Furthermore, regarding redundancy, PDVC and GVL tend to generate captions for events that overlap in time, resulting in a large amount of repetition in the generated descriptions. In contrast, this invention can generate caption descriptions over a longer period, reducing redundancy.

[0168] Simulation Experiment 2 uses the present invention and four existing video captioning methods—TAD-CG, MT, PDVC, and GVL—to test on the CholecT50 dataset. It generates captions and timestamps corresponding to the videos, and evaluates the results using the same evaluation metrics as Simulation Experiment 1, obtaining the evaluation scores for each method. The results are shown in Table 2.

[0169] Table 2. Experimental evaluation results for the CholecT50 dataset.

[0170]

[0171] As shown in Table 2, the present invention achieves better scores on every indicator compared to other existing methods, indicating that the present invention can effectively improve the accuracy of the generated subtitles.

[0172] The simulation results above demonstrate that the present invention, through its method for automatically generating surgical records based on dense video descriptions, can effectively improve the accuracy and simplicity of surgical video description generation.

Claims

1. A method for automatically generating surgical records based on dense video description, characterized in that, include: Obtain surgical videos and randomly divide them into training, validation, and test sets in a 6:2:2 ratio; A network for automatically generating surgical records based on dense video description is constructed, comprising: a surgical instrument-guided feature extraction module, a matching enhancement encoding / decoding module, and a redundancy removal output module; the surgical instrument-guided feature extraction module includes a component for extracting basic visual features from the input video. CLIP image encoder and underlying text features for extracting input video The CLIP text encoder, and the two encoders output in parallel, as well as the output for visual features. and text features A feature fusion module is used to perform feature fusion to obtain enhanced global visual features f; and the total loss function L is obtained according to the following formula: ； in, It is a reconstruction loss. It is the result of generating loss. It is the query initialization loss. , These are the weights of these three types of losses; The matching enhancement encoding / decoding module includes: an event encoder, which is composed of a deformable Transformer network, used to process global features of the video. Encode and extract fragment-level features The query initialization module, composed of a fully connected neural network, is used to evaluate fragment-level features. Does the key event feature exist? Select the top-ranked events with the highest confidence level. Features Generate initialization event query ;in , This represents a regression head composed of multiple perceptrons. It is the center of time. It is the duration; the selected... Mapped to query feature E; denoising and reconstruction module, used to process the real event time labels ( , Inject random noise to transform it into a noisy event query. , Event decoder, used for initial query Noisy event query ( , Decode the event to generate event feature set D and denoised event feature set D. Then query the initialization event. Noisy event query ( , The data is concatenated and mapped to query features. The Multi-Scale Deformable Attention Module (MSDA) is used to combine query features with fragment-level features. Interact and decode to generate event-level features : ； in, , , These are three learnable weight matrices for the multi-scale deformable attention module in the decoder; The denoising and reconstruction module injects random noise into the real event time stamps, transforming them into noisy event queries. Its implementation includes: By setting two hyperparameters and Control over real event labels ( , Random noise is injected to obtain noisy event time labels. , ),in This indicates the number of queries that generated with added noise. The threshold representing the noise offset over time; Noisy event time tags ( , Make a judgment: if the noise offset and None exceeded If the sample is positive, it is labeled as a positive sample; otherwise, it is labeled as a negative sample. Noisy event time tags ( , The event features are obtained after adding noise through the embedding layer. This is then combined with time tags to form a noisy event query. , ); The redundancy removal output module includes: A counter, consisting of a fully connected network, is used to determine the number of clusters K; The event redundancy removal module is used to aggregate similar events in the event feature set D to reduce the number of events and obtain an aggregated event feature set F; its implementation includes the following: (7a) Calculate the event feature set The mean u; (7b) Select the feature furthest from u as the first cluster center. ; (7c) Based on the number of clusters K, select the point closest to the current cluster center step by step. All that will be obtained As initial cluster centers ; (7d) Calculate the cosine similarity between each data point in the event feature set and the cluster center U, assign each data point to the cluster center cluster with the maximum cosine similarity, and select the data point closest to the centroid of the cluster in each cluster center cluster as the new cluster center of the cluster. (7e) Iterate through the above (7d) process multiple times until the clustering classification results are stable; (7f) Perform max pooling on the similar event features within the same cluster in the clustering results to obtain aggregated event features. And aggregate all K types of event features The features of the aggregated events are merged to obtain the aggregated event feature set. ; The output module, consisting of a locator and a subtitle generator, is used to determine the subtitle based on the aggregated event feature set F and the denoised event feature set. Generate the final output ( ),in The start time of the event, It is the end time of the event. This corresponds to the subtitle description; The network for automatically generating surgical records based on dense video descriptions was iteratively trained and validated using training and validation sets: The test set videos are input into a pre-trained surgical record automatic generation network based on dense video description to obtain subtitles and timestamps corresponding to the surgical videos.

2. The method according to claim 1, characterized in that, The feature fusion module performs visual feature analysis. and text features Feature fusion is performed to obtain enhanced global visual features f, the implementation of which includes: Calculate using the Softmax function and the cosine similarity function. and Similarity matrix ; Based on the similarity matrix Filter out the text features most relevant to visual features. and its visual characteristics The features are then concatenated, and temporal information is added to the concatenated features using sinusoidal positional encoding to enhance the visual feature representation, resulting in the visual global feature f.

3. The method according to claim 1, characterized in that, The reconstruction loss The result is a loss. and query initialization loss Specifically, it is obtained in the following way: To rebuild the losses, It is the denoising reconstruction localization loss, which is composed of the Giou loss function; It is the denoising and reconstruction classification loss, by The classification loss function is composed of; It is the loss for denoising and reconstructing subtitles, which is composed of the cross-entropy loss function; , and Respectively represent and , and The corresponding weights; To generate a loss for the result, The localization loss is composed of the Giou loss function; For classification loss, by The classification loss function is composed of; Loss due to the number of events; The subtitle loss is composed of the cross-entropy loss function; , , and Respectively represent and , , and The corresponding weights; To query the initial loss, It is the query initialization location loss, which is composed of the Giou loss function; It is the initial classification loss of the query, by The classification loss function is composed of; and Respectively represent and and The corresponding weights.

4. The method according to claim 1, characterized in that, The iterative training and validation of the surgical record automatic generation network based on dense video description using training and validation sets includes the following implementation: (9a) Set the maximum number of iterations for network training Initial learning rate ; (9b) Extract frame sequences from the unedited surgical videos in the training set in chronological order, and input them into the surgical instrument-guided feature extraction module to obtain a global video feature representation. Then, it is input into the matching enhancement codec module to obtain the event feature set D and the denoised event feature set. The predicted result is then output through the redundancy removal output module. (9c) Calculate the training loss L based on the prediction results and the true values, backpropagate it using gradient descent, update the network parameters, and validate it on the validation set. (9d) Repeat (9b) to (9c) until the maximum number of iterations is reached. We select the model parameters that perform best on the validation set to obtain a trained network for automatically generating surgical records based on dense video descriptions.

5. A surgical record automatic generation system for implementing the method of any one of claims 1-4, characterized in that, include: The data preprocessing module is used to randomly divide the surgical videos into training, validation, and test sets. The feature extraction module is used to fuse video features and text features to obtain global video features; The encoding / decoding module is used to convert global video features into event features; The output module is used to aggregate event features and generate output for each event feature; The model building module is used to connect the feature extraction module, the encoding / decoding module and the output module to form an automatic surgical record generation network based on dense video description. The model training module is used to iteratively train the constructed surgical record automatic generation network using the training set and the total loss function L, and obtain the model parameters that perform best on the validation set to obtain the trained surgical record automatic generation network. The model testing module is used to test the trained surgical record automatic generation network using test set videos to obtain subtitles corresponding to the surgical videos.

Images