Minimally invasive surgery video abstract generation method

Abstract

The application provides a minimally invasive surgery video abstract generation method and relates to the technical field of video abstract generation.The application proposes frame-level importance estimation based on a surgery behavior semantic label, estimates a frame-level importance score according to the semantic label change of adjacent frames of a minimally invasive surgery video, captures more valuable surgery information contained in the video, and breaks through the limitation of the prior art method for extracting key content of a surgery video.Meanwhile, the application proposes a shot segmentation method based on change point detection, which takes the disappearance or appearance of a surgery instrument as the principle to segment the surgery video into shots, and overcomes the defect that the surgery video has no shot boundary in vision.Furthermore, the multi-target 0,1 programming method for selecting shots can ensure that the generated minimally invasive surgery video abstract has good surgery stage integrity, and can better meet the needs of doctors for the surgery process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video summary generation, and particularly relates to a minimally invasive surgery video summary generation method. BACKGROUND

[0002] The minimally invasive surgery video summary generation extracts the key content in the minimally invasive surgery, enables the doctor to quickly understand the minimally invasive surgery process and accelerates the sharing of minimally invasive surgery knowledge, helps the doctor to quickly understand the minimally invasive surgery process, and saves the time cost of watching the minimally invasive surgery video.

[0003] The existing minimally invasive surgery video summary generation is mostly important evaluation only for the image features (color, brightness, etc.) of the video, and then obtains the video summary.

[0004] Therefore, the existing video summary generation method does not consider the constraint of the prior information such as the minimally invasive surgery stage, and the video summary may miss part of the minimally invasive surgery stage information, and cannot guarantee the integrity of the minimally invasive surgery stage in the video summary. SUMMARY

[0005] (I) Technical problems solved

[0006] In view of the defects of the prior art, the present application provides a minimally invasive surgery video summary generation method, which solves the problem that the video summary generated by the prior art cannot guarantee the integrity of the minimally invasive surgery stage.

[0007] (II) Technical scheme

[0008] To achieve the above purpose, the present application is realized by the following technical scheme:

[0009] A minimally invasive surgery video summary generation method, the method comprising:

[0010] Based on the minimally invasive surgery video, the minimally invasive surgery behavior semantic label and the minimally invasive surgery stage label of each frame are obtained; wherein the minimally invasive surgery behavior semantic label comprises an instrument label, an action label and a target label;

[0011] Based on the minimally invasive surgery behavior semantic label, the importance score of each frame of the minimally invasive surgery video is obtained;

[0012] Based on the minimally invasive surgery stage label, the minimally invasive surgery video is divided into a plurality of corresponding video segments, and each video segment is divided into a plurality of shots according to the disappearance or appearance of the instrument; the shot comprises a plurality of minimally invasive surgery video frames;

[0013] Based on the minimally invasive surgery behavior semantic label, the semantic similarity between the shots is obtained;

[0014] Select several shots to compose a minimally invasive surgery video summary based on semantic similarity between shots and importance score of each frame of the minimally invasive surgery video.

[0015] Further, the minimally invasive surgery behavior semantic label and the minimally invasive surgery stage label of each frame of the minimally invasive surgery video are obtained based on the minimally invasive surgery video, and the obtaining includes:

[0016] The minimally invasive surgery image is obtained by preprocessing the minimally invasive surgery video.

[0017] Video frame image features of the minimally invasive surgery image are extracted based on a pre-trained convolutional neural network.

[0018] The video frame image features are respectively input into an instrument classification network, an action classification network, a target classification network, and a minimally invasive surgery stage classification network to obtain the minimally invasive surgery behavior semantic label and the minimally invasive surgery stage label.

[0019] Further, the importance score of each frame of the minimally invasive surgery video is obtained based on the minimally invasive surgery behavior semantic label, and the obtaining includes:

[0020] The importance score of each frame is calculated by a weighted sum of the Euclidean distance of the minimally invasive surgery behavior semantic labels of adjacent two frames.

[0021] Further, the minimally invasive surgery video is split into corresponding multiple video segments based on the minimally invasive surgery stage label, and each video segment is divided into several shots according to disappearance or appearance of an instrument, and the splitting includes:

[0022] The minimally invasive surgery video is split into corresponding multiple video segments based on the minimally invasive surgery stage label.

[0023] An average probability of existence of an instrument in each frame is obtained by averaging the probabilities of existence of all instrument categories in the instrument semantic label of each frame;

[0024] An instrument existence average probability of all frames in each video segment is input into a change point detection model to obtain a shot boundary set of each video segment.

[0025] Each video segment is divided into several shots based on the shot boundary set.

[0026] Further, the calculation method of the semantic similarity between shots is:

[0027]

[0028]

[0029] wherein, l(M α , N β ) represents a shot M = [M1, …, M α , …, Mm ] and semantic similarity of lens N = [N1, …, N β , …, N n ]; m and n represent the number of frames contained in lens M and N, respectively;

[0030] represents the minimally invasive surgery behavior semantic label of the αth frame in lens M;

[0031] N β represents the minimally invasive surgery behavior semantic label of the βth frame in lens N.

[0032] Further, the minimally invasive surgery video summary is selected based on the semantic similarity between the lenses and the importance score of each frame of the minimally invasive surgery video, comprising:

[0033] obtaining a multi-objective programming model aiming to maximize the sum of the importance scores of the selected lenses and minimize the cosine similarity of the semantic labels of the selected lenses;

[0034] solving the multi-objective programming model to screen a plurality of lens combinations to form a video summary.

[0035] Further, the objective function of the multi-objective programming model comprises:

[0036]

[0037] minf2(x) = XBX T

[0038] And the constraint conditions comprise:

[0039]

[0040] Wherein, maxf1(x) represents the maximization of the sum of the importance scores of the selected lenses;

[0041] minf2(x) represents the minimization of the cosine similarity of the semantic labels of the selected lenses;

[0042] E represents the number of video segments divided by stages;

[0043] N a represents the number of lenses of the ath video segment;

[0044] s ab represents the importance score of the bth lens of the ath video segment, and s ab is obtained from the mean value of the importance scores si of all frames in the lens;

[0045] l ab represents the length of the bth lens of the ath video segment;

[0046] B represents a similarity matrix of inter-lens minimally invasive surgical behavior, and each element in B is obtained by similarity calculation;

[0047] x ab ∈{0,1} represents whether to select the shot, 1 represents selecting the shot, and 0 represents not selecting;

[0048] X represents a vector about x ab , X = [x 00 , x 01 , …, x ab , …];

[0049] β represents the proportion of the length of the abstract to the length of the original video.

[0050] Further, the video frame image features are respectively input into the instrument classification network, the action classification network and the target classification network to obtain the minimally invasive surgical behavior semantic labels, including:

[0051] The video frame image features are respectively input into the pre-trained instrument convolutional network, the action convolutional network and the target classification network to obtain corresponding instrument convolutional features, action convolutional features and target convolutional features;

[0052] The instrument convolutional features, the action convolutional features and the target convolutional features are spliced and then input into a transformer model to be matched with label embedding vectors to obtain the minimally invasive surgical behavior semantic labels.

[0053] Further, the video frame image features are input into the minimally invasive surgical stage classification network to obtain the minimally invasive surgical stage labels, including:

[0054] The video frame image features are input into the pre-trained long short-term memory network to capture the time sequence features, and then the preliminary recognition results are obtained through two fully connected layers;

[0055] The output results of the fully connected layers are then input into a conditional random field to correct unreasonable minimally invasive surgical stage recognition results to obtain the final recognition results.

[0056] (Three) beneficial effects

[0057] The application provides a minimally invasive surgical video abstract generation method. Compared with the prior art, the application has the following beneficial effects:

[0058] 1. The application proposes frame-level importance estimation based on surgical behavior semantic labels, estimates frame-level importance scores according to the semantic label changes of adjacent frames of the minimally invasive surgical video, captures more valuable surgical information in the video, and breaks through the limitations of the prior art in extracting key contents of the surgical video.

[0059] 2. This invention proposes a shot segmentation method based on change point detection, which segments the surgical video based on the disappearance or appearance of surgical instruments, overcoming the defect that the surgical video has no shot boundaries in terms of visual perception.

[0060] 3. This invention proposes a multi-objective 0,1 planning method for selecting lenses, ensuring that the generated minimally invasive surgical video summary has good surgical stage integrity and can better meet the needs of doctors for the surgical process.

[0061] 4. This invention generates a summary of minimally invasive surgical videos, retaining key content from the videos, which helps doctors quickly understand the surgical process and saves time spent watching surgical videos. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of an embodiment of the present invention;

[0064] Figure 2 This is a flowchart illustrating the semantic tags for minimally invasive surgical behaviors, the tags for minimally invasive surgical stages, and the process of obtaining importance scores, as described in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] This application provides a method for generating video summaries of minimally invasive surgeries, which solves the problem that video summaries generated by existing technologies cannot guarantee the integrity of the minimally invasive surgical stages.

[0067] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0068] This invention constructs semantic tags for surgical behavior by identifying fine-grained information about the surgical process, such as instruments, actions, and target sites. It estimates the importance of video frames by utilizing changes in the semantic tags for surgical behavior between adjacent frames. Based on the identification of minimally invasive surgical stages, it constructs a shot segmentation algorithm based on the disappearance or appearance of surgical instruments to segment the video into sets with shots as the basic unit. Finally, it selects shots from each surgical stage through a multi-objective programming method to form a video summary.

[0069] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0070] Example 1:

[0071] This invention provides a method for generating video summaries of minimally invasive surgeries, which is executed by a computer and includes:

[0072] Semantic tags for minimally invasive surgical behavior and minimally invasive surgical stage are obtained for each frame of minimally invasive surgical video. The semantic tags for minimally invasive surgical behavior include instrument tags, action tags, and target tags.

[0073] Importance scores for each frame of a minimally invasive surgical video are obtained based on semantic tags of minimally invasive surgical behavior.

[0074] Based on the minimally invasive surgery stage tags, the minimally invasive surgery video is divided into multiple corresponding video segments, and each video segment is further divided into several shots according to the disappearance or appearance of instruments; each shot includes several minimally invasive surgery video frames.

[0075] Semantic similarity between shots is obtained based on semantic tags of minimally invasive surgical behaviors;

[0076] Based on the semantic similarity between shots and the importance score of each frame in the minimally invasive surgery video, several shots are selected to form a summary of the minimally invasive surgery video.

[0077] The beneficial effects of this embodiment are:

[0078] This invention proposes a frame-level importance estimation method based on semantic tags of surgical behavior. By estimating the frame-level importance score based on the changes in semantic tags of surgical behavior in adjacent frames of a minimally invasive surgical video, it captures more valuable surgical information contained in the video, overcoming the limitations of existing methods in extracting key content from surgical videos.

[0079] like Figure 1 As shown below, the implementation process of the embodiments of the present invention will be described in detail:

[0080] S100: Obtain semantic labels for minimally invasive surgical behaviors and minimally invasive surgical stage labels for each frame of the minimally invasive surgical video.

[0081] In practical implementation, a feasible method for obtaining semantic tags for minimally invasive surgical behaviors and tags for minimally invasive surgical stages can be adopted, the steps of which include S101 to S103:

[0082] S101. The minimally invasive surgical video is downsampled into images, and the images are preprocessed by scaling, normalization and other data processing so that they can be input into the subsequent multi-task learning model.

[0083] Specifically, the minimally invasive surgery video is converted into images using ffmpeg, and the images are extracted at a downsampling rate of 1 frame per second. Where c is the number of channels in the image, h is the height of the image, and w is the width of the image. After scaling the height and width of the image to 256*448 pixels, normalization is then performed.

[0084]

[0085] The values ​​for mean and std are typically mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225], respectively.

[0086] S102. Input the preprocessed series of image data into a pre-trained convolutional neural network to extract video frame image features of the minimally invasive surgical images.

[0087] Specifically, the preprocessed image data can be represented as:

[0088]

[0089] Where H0 represents the height of the image, W0 represents the width of the image, and C0 represents the number of channels of the image.

[0090] Inputting it into a pre-trained convolutional network CNN0 to extract video frame image features of minimally invasive surgery images can be represented as:

[0091]

[0092] Where C is the number of channels of feature f, and H and W are the height and width of feature f, respectively.

[0093] S103. Input the video frame image features f into the instrument classification network, action classification network, target classification network, and minimally invasive surgery stage classification network respectively to obtain the semantic labels of minimally invasive surgery behavior and the labels of minimally invasive surgery stages.

[0094] Specifically, the following methods can be used to identify semantic tags for minimally invasive surgical procedures:

[0095] The video frame image features f are input into a pre-trained multi-task learning model to extract convolutional features of fine-grained minimally invasive surgical behavior information such as instruments, actions, and target sites. This can be represented as:

[0096]

[0097]

[0098]

[0099] Where I, V, and T represent the three tasks of instrument recognition, motion recognition, and target part recognition, respectively, and B represents the batch data volume.

[0100] To improve the accuracy of identifying minimally invasive surgical procedures, the features H from the three tasks were combined. I H V H T The splicing can be represented as:

[0101]

[0102] Wherein, concat represents the concatenation operation;

[0103] Then, F is input into the transformer and matched with the label embedding vector l to obtain the output result R, which can be expressed as:

[0104]

[0105]

[0106] Where, n I n V n T This indicates the number of equipment categories, the number of motion categories, and the number of target body part categories.

[0107] Then The procedure is broken down into three parts: instruments, actions, and target sites. These parts are then input into a fully connected layer to obtain semantic tags for the minimally invasive surgical behavior.

[0108]

[0109]

[0110]

[0111] Among them, A I A V A T Semantic tags representing equipment, actions, and targets, respectively.

[0112] In addition, the following methods can be used to identify the stages of minimally invasive surgery:

[0113] While performing semantic label recognition of minimally invasive surgical actions, the video frame image features f can also be input into a pre-trained Long Short-Term Memory (BiLSTM) network to capture temporal features. Then, through two fully connected layers, a preliminary recognition result is obtained, which can be represented as:

[0114]

[0115] Where P represents the minimally invasive surgery stage, n p This represents the number of categories in the minimally invasive surgery stage, and n represents the number of video frames.

[0116] Then convert the output result A of the fully connected layer P The input is fed into a Conditional Random Field (CRF) to calibrate the identification results of the minimally invasive surgery stage, correcting unreasonable identification results, and obtaining the final identification result, which can be represented as:

[0117]

[0118] Where, r i r represents the minimally invasive surgical stage to which the i-th frame belongs. i ∈{0, 1, ..., n P}

[0119] S200: Obtain the importance score of each frame of a minimally invasive surgical video based on semantic tags of minimally invasive surgical behavior.

[0120] Specifically, the importance score of each frame can be evaluated by weighting the Euclidean distance between the semantic labels of minimally invasive surgical behaviors in two adjacent frames. This can be used to represent the changes in minimally invasive surgical behaviors between adjacent frames, as follows:

[0121]

[0122]

[0123] Where, m I m V m T Indicates the recognition accuracy for different tasks;

[0124] p j Indicates the weight of different tasks;

[0125] This represents the semantic label corresponding to task j in frame i.

[0126] S300: Based on the minimally invasive surgery stage tags, the minimally invasive surgery video is split into multiple corresponding video segments, and each video segment is divided into several shots according to the disappearance or appearance of instruments.

[0127] Specifically, because the content of minimally invasive surgical videos is very similar, it is difficult to determine the camera boundaries from a visual perspective alone. Therefore, we segment the shots based on the disappearance or appearance of instruments. The specific steps include S301 to S304:

[0128] S301. Based on the minimally invasive surgery stage tags, the minimally invasive surgery video is split into multiple corresponding video segments, which can be represented as:

[0129]

[0130] in, This represents the θ-th video segment;

[0131] n θ This represents the total number of frames in the θ-th video segment;

[0132] t1 represents the index of the first frame of the θ-th video segment;

[0133] Represents the nth video segment of the θ-th segment. θ The index of the frame (the last frame).

[0134] S302, Based on the obtained instrument semantic labels Describes all n for each frame of the image I The probability of each type of device appearing within each device category is calculated by averaging the probabilities of all device categories. The average probability of device presence is obtained when the average probability of device presence suddenly increases or decreases in consecutive frames, indicating that a device has appeared or disappeared. The calculation formula can be expressed as:

[0135]

[0136] The semantic tag for the device in a particular frame is:

[0137] n I Indicates the number of medical device categories.

[0138] d1 represents the probability of a Class 1 device appearing.

[0139] S303, the θ-th video segment o θ The average probability of the instruments existing in all frames is input into the PELT change point detection model to obtain the set of shot boundaries for the θ-th video segment, which can be represented as:

[0140]

[0141] Among them, t l This indicates that the l-th frame is considered as a shot boundary;

[0142] S304. Divide the θ-th video segment into several shots based on the obtained shot boundary set J.

[0143] For example, t1 and t 10 If two adjacent shot boundaries are in the shot boundary set J, then frames 1 to 10 can be combined into one shot.

[0144] S400: Obtain semantic similarity between shots based on semantic tags of minimally invasive surgical behavior.

[0145] Specifically, in order to select shots that are less similar to minimally invasive surgical procedures during the shot selection step, thereby ensuring low redundancy in the video summary content, this embodiment chooses to measure the semantic similarity between shots using cosine similarity, with the specific formula as follows:

[0146]

[0147]

[0148] Among them, l(M) α N β ) represents the lens M = [M1, ..., M a M m ] and lens N = [N1, ..., N β ,…,N n Semantic similarity;

[0149] m and n represent the number of frames contained in shots M and N, respectively;

[0150] This represents the semantic label of the αth frame in shot M;

[0151] N β This represents the semantic label for the minimally invasive surgical procedure in the βth frame of shot N.

[0152] S500: Based on the semantic similarity between shots and the importance score of each frame in the minimally invasive surgery video, several shots are selected to form a summary of the minimally invasive surgery video.

[0153] Specifically, a multi-objective 0,1 programming model can be designed to select shots from the obtained shot set to form a video summary, and constraints can be imposed on the minimally invasive surgery stages to ensure the completeness of the minimally invasive surgery stages in the video summary. Since the video summary of minimally invasive surgery extracts key content from the minimally invasive surgery, allowing doctors to quickly understand the minimally invasive surgery process and accelerate the sharing of minimally invasive surgery knowledge, the video summary should contain a large amount of minimally invasive surgery knowledge, and the similarity between minimally invasive surgery knowledge should be low. In this example, minimally invasive surgery behavior is used to represent minimally invasive surgery knowledge. Therefore, this dynamic programming model has two objectives: maximizing the sum of importance scores of the selected shots and minimizing the cosine similarity of the semantic labels of the selected shots. At the same time, in order to ensure that the summary video does not lack information on the minimally invasive surgery stages, the constraints should include: each minimally invasive surgery stage has a shot selected; the sum of the lengths of the selected shots should be less than γ times the length of the original video (the value of γ is determined based on whether the model has a solution, usually a real number within 15%-30%). The objective function can be expressed as:

[0154]

[0155] minf2(x)=XBX T

[0156] And the constraints are:

[0157]

[0158] Where maxf1(x) represents maximizing the sum of importance scores for the selected shots;

[0159] maxf2(x) represents minimizing the cosine similarity of the semantic labels of the selected shots;

[0160] E represents the number of video segments divided into stages;

[0161] N a This represents the number of shots in the a-th video segment;

[0162] s ab Let represent the importance score of the b-th shot in the a-th video segment, and s ab The importance score s of all frames within the shot i The average value is obtained;

[0163] l ab This represents the length of the b-th shot in the a-th video segment;

[0164] B represents the dissimilarity matrix of minimally invasive surgical behaviors between lenses, and each element in B is obtained through similarity calculation;

[0165] x ab∈{0,1} indicates whether to select the shot, where 1 indicates selecting the shot and 0 indicates not selecting it;

[0166] X represents x ab The vector, X = [x 00 x 01 , ..., x ab ,...];

[0167] β represents the ratio of the summary length to the original video length.

[0168] By solving the above planning model, a number of shots are finally selected to form a video summary.

[0169] To verify the method proposed in this invention, the specific experimental procedure is as follows:

[0170] 1. Data Preparation

[0171] This invention uses the CholecT45 dataset, which contains 45 cholecystectomy videos collected in Strasbourg, France. Images are extracted from the videos at a rate of 1 frame per second, and the surgical actions are labeled with triples in the format of <instrument, verb, target>. The dataset contains a total of 90,489 frames and 127,385 triple instances.

[0172] 2. Evaluation Indicators

[0173] The purpose of this video summarization invention is to extract more important surgical information. We evaluate the proposed method using the self-information of the triplet of each frame. The difference in self-information between adjacent frames reflects the change in information. The greater the change, the more important the information between the two adjacent frames, and the higher the value of the compactness evaluation index. On the other hand, it also reflects that the surgical information in the summarized video has a large dissimilarity. The compactness index is calculated as follows:

[0174]

[0175] Where T is the number of video frames in the summary video;

[0176] n is the number of video frames in the original video;

[0177] The self-information of the l-th frame of the summary video;

[0178] Ent(F i ) represents the self-information of the i-th frame of the original video;

[0179] 3. Analysis of Experimental Results

[0180] This invention sets the summary length to the original video length ratio β to 30%, and divides the 45 videos into training and testing sets in a 4:1 ratio. Five-fold cross-validation is then performed, and the results are compared with existing methods SUMGAN, DSN, and SUMGAN_ATT. The experimental results are shown in Table 1 below.

[0181] Table 1

[0182]

[0183]

[0184] Comparing the present invention with existing methods, as shown in Table 1, the minimally invasive surgical video summarization method proposed in this invention outperforms other advanced methods on the CholecT45 dataset, indicating that the summary video generated by this invention extracts more important surgical information, further verifying the effectiveness of the present invention.

[0185] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0186] 1. This invention proposes a frame-level importance estimation based on semantic tags of surgical behavior. It estimates the frame-level importance score based on the changes in semantic tags of surgical behavior in adjacent frames of a minimally invasive surgical video, thereby capturing more valuable surgical information contained in the video and overcoming the limitations of existing methods in extracting key content from surgical videos.

[0187] 2. This invention proposes a shot segmentation method based on change point detection, which segments the surgical video based on the disappearance or appearance of surgical instruments, overcoming the defect that the surgical video has no shot boundaries in terms of visual perception.

[0188] 3. This invention proposes a multi-objective 0,1 planning method for selecting lenses, ensuring that the generated minimally invasive surgical video summary has good surgical stage integrity and can better meet the needs of doctors for the surgical process.

[0189] 4. This invention generates a summary of minimally invasive surgical videos, retaining key content from the videos, which helps doctors quickly understand the surgical process and saves time spent watching surgical videos.

[0190] It should be noted that, through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the prior art, can be embodied in the form of software products. These computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain portions of the embodiments. In this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0191] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating video summaries of minimally invasive surgeries, characterized in that, The method includes: Semantic tags for minimally invasive surgical behaviors and minimally invasive surgical stage tags are obtained for each frame of the minimally invasive surgical video; wherein, the semantic tags for minimally invasive surgical behaviors include instrument tags, action tags, and target tags; The importance score of each frame of the minimally invasive surgery video is obtained by calculating the weighted sum of Euclidean distance based on the semantic tags of the minimally invasive surgical behavior in two adjacent frames. Based on the minimally invasive surgery stage tags, the minimally invasive surgery video is divided into multiple corresponding video segments, and each video segment is further divided into several shots based on the disappearance or appearance of instruments; each shot includes several minimally invasive surgery video frames; specifically, this step includes: S301. Based on the minimally invasive surgery stage tags, the minimally invasive surgery video is split into multiple corresponding video segments; S302. The average probability of the existence of a device at the frame level is obtained by averaging the probability of all device categories appearing in the device semantic tag of each frame. S303. Input the average probability of the existence of the instrument in all frames of each video segment into the change point detection model to obtain the set of shot boundaries of each video segment. S304. Divide each video segment into several shots based on the shot boundary set; Semantic similarity between shots is obtained based on semantic tags of minimally invasive surgical behaviors; Based on the semantic similarity between shots and the importance score of each frame in the minimally invasive surgery video, several shots are selected to form a summary of the minimally invasive surgery video; specifically, this step includes: S501. Obtain a multi-objective programming model that aims to maximize the sum of importance scores of selected shots and minimize the cosine similarity of semantic labels of selected shots. S502. Solve the multi-objective programming model to select several shots to form a video summary.

2. The method for generating a minimally invasive surgical video summary as described in claim 1, characterized in that, The method of obtaining semantic tags for minimally invasive surgical behavior and minimally invasive surgical stage tags for each frame based on minimally invasive surgical video includes: Preprocessing of minimally invasive surgical videos to obtain minimally invasive surgical images; Based on a pre-trained convolutional neural network, video frame image features of minimally invasive surgical images are extracted; The video frame image features are input into the instrument classification network, action classification network, target classification network, and minimally invasive surgery stage classification network, respectively, to obtain semantic labels for minimally invasive surgery behavior and labels for minimally invasive surgery stages.

3. The method for generating a minimally invasive surgical video summary as described in claim 1, characterized in that, The method for calculating the semantic similarity between the shots is as follows: in, Indicates the lens and the lens semantic similarity; and They represent lenses respectively. and The number of frames included; Indicates the lens The Middle Semantic tags for minimally invasive surgical procedures in frames; Indicates the number of categories of instruments, Indicates the number of action categories, Indicates the number of target categories; Indicates the lens The Middle Semantic tags for minimally invasive surgical behaviors in frames.

4. The method for generating a minimally invasive surgical video summary as described in claim 1, characterized in that, The objective function of the multi-objective programming model includes: And the constraints include: in, This represents the sum of importance scores for maximizing the selection of lenses; This represents minimizing the cosine similarity of the semantic labels of the selected shots; This indicates the number of video segments divided into stages; Indicates the first The number of shots in a video clip; Indicates the first The first video clip The importance score of each shot, and The importance score of all frames within the shot The average value is obtained; Indicates the first The first video clip The length of a single lens; This represents the similarity matrix of minimally invasive surgical procedures across different camera shots, and Each element in the dataset is obtained through similarity calculation; This indicates whether to select a lens; 1 indicates that the lens is selected, and 0 indicates that it is not selected. Indicates about The vector, ; This indicates the proportion of the summary length to the original video length.

5. The method for generating a minimally invasive surgical video summary as described in claim 2, characterized in that, The video frame image features are input into the instrument classification network, action classification network, and target classification network, respectively, to obtain semantic labels for minimally invasive surgical behaviors, including: The video frame image features are input into the pre-trained convolutional network for equipment, convolutional network for action, and convolutional network for target classification to obtain the corresponding convolutional features for equipment, action, and target. The convolutional features of the device, the action, and the target are concatenated and then input into... The model is matched with the label embedding vector to obtain semantic labels for minimally invasive surgical behaviors.

6. The method for generating a minimally invasive surgical video summary as described in claim 2, characterized in that, The video frame image features are input into the minimally invasive surgery stage classification network to obtain minimally invasive surgery stage labels, including: The video frame image features are input into a pre-trained long short-term memory network to capture temporal features, and then two fully connected layers are used to obtain preliminary recognition results. The output of the fully connected layer is then input into the conditional random field to correct unreasonable identification results for the minimally invasive surgery stage, thus obtaining the final identification result.

Images