Operation video editing method and device, computer equipment and storage medium

Abstract

The invention relates to an operation video editing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a to-be-edited original operation video shot by an endoscope system, wherein the original operation video comprises original video frames; acquiring a first to-be-deleted video frame of which the video frame category is a to-be-deleted video frame category from the original video frames, and taking the video frames except the first to-be-deleted video frame in the original video frames as cleaned video frames; acquiring a second to-be-deleted video frame of which the video frame motion amount is smaller than a preset motion amount threshold from the cleaned video frame, and determining a to-be-deleted video frame according to the first to-be-deleted video frame and the second to-be-deleted video frame; deleting the video frame to be deleted to obtain an initial edited video; and inputting the initial edited video into a pre-trained video editing model, and editing the initial edited video through the video editing model to obtain a target edited video corresponding to the original operation video. By adopting the method, the operation video editing quality can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of medical image processing, and in particular to a surgical video editing method, device, computer equipment and storage medium. Background Art

[0002] With the development of medical image processing technology, a technology has emerged that uses an endoscope to shoot surgical videos. At the same time, since the operation time is usually long and the video may contain a large number of invalid frames such as blurred frames, smoke frames and external frames, if the surgical process needs to be taught, reviewed or presented in a meeting, users usually need to edit the surgical video to make it easier to display the surgical video.

[0003] In traditional technology, editing of surgical videos usually requires first identifying valid frames and invalid frames in the surgical videos, and deleting the invalid frames from the surgical videos to achieve editing of the surgical videos.

[0004] However, the current surgical video editing method usually requires users to manually screen each video frame in the surgical video to determine the valid frames and invalid frames. This method consumes huge human resources. If the video frames are directly predicted through a neural network model, the screened valid frames and invalid frames may not meet user needs. Therefore, the current surgical video editing method produces poor surgical videos. Summary of the invention

[0005] Based on this, it is necessary to provide a surgical video editing method, device, computer equipment and storage medium that can improve the quality of surgical video editing in response to the above technical problems.

[0006] In a first aspect, the present application provides a surgical video editing method, the method comprising:

[0007] Acquire the original surgical video to be edited and captured by the endoscope system, as well as the original video frames contained in the original surgical video;

[0008] From the original video frames, a first video frame to be deleted whose video frame category is a video frame to be deleted category is obtained, and video frames other than the first video frame to be deleted in the original video frames are used as cleaning video frames; the video frame category to be deleted is obtained based on the selection of the video clip user;

[0009] From the cleaned video frames, a second video frame to be deleted whose video frame motion is less than a preset motion threshold is obtained, and a video frame to be deleted is determined according to the first video frame to be deleted and the second video frame to be deleted;

[0010] Deleting the video frames to be deleted from the original surgical video to obtain an initial clipped video;

[0011] The initial clipped video is input into a pre-trained video clipping model, and the initial clipped video is clipped by the video clipping model to obtain a target clipped video corresponding to the original surgical video.

[0012] In one of the embodiments, the video frame category is characterized by a combination of video frame labels of the original video frame; the video frame category to be deleted is characterized by a combination of video frame labels to be deleted;

[0013] Obtaining the first to-be-deleted video frame whose video frame category is the to-be-deleted video frame category, including:

[0014] Obtain the video frame labels contained in each original video frame, and obtain the video frame label combination corresponding to each original video frame;

[0015] In response to a video editing instruction triggered by a video editing user, obtaining a combination condition of video frame labels to be deleted that matches the video editing instruction;

[0016] The original video frame whose video frame label combination satisfies the label combination condition of the video frame to be deleted is used as the first video frame to be deleted.

[0017] In one embodiment, obtaining the video frame label contained in each original video frame includes:

[0018] Obtain the video frame levels corresponding to each original video frame;

[0019] Each original video frame is input into a pre-trained video frame label recognition model corresponding to each video frame level, and the video frame label contained in each original video frame is obtained through the video frame label recognition model.

[0020] In one of the embodiments, before obtaining, from the cleaned video frames, a second video frame to be deleted whose video frame motion amount is less than a preset motion amount threshold, the method further includes:

[0021] Obtain a current cleaning video frame and a previous cleaning video frame adjacent to the current cleaning video frame; the current cleaning video frame is any cleaning video frame;

[0022] By using a key point detection algorithm, video frame feature points are obtained from the previous cleaned video frame, and the feature point positions of each video frame feature point in the previous cleaned video frame are determined;

[0023] Perform optical flow tracking on the feature points of each video frame to obtain the feature point position of each video frame feature point in the current cleaning video frame;

[0024] Based on the feature point position of each video frame feature point in the current cleaning video frame and the feature point position of each video frame feature point in the previous cleaning video frame, the feature point displacement of each video frame feature point corresponding to the current cleaning video frame is obtained;

[0025] According to the displacement of each feature point, the video frame motion corresponding to the current cleaning video frame is obtained.

[0026] In one embodiment, determining the video frame to be deleted according to the first video frame to be deleted and the second video frame to be deleted includes:

[0027] The video frames other than the first video frame to be deleted and the second video frame to be deleted in the original video frames are used as the video frames to be retained;

[0028] Sorting the to-be-retained video frames in chronological order to form an initial to-be-retained video segment consisting of temporally continuous to-be-retained video frames;

[0029] According to the initial video segment to be retained, a target video segment to be retained is determined, and original video frames other than the target video segment to be retained are used as video frames to be deleted.

[0030] In one embodiment, there are multiple initial video segments to be retained; determining target video segments to be retained according to the initial video segments to be retained includes:

[0031] When the video segment spacing between adjacent initial video segments to be retained is less than the preset video segment spacing, the video segments between the adjacent initial video segments to be retained are used as new video segments to be retained, and the adjacent initial video segments to be retained and the video segments to be retained are spliced ​​to obtain new initial video segments to be retained;

[0032] When the number of to-be-retained video frames contained in each initial to-be-retained video segment is greater than the preset number of video frames, the initial to-be-retained video segment is used as the target to-be-retained video segment.

[0033] In one embodiment, after obtaining the target clipped video corresponding to the original surgical video, the method further includes:

[0034] Obtaining evaluation information of the video editing user on the target editing video, and obtaining a reward value of the video editing model based on the evaluation information;

[0035] Based on a preset reinforcement learning method, the model parameters of the video editing model are updated using the reward value to obtain an updated video editing model.

[0036] In one embodiment, before inputting the initial clip video into a pre-trained video clip model, the method further includes:

[0037] Obtaining a sample original surgical video captured by an endoscope system and a target clipped video corresponding to the sample original surgical video;

[0038] The sample original surgical video is input into the video editing model to be trained, and the predicted editing video corresponding to the sample original surgical video is obtained through the video editing model;

[0039] According to the difference between the target editing video corresponding to the sample original surgical video and the predicted editing video corresponding to the sample original surgical video, the video editing model is trained to obtain a pre-trained video editing model.

[0040] In a second aspect, the present application also provides a surgical video editing device, the device comprising:

[0041] An original video acquisition module is used to acquire the original surgical video to be edited and obtained by the endoscope system, as well as the original video frames contained in the original surgical video;

[0042] The video frame distinguishing module is used to obtain a first to-be-deleted video frame whose video frame category is a to-be-deleted video frame category from the original video frame, and use the video frames other than the first to-be-deleted video frame in the original video frame as cleaning video frames; the to-be-deleted video frame category is obtained based on the selection of the video clip user;

[0043] A to-be-deleted frame determination module is used to obtain a second to-be-deleted video frame whose video frame motion amount is less than a preset motion amount threshold from the cleaned video frames, and determine the to-be-deleted video frame according to the first to-be-deleted video frame and the second to-be-deleted video frame;

[0044] An initial editing video determination module is used to delete the video frames to be deleted from the original surgical video to obtain an initial editing video;

[0045] The target clipping video acquisition module is used to input the initial clipping video into a pre-trained video clipping model, and to clip the initial clipping video through the video clipping model to obtain the target clipping video corresponding to the original surgical video.

[0046] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0047] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0048] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0049] The above-mentioned surgical video editing method, device, computer equipment and storage medium obtain the original surgical video to be edited obtained by an endoscopic system, and the original video frames contained in the original surgical video; from the original video frames, obtain the first video frame to be deleted whose video frame category is the category of the video frame to be deleted, and use the video frames in the original video frames except the first video frame to be deleted as cleaning video frames; the category of the video frame to be deleted is obtained based on the selection of the video editing user; from the cleaning video frames, obtain the second video frame to be deleted whose video frame motion is less than a preset motion threshold, and determine the video frame to be deleted based on the first video frame to be deleted and the second video frame to be deleted; delete the video frame to be deleted from the original surgical video to obtain an initial editing video; input the initial editing video into a pre-trained video editing model, and edit the initial editing video through the video editing model to obtain a target editing video corresponding to the original surgical video. Compared with the traditional technology, the present application determines the first video frame to be deleted by the video editing user as the category of the video frame to be deleted, and determines the second video frame to be deleted by performing motion detection on the cleaned video frame. By deleting the first video frame to be deleted and the second video frame to be deleted, rough editing is achieved to obtain an initial edited video, and the initial edited video is further fine-edited using a pre-trained video editing model to obtain a target edited video. In this way, the specific rough editing and fine editing methods provided by the present application can improve the quality of surgical video editing and improve the editing effect of surgical videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram showing an application environment of a surgical video editing method in one embodiment;

[0051] Figure 2 A schematic diagram of a process of a surgical video editing method in one embodiment;

[0052] Figure 3 A schematic diagram of a flow chart of a step of obtaining a first to-be-deleted video frame whose video frame category is a to-be-deleted video frame category in an embodiment;

[0053] Figure 4 A flowchart of the steps before obtaining a second video frame to be deleted whose motion amount of the video frame is less than a preset motion amount threshold in one embodiment;

[0054] Figure 5 A first flow chart of a step of determining a target video segment to be retained in one embodiment;

[0055] Figure 6A second flow chart of a step of determining a target video segment to be retained in one embodiment;

[0056] Figure 7 A schematic diagram of a process of an intelligent minimally invasive surgery video editing method in one embodiment;

[0057] Figure 8 A schematic diagram of a flow chart of a step of performing motion detection on a cleaned video frame in one embodiment;

[0058] Fig. 9 is a structural block diagram of a surgical video editing device in one embodiment;

[0059] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] Minimally invasive surgeries use endoscope systems, so corresponding endoscopic surgery videos are naturally produced. Since the surgery lasts a long time and the video contains a large number of invalid frames such as blurred frames, smoke frames, and in vitro frames, it takes a lot of manpower for users to edit a video that can be used for teaching, review, or conference presentations, which exacerbates the shortage of medical resources. At the same time, minimally invasive surgery videos have special editing requirements compared to conventional videos, such as the completeness of key content, including important pathological sites, key operation actions, etc.; it is necessary to select materials, delete shaky images, laparoscopic images, redundant images, etc.; the simplicity requirements are high, and the video length is compressed. Surgical videos that are up to 1 to 3 hours or even longer are finally compressed to about 5 to 30 minutes. These requirements make the editing of surgical videos more difficult than conventional videos.

[0062] This application takes into account the characteristics of surgical video editing, such as high compression ratio, low continuity requirements, high image quality requirements, and editing preferences that are different from conventional videos, and proposes a surgical video editing method that can flexibly filter and delete to obtain high-quality video frames.

[0063] The surgical video editing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server can obtain the original surgical video from the data storage system, and the data storage system can store the target clip video. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0064] In one embodiment, Figure 2 As shown, a surgical video editing method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0065] S202, obtaining the original surgical video to be edited and captured by the endoscope system, and the original video frames contained in the original surgical video.

[0066] The endoscope system may include an endoscope, and the required surgical video may be captured by the endoscope. The endoscope system is a surgical system used for minimally invasive surgery. The original surgical video may be an unedited surgical video captured by the endoscope system. The original video frame may be a surgical image corresponding to each preset time in the original surgical video in units of time; for example, one second may correspond to 12 frames of surgical images, and one frame of surgical image may be 1 / 12 second.

[0067] Exemplarily, the original unedited surgical video captured by the endoscope system can be uploaded to the data storage system, and the server can obtain the original surgical video from the data storage system and determine each original video frame contained in the original surgical video.

[0068] S204, obtaining a first video frame to be deleted whose video frame category is a video frame to be deleted category from the original video frame, and using video frames other than the first video frame to be deleted in the original video frame as cleaning video frames; the video frame category to be deleted is obtained based on the selection of the video clip user.

[0069] The video frame category may be a category pre-set for the video frame. The video frame category to be deleted may be a video frame category that needs to be deleted. For example, the original video frame may be divided into frames with poor visual field, occluded frames, blurred frames, clear frames, etc., and the frames with poor visual field, occluded frames, and blurred frames may be selected as the original video frames corresponding to the video frame category to be deleted, that is, the frames with poor visual field, occluded frames, and blurred frames may be selected as the first video frames to be deleted. The video frame category to be deleted may be determined by the video clip user. The cleaned video frame may be a video frame that needs to be cleaned.

[0070] Exemplarily, the server can obtain the category of the video frame to be deleted input by the video editing user, and can determine the video frame category corresponding to the original video frame. If the video frame category of the original video frame is the category of the video frame to be deleted, the original video frame is used as the first video frame to be deleted, and the video frames in the original video frame except the first video frame to be deleted are used as cleaning video frames. Further, the cleaning video frame can be cleaned to determine other video frames that need to be deleted. In this way, the first video frame to be deleted can be determined based on the category of the video frame to be deleted determined by the video editing user, and the video frame to be deleted can be quickly obtained, thereby improving the quality and efficiency of surgical video editing.

[0071] Optionally, the category of the video frames to be deleted may be set according to actual conditions, and the category of the video frames to be deleted may be determined according to the video frames that the video clip user needs to delete.

[0072] S206, obtaining a second video frame to be deleted whose video frame motion amount is less than a preset motion amount threshold from the cleaned video frames, and determining a video frame to be deleted according to the first video frame to be deleted and the second video frame to be deleted.

[0073] The video frame motion amount may be the motion amount of the cleaned video frames between the preset frame numbers. The motion amount of the cleaned video frames may be the displacement amount between the cleaned video frames; for example, the displacement amount (motion amount) between the cleaned video frames may be determined based on the displacement amount between the same pixels between the cleaned video frames. The preset motion amount threshold refers to a pre-set motion amount threshold, and the motion amount threshold may be a threshold used to compare the motion amounts of the video frames. The video frame to be deleted may be a video frame that needs to be deleted.

[0074] Exemplarily, the server may determine the video motion amount between the cleaning video frames. If the video frame motion amount is less than a preset motion amount threshold, the cleaning video frame corresponding to the video frame motion amount may be used as the second video frame to be deleted. As an example, the cleaning video frames corresponding to the video frame motion amount may be two cleaning video frames or more than two cleaning video frames. For example, the video motion amount between the cleaning video frames may be determined based on three cleaning video frames.

[0075] Further, the video frames to be deleted that need to be deleted can be determined based on the first video frame to be deleted and the second video frame to be deleted. For example, the first video frame to be deleted and the second video frame to be deleted can be used as the video frames to be deleted, or the first video frame to be deleted and the second video frame to be deleted can be further screened to obtain the video frames to be deleted.

[0076] S208, deleting the video frames to be deleted from the original surgical video to obtain an initial edited video.

[0077] The initial clipping video may be a video that has undergone initial clipping; and the initial clipping step may be deleting a video frame to be deleted.

[0078] For example, the video frames to be deleted contained in the original surgical video can be deleted to obtain the surgical video after deleting the video frames to be deleted as the initial editing video. In this way, based on the above steps S204-S208, the original surgical video can be "roughly edited" to achieve preliminary editing, thereby improving the efficiency of editing the surgical video. The initial editing video after preliminary editing can be further edited, thereby improving the effect of editing the surgical video and improving the quality of surgical video editing.

[0079] S210, inputting the initial clipped video into a pre-trained video clipping model, and performing the video clipping on the initial clipped video through the video clipping model to obtain a target clipped video corresponding to the original surgical video.

[0080] The video editing model may be a model for editing surgical videos. The video editing model may be a model obtained through reinforcement learning, and the video editing model may have the ability to edit surgical videos. The target editing video may be an edited video that the video editing user expects to obtain.

[0081] For example, the server can input the initial edited video obtained by "rough editing" into a pre-trained video editing model, and edit the initial edited video through the video editing model to obtain the target edited video corresponding to the original surgical video after editing. In this way, the initial edited video can be edited more accurately through the video editing model, which can improve the accuracy of surgical video editing and the quality of the edited surgical video.

[0082] In this embodiment, an original surgical video to be edited, which is taken by an endoscopic system, and original video frames contained in the original surgical video are obtained; a first video frame to be deleted whose video frame category is a video frame to be deleted category is obtained from the original video frames, and video frames other than the first video frame to be deleted in the original video frames are used as cleaning video frames; the category of the video frame to be deleted is obtained based on the selection of the video editing user; a second video frame to be deleted whose video frame motion is less than a preset motion threshold is obtained from the cleaning video frames, and a video frame to be deleted is determined based on the first video frame to be deleted and the second video frame to be deleted; the video frame to be deleted is deleted from the original surgical video to obtain an initial editing video; the initial editing video is input into a pre-trained video editing model, and the initial editing video is edited by the video editing model to obtain a target editing video corresponding to the original surgical video. Compared with the traditional technology, the present application determines the first video frame to be deleted by the video editing user as the category of the video frame to be deleted, and determines the second video frame to be deleted by performing motion detection on the cleaned video frame. By deleting the first video frame to be deleted and the second video frame to be deleted, rough editing is achieved to obtain an initial edited video, and the initial edited video is further fine-edited using a pre-trained video editing model to obtain a target edited video. In this way, the specific rough editing and fine editing methods provided by the present application can improve the quality of surgical video editing and improve the editing effect of surgical videos.

[0083] In this embodiment, flexible filtering and deletion can be performed to obtain high-quality video frames; and the surgical video editing preferences and the video editing user's editing preferences can be learned through fine editing, and finally an edited version of the video that meets the surgical video editing requirements can be output.

[0084] In one embodiment, the video frame category is represented by a combination of video frame labels of the original video frame; the video frame category to be deleted is represented by a combination of video frame labels to be deleted;

[0085] like Figure 3 As shown, obtaining the first to-be-deleted video frame whose video frame category is the to-be-deleted video frame category includes:

[0086] S302, obtaining the video frame labels contained in each original video frame, and obtaining the video frame label combinations corresponding to each original video frame.

[0087] The video frame label may be a category label of the video frame, and the category label set of the video frame may be predefined and determined by the algorithm. For example, the video frame labels may be divided into in vitro frames, non-instrument frames, blurry frames, smoke frames, poor field of view frames, occlusion frames, lens motion frames, no operation frames, etc. The video frame label combination may be a logical combination of multiple video frame labels defined by the video clip user.

[0088] Exemplarily, the video frame tag carried by each original video frame can be obtained, each original video frame can carry multiple video frame tags, and the video frame tag combination corresponding to each original video frame is obtained. If the original video frame carries only one video frame tag, the video frame tag combination corresponding to the original video frame includes one video tag.

[0089] S304, in response to a video editing instruction triggered by a video editing user, obtaining a combination condition of video frame labels to be deleted that matches the video editing instruction.

[0090] The video editing instruction may be an instruction triggered by a video editing user in the terminal to edit the original surgical video. The video frame label combination condition to be deleted may be a condition set for the video frame label combination corresponding to the original video to be deleted.

[0091] For example, a video clipping user may trigger a video clipping instruction in a terminal, and the terminal sends the video clipping instruction to a server. The server may obtain a combination condition of video frame labels to be deleted that matches the video clipping instruction.

[0092] Optionally, the video editing user can select 0 to n video frame categories to be deleted according to actual needs, and combine them to obtain the label combination conditions of the video frames to be deleted. The video frame labels may include labels a, b, c, d, e, f, and g. The video editing user can define the advanced expression as follows:

[0093] a or c or f can be represented as deleting the original video frames that match one or more of the video frame labels a, c, and f; (b and c) or (e and f) not a can be represented as when the original video frame has both the b and c video frame labels, or both the e and f video frame labels, and does not have the a video frame label, the original video frame needs to be deleted. Video editing users can use the logical symbols and, or, and not to define the video frame labels of the original video frames that need to be deleted, so as to achieve more personalized, flexible, and free surgical video editing.

[0094] Among them, a or c or f and (b and c) or (e and f) not a can both be used as the label combination conditions of the video frame to be deleted in this embodiment.

[0095] S306: The original video frame whose video frame label combination satisfies the to-be-deleted video frame label combination condition is used as the first to-be-deleted video frame.

[0096] Exemplarily, if the video frame label combination corresponding to the original video frame meets the video frame label combination condition to be deleted, the original video frame can be used as the first video frame to be deleted. For example, if the video frame label combination corresponding to the original video frame is a video frame label combination that is predefined by the video clip user and needs to be deleted, the original video frame can be used as the first video frame to be deleted.

[0097] Optionally, if the video frame label combination corresponding to the original video frame satisfies a or c or f, it can be considered that the video frame label combination satisfies the video frame label combination condition to be deleted.

[0098] In this embodiment, by obtaining the video frame labels contained in each original video frame, the video frame label combination corresponding to each original video frame is obtained; and in response to the video editing instruction triggered by the video editing user, the video frame label combination condition to be deleted that matches the video editing instruction is obtained, and the original video frame whose video frame label combination satisfies the video frame label combination condition to be deleted is used as the first video frame to be deleted. In this way, the video editing user to be deleted is determined based on the video editing user, which can realize personalized, more flexible and free surgical video editing, so that the video editing user can edit the desired video, thereby improving the quality of surgical video editing.

[0099] In one embodiment, obtaining the video frame label contained in each original video frame includes:

[0100] Obtain the video frame levels corresponding to each original video frame;

[0101] Each original video frame is input into a pre-trained video frame label recognition model corresponding to each video frame level, and the video frame label contained in each original video frame is obtained through the video frame label recognition model.

[0102] The video frame level may be a level corresponding to a predefined video frame; for example, the original video frame may be divided into an image level and a timing level, and each original video frame may correspond to one or more video frame levels. As an example, the video frames at the image level may include in vitro frames, non-instrument frames, blurry frames, smoke frames, poor field of view frames, and occlusion frames, and the video frames at the timing level may include lens motion frames and no-operation frames.

[0103] Among them, the video frame label recognition model can be a modified ResNet model, ADDS, Query2Label using the transformer module, CRSA introducing residual attention, etc.

[0104] Exemplarily, the video frame levels corresponding to the original video frames may be determined, and a video frame label recognition model corresponding to the video frame level may be used to perform label recognition on the original video frames to obtain video frame labels respectively contained in the original video frames.

[0105] Optionally, if the original video frame corresponds to multiple video frame levels, the original video frame may be input into video frame label recognition models corresponding to the multiple video frame levels respectively to obtain multiple video frame labels corresponding to the original video frame.

[0106] Optionally, for image-level video frame labels, commonly used traditional multi-label multi-classification models include ML-KNN (multi-label-k-neighbor), ML-DT (multi-label-decision tree), Rank-SVM, CML, etc. Deep learning can be used to: obtain a multi-label classification model by modifying a single-label multi-classification model, such as modifying a loss function, an activation function, or a detection head; establish a correlation between the original video frame and the video frame label or between the video frame label and the video frame label, introduce an attention mechanism, and realize multi-label classification.

[0107] Optionally, for the video frame labels at the time-series level, the key original video frame sequence can be input and predicted using deep learning to obtain the time-series labels of the current original video frame. Common algorithms include three-dimensional network models such as 3D ResNet, and image and time-series combined detection such as CNN-RNN architecture networks.

[0108] In this embodiment, by obtaining the video frame level corresponding to each original video frame, and inputting each original video frame into the pre-trained video frame label recognition model corresponding to each video frame level, the video frame label contained in each original video frame is obtained through the video frame label recognition model. In this way, the video frame label corresponding to each original video frame can be identified through the pre-trained video frame label recognition model, which can improve the accuracy of identifying the video frame label of the original video frame, thereby improving the quality of surgical video editing.

[0109] In one embodiment, Figure 4 As shown, before obtaining the second to-be-deleted video frame whose video frame motion amount is less than a preset motion amount threshold from the cleaned video frames, the method further includes:

[0110] S402, obtaining a current cleaning video frame and a previous cleaning video frame adjacent to the current cleaning video frame; the current cleaning video frame is any cleaning video frame.

[0111] The current cleaning video frame may be any one of the cleaning video frames; any cleaning video frame may be processed using the method provided in this embodiment. The previous cleaning video frame may be the previous cleaning video frame of the current cleaning video frame.

[0112] Exemplarily, any cleaning video frame can be used as the current cleaning video frame, and the previous cleaning video frame adjacent to the current cleaning video frame is obtained; further, the motion amount detection can be performed using the current cleaning video frame and the corresponding previous cleaning video frame.

[0113] S404, obtaining video frame feature points from the previous cleaned video frame through a key point detection algorithm, and determining the feature point position of each video frame feature point in the previous cleaned video frame.

[0114] The key point detection algorithm may include a corner point detection algorithm. The key point detection algorithm may be a key point detection algorithm such as Harris, Shitomasi, SUSAN, FAST, SIFT, ORB, etc. The video frame feature point may be a point identified as having a characteristic. The feature point position may be a position of the video frame feature point in the cleaned video frame.

[0115] For example, key point detection processing may be performed on a previous cleaning video frame adjacent to the current cleaning video frame to obtain video frame feature points contained in the previous cleaning video frame, and feature point positions of the video frame feature points in the previous cleaning video frame may be determined.

[0116] Optionally, the Shitomasi key point detection algorithm (Shi-Tomasi corner point detection) can be used to obtain the video frame feature points contained in the previous cleaning video frame, which can improve the robustness of the key point detection and thus improve the quality of the edited surgical video.

[0117] S406 , performing optical flow tracking on the feature points of each video frame to obtain the feature point position of each video frame feature point in the current cleaning video frame.

[0118] For example, the optical flow tracking algorithm can be used to perform optical flow tracking processing on the feature points of each video frame to determine the feature point position of each video frame feature point in the current cleaned video frame. Further, the tracked video frame feature points can be screened to screen the video frame feature points in a normal tracking state to improve the accuracy of optical flow tracking.

[0119] Optionally, a KLT (Kanade-Lucas-Tomasi) tracking algorithm may be used to perform optical flow tracking on the feature points of each video frame to the feature point position of each video frame feature point in the current cleaned video frame.

[0120] S408, based on the feature point position of each video frame feature point in the current cleaning video frame and the feature point position of each video frame feature point in the previous cleaning video frame, obtain the feature point displacement of each video frame feature point corresponding to the current cleaning video frame.

[0121] The feature point displacement refers to the displacement of the feature point of the video frame between the previous cleaning video frame and the current cleaning video frame.

[0122] Exemplarily, for any video frame feature point, the feature point displacement of the video frame feature point corresponding to the current cleaning video frame can be obtained based on the feature point position of the video frame feature point in the current cleaning video frame and the feature point position of the video frame feature point in the previous cleaning video frame.

[0123] Furthermore, the displacement of feature points between the feature points of the video frames in the normal tracking state may be calculated and screened.

[0124] S410, obtaining the video frame motion amount corresponding to the current cleaning video frame according to the displacement amount of each feature point.

[0125] Exemplarily, the video frame motion corresponding to the current cleaning video frame can be obtained according to the displacement of feature points corresponding to multiple video frame feature points. For example, the average displacement of feature point displacements corresponding to multiple video frame feature points can be used as the video frame motion corresponding to the current cleaning video frame. Further, a threshold determination can be performed to identify the cleaning video frame whose video frame motion is lower than a preset motion threshold as a redundant frame, and the redundant frame is marked as the second frame to be deleted.

[0126] In this embodiment, the video frame feature points of the previous cleaning video frame of the current cleaning video frame are determined by a key point detection algorithm, and optical flow tracking is performed based on the video frame feature points to obtain the feature point positions of the video frame feature points of the current cleaning video frame. Based on the feature point positions of each video frame feature point in the current cleaning video frame and the feature point positions of each video frame feature point in the previous cleaning video frame, the feature point displacement of each video frame feature point corresponding to the current cleaning video frame is obtained. In this way, the video frame motion amount corresponding to the current cleaning video frame can be accurately determined, which is conducive to realizing the judgment of the second frame to be deleted that needs to be deleted, thereby improving the quality of surgical video editing.

[0127] In one embodiment, determining the video frame to be deleted according to the first video frame to be deleted and the second video frame to be deleted includes:

[0128] The video frames other than the first video frame to be deleted and the second video frame to be deleted in the original video frames are used as the video frames to be retained;

[0129] Sorting the to-be-retained video frames in chronological order to form an initial to-be-retained video segment consisting of temporally continuous to-be-retained video frames;

[0130] According to the initial video segment to be retained, a target video segment to be retained is determined, and original video frames other than the target video segment to be retained are used as video frames to be deleted.

[0131] The video frames to be retained may be video frames that need to be retained during the surgical video editing process. The initial video segments to be retained may be video frames to be retained arranged in time sequence. The target video segments to be retained may be video frames that are finally determined to be retained.

[0132] Exemplarily, video frames other than the first video frame to be deleted and the second video frame to be deleted in the original video frame can be used as the video frames to be retained. The video frames to be retained can be sorted in time order to form an initial video segment to be retained, and the initial video segment to be retained includes time-continuous video frames to be retained.

[0133] The initial video segment to be retained may be further time-series smoothed to obtain a target video segment to be retained, and the original video frames in the original surgical video except the target video segment to be retained are used as video frames to be deleted.

[0134] In this embodiment, by further performing timing-related processing on the original video frames except the first video frame to be deleted and the second video frame to be deleted, the original video frames to be deleted can be further selected, thereby improving the quality of surgical video editing.

[0135] In one embodiment, Figure 5 As shown, the number of initial video segments to be retained is multiple; according to the initial video segments to be retained, determining the target video segments to be retained includes:

[0136] S502, when the video segment spacing between adjacent initial video segments to be retained is less than the preset video segment spacing, taking the video segments between the adjacent initial video segments to be retained as new video segments to be retained, and splicing the adjacent initial video segments to be retained and the video segments to be retained to obtain new initial video segments to be retained;

[0137] S504: When the number of to-be-retained video frames contained in each initial to-be-retained video segment is greater than a preset number of video frames, the initial to-be-retained video segment is used as a target to-be-retained video segment.

[0138] The video segment spacing may be a time spacing between the initial video segments to be retained. The preset video segment spacing may be a threshold set for comparison with the video segment spacing. The number of video frames to be retained refers to the number of video frames to be retained. The preset number of video frames may be a threshold set for comparison with the number of video frames to be retained.

[0139] Exemplarily, the video segment spacing between two adjacent initial video segments to be retained can be determined. If the video segment spacing is less than the preset video segment spacing, the video segment between the adjacent initial video segments to be retained is used as a new video segment to be retained, that is, the video segment corresponding to the video segment spacing is used as a new video segment to be retained. The adjacent initial video segments to be retained and the new video segments to be retained can be spliced ​​to obtain new initial video segments to be retained. If the number of video frames to be retained contained in each initial video segment to be retained is greater than the preset number of video frames, the initial video segment to be retained can be used as the target video segment to be retained.

[0140] Optionally, the video frames to be deleted can be marked as frames to be deleted, and the remaining video frames are valid frames. Since the detection accuracy of the algorithm cannot reach 100%, in order to reduce false detection, the double threshold method can be used to smooth the final detection results in time series and correct the wrongly predicted labels. As an example, the flowchart is as follows Figure 6 As shown: S601, obtain the video frame labels at the time level; S602, classify and sort the cleaned video frames; S603, obtain the initial video segment to be retained; S604, whether the video segment spacing is less than the minimum preset video segment spacing; if so, S605, connect the two initial video segments to be retained to obtain a new initial video segment to be retained; S606, whether the segment length of the initial video segment to be retained is greater than the minimum continuous frame; if so, S607, use the initial video segment to be retained as the target video segment to be retained; if not, S608, use the initial video segment to be retained as the video segment to be deleted. S609, output the target video segment to be retained.

[0141] The original video frames and the video frames to be retained can be sorted in chronological order to form the initial video segments to be retained, and the initial video segments to be retained are marked as 1, and the video frames that do not belong to the initial video segments to be retained are marked as 0. If the video segment interval between the initial video segments to be retained is less than min_interval (preset video segment interval), the video segments between adjacent initial video segments to be retained are used as new video segments to be retained, that is, the labels of the video frames between adjacent initial video segments to be retained are set to 1. If the segment length of the merged initial video segments to be retained is greater than the minimum continuous frame number min_n (preset number of video frames), the initial video segments to be retained are used as the target video segments to be retained, otherwise the initial video segments to be retained are discarded, that is, the initial video segments to be retained are set to 0, and the original video frames contained in the initial video segments to be retained are used as video frames to be deleted.

[0142] In this embodiment, the initial video segment to be retained can be time-series smoothed by means of a dual threshold value, and the target video segment to be retained can be accurately determined, thereby improving the effect and quality of surgical video editing.

[0143] In one embodiment, after obtaining the target clipped video corresponding to the original surgical video, the method further includes:

[0144] Obtaining evaluation information of the video editing user on the target editing video, and obtaining a reward value of the video editing model based on the evaluation information;

[0145] Based on a preset reinforcement learning method, the model parameters of the video editing model are updated using the reward value to obtain an updated video editing model.

[0146] The evaluation information may be information for evaluating the evaluation index, and the evaluation index may be an index defined according to the actual editing requirements of the surgical video. The reward value may be a value obtained by a reward function.

[0147] For example, a video editing user can input evaluation information for a target video clip in a terminal, and the server can obtain the evaluation information sent by the terminal, and can calculate the reward value corresponding to the evaluation information through a reward function. And the server can update the model parameters of the video editing model based on a preset reinforcement learning method and the reward value to obtain an updated video editing model.

[0148] Optionally, the video editing user can evaluate the edited version of the target video. According to the requirements of surgical video editing, the scoring indicators can be defined as follows:

[0149]

[0150] By setting the reward function, calculate the reward value:

[0151] After the video editing user scores, the preset content integrity score is Score_Q, the material deletion score is Score_D, and the personal preference score is Score_I. Then there will be corresponding rewards, where c1, c2, and c3 are the corresponding weights respectively:

[0152] Reward_Q=Score_Q*c1; Reward_D=Score_D*c2; Reward_I=Score_I*c3;

[0153] The reward score (reward value) of the clip length is as follows, [a, b] is the ideal clip range defined by the video clip user, Len is the actual clip length, when Len is within this range, the reward is positive, otherwise it is 0, and the closer the reward Len is to the center of the ideal range, the greater the value:

[0154] Reward_L=max(0,1-abs(Len-m) / (ba))*c2;

[0155] In summary, the overall reward function is as follows:

[0156] Reward=Reward_Q+Reward_D+Reward_I+Reward_L

[0157] Since the reward function is not differentiable, the following methods can be used to update based on the calculated reward value: REINFORCE: Estimate parameter gradients based on reward values. Actor-Critic: Classic reinforcement learning method, Actor generates output, Critic is used to evaluate the output and generate the corresponding Q value (representing the state-action value function), and the stable Q value is used to update the parameters so that the model continues to accept new data after deployment. The model is immediately updated with manual feedback so that the model can quickly adapt to changes in the environment and user preferences. The video editing user decides whether he is satisfied with the editing output, outputs the edited video, and saves the new data. If the video editing user is satisfied with the fine editing result, the edited video is output; if the video editing user is not satisfied, the video editing user can manually edit again, and the output after editing is the final edited video. At the same time, the video edited by the video editing user is saved as new data for model update.

[0158] In this embodiment, the evaluation information of the video editing user on the target video clip is obtained, and the reward value of the video editing model is obtained based on the evaluation information; based on the preset reinforcement learning method, the model parameters of the video editing model are updated using the reward value to obtain an updated video editing model. In this way, by updating the video editing model, a better edited surgical video can be output through the video editing model, thereby improving the quality of surgical editing.

[0159] In one embodiment, before inputting the initial clip video into a pre-trained video clip model, the method further includes:

[0160] Obtaining a sample original surgical video captured by an endoscope system and a target clipped video corresponding to the sample original surgical video;

[0161] The sample original surgical video is input into the video editing model to be trained, and the predicted editing video corresponding to the sample original surgical video is obtained through the video editing model;

[0162] According to the difference between the target editing video corresponding to the sample original surgical video and the predicted editing video corresponding to the sample original surgical video, the video editing model is trained to obtain a pre-trained video editing model.

[0163] The sample original surgical video can be used as a sample for training the video editing model. The target editing video can be the actual editing result obtained after editing the original surgical video. The predicted editing video can be the predicted result of the video editing model editing the original surgical video.

[0164] For example, a sample original surgical video can be input into a video editing model to be trained, and the video editing model can predict the predicted editing video corresponding to the original surgical video. Further, the video editing model can be iteratively trained based on the difference between the target editing video corresponding to the sample original surgical video and the predicted editing video corresponding to the sample original surgical video to obtain a pre-trained video editing model.

[0165] In this embodiment, the sample original surgical video is input into the video editing model to be trained, and the predicted editing video corresponding to the sample original surgical video is predicted. The video editing model is trained according to the difference between the target editing video corresponding to the sample original surgical video and the predicted editing video corresponding to the sample original surgical video. In this way, the video editing model can be trained, providing a video editing model foundation for the "fine editing" step of this application, improving the accuracy of "fine editing", and thus improving the quality of editing the surgical video.

[0166] In one embodiment, Figure 7 As shown, an intelligent minimally invasive surgery video editing method is provided, including: S701, video input; S702, rough editing; S703, fine editing through a video editing model; S704, video editing user evaluation; S705, setting a reward function; S706, updating the video editing model; S707, if the editing user is not satisfied, fine-tuning the editing version; S708, outputting the fine-tuned target editing video or the target editing video that the editing user is satisfied with; S709, saving the data for fine-tuning the editing version, and fine-tuning and updating the video editing model.

[0167] Among them, S702, rough editing includes: S7021, structural processing through a multi-label classification algorithm and predefined free combination video frame label categories; S7022, motion detection of the cleaned video frames.

[0168] S7021, structured processing is performed through a multi-label classification algorithm and predefined freely combined video frame label categories, including: obtaining the first video frame to be deleted whose video frame category is the video frame category to be deleted from the original video frame; that is, deleting the material in the original surgical video, removing the invalid frame category (predefined category, freely combined by the user), and the video frame of the invalid frame category is the first video frame to be deleted.

[0169] Predefined category labels (video frame labels) are used to perform multi-label classification on the original video frames and label them accordingly. The predefined categories (video frame labels) can be divided into image level and video level. The image level refers to the information that can be directly obtained from the original video frame, such as "the original video frame is outside the body of the surgical object"; the timing level refers to the information that can only be obtained by combining the previous and next frames in the timing, such as "the endoscope lens is moving".

[0170] For image-level video frame labels, commonly used traditional video frame label recognition models include ML-KNN (multi-label-k-neighbor), ML-DT (multi-label-decision tree), Rank-SVM, CML, etc. Deep learning can be used to obtain a multi-label classification model by modifying a single-label multi-classification model, such as modifying the loss function, activation function, or detection head; establish the correlation between the original video frame and the video frame label or between the video frame label and the video frame label, introduce an attention mechanism, and realize multi-label classification. Among them, the video frame label recognition model can be a modified ResNet model, ADDS, Query2Label using the transformer module, CRSA introducing residual attention, etc.

[0171] For the time-series video frame labels, we can input the key original video frame sequence and use deep learning to predict the key original video frame sequence to obtain the time-series labels of the current original video frame. Commonly used algorithms include three-dimensional network models such as 3D ResNet, and image and time-series combined detection such as CNN-RNN architecture network.

[0172] Multi-label classification can classify video frame label categories into 8 categories. The definition of the video frame label category can be as shown in Table 2:

[0173] Table 2 Tag category definition

[0174]

[0175] Video editing users can select 0 to n video frame categories to be deleted according to actual needs, and combine them to obtain the label combination conditions of the video frames to be deleted. Video frame labels can include labels a, b, c, d, e, f, and g. Video editing users can define advanced expressions such as:

[0176] ①a or c or f, which can be represented as deleting the original video frames that match one or more video frame labels in a, c, and f;

[0177] ②(b and c)or(e and f)not a, which means that when the original video frame has both b and c video frame labels, or both e and f video frame labels, and does not have a video frame label, the original video frame needs to be deleted. Video editing users can use logical symbols and, or, and not to define the video frame labels of the original video frames that need to be deleted, so as to achieve more personalized, flexible, and free surgical video editing.

[0178] S7022, performing motion detection on the cleaned video frames, including: taking the video frames in the original video frames except the first video frame to be deleted as the cleaned video frames, and obtaining the second video frame to be deleted whose video frame motion is less than a preset motion threshold from the cleaned video frames; that is, removing redundant static original video frames (the second video frame to be deleted) to obtain a coarsely edited version.

[0179] The key point detection algorithm is used to detect the video frame feature points of the previous cleaning video frame of the current cleaning video frame to obtain its video frame feature points. The key point detection algorithm can be a key point detection algorithm such as Harris, Shitomasi, SUSAN, FAST, SIFT, ORB, etc. The KLT (Kanade-Lucas-Tomasi) tracking algorithm is used to perform optical flow tracking on the feature points of each video frame to the feature point position of each video frame feature point in the current cleaning video frame. The tracked video frame feature points are screened, and the video frame feature points in the normal tracking state are screened, and the average displacement of the feature points in the two original video frames before and after is calculated as the motion amount of the current clear video frame. Threshold judgment, the cleaning video frame whose video frame motion amount is lower than the preset motion amount threshold is identified as a redundant frame, and the redundant frame is marked as the second frame to be deleted.

[0180] The video frames to be deleted can be marked as frames to be deleted, and the remaining video frames are valid frames. Since the detection accuracy of the algorithm cannot reach 100%, in order to reduce false detection, the double threshold method can be used to smooth the final detection results in time series and correct the wrongly predicted labels.

[0181] The original video frames and the video frames to be retained can be sorted in chronological order to form the initial video segments to be retained, and the initial video segments to be retained are marked as 1, and the video frames that do not belong to the initial video segments to be retained are marked as 0. If the video segment interval between the initial video segments to be retained is less than min_interval (preset video segment interval), the video segments between adjacent initial video segments to be retained are used as new video segments to be retained, that is, the labels of the video frames between adjacent initial video segments to be retained are set to 1. If the segment length of the merged initial video segments to be retained is greater than the minimum continuous frame number min_n (preset number of video frames), the initial video segments to be retained are used as the target video segments to be retained, otherwise the initial video segments to be retained are discarded, that is, the initial video segments to be retained are set to 0, and the original video frames contained in the initial video segments to be retained are used as video frames to be deleted.

[0182] Through double thresholds (minimum interval between segments min_interval (preset video segment spacing) and minimum segment duration frame min_n (preset number of video frames)), each category is processed separately, and finally the smoothed result of each segment is output.

[0183] like Figure 8 As shown, S801, input the cleaned video frame; S802, Shitomasi algorithm is used to update the corner points; S803, the video frame feature points of the previous cleaned video frame are obtained, S804, KLT algorithm is used to perform optical flow tracking; S805, the displacement of the video frame feature points is screened and calculated; S806, threshold determination is performed; S807, the second video frame to be deleted is output.

[0184] S703, fine editing by a video editing model includes: inputting the initial editing video into a pre-trained video editing model, and editing the initial editing video by the video editing model to obtain a target editing video corresponding to the original surgical video.

[0185] Compared with regular videos, surgical videos do not pursue video continuity. They require that important content be complete, well-regulated, academic, and evidence-based. They require that the essentials be eliminated, that the logic be clear, that the key points be highlighted, that they be direct and decisive, and that the big picture be focused and the details be ignored.

[0186] The video summarization algorithm is used to fine-tune the initial video clips. Transfer learning is combined with reinforcement learning based on manual feedback. Based on the feedback from the video editing users, the model is updated online so that the model can learn the editing preferences of the video editing users. At the same time, the data of the video editing users' fine-tune is saved for model transfer learning and updating model parameters. The more data is accumulated, the better the model effect.

[0187] According to the offline trained video clipping model, the clipped version video is obtained to realize the editing of a shorter video containing important information from the original video from the original lengthy initial clipped video. The video clipping model can be a model that uses GRU and attention mechanism to generate video summaries, such as DR-DSN. The video clipping model can use RVM, Gone in 60ms, MM-GA of recurrent neural network, etc. The video clipping model can use Deep VideoCondenseNet of 3D-CNN, and DSNet, etc., which are anchor-free networks. Table 1 Scoring indicators

[0188]

[0189] Video editing users can evaluate the edited version of the target video. According to the requirements of surgical video editing, the scoring indicators can be defined as shown in Table 1.

[0190] By setting the reward function, calculate the reward value:

[0191] After the video editing user scores, the preset content integrity score is Score_Q, the material deletion score is Score_D, and the personal preference score is Score_I. Then there will be corresponding rewards, where c1, c2, and c3 are the corresponding weights respectively:

[0192] Reward_Q=Score_Q*c1; Reward_D=Score_D*c2; Reward_I=Score_I*c3;

[0193] The reward score (reward value) of the clip length is as follows, [a, b] is the ideal clip range defined by the video clip user, Len is the actual clip length, when Len is within this range, the reward is positive, otherwise it is 0, and the closer the reward Len is to the center of the ideal range, the greater the value:

[0194] Reward_L=max(0,1-abs(Len-m) / (ba))*c2;

[0195] In summary, the overall reward function is as follows:

[0196] Reward=Reward_Q+Reward_D+Reward_I+Reward_L

[0197] Since the reward function is not differentiable, the following methods can be used to update based on the calculated reward value: REINFORCE: Estimate parameter gradients based on reward values. Actor-Critic: Classic reinforcement learning method, Actor generates output, Critic is used to evaluate the output and generate the corresponding Q value (representing the state-action value function), and the stable Q value is used to update the parameters so that the model continues to accept new data after deployment. The model is immediately updated with manual feedback so that the model can quickly adapt to changes in the environment and user preferences. The video editing user decides whether he is satisfied with the editing output, outputs the edited video, and saves the new data. If the video editing user is satisfied with the fine editing result, the edited video is output; if the video editing user is not satisfied, the video editing user can manually edit again, and the output after editing is the final edited video. At the same time, the video edited by the video editing user is saved as new data for model update.

[0198] In a specific embodiment, the evaluation index and the corresponding reward function setting can be: Classification scoring: The video editing user gives a classification score of 1-5 points or 1-10 points according to the quality of the generated result, and then the classification score is mapped to a reward value. The higher the category score, the greater the reward value, such as: 5 points → +5, 4 points → +3, 3 points → +1, 2 points → -1, 1 point → -3. Satisfaction scoring: The video editing user scores the satisfaction of the generated result, ranging from 0 to 100. The higher the satisfaction, the greater the reward. Then linearly map to the interval [0, 1] or [-1, 1]. Binary classification: The video editing user gives a binary judgment of whether it is satisfied, with satisfaction being +1 and dissatisfaction being -1. Segment or fine classification: The video editing user gives a score by segment or finer classification, and then calculates the average or comprehensive reward. Segment subdivision, for example: Scoring is performed based on the results of manual editing by the video editing user. The video editing user edits the video, and for each frame, there are only two situations - keep the frame, or delete the frame. The algorithm also detects frames, and each frame is divided into a retained frame or a deleted frame. For each frame: if the algorithm detects it as retained, the video editor user also defines it as retained: +10 points; if the algorithm detects it as deleted, the video editor user also defines it as deleted: +20 points; if the algorithm detects it as retained, the video editor user defines it as deleted: -10 points; if the algorithm detects it as deleted, the video editor user defines it as retained: -20 points. The final score is the total score of all frames, where 10 points and 20 points can be custom values, indicating whether the model should focus more on video compression or retaining the target frame. Custom reward: The custom reward value is directly given by the video editor user.

[0199] In a specific embodiment, the reinforcement learning strategy can be: Q-Learning - finding the optimal strategy by iteratively updating the Q function, Policy-based method - directly iterating the policy parameters to find the optimal strategy, model-based method - first learning the ring model, and then planning the optimal strategy based on the model, etc.

[0200] In this embodiment, an intelligent minimally invasive surgery video editing method is proposed, which is divided into two modules: rough editing and fine editing. It has the advantages of high editing precision, high video compression rate, flexible material selection, more in line with surgical video editing preferences, and is suitable for most minimally invasive surgery videos. In the rough editing module, the video structuring algorithm and motion detection algorithm are used to screen the materials, simulating the video editing user to choose the materials. The video structuring algorithm and motion detection delete the video frames in different aspects. The structured information has multi-scale granularity. Motion detection can remove redundant frames. Through this rough editing module, the material can be greatly streamlined. The fine editing module includes three main components: video summary algorithm, manual fine-tuning editing, and reinforcement learning algorithm based on manual feedback. Based on the evaluation feedback of video editing users, the editing preferences of video editing users are learned online, so that the editing effect of surgical videos is more in line with their special needs. When the model is not trained to a satisfactory effect, the final version of the video is obtained by manual fine-tuning editing, and the video edited by the video editing user is retained as a data set for data expansion and fine-tuning, which is used to fine-tune and optimize the video summary model. As the amount of video editing user data increases, the editing accuracy of the model will continue to improve and will be more in line with user editing preferences.

[0201] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0202] Based on the same inventive concept, the embodiment of the present application also provides a surgical video editing device for implementing the surgical video editing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more surgical video editing device embodiments provided below can refer to the limitations of the surgical video editing method above, and will not be repeated here.

[0203] In one embodiment, Fig. 9 As shown, a surgical video editing device 900 is provided, comprising: an original video acquisition module 910, a video frame distinction module 920, a to-be-deleted frame determination module 930, an initial editing video determination module 940 and a target editing video acquisition module 950, wherein:

[0204] The original video acquisition module 910 is used to acquire the original surgical video to be edited, which is captured by the endoscope system, and the original video frames contained in the original surgical video.

[0205] The video frame distinguishing module 920 is used to obtain the first to-be-deleted video frame whose video frame category is the to-be-deleted video frame category from the original video frame, and use the video frames in the original video frame except the first to-be-deleted video frame as cleaning video frames; the to-be-deleted video frame category is obtained based on the selection of the video clip user.

[0206] The to-be-deleted frame determination module 930 is used to obtain a second to-be-deleted video frame whose video frame motion is less than a preset motion threshold from the cleaned video frames, and determine a to-be-deleted video frame according to the first to-be-deleted video frame and the second to-be-deleted video frame.

[0207] The initial editing video determination module 940 is used to delete the video frames to be deleted from the original surgical video to obtain the initial editing video.

[0208] The target clipping video acquisition module 950 is used to input the initial clipping video into a pre-trained video clipping model, and perform clipping processing on the initial clipping video through the video clipping model to obtain a target clipping video corresponding to the original surgical video.

[0209] In one embodiment, the video frame category is characterized by a combination of video frame labels of the original video frame; the video frame category to be deleted is characterized by a combination of video frame labels to be deleted. The video frame distinguishing module includes a video frame label acquisition unit, a video frame label response unit and a first frame to be deleted unit.

[0210] The video frame label acquisition unit is used to acquire the video frame labels contained in each original video frame, and obtain the video frame label combination corresponding to each original video frame. The video frame label response unit is used to respond to the video editing instruction triggered by the video editing user, and obtain the video frame label combination condition to be deleted that matches the video editing instruction. The first frame to be deleted unit is used to combine the video frame labels to the original video frame that meets the video frame label combination condition to be deleted as the first video frame to be deleted.

[0211] In one embodiment, the video frame label acquisition unit includes a video frame level acquisition unit and a video frame label identification unit.

[0212] The video frame level acquisition unit is used to acquire the video frame level corresponding to each original video frame. The video frame label recognition unit is used to input each original video frame into a pre-trained video frame label recognition model corresponding to each video frame level, and obtain the video frame label contained in each original video frame through the video frame label recognition model.

[0213] In one embodiment, the surgical video editing device further includes a current video frame determination module, a key point detection module, an optical flow tracking module, a feature point displacement determination module and a video frame motion determination module.

[0214] The current video frame determination module is used to obtain the current cleaning video frame and the previous cleaning video frame adjacent to the current cleaning video frame; the current cleaning video frame is any cleaning video frame. The key point detection module is used to obtain the video frame feature points from the previous cleaning video frame through the key point detection algorithm, and determine the feature point position of each video frame feature point in the previous cleaning video frame. The optical flow tracking module is used to perform optical flow tracking on each video frame feature point to obtain the feature point position of each video frame feature point in the current cleaning video frame. The feature point displacement determination module is used to obtain the feature point displacement of each video frame feature point corresponding to the current cleaning video frame based on the feature point position of each video frame feature point in the current cleaning video frame and the feature point position of each video frame feature point in the previous cleaning video frame. The video frame motion determination module is used to obtain the video frame motion corresponding to the current cleaning video frame according to the displacement of each feature point.

[0215] In one embodiment, the to-be-deleted frame determination module includes a to-be-retained video frame unit, a time sorting unit, and a target to-be-retained video segment unit.

[0216] The video frame unit to be retained is used to take the video frames other than the first video frame to be deleted and the second video frame to be deleted in the original video frame as the video frames to be retained. The time sorting unit is used to sort the video frames to be retained in time order to form an initial video segment to be retained consisting of time-continuous video frames to be retained. The target video segment to be retained unit is used to determine the target video segment to be retained based on the initial video segment to be retained, and take the original video frames other than the target video segment to be retained as the video frames to be deleted.

[0217] In one embodiment, the number of the initial video segments to be retained is multiple. The target video segment unit to be retained includes a video segment spacing comparison unit and a video frame number comparison unit.

[0218] The video segment spacing comparison unit is used to, when the video segment spacing between adjacent initial video segments to be retained is less than the preset video segment spacing, use the video segments between adjacent initial video segments to be retained as new video segments to be retained, and splice the adjacent initial video segments to be retained and the video segments to be retained to obtain new initial video segments to be retained. The video frame number comparison unit is used to, when the number of video frames to be retained contained in each initial video segment to be retained is greater than the preset number of video frames, use the initial video segment to be retained as the target video segment to be retained.

[0219] In one embodiment, the surgical video editing device further includes an evaluation information acquisition module and a video editing model updating module.

[0220] The evaluation information acquisition module is used to obtain the evaluation information of the video editing user for the target video clip, and obtain the reward value of the video editing model based on the evaluation information. The video editing model update module is used to update the model parameters of the video editing model based on the preset reinforcement learning method and the reward value to obtain an updated video editing model.

[0221] In one embodiment, the surgical video editing device further includes a sample surgical video acquisition module, a predicted editing video acquisition module and a video editing model training module.

[0222] The sample surgical video acquisition module is used to acquire the sample original surgical video captured by the endoscope system, and the target clip video corresponding to the sample original surgical video. The predicted clip video acquisition module is used to input the sample original surgical video into the video clip model to be trained, and obtain the predicted clip video corresponding to the sample original surgical video through the video clip model. The video clip model training module is used to train the video clip model according to the difference between the target clip video corresponding to the sample original surgical video and the predicted clip video corresponding to the sample original surgical video, so as to obtain a pre-trained video clip model.

[0223] Each module in the above surgical video editing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0224] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store original surgical video data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a surgical video editing method is implemented.

[0225] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0226] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0227] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0228] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0229] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0230] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0231] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A surgical video editing method, characterized in that: The method comprises: Acquire an original surgical video to be edited and captured by an endoscope system, and original video frames contained in the original surgical video; From the original video frame, a first video frame to be deleted whose video frame category is a video frame to be deleted category is obtained, and video frames other than the first video frame to be deleted in the original video frame are used as cleaning video frames; the video frame category to be deleted is obtained based on the selection of the video clip user; Acquire a second to-be-deleted video frame whose video frame motion amount is less than a preset motion amount threshold from the cleaned video frames, and determine a to-be-deleted video frame according to the first to-be-deleted video frame and the second to-be-deleted video frame; Deleting the video frame to be deleted from the original surgical video to obtain an initial edited video; The initial clipped video is input into a pre-trained video clipping model, and the initial clipped video is clipped by the video clipping model to obtain a target clipped video corresponding to the original surgical video.

2. The method according to claim 1, characterized in that The video frame category is represented by a combination of video frame labels of the original video frame; the video frame category to be deleted is represented by a combination condition of video frame labels to be deleted; The step of obtaining a first to-be-deleted video frame of a to-be-deleted video frame category includes: Obtaining the video frame labels contained in each of the original video frames, and obtaining the video frame label combinations corresponding to each of the original video frames; In response to a video editing instruction triggered by a video editing user, obtaining a combination condition of video frame labels to be deleted that matches the video editing instruction; The original video frame whose video frame label combination satisfies the label combination condition of the video frame to be deleted is used as the first video frame to be deleted.

3. The method according to claim 2, characterized in that The obtaining of the video frame labels contained in each of the original video frames includes: Obtaining the video frame levels corresponding to the original video frames; Each of the original video frames is respectively input into a pre-trained video frame label recognition model corresponding to each of the video frame levels, and the video frame label respectively contained in each of the original video frames is obtained through the video frame label recognition model.

4. The method according to claim 1, characterized in that: Before obtaining the second to-be-deleted video frame whose video frame motion amount is less than a preset motion amount threshold from the cleaning video frame, the method further includes: Acquire a current cleaning video frame and a previous cleaning video frame adjacent to the current cleaning video frame; the current cleaning video frame is any cleaning video frame; Obtaining video frame feature points from the previous cleaning video frame through a key point detection algorithm, and determining the feature point position of each video frame feature point in the previous cleaning video frame; Performing optical flow tracking on the feature points of each video frame to obtain the feature point position of each video frame feature point in the current cleaning video frame; Based on the feature point position of each of the video frame feature points in the current cleaning video frame and the feature point position of each of the video frame feature points in the previous cleaning video frame, obtaining the feature point displacement of each of the video frame feature points corresponding to the current cleaning video frame; According to the displacement of each feature point, the video frame motion amount corresponding to the current cleaning video frame is obtained.

5. The method according to any one of claims 1 to 4, characterized in that: The determining the video frame to be deleted according to the first video frame to be deleted and the second video frame to be deleted includes: Using video frames other than the first to-be-deleted video frame and the second to-be-deleted video frame in the original video frames as video frames to be retained; Sorting the to-be-retained video frames in chronological order to form an initial to-be-retained video segment consisting of temporally continuous to-be-retained video frames; According to the initial video segment to be retained, a target video segment to be retained is determined, and original video frames other than the target video segment to be retained are used as the video frames to be deleted.

6. The method according to claim 5, characterized in that The number of the initial video segments to be retained is multiple; and determining the target video segments to be retained according to the initial video segments to be retained includes: When the video segment spacing between adjacent initial video segments to be retained is less than the preset video segment spacing, the video segments between the adjacent initial video segments to be retained are used as new video segments to be retained, and the adjacent initial video segments to be retained and the video segments to be retained are spliced ​​to obtain new initial video segments to be retained; When the number of to-be-retained video frames contained in each of the initial to-be-retained video segments is greater than the preset number of video frames, the initial to-be-retained video segment is used as the target to-be-retained video segment.

7. The method according to claim 1, characterized in that After obtaining the target clipped video corresponding to the original surgical video, the method further includes: Obtaining evaluation information of the video editing user for the target editing video, and obtaining a reward value of the video editing model based on the evaluation information; Based on a preset reinforcement learning method, the model parameters of the video editing model are updated using the reward value to obtain an updated video editing model.

8. The method according to claim 1, characterized in that Before inputting the initial clipped video into the pre-trained video clipping model, the method further includes: Acquire a sample original surgical video captured by the endoscope system and a target clipped video corresponding to the sample original surgical video; Inputting the sample original surgical video into the video editing model to be trained, and obtaining the predicted editing video corresponding to the sample original surgical video through the video editing model; The video editing model is trained according to the difference between the target editing video corresponding to the sample original surgical video and the predicted editing video corresponding to the sample original surgical video to obtain the pre-trained video editing model.

9. A surgical video editing device, characterized in that: The device comprises: An original video acquisition module, used to acquire the original surgical video to be edited, which is captured by the endoscope system, and the original video frames contained in the original surgical video; The video frame distinguishing module is used to obtain a first to-be-deleted video frame whose video frame category is a to-be-deleted video frame category from the original video frame, and use the video frames other than the first to-be-deleted video frame in the original video frame as cleaning video frames; the to-be-deleted video frame category is obtained based on the selection of the video clip user; a to-be-deleted frame determination module, configured to obtain, from the cleaned video frames, a second to-be-deleted video frame whose video frame motion amount is less than a preset motion amount threshold, and determine a to-be-deleted video frame according to the first to-be-deleted video frame and the second to-be-deleted video frame; An initial editing video determination module is used to delete the video frames to be deleted from the original surgical video to obtain an initial editing video; The target clipping video acquisition module is used to input the initial clipping video into a pre-trained video clipping model, and to clip the initial clipping video through the video clipping model to obtain a target clipping video corresponding to the original surgical video.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images