Video abstract generation method and device, computer equipment and storage medium

Abstract

The invention relates to a video abstract generation method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a plurality of frames of video key images corresponding to key part features and key motion features in video data; respectively inputting the video key images into a plurality of pre-trained first network models to obtain corresponding prediction results; simultaneously extracting spatio-temporal information from multiple frames of video key images by using a first network model, and converting each frame of video key image into two-dimensional image features related to a time sequence; and combining the plurality of prediction results to obtain a video abstract of the video data. Through the stage recognition method and device, the problem of low accuracy of stage recognition in related technologies is solved, complementary information of key images of different frames of videos is provided by using two-dimensional image features and spatio-temporal information related to a time sequence, and then the accuracy of stage recognition is improved.

Description

Technical Field

[0001] The present application relates to the technical field of computer graphics and image video processing, and in particular to a video summary generation method, device, computer equipment and storage medium. Background Art

[0002] In the modern medical industry, automated surgical video analysis and summary generation technology is playing an increasingly important role. This technology can deeply analyze the surgical process, intelligently extract key information, and form real-time surgical step descriptions, guidance suggestions, and postoperative analysis reports, which not only significantly reduces the workload of surgeons, but also provides strong support for medical education and the training of novice doctors.

[0003] The current solution is to collect minimally invasive surgical videos and process them to obtain a picture sequence dataset; use the Backbone network shared intermediate layer to perform preliminary feature extraction on the surgical tools and surgical stages in the picture sequence dataset, and then use the classifier to obtain the recognition results of surgical tools and surgical stages to generate a video summary. The disadvantage of this solution is that it does not consider the information relationship between different frames in the video, resulting in low accuracy in surgical stage recognition.

[0004] With regard to the problem of low accuracy in surgical stage recognition in related technologies, no effective solution has been proposed so far. Summary of the invention

[0005] In this embodiment, a video summary generation method, apparatus, computer device and storage medium are provided to solve the problem of low accuracy of stage recognition in the related art.

[0006] In a first aspect, a video summary generation method is provided in this embodiment, including:

[0007] Acquire multiple frames of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features;

[0008] The video key images are respectively input into a plurality of pre-trained first network models to obtain corresponding prediction results; the first network model is used to simultaneously extract spatiotemporal information from a plurality of frames of the video key images, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model;

[0009] A plurality of the prediction results are combined to obtain a video summary of the video data.

[0010] In some embodiments, obtaining a plurality of video key images corresponding to key part features and key action features in video data includes:

[0011] Processing the acquired video data to obtain video image sequences corresponding to different scenes in the video data;

[0012] After randomly sorting the video image sequences between different scenes, the sequences are input into a pre-trained second network model to obtain candidate key images corresponding to key part features and key action features;

[0013] The video key frame is determined based on the local similarity between the candidate key images.

[0014] In some of the embodiments, the video data is processed to obtain video image sequences corresponding to different scenes in the video data, including:

[0015] Preprocessing the video data based on a sliding window with a preset number of sampling frames to obtain a continuous video image sequence; there is an overlapping area between two adjacent groups of video image sequences;

[0016] Based on the local difference between two adjacent frames of video images in each of the video image sequences, video image sequences corresponding to different scenes in the video data are determined.

[0017] In some of the embodiments, the expression corresponding to the local similarity is:

[0018] sim(i,j)=min(SIM[F(i),F(j)]);

[0019] Where sim(i,j) represents the minimum local similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame; SIM[F(i),F(j)] represents the cosine similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame.

[0020] In some of the embodiments, the first network model includes a Swin Transformer architecture and functional modules;

[0021] The prediction layer of the Swin Transformer architecture is a 2 / 3d temporal correlation convolution module;

[0022] The first network model is the action classification network model, and the functional module in the action classification network model is an action classification unit;

[0023] The first network model is the device classification network model, and the functional module in the device classification network model is a device classification unit;

[0024] The first network model is the target object detection network model, and the functional module in the target object detection network model is a target object detection unit.

[0025] In some embodiments, the video key images are respectively input into a plurality of pre-trained first network models to obtain corresponding prediction results, including:

[0026] Using the Swin Transformer architecture, spatiotemporal information is extracted from multiple frames of the video key image simultaneously, and each frame of the video key image is converted into a time-series-related two-dimensional image feature;

[0027] When the functional module is an action classification unit, action classification is performed on the time-series-related two-dimensional image features to obtain a prediction result of the action classification;

[0028] When the functional module is an instrument classification unit, performing instrument classification on the time-series-related two-dimensional image features to obtain an instrument classification prediction result;

[0029] When the functional module is a target object detection unit, target object detection is performed on the time-series-related two-dimensional image features to obtain a prediction result of the target object.

[0030] In some embodiments, the method further comprises:

[0031] Fine-tuning the functional module of the first network model;

[0032] The fine-tuned first network model is pre-trained to obtain the action classification network model, the device classification network model and the target object detection network model.

[0033] In some of the embodiments, the processing process of the 2 / 3D time series correlation convolution module is:

[0034] Performing a convolution operation on the feature map dimension corresponding to the key image of each frame of the video to obtain a corresponding first feature map;

[0035] Performing shape transformation on each of the first feature maps to obtain a second feature map Q, a second feature map K, and a second feature map V;

[0036] Perform matrix multiplication on the second feature map Q and the second feature map K corresponding to the first feature map of two adjacent frames, and use Softmax to determine the normalized attention weight coefficient;

[0037] Applying the attention weight coefficient to the second feature map V to obtain a first matrix; and concatenating all the first matrices to obtain a second matrix;

[0038] Based on the second matrix, a third matrix is ​​determined; and based on the third matrix, time-series-related two-dimensional image features are determined.

[0039] In a second aspect, a video summary generation device is provided in this embodiment, including: an acquisition module, a prediction module and a generation module;

[0040] The acquisition module is used to acquire multiple frames of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features;

[0041] The prediction module is used to input the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results; using the first network model, the temporal and spatial information is extracted from multiple frames of the video key images at the same time, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model;

[0042] The generating module is used to combine the multiple prediction results to obtain a video summary of the video data.

[0043] In a third aspect, a computer device is provided in this embodiment, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video summary generation method described in the first aspect when executing the computer program.

[0044] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the video summary generation method described in the first aspect is implemented.

[0045] Compared with the related art, the video summary generation method, device, computer equipment and storage medium provided in this embodiment obtain multiple frames of video key images corresponding to key part features and key action features in video data; the video key images are three-dimensional image features; the video key images are respectively input into multiple pre-trained first network models to obtain corresponding prediction results; using the first network model, spatiotemporal information is simultaneously extracted from the multiple frames of video key images, and each frame of video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model; multiple prediction results are combined to obtain a video summary of the video data, which solves the problem of low accuracy of stage recognition in the related art. Using the time-series-related two-dimensional image features and spatiotemporal information, the dependency between each element in different frames in the video data or different positions in different frames in the video data can be effectively captured to provide complementary information of different frames of video key images, thereby improving the accuracy of stage recognition.

[0046] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 It is a hardware structure block diagram of a terminal device of a video summary generation method provided by an embodiment of the present application;

[0049] Figure 2 is a flowchart of a video summary generation method provided by an embodiment of the present application;

[0050] Figure 3 It is a flowchart for obtaining key images of multiple frames of video;

[0051] Figure 4 It is a flow chart to obtain the prediction results;

[0052] Figure 5 It is a schematic diagram of the structure of a 2 / 3D time-series correlation convolution module provided in an embodiment of the present application;

[0053] Figure 6 It is a structural block diagram of a video summary generation device provided in an embodiment of the present application.

[0054] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, acquisition module; 220, prediction module; 230, generation module. DETAILED DESCRIPTION

[0055] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0056] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0057] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 FIG. 1 is a hardware structure diagram of a terminal of the video summary generation method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0058] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video summary generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0059] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0060] In this embodiment, a video summary generation method is provided. Figure 2 is a flow chart of the video summary generation method of this embodiment. Figure 2 As shown, the process includes the following steps:

[0061] Step S210, obtaining a plurality of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features;

[0062] Step S220, inputting the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results; using the first network model, extracting spatiotemporal information from multiple frames of video key images simultaneously, and converting each frame of video key image into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model, and a target object detection network model;

[0063] Step S230: combining multiple prediction results to obtain a video summary of the video data.

[0064] Specifically, in practical applications, the methods for obtaining the video key images in the embodiments of the present application include, but are not limited to, obtaining the video key images that meet the above requirements from pre-stored data in a database; or downloading the video key images that meet the requirements from a network platform; or first obtaining video data, and then processing the video data according to requirements to generate corresponding video key images, etc. The embodiments of the present application do not limit the methods for obtaining the video key images. Among them, the video key images are three-dimensional image features, which contain spatiotemporal information that characterizes the dependencies between elements in different frames or different positions.

[0065] Among them, the first network model is pre-trained, and the training samples used in the training are a large number of annotated video images with key part features and key action features; the network architecture of the first network model includes but is not limited to the Vision Transformer architecture, the Swin Transformer architecture, the LeNers architecture, the NASNet architecture, etc. If a different network architecture is used (the difference between the first network models is mainly in the functional modules), the training method can be adaptively adjusted, and there is no restriction on this. The first network model can be considered as three types of network models, namely, the action classification network model, the instrument classification network model, and the target object detection network model. That is, each first network model can simultaneously extract spatiotemporal information from multiple frames of video key images, and convert each frame of video key images into time-related two-dimensional image features, and then combine their own different functional modules to output the corresponding prediction results; the prediction results also have three categories, namely, the prediction results of action classification, the prediction results of instrument classification, and the prediction results of target objects. In other words: each first network model can obtain spatiotemporal information and time-series related two-dimensional image features, that is, each first network module can effectively capture the dependencies between elements in different frames of the video data or in different positions of different frames of the video data, so as to provide complementary information of key images of different frames of video, which can improve the accuracy of prediction result recognition.

[0066] Finally, these prediction results are combined to obtain a video summary of the video data; for example, the prediction results are input into an existing text processor to generate a video summary of the video data. After obtaining the video summary, information processing or analysis can be further performed based on the video summary, and the processing results or analysis results can be applied in different scenarios or used for medical education. If it is a surgical scene, the video data is surgical video data; the video summary is a surgical video summary.

[0067] The scheme adopted in the related art is: collect minimally invasive surgical videos and process them to obtain a picture sequence data set; use the Backbone network shared intermediate layer to perform preliminary feature extraction on the surgical tools and surgical stages in the picture sequence data set, and then use the classifier to obtain the recognition results of the surgical tools and surgical stages to generate a video summary. The disadvantage of this scheme is that it does not consider the information relationship between different frames in the video, resulting in low accuracy in surgical stage recognition. In this embodiment, the video key image itself has key part features and key action features that can determine the stage where the video key image is located. And because each first network model can obtain spatiotemporal information and time-series related two-dimensional image features, that is, each first network module can effectively capture the dependency between each element in different frames in the video data or different positions in different frames in the video data, so as to provide complementary information of video key images of different frames, which can improve the accuracy of prediction result recognition and thus improve the accuracy of stage recognition.

[0068] The following is a detailed description of each of the above steps:

[0069] In some of the embodiments, Figure 3 As shown, the step S210 of obtaining multiple frames of video key images corresponding to key part features and key action features in the video data includes the following steps:

[0070] Step S211, processing the acquired video data to obtain video image sequences corresponding to different scenes in the video data;

[0071] Step S212, randomly sorting the video image sequences between different scenes and inputting them into the pre-trained second network model to obtain candidate key images corresponding to key part features and key action features;

[0072] Step S213: determining the video key frame based on the local similarity between the candidate key images.

[0073] Specifically, the acquired video data can be processed using video image processing technology to obtain video image sequences corresponding to different scenes in the video data. For example: the video data is divided into a large number of video images; if there are multiple scenes in the video data, there will be relatively large differences between the video images of the scenes (the similarity between the calculated images can be used to determine whether the scenes have changed), and different scenes can be determined; for each scene, a preset number of video images are screened to obtain the corresponding video image sequence.

[0074] Among them, the second network model, similar to the first network model, is also pre-trained; the training samples used in the training are a large number of annotated video images with key part features and key action features; the network architecture of the first network model includes but is not limited to the Vision Transformer architecture, the Swin Transformer architecture, the LeNers architecture, the NASNet architecture, etc. If a different network architecture is used (the difference between the first network models is mainly in the functional modules), the training method can be adaptively adjusted, and there is no restriction on this. The second network model can also be considered as three types of network models, namely, the action classification network model, the equipment classification network model, and the target object detection network model. The difference between the second network model and the first network model is that the second network model may not have a 2 / 3d temporal correlation convolution module; for example, the second network model is a classic Swin Transformer architecture.

[0075] The video image sequences between different scenes are randomly sorted and input into the pre-trained second network model to obtain candidate key images corresponding to key part features and key action features; the video images still maintain temporal coherence within the same scene sequence, while presenting a disordered state between global sequences; thereby retaining the inherent logic of the video sequence and breaking the overall temporal linearity, effectively improving the generalization ability of the network model and preventing the occurrence of overfitting. It can be considered that the candidate key frames output by the second network model include frame-level importance scores, and the importance of video frames is evaluated according to the scores to finally obtain candidate key frames.

[0076] Finally, the local similarity between the candidate key images is calculated, and redundant frames with large similarity are removed according to the local similarity to obtain the video key frame. For example, if the local similarity between the candidate key images is greater than the similarity threshold, one of them is removed and the other is retained as the video key frame. Preferably, the similarity threshold is 0.6, which can be set according to the scene.

[0077] Through this embodiment, not only the generalization ability of the network model is effectively improved and the overfitting phenomenon is prevented; but also the local similarity is used to avoid the possibility of repeated descriptions in the final generated video summary.

[0078] In some embodiments, the step S211 of processing the video data to obtain video image sequences corresponding to different scenes in the video data includes the following steps:

[0079] Based on a sliding window with a preset number of sampling frames, the video data is preprocessed to obtain a continuous video image sequence; there is an overlapping area between two adjacent groups of video image sequences;

[0080] Based on the local difference between two adjacent frames of video images in each video image sequence, video image sequences corresponding to different scenes in the video data are determined.

[0081] Specifically, the sliding window is pre-set, and the size of the sliding window and the number of sampling frames can be selected by the user. According to the sliding window, N frames are continuously sampled by shifting backward one frame each time to obtain a group of continuous video image sequences of interest; repeat multiple times to obtain multiple groups of video image sequences. Among them, each group of video image sequences includes N frames of continuous video images, so there are N-1 frames of overlapping video images. And because the background changes are small during continuous acquisition, there are also overlapping areas between two adjacent groups of video image sequences. Preferably, the number of sampling frames can be 5 frames.

[0082] For surgical application scenarios, since the application background changes of video images are generally small, local frame difference is used to perform shot segmentation. The video images corresponding to each group of video image sequences are divided into grids of equal size, and then the previous and next frame differences are calculated according to the corresponding grid positions. The local maximum difference absolute value is used as the local difference between the previous and next frames. If the local difference exceeds the set threshold, it is considered that the surgical scene has changed, thereby determining the video image sequences corresponding to different scenes in the video data.

[0083] Through this embodiment, video image sequences corresponding to different scenes can be quickly and effectively screened out.

[0084] In some of the embodiments, the expression corresponding to the local similarity is:

[0085] sim(i,j)=min(SIM[F(i),F(j)]);

[0086] Where sim(i,j) represents the minimum local similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame; SIM[F(i),F(j)] represents the cosine similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame.

[0087] Specifically, before calculating the local similarity, the network can be equally divided similar to step S211, and then the above expression can be used to calculate the minimum local similarity between two frames. If the minimum local similarity between the two frames is higher than the set threshold of 0.6, redundant frames with large similarity are removed.

[0088] Through this embodiment, local similarity is used to avoid the possibility of repeated descriptions in the final generated video summary.

[0089] In some of the embodiments, the first network model includes a Swin Transformer architecture and functional modules;

[0090] The prediction layer of the Swin Transformer architecture is a 2 / 3d temporal correlation convolution module;

[0091] The first network model is an action classification network model, and the functional module in the action classification network model is an action classification unit;

[0092] The first network model is an equipment classification network model, and the functional module in the equipment classification network model is an equipment classification unit;

[0093] The first network model is a target object detection network model, and the functional module in the target object detection network model is a target object detection unit.

[0094] Specifically, each first network model has the same Swin Transformer architecture, but uses different functional modules, which are divided into action classification units, instrument classification units, or target object detection units. That is, it can be considered that the functional module in the action classification network model is the action classification unit; the functional module in the instrument classification network model is the instrument classification unit; and the functional module in the target object detection network model is the target object detection unit.

[0095] Among them, the Swin Transformer architecture uses the attention mechanism to drive and adopts the transfer learning strategy to fine-tune the pre-trained Swin Transformer model, remove the last prediction layer, and add a 2 / 3d temporal correlation convolution module as a new prediction layer. That is, the 2 / 3d temporal correlation convolution module is to capture and understand the temporal dynamics in the video. The feature map dimensions of adjacent frames extracted by the Swin Transformer architecture (excluding the prediction layer) are seamlessly transferred to the 2 / 3d temporal correlation convolution module to simultaneously analyze the three-dimensional data in the video data. It can simultaneously extract spatiotemporal information and effectively capture the dependencies between elements in different frames in the video data or in different positions in different frames in the video data. It not only maximizes the utilization efficiency of visual and temporal information, but also ensures that the network model can accurately capture and utilize the most representative features, thereby significantly improving the accuracy of analysis.

[0096] In some of the embodiments, Figure 4 As shown, in step S220, inputting the key images of the video into a plurality of pre-trained first network models respectively to obtain corresponding prediction results includes the following steps:

[0097] Step S221, using the Swin Transformer architecture, extracting spatiotemporal information from multiple video key images simultaneously, and converting each video key image into a time-series-related two-dimensional image feature;

[0098] Step S222, when the functional module is an action classification unit, action classification is performed on the time-series-related two-dimensional image features to obtain a prediction result of the action classification;

[0099] Step S223, when the functional module is an instrument classification unit, performing instrument classification on the time-series-related two-dimensional image features to obtain an instrument classification prediction result;

[0100] Step S224, when the functional module is a target object detection unit, target object detection is performed on the time-series-related two-dimensional image features to obtain a prediction result of the target object.

[0101] Specifically, the feature map dimensions extracted from the multi-frame video key images by the Swin Transformer architecture (excluding the prediction layer) are seamlessly transferred to the 2 / 3D temporal correlation convolution module. Figure 5 The figure shows the structure of the 2 / 3D time series correlation convolution module; the processing process of the 2 / 3D time series correlation convolution module is described:

[0102] If the feature map dimension corresponding to each frame of video key image (extracted by Swin Transformer architecture (excluding prediction layer)) is H×W×C; where H, W, and C represent the height, width, and number of channels of the map dimension respectively; if the number of channels is 512; then the key image input of this frame of video is a H×W×512-dimensional tensor.

[0103] Among them, the processing process of the 2 / 3D time series related convolution module is:

[0104] Performing a convolution operation on the feature map dimension corresponding to the key image of each frame of video to obtain a corresponding first feature map;

[0105] Performing shape transformation on each first feature map to obtain a second feature map Q, a second feature map K, and a second feature map V;

[0106] Perform matrix multiplication on the second feature map Q and the second feature map K corresponding to the first feature map of two adjacent frames, and use Softmax to determine the normalized attention weight coefficient;

[0107] Apply the attention weight coefficient to the second feature map V to obtain the first matrix; and concatenate all the first matrices to obtain the second matrix;

[0108] Based on the second matrix, a third matrix is ​​determined; and based on the third matrix, time-series-related two-dimensional image features are determined.

[0109] Specifically, 1. Perform three 1×1 convolution operations on the dimension of the feature map corresponding to each key image of the input video (the number of channels of the output feature map is C / 2) to obtain three first feature maps of size H×W×512. 2. Reshape the three first feature maps into the second feature map Q, the second feature map K and the second feature map V. The dimensions of the second feature map Q, the second feature map K and the second feature map V are HW×512, 512×HW and HW×512 respectively. Assuming that there are N frames in the same group of input video image sequences, the feature map dimensions of each frame after the feature map shape transformation are Q, K and V are HWi×512, 512×HWi and HWi×512 (i=1,2,3,4,5). 3. Perform matrix multiplication on Q and K corresponding to the feature maps of two adjacent frames, such as multiplying HW1×512 and 512×HW2 to obtain HW1×HW2, and then use Softmax to calculate the normalized attention weight coefficient. That is, the feature correlation coefficient of each frame image and the next adjacent frame image, where the last frame repeats the correlation coefficient with the previous frame image to maintain the number N. 4. Apply the attention weight coefficient to V, that is, calculate Softmax(Q, K)V, and obtain a HW×512 first matrix. 5. Concatenate the N HW×512 first matrices obtained in step 4 to obtain a N×HW512 second matrix, and then determine the third matrix based on the second matrix. Specifically: perform a shape transformation on the second matrix to obtain a HW512×N dimensional tensor. For example: N, H, and W are 5, 16, and 16 respectively, then the N×HW512 second matrix is ​​5×131,072 (16×16×512) in size, and the feature matrix is ​​transposed to 131,072×5 after shape transformation. Then perform matrix multiplication to obtain an N×N third matrix, and then use Softmax calculation to normalize and obtain the correlation coefficient between each frame of the key image in the video image sequence. 6. Based on the third matrix, determine the time-related two-dimensional image features; specifically: calculate the mean of the N×N third matrix to obtain an N×1-dimensional tensor, then perform matrix multiplication with the feature HW512×N to obtain the HW512×1 fourth matrix, and finally transform the fourth matrix into a shape to output the H×W×512 time-related two-dimensional image features.

[0110] Through the above process, multiple frames of video key images (with three-dimensional data) can be converted into time-series-related two-dimensional image features.

[0111] After obtaining the two-dimensional image features, they are input into the corresponding functional modules to output the prediction results of action classification, device classification and target object.

[0112] Through this embodiment, the execution logic is simplified, and the conversion of two-dimensional image features only needs to be executed once, and then input into different functional modules to output corresponding prediction results, thereby improving prediction efficiency.

[0113] In some of the embodiments, the video summary generation method further includes the following steps:

[0114] Fine-tune the functional modules of the first network model;

[0115] The fine-tuned first network model is pre-trained to obtain an action classification network model, an equipment classification network model, and a target object detection network model.

[0116] Specifically, the first network model is based on the Swin Transformer architecture, and the functional module is adjusted to be one of an action classification unit, an instrument classification unit, and a target object detection unit. Among them, the action classification unit and the instrument classification unit can be corresponding classifiers, including action classifiers and instrument classifiers; the target object detection unit can be a detector. The classifier includes a D×C fully connected layer, where D represents the dimension of the feature vector obtained after linear transformation, and C represents the total number of corresponding instrument categories or action categories. Among them, the detector can accurately detect the lesion location, lesion size, and confidence in each two-dimensional image feature.

[0117] After adjustment, the network of the added functional modules will be initialized to a zero-value matrix, avoiding the deviation that may be caused by the pre-trained model and ensuring the adaptability and generalization ability of the first network model to the new data set; the network parameters can be back-propagated and updated based on the network's prediction error for surgical video data.

[0118] Through this embodiment, based on the unchanged Swin Transformer architecture, the functional modules are adjusted to quickly match different requirements, reduce the difficulty of model training, and improve the efficiency of model training.

[0119] Figure 6 is a structural block diagram of the video summary generation device of this embodiment. Figure 6 As shown, the device includes: an acquisition module 210, a prediction module 220 and a generation module 230;

[0120] The acquisition module 210 is used to acquire multiple frames of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features;

[0121] The prediction module 220 is used to input the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results; using the first network model, the temporal and spatial information is extracted from the multiple frames of the video key images at the same time, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model;

[0122] The generation module 230 is used to combine multiple prediction results to obtain a video summary of the video data.

[0123] Through the above-mentioned device, the problem of low accuracy of stage recognition in the related technology is solved. By utilizing the time-related two-dimensional image features and spatiotemporal information, the dependency between various elements in different frames of video data or different positions of different frames in video data can be effectively captured to provide complementary information of key images of different frames of video, thereby improving the accuracy of stage recognition.

[0124] In some of the embodiments, the acquisition module 210 is further used to process the acquired video data to obtain video image sequences corresponding to different scenes in the video data;

[0125] After randomly sorting the video image sequences between different scenes, the sequences are input into the pre-trained second network model to obtain candidate key images corresponding to key part features and key action features;

[0126] Based on the local similarity between the candidate key images, the video key frames are determined.

[0127] In some of the embodiments, the acquisition module 210 is further used to pre-process the video data based on a sliding window of a preset sampling frame number to obtain a continuous video image sequence; there is an overlapping area between two adjacent groups of video image sequences;

[0128] Based on the local difference between two adjacent frames of video images in each video image sequence, video image sequences corresponding to different scenes in the video data are determined.

[0129] In some of the embodiments, the expression corresponding to the local similarity is:

[0130] sim(i,j)=min(SIM[F(i),F(j)]);

[0131] Where sim(i,j) represents the minimum local similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame; SIM[F(i),F(j)] represents the cosine similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame.

[0132] In some of these embodiments, the first network model includes a Swin Transformer architecture and functional modules;

[0133] The prediction layer of the Swin Transformer architecture is a 2 / 3d temporal correlation convolution module;

[0134] The first network model is an action classification network model, and the functional module in the action classification network model is an action classification unit;

[0135] The first network model is an equipment classification network model, and the functional module in the equipment classification network model is an equipment classification unit;

[0136] The first network model is a target object detection network model, and the functional module in the target object detection network model is a target object detection unit.

[0137] The functional module is an action classification unit, an instrument classification unit or a target object detection unit.

[0138] In some of the embodiments, the prediction module 220 is further used to simultaneously extract spatiotemporal information from multiple video key images using a Swin Transformer architecture, and convert each video key image into a time-series-related two-dimensional image feature;

[0139] When the functional module is an action classification unit, action classification is performed on the time-series-related two-dimensional image features to obtain a prediction result of the action classification;

[0140] When the functional module is an instrument classification unit, the time-series-related two-dimensional image features are used to classify instruments to obtain a prediction result of the instrument classification;

[0141] When the functional module is a target object detection unit, target object detection is performed on time-series-related two-dimensional image features to obtain a prediction result of the target object.

[0142] In some of the embodiments, the video summary generation apparatus further includes a fine-tuning module;

[0143] A fine-tuning module, used for fine-tuning the functional module of the first network model;

[0144] The fine-tuned first network model is pre-trained to obtain an action classification network model, an equipment classification network model, and a target object detection network model.

[0145] In some of the embodiments, the processing process of the 2 / 3D time-series correlation convolution module is:

[0146] Performing a convolution operation on the feature map dimension corresponding to the key image of each frame of video to obtain a corresponding first feature map;

[0147] Performing shape transformation on each first feature map to obtain a second feature map Q, a second feature map K, and a second feature map V;

[0148] Perform matrix multiplication on the second feature map Q and the second feature map K corresponding to the first feature map of two adjacent frames, and use Softmax to determine the normalized attention weight coefficient;

[0149] Apply the attention weight coefficient to the second feature map V to obtain the first matrix; and concatenate all the first matrices to obtain the second matrix;

[0150] Based on the second matrix, a third matrix is ​​determined; and based on the third matrix, time-series-related two-dimensional image features are determined.

[0151] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0152] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0153] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0154] S1, obtaining a multi-frame video key image corresponding to key part features and key action features in the video data; the video key image is a three-dimensional image feature;

[0155] S2, inputting the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results; using the first network model, extracting spatiotemporal information from multiple frames of video key images simultaneously, and converting each frame of video key images into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model;

[0156] S3, combining multiple prediction results to obtain a video summary of the video data.

[0157] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0158] In addition, in combination with the video summary generation method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the video summary generation methods in the above embodiments is implemented.

[0159] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0160] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0161] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A video summary generation method, characterized in that: include: Acquire multiple frames of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features; The video key images are respectively input into a plurality of pre-trained first network models to obtain corresponding prediction results; the first network model is used to simultaneously extract spatiotemporal information from a plurality of frames of the video key images, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model; A plurality of the prediction results are combined to obtain a video summary of the video data.

2. The video summary generation method according to claim 1, characterized in that: Obtaining multi-frame video key images corresponding to key part features and key action features in the video data, including: Processing the acquired video data to obtain video image sequences corresponding to different scenes in the video data; After randomly sorting the video image sequences between different scenes, the sequences are input into a pre-trained second network model to obtain candidate key images corresponding to key part features and key action features; The video key frame is determined based on the local similarity between the candidate key images.

3. The video summary generation method according to claim 2, characterized in that: Processing the video data to obtain video image sequences corresponding to different scenes in the video data includes: Preprocessing the video data based on a sliding window with a preset number of sampling frames to obtain multiple groups of video image sequences; there are overlapping areas between two adjacent groups of video image sequences; Based on the local difference between two adjacent frames of video images in each of the video image sequences, video image sequences corresponding to different scenes in the video data are determined.

4. The video summary generation method according to claim 2, characterized in that: The expression corresponding to the local similarity is: sim(i,j)=min(SIM[F(i),F(j)]); Where sim(i,j) represents the minimum local similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame; SIM[F(i),F(j)] represents the cosine similarity between the candidate key image of the i-th frame and the candidate key image of the j-th frame.

5. The video summary generation method according to claim 1, characterized in that: The first network model includes a SwinTransformer architecture and functional modules; The prediction layer of the Swin Transformer architecture is a 2 / 3d temporal correlation convolution module; The first network model is the action classification network model, and the functional module in the action classification network model is an action classification unit; The first network model is the device classification network model, and the functional module in the device classification network model is a device classification unit; The first network model is the target object detection network model, and the functional module in the target object detection network model is a target object detection unit.

6. The video summary generation method according to claim 5, characterized in that: Inputting the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results, including: Using the Swin Transformer architecture, spatiotemporal information is extracted from multiple frames of the video key image simultaneously, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; When the functional module is an action classification unit, action classification is performed on the time-series-related two-dimensional image features to obtain a prediction result of the action classification; When the functional module is an instrument classification unit, performing instrument classification on the time-series-related two-dimensional image features to obtain an instrument classification prediction result; When the functional module is a target object detection unit, target object detection is performed on the time-series-related two-dimensional image features to obtain a prediction result of the target object.

7. The video summary generation method according to claim 5, characterized in that: The method further comprises: Fine-tuning the functional module of the first network model; The fine-tuned first network model is pre-trained to obtain the action classification network model, the device classification network model and the target object detection network model.

8. The video summary generation method according to claim 5, characterized in that: The processing process of the 2 / 3D time-series correlation convolution module is: Performing a convolution operation on the feature map dimension corresponding to the key image of each frame of the video to obtain a corresponding first feature map; Performing shape transformation on each of the first feature maps to obtain a second feature map Q, a second feature map K, and a second feature map V; Perform matrix multiplication on the second feature map Q and the second feature map K corresponding to the first feature map of two adjacent frames, and use Softmax to determine the normalized attention weight coefficient; Applying the attention weight coefficient to the second feature map V to obtain a first matrix; and concatenating all the first matrices to obtain a second matrix; Based on the second matrix, a third matrix is ​​determined; and based on the third matrix, time-series-related two-dimensional image features are determined.

9. A video summary generation device, characterized in that: include: Acquisition module, prediction module and generation module; The acquisition module is used to acquire multiple frames of video key images corresponding to key part features and key action features in the video data; the video key images are three-dimensional image features; The prediction module is used to input the video key images into a plurality of pre-trained first network models respectively to obtain corresponding prediction results; using the first network model, the temporal and spatial information is extracted from multiple frames of the video key images at the same time, and each frame of the video key image is converted into a time-series-related two-dimensional image feature; the first network model is divided into an action classification network model, an equipment classification network model and a target object detection network model; The generating module is used to combine the multiple prediction results to obtain a video summary of the video data.

10. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the steps of the video summary generation method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video summary generation method according to any one of claims 1 to 8 are implemented.

Images