Key frame extraction method and device, electronic device and storage medium

Abstract

The invention relates to a key frame extraction method and device, an electronic device and a storage medium, and the method comprises the steps: obtaining a to-be-processed video stream, and obtaining each video frame image in the to-be-processed video stream; inputting each video frame image into the trained image reconstruction network to generate a reconstructed image; the reconstructed image carries depth generation prior information of the video frame image; inputting the reconstructed image into the trained image classification network, and performing recognition processing based on depth generation prior information to generate a key frame recognition result; and based on the key frame identification result, generating a key frame extraction result for the to-be-processed video stream. Through the method and the device, the problem of low accuracy of key frame extraction is solved.

Description

Technical Field

[0001] The present application relates to the field of video image processing, and in particular to a key frame extraction method, device, electronic device and storage medium. Background Art

[0002] Keyframe extraction is of great significance in the field of computer vision. It refers to extracting frame images with significant features from image sequences with a large amount of information. Keyframe extraction can effectively reduce the amount of calculation. Existing video keyframe extraction technologies mainly rely on manual selection or use algorithms based on video content. However, these methods are either time-consuming and labor-intensive and it is difficult to guarantee the extraction of the most representative keyframes, or they cannot accurately distinguish between important and non-important information in surgery, resulting in low accuracy of keyframe extraction.

[0003] Currently, no effective solution has been proposed for the problem of low accuracy of key frame extraction in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a key frame extraction method, device, electronic device and storage medium to at least solve the problem of low accuracy of key frame extraction in the related art.

[0005] In a first aspect, an embodiment of the present application provides a key frame extraction method, the method comprising:

[0006] Obtaining a video stream to be processed, and obtaining each video frame image in the video stream to be processed;

[0007] Inputting each of the video frame images into a trained image reconstruction network to generate a reconstructed image; the reconstructed image carries a priori information on the depth generation of the video frame image;

[0008] Inputting the reconstructed image into a trained image classification network, and performing recognition processing based on the depth-generated prior information to generate a key frame recognition result;

[0009] Based on the key frame recognition result, a key frame extraction result for the video stream to be processed is generated.

[0010] In some embodiments, the method further comprises:

[0011] Get training images;

[0012] Inputting the training image into an initial reconstruction network for training to obtain a predicted image;

[0013] A reconstruction loss function is calculated according to the training image and the predicted image; and a gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate the image reconstruction network.

[0014] In some embodiments, the step of back-transferring the gradient of the reconstruction loss function to the initial reconstruction network for iteration to generate the image reconstruction network includes:

[0015] Extracting first image features of the prediction image and second image features of the training image respectively, and calculating texture similarity between the first image features and the second image features;

[0016] Determining the number of iteration steps according to the texture similarity;

[0017] Based on the number of iteration steps, the gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate the image reconstruction network.

[0018] In some embodiments, after inputting the training image into the initial reconstruction network for training to obtain the predicted image, the method further comprises:

[0019] Obtaining image label data corresponding to the predicted image;

[0020] Inputting the predicted image into an initial classification network for training to obtain a classification prediction result;

[0021] A classification loss function is calculated according to the classification prediction result and the image label data; and the gradient of the classification loss function is back-transferred to the initial classification network for iteration to generate the image classification network.

[0022] In some embodiments, generating a key frame extraction result for the video stream to be processed based on the key frame recognition result includes:

[0023] Traversing the key frame recognition results corresponding to the reconstructed image;

[0024] If the key frame recognition result indicates that the traversed reconstructed image has key information, then the corresponding video frame image is determined to be a key frame; otherwise, then the video frame image is determined to be a non-key frame image;

[0025] The key frame recognition result corresponding to the next reconstructed image that is traversed is detected until the key frame recognition results corresponding to all the reconstructed images are traversed, and the key frame extraction result is generated.

[0026] In some embodiments, the step of obtaining each video frame image in the video stream to be processed includes:

[0027] Acquire each initial frame image in the video stream to be processed;

[0028] Performing bright channel processing on each of the initial frame images to determine a maximum pixel value in a preset color channel; and generating a grayscale image based on the maximum pixel value;

[0029] Perform maximum value filtering on the grayscale image to obtain the corresponding video frame image.

[0030] In some embodiments, performing maximum value filtering on the grayscale image to obtain the corresponding video frame image includes:

[0031] Performing maximum value filtering on the grayscale image to obtain a filtered image; performing normalization processing based on the filtered image to obtain a weight matrix;

[0032] The cumulative distribution function of the initial frame image is calculated; and according to the weight matrix and the cumulative distribution function, the initial frame image is subjected to weighted equalization processing to obtain the video frame image.

[0033] In a second aspect, an embodiment of the present application provides a key frame extraction device, the device comprising:

[0034] An acquisition module, used to acquire a video stream to be processed, and acquire each video frame image in the video stream to be processed;

[0035] A reconstruction module, used for inputting each of the video frame images into a trained image reconstruction network to generate a reconstructed image; the reconstructed image carries a priori information on the depth generation of the video frame image;

[0036] A recognition module, used for inputting the reconstructed image into a trained image classification network, and performing recognition processing based on the depth-generated prior information to generate a key frame recognition result;

[0037] The extraction module is used to generate a key frame extraction result for the video stream to be processed based on the key frame recognition result.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the key frame extraction method described in the first aspect above is implemented.

[0039] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the key frame extraction method described in the first aspect above is implemented.

[0040] Compared with the related art, the key frame extraction method, device, electronic device and storage medium provided in the embodiment of the present application obtain the video stream to be processed, and obtain each video frame image in the video stream to be processed; input each video frame image into the trained image reconstruction network to generate a reconstructed image; the reconstructed image carries the depth generation prior information of the video frame image; input the reconstructed image into the trained image classification network, and perform recognition processing based on the depth generation prior information to generate a key frame recognition result; based on the key frame recognition result, generate a key frame extraction result for the video stream to be processed. Based on this, through the image reconstruction technology based on the depth prior, the key information (i.e., the depth generation prior information) of the video stream to be processed is extracted, a reconstructed image is generated, and key frame recognition is performed based on the reconstructed image, so that the accuracy of key frame judgment is significantly improved, so that important information and non-important information in the video can be accurately distinguished, and the problem of low accuracy of key frame extraction caused by unclear key frame judgment targets in the related art is effectively solved, and an accurate and efficient key frame extraction method is realized.

[0041] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0043] Figure 1 It is a hardware structure block diagram of a terminal of a key frame extraction method according to an embodiment of the present application;

[0044] Figure 2 is a flowchart of a key frame extraction method according to an embodiment of the present application;

[0045] Figure 3 is a flowchart of another key frame extraction method according to an embodiment of the present application;

[0046] Figure 4 is a schematic diagram of a reconstruction network training process according to an embodiment of the present application;

[0047] Figure 5 is a flowchart of another key frame extraction method according to an embodiment of the present application;

[0048] Figure 6 It is a structural block diagram of a key frame extraction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0050] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 1 is a hardware structure block diagram of a terminal according to a key frame extraction method of an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a key frame extraction method in an embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0055] This embodiment provides a key frame extraction method. Figure 2 is a flowchart of a key frame extraction method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0056] Step S210, obtaining a video stream to be processed, and obtaining each video frame image in the video stream to be processed.

[0057] The video stream to be processed can be obtained from a variety of sources, including local files (such as video files stored on a hard disk), webcams or recording devices. In order to process the video data, it is necessary to first decode it into a sequence of raw video frames. The decoding process is usually completed by a video decoder or a library that supports a specific video format. Among them, for real-time video streams (such as videos from webcams), it may be necessary to use a specific application programming interface (Application Programming Interface, API) or library to capture and read the video stream. For local video files, file I / O operations can be used to read video data, or a video processing library can be used to simplify the process.

[0058] After the video stream is decoded or read, the video frame images can be extracted frame by frame. Specifically, the video frame data is separated from the video stream and converted into an image format. The extracted video frame images can be stored in a local file system, database, or cloud storage for subsequent processing or analysis.

[0059] It should be understood that the above-obtained video stream to be processed may be video stream data collected in different application scenarios. For example, the video stream to be processed may be video stream data collected in a surgical scenario, or video stream data collected in a scenario such as a traffic intersection.

[0060] Step S220, input each video frame image into the trained image reconstruction network to generate a reconstructed image; the reconstructed image carries a priori information on the depth generation of the video frame image.

[0061] Usually, a key frame is a frame that contains rich semantic information. Assuming that a complex deep neural network itself contains prior information of complex image semantics, and in the image generation task based on the deep neural network, the network often generates or reconstructs the texture structure containing the main semantic information.

[0062] Based on this theoretical basis, the image reconstruction network can be trained. The process of inputting each video frame image into the trained image reconstruction network to generate a reconstructed image is an important step in the field of computer vision and image processing. This process usually involves the application of deep learning and neural networks, where the image reconstruction network has been optimized through a large amount of training data to learn how to recover or generate high-quality images from the input video frame images.

[0063] In this process, the reconstructed image is not just a simple copy or transformation of the input video frame, but carries "deep generative prior information" about the input video frame. The "deep generative prior information" here refers to the deep understanding and representation of image structure, content, texture and other characteristics obtained by the network through learning a large amount of data. This information makes the reconstructed image visually clearer and more realistic, and is superior to the original input video frame in terms of highlighting key information.

[0064] Specifically, when generating a reconstructed image, the image reconstruction network uses the deep generative prior information learned during training to guide the image generation process. The image reconstruction network is usually a deep learning model that has been trained with a large amount of data to learn how to extract key features from the input image and generate new images based on these features; the image reconstruction network can include a U-shaped network (Unet), a fully convolutional neural network (FCN), or a neural network model based on a self-attention mechanism (Transfromer).

[0065] When a video frame image is input into the trained image reconstruction network, the network performs the following steps: First, the input image is analyzed to extract key features; these features may include color, texture, shape, etc., as well as deeper semantic information, such as the category, location, and relationship of objects. The network further selects the most critical features for image understanding and reconstruction, and enhances these features through nonlinear transformations to ensure that key information is given sufficient attention and accurately expressed during the reconstruction process. In the process of generating the reconstructed image, the network integrates the selected and enhanced features to effectively highlight the key information in the reconstructed image (i.e., the deep generation prior information mentioned above).

[0066] Step S230, input the reconstructed image into the trained image classification network, and perform recognition processing based on the depth-generated prior information to generate a key frame recognition result.

[0067] Specifically, reconstructed images are generated from the original video frame images through an image reconstruction network. These images carry deep generative prior information that helps with subsequent classification tasks. The reconstructed images output by the image reconstruction network are passed to a trained image classification network. This network is usually a deep convolutional neural network (CNN), which can extract useful features from the image for subsequent classification tasks.

[0068] More specifically, the first few layers of the image classification network (usually convolutional layers) extract low-level features in the image, such as edges, textures, etc. As the network goes deeper, these low-level features are further combined and abstracted to form higher-level feature representations. In the later layers of the network (usually fully connected layers or classification layers), these high-level features are used to make classification decisions. Based on the deep generation of prior information, the network can more accurately identify the key content in the image, thereby helping to determine whether the reconstructed image already has the key information required for practical applications.

[0069] The final image classification network outputs a probability value or category label (keyframe or non-keyframe). Based on the output of the classification network, a keyframe recognition result indicating whether the reconstructed image is a keyframe or a non-keyframe is generated. The result can be displayed in text form (such as "keyframe" or "non-keyframe"), digital label (such as 1 for keyframe, 0 for non-keyframe) or visual form (such as annotation on the image).

[0070] In addition, deep generative prior information plays a crucial role in this process. It provides a deep understanding of the structure and content of the image, allowing the image classification network to more accurately identify key frames. This prior information is obtained through learning from a large amount of training data. Therefore, in practical applications, it is necessary to ensure the quality and diversity of training data to improve the generalization ability and recognition performance of the network.

[0071] Step S240: generating a key frame extraction result for the video stream to be processed based on the key frame recognition result.

[0072] Through the above steps, the image classification network outputs a key frame recognition result, which may be a label sequence, each label corresponding to a video frame, indicating whether the frame is considered a key frame. Based on the key frame recognition result, a key frame extraction result is finally generated. This usually involves selecting those frames marked as key frames from the video stream to form a key frame set.

[0073] In the above-mentioned key frame extraction method, the key information of the video stream to be processed (i.e., the depth generated prior information) is extracted through the image reconstruction technology based on the depth prior, a reconstructed image is generated, and key frame recognition is performed based on the reconstructed image, so that the accuracy of key frame judgment is significantly improved, so that important information and unimportant information in the video can be accurately distinguished, and the problem of low accuracy of key frame extraction caused by unclear key frame judgment targets in related technologies is effectively solved, thereby realizing an accurate and efficient key frame extraction method.

[0074] In some of these examples, see Figure 3 , training the image reconstruction network can also include the following steps:

[0075] Step S310, obtaining a training image.

[0076] A set of image datasets for training the image reconstruction network is collected. The collection method can be to obtain high-quality, clearly labeled images from various sources (such as public databases, custom datasets, etc.). These images should cover various scenes and objects that the reconstruction network can handle.

[0077] Step S320: input the training image into the initial reconstruction network for training to obtain a predicted image.

[0078] In this step, the training image is processed using the initial reconstruction network (which may be an untrained or pre-trained model) to generate a predicted image. Specifically, the training image is input into the network, and the predicted image is calculated through the network's forward propagation process. These predicted images are the initial reconstruction results of the reconstruction network for the input image.

[0079] Step S330, calculating the reconstruction loss function according to the training image and the predicted image; transferring the gradient of the reconstruction loss function back to the initial reconstruction network for iteration to generate an image reconstruction network.

[0080] The difference between the predicted image and the training image is evaluated by calculating the reconstruction loss function, and these differences are used to optimize the reconstruction network. Among them, an appropriate loss function can be selected to measure the difference between the predicted image and the training image, and the gradient of the loss function with respect to the network parameters is calculated, and the gradient is passed back to each layer of the network. The reconstruction loss function can generally use the mean square error (MSE) or mean absolute error (L1) loss function.

[0081] Next, we use optimization algorithms, such as Adaptive Moment Estimation (Adam) based on first-order moment estimation and second-order moment estimation, Stochastic Gradient Descent (SGD), etc., to update the network parameters according to the gradient to reduce the reconstruction loss. Repeat the above process until the preset number of training rounds is reached or the reconstruction loss converges.

[0082] Taking the nasopharyngeal endoscopic surgery scenario as an example, please refer to Figure 4 , a training and reasoning process using the Unet network as the reconstruction network structure is provided. In which, the training image in the nasopharyngeal endoscopic surgery scene is input into the untrained Unet network for reconstruction, and the predicted image is output; according to the reconstruction loss function between the training image and the predicted image, the difference between the two is determined to iteratively optimize the Unet network, and finally generate an image reconstruction network applied in the nasopharyngeal endoscopic surgery scene.

[0083] Through the above steps S310 to S330, the training image is reconstructed and predicted by the initial reconstruction network to obtain a predicted image, and the initial reconstruction network is iteratively optimized based on the reconstruction loss function calculated based on the predicted image to obtain a fully trained image reconstruction network, which can effectively improve the accuracy of image reconstruction, thereby facilitating further improving the accuracy of key frame extraction.

[0084] In some embodiments, the above-mentioned back-transferring the gradient of the reconstruction loss function to the initial reconstruction network for iteration to generate the image reconstruction network may also include the following steps:

[0085] The first image feature of the predicted image and the second image feature of the training image are extracted respectively, and the texture similarity between the first image feature and the second image feature is calculated; the number of iteration steps is determined according to the texture similarity; based on the number of iteration steps, the gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate an image reconstruction network.

[0086] It should be noted that if the training data used in the above steps is only one frame of input image, overfitting will soon occur. Therefore, it is very important to choose a model with appropriate depth and specify a fixed number of iteration steps, which directly affects the algorithm effect.

[0087] In this embodiment, a large amount of training image data in similar scenes, such as other video images of the same operation, is used to find a suitable network model and number of iterations through an experimental method. The process of determining the number of iterations and training the network model based on the number of iterations is described below.

[0088] First, extract the image features of the prediction image and the training image respectively. The feature extraction method for the prediction image can be: use a pre-trained feature extraction model or a custom convolutional neural network layer to extract the first image features from the prediction image. These features usually reflect the texture, edge, shape and other information of the image. Similarly, for the training image, the above feature extraction model or network layer can also be used to extract the second image features from the training image.

[0089] Next, select an appropriate similarity measurement method, such as cosine similarity, Euclidean distance, structural similarity, etc., to calculate the texture similarity between the first image feature of the predicted image and the second image feature of the training image. Then determine the number of iterations based on the texture similarity. Specifically, a similarity threshold for texture similarity can be pre-set according to actual needs. The threshold determines when to stop training; when the texture similarity between the predicted image and the training image reaches or exceeds the similarity threshold, it can be considered that the reconstructed network is accurate enough. That is, during the training process, the texture similarity is continuously calculated, and whether to continue training is determined based on the comparison result with the threshold. If the texture similarity has reached or exceeded the threshold, stop training; otherwise, continue training until the predetermined maximum number of iterations is reached or the texture similarity meets the requirements.

[0090] Finally, according to the determined number of iterations (or until the maximum number of iterations is reached and the texture similarity still does not meet the requirements), the original gradient back propagation and parameter update steps are continued. After multiple iterations of training, an image reconstruction network is generated that can accurately reconstruct the input image with similar texture.

[0091] For example, the first image feature and the second image feature extracted above can be the Canny edge features of the predicted image and the Canny edge features of the training image respectively extracted using the Canny detection algorithm (a multi-level edge detection algorithm). That is, when determining the number of iterations of the image reconstruction model, the Canny edge features are used to evaluate the texture similarity between the input image and the reconstructed image. When the edge features of the reconstructed image are similar enough to the edge features of the input image, it can be considered that the reconstruction quality has reached an acceptable level, and the iteration process of the network model can be stopped at this time.

[0092] Through the above embodiment, by comparing the texture similarity between the training image and the texture image to specify a suitable number of iteration steps, overfitting of the network model is effectively avoided, which can help improve the performance and accuracy of the image reconstruction network, and further improve the accuracy of key frame extraction.

[0093] In some embodiments, after the training image is input into the initial reconstruction network for training to obtain the predicted image, the key frame extraction method further includes the following steps:

[0094] Obtain the image label data corresponding to the predicted image; input the predicted image into the initial classification network for training to obtain the classification prediction result; calculate the classification loss function based on the classification prediction result and the image label data; transfer the gradient of the classification loss function back to the initial classification network for iteration to generate an image classification network.

[0095] Among them, after the above training image is input into the reconstruction network and the predicted image is output, the predicted image is labeled according to the classification of key frames and non-key frames. Then, the predicted image carrying the image label data indicating whether the corresponding image is a key frame is input into the initial classification network for training, and forward propagated through the classification network to obtain the classification prediction result. According to the classification prediction result and the image label data, the classification loss function is calculated, and the gradient of the classification loss function is back-propagated to the initial classification network for iteration: the optimization algorithm is used to update the parameters of the classification network according to the gradient of the classification loss function, and multiple iterative training is performed until the classification performance reaches the predetermined requirements or the maximum number of iterations is reached.

[0096] It should also be understood that the training process of the above classification network can be performed after the image reconstruction network training, or the two can be processed in parallel to improve efficiency.

[0097] Through the above embodiment, the classification network is iteratively optimized through the classification prediction result output by the initial classification network and the loss function result between the image label data corresponding to the predicted image, so as to obtain an optimized image classification network, thereby effectively improving the accuracy of image classification and further improving the accuracy of key frame extraction.

[0098] In some embodiments, the above-mentioned step of generating a key frame extraction result for a video stream to be processed based on the key frame recognition result further includes the following steps:

[0099] The key frame recognition results corresponding to the reconstructed image are traversed; if the key frame recognition result indicates that the traversed reconstructed image has key information, the corresponding video frame image is determined to be a key frame; otherwise, the video frame image is determined to be a non-key frame image; the key frame recognition result corresponding to the next traversed reconstructed image is detected until the key frame recognition results corresponding to all reconstructed images are traversed, and a key frame extraction result is generated.

[0100] Specifically, the input video stream is processed frame by frame, and each frame of the image is input into a trained image reconstruction network for reconstruction to obtain a reconstructed image sequence.

[0101] Apply the image classification network to each reconstructed frame to obtain the key frame recognition result. Traverse the key frame recognition results corresponding to each reconstructed image. For each recognition result, if it indicates that the current image contains key information (that is, the model predicts it to be a key frame), the video frame is determined to be a key frame. If the recognition result indicates that the current image does not contain key information (that is, the model predicts it to be a non-key frame), the video frame is determined to be a non-key frame image. Repeat this process until the key frame recognition results corresponding to all reconstructed images (or original video frames) are traversed. Finally, all images determined to be key frames are extracted from the video stream to generate a key frame extraction result. This result can be an image sequence containing all key frames, or an index list marking which frames are key frames.

[0102] Through the above steps, key frames can be effectively extracted from the video stream to provide useful information for subsequent video analysis, summary generation or other tasks.

[0103] In some embodiments, the above-mentioned obtaining of each video frame image in the video stream to be processed further includes the following steps:

[0104] Acquire each initial frame image in the video stream to be processed; perform bright channel processing on each initial frame image to determine the maximum pixel value in a preset color channel; generate a grayscale image based on the maximum pixel value; perform maximum value filtering on the grayscale image to obtain the corresponding video frame image.

[0105] It should be noted that in some application scenarios, factors such as lighting and environment may affect the accuracy of identifying key information in the image. Especially in the scenario of surgical endoscopy, which is usually performed in an internal environment with limited light, and the light source mainly comes from external devices (such as surgical lights), the bright areas in the image often contain more important surgical details. Therefore, it is necessary to extract the brightest areas or features from these images to highlight the details of the surgical area and improve the clarity and visibility of the image.

[0106] Among them, the above-mentioned color channels are used to describe the color composition of the image. The color channels are different in different color spaces. For example, if the image to be tested is an image in RGB format, the corresponding preset color channels include R (red) channel, G (green) channel and B (blue) channel; if the image to be tested is a LAB image, the corresponding color channels include L (brightness) channel, A (color change from green to red) channel and B (color change from blue to yellow) channel. Specifically, taking the RGB channel as an example, one of the calculation methods of the bright channel is shown in the following formula:

[0107]

[0108] In the above formula, max() is used to represent the maximum value function. J is used to represent the pixel value of each pixel in the image to be tested, x and y represent the pixel coordinates, and c represents the channel.

[0109] Through the above embodiment, the maximum pixel value of the image to be tested in the color channel is determined to form a grayscale image, and then a maximum value filtering is performed to achieve bright channel calculation; since the pixel value of the highlight area in the image in the color channel is often large, the maximum value filtering can effectively retain the highlight area in the image. Therefore, by converting the maximum pixel value into a grayscale image and then performing the maximum value filtering, the difference between different brightness areas in the image can be increased, and the highlight parts in the image can be further emphasized, so that these areas are more prominent in the final image.

[0110] In some embodiments, the above-mentioned maximum value filtering process is performed on the grayscale image to obtain the corresponding video frame image, and further includes the following steps:

[0111] Perform maximum filtering on the grayscale image to obtain a filtered image; perform normalization based on the filtered image to obtain a weight matrix; calculate the cumulative distribution function of the initial frame image; perform weighted equalization on the initial frame image based on the weight matrix and the cumulative distribution function to obtain a video frame image.

[0112] After obtaining the bright channel image of the bright channel processing, the weight matrix can be obtained by normalization processing. The calculation formula of the normalization processing is as follows:

[0113] Normalized value = (pixel value - minimum value) / (maximum value - minimum value)

[0114] In this way, each pixel in the calculated image has a value between 0 and 1. The purpose of normalization is to scale the pixel values ​​to a specific range (usually 0 to 1) so that they can be used as weights. Each element in the weight matrix represents the relative brightness of the corresponding pixel in the original bright channel image.

[0115] After that, weighted image histogram equalization is performed based on the weight matrix W. Specifically, for an input image, the grayscale histogram of the input original image to be tested is first counted. Then the cumulative distribution function (CDF) of the original image is calculated, that is, the cumulative sum of the number of pixels from gray level 0 to the current gray level. Assuming that the current pixel value is x', the formula for calculating the new pixel value new_x' is as follows:

[0116] new_x'=round(255×CDF(x')×W(x'))

[0117] In the above formula, round() is used to round the number to a given number of digits. Using this formula, the original values ​​of all pixels are replaced with new pixel values ​​to form a weighted equalized image.

[0118] Through the above-mentioned embodiment, in the preprocessing stage of the image to be tested, after the bright channel calculation, the image normalization weight matrix is ​​calculated and the weighted image histogram equalization processing is performed; by redistributing the pixel values ​​of the image, the brightness distribution of the image is made more uniform, thereby enhancing the details and edges of the image, and in subsequent processing and analysis, important features in the image can be better identified, which helps to improve the accuracy and stability of the algorithm.

[0119] The present application is described below in conjunction with specific embodiments. Taking the application in surgical scene recording as an example, Figure 5 is a flowchart of another key frame extraction method according to an embodiment of the present application. Figure 5 As shown, the process includes the following steps:

[0120] Step S501, obtaining the surgical video (i.e., the video stream to be processed). First, for the input surgical video, in order to improve the recognition effect of the subsequent algorithm, combined with the particularity of the in vivo scene, the following image preprocessing operations are performed on each frame of the surgical video:

[0121] Calculate the bright channel; after obtaining the bright channel image, calculate the image normalization weight matrix according to the normalization method, and use the weight matrix to perform weighted image histogram equalization processing on the input frame image.

[0122] Step S502, deep generation a priori to obtain key frames. After obtaining the video frame image preprocessed in step S501, the key detail features have been highlighted. At this time, the deep generation a priori technology can be used to determine whether the current frame is a key frame.

[0123] Specifically, for an input image, a reconstructed image is obtained after being processed by the image reconstruction network. Then, the image classification network generated by training is used to determine whether the reconstructed image has key information. If so, the corresponding input image is judged as a key frame, otherwise it is judged as a non-key frame.

[0124] Step S503, output key frames. Finally, the key frames of each frame in the surgical video are extracted and output.

[0125] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0126] This embodiment also provides a key frame extraction device, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0127] Figure 6 is a structural block diagram of a key frame extraction device according to an embodiment of the present application, such as Figure 6 As shown, the device includes an acquisition module 61, a reconstruction module 62, a recognition module 63 and an extraction module 64; wherein:

[0128] The acquisition module 61 is used to acquire the video stream to be processed and acquire each video frame image in the video stream to be processed.

[0129] The reconstruction module 62 is used to input each video frame image into the trained image reconstruction network to generate a reconstructed image; the reconstructed image carries the depth generation prior information for the video frame image.

[0130] The recognition module 63 is used to input the reconstructed image into the trained image classification network, and perform recognition processing based on the depth-generated prior information to generate a key frame recognition result;

[0131] The extraction module 64 is used to generate a key frame extraction result for the video stream to be processed based on the key frame recognition result.

[0132] In some embodiments, the key frame extraction device further includes: a training module for acquiring training images.

[0133] The training module is also used to input the training image into the initial reconstruction network for training to obtain a predicted image.

[0134] The training module is also used to calculate the reconstruction loss function based on the training image and the predicted image; the gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate an image reconstruction network.

[0135] In some of the embodiments, the training module is further used to extract first image features of the prediction image and second image features of the training image, respectively, and calculate texture similarity between the first image features and the second image features.

[0136] The training module is also used to determine the number of iteration steps based on texture similarity.

[0137] The training module is also used to transfer the gradient of the reconstruction loss function back to the initial reconstruction network for iteration based on the number of iteration steps to generate an image reconstruction network.

[0138] In some of the embodiments, the training module is further used to obtain image label data corresponding to the predicted image.

[0139] The training module is also used to input the predicted image into the initial classification network for training to obtain the classification prediction result.

[0140] The training module is also used to calculate the classification loss function based on the classification prediction results and image label data; the gradient of the classification loss function is back-transferred to the initial classification network for iteration to generate an image classification network.

[0141] In some embodiments, the extraction module 64 is further configured to traverse key frame recognition results corresponding to the reconstructed image.

[0142] The extraction module 64 is further configured to determine that the corresponding video frame image is a key frame if the key frame recognition result indicates that the traversed reconstructed image has key information; otherwise, determine that the video frame image is a non-key frame image.

[0143] The extraction module 64 is further used to detect the key frame recognition result corresponding to the next reconstructed image frame traversed, until the key frame recognition results corresponding to all reconstructed images are traversed, and a key frame extraction result is generated.

[0144] In some embodiments, the acquisition module 61 is further used to acquire each initial frame image in the video stream to be processed.

[0145] The acquisition module 61 is further used to perform bright channel processing on each initial frame image, determine the maximum pixel value in a preset color channel, and generate a grayscale image based on the maximum pixel value.

[0146] The acquisition module 61 is also used to perform maximum filtering on the grayscale image to obtain a corresponding video frame image.

[0147] In some embodiments, the acquisition module 61 is further used to perform maximum filtering on the grayscale image to obtain a filtered image; and perform normalization processing based on the filtered image to obtain a weight matrix.

[0148] The acquisition module 61 is also used to calculate the cumulative distribution function of the initial frame image; according to the weight matrix and the cumulative distribution function, the initial frame image is subjected to weighted equalization processing to obtain a video frame image.

[0149] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0150] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0151] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0152] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0153] S1, obtaining a video stream to be processed, and obtaining each video frame image in the video stream to be processed.

[0154] S2, input each video frame image into the trained image reconstruction network to generate a reconstructed image; the reconstructed image carries the depth generation prior information of the video frame image.

[0155] S3, input the reconstructed image into the trained image classification network, and perform recognition processing based on the deep generation prior information to generate key frame recognition results.

[0156] S4, based on the key frame recognition result, generating a key frame extraction result for the video stream to be processed.

[0157] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0158] In addition, in combination with the key frame extraction method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any key frame extraction method in the above embodiment is implemented.

[0159] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0161] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A key frame extraction method, characterized in that: The method comprises: Obtaining a video stream to be processed, and obtaining each video frame image in the video stream to be processed; Inputting each of the video frame images into a trained image reconstruction network to generate a reconstructed image; the reconstructed image carries a priori information on the depth generation of the video frame image; Inputting the reconstructed image into a trained image classification network, and performing recognition processing based on the depth-generated prior information to generate a key frame recognition result; Based on the key frame recognition result, a key frame extraction result for the video stream to be processed is generated.

2. The key frame extraction method according to claim 1, characterized in that: The method further comprises: Get training images; Inputting the training image into an initial reconstruction network for training to obtain a predicted image; A reconstruction loss function is calculated according to the training image and the predicted image; and a gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate the image reconstruction network.

3. The key frame extraction method according to claim 2, characterized in that: The step of back-transferring the gradient of the reconstruction loss function to the initial reconstruction network for iteration to generate the image reconstruction network comprises: Extracting first image features of the prediction image and second image features of the training image respectively, and calculating texture similarity between the first image features and the second image features; Determining the number of iteration steps according to the texture similarity; Based on the number of iteration steps, the gradient of the reconstruction loss function is back-transferred to the initial reconstruction network for iteration to generate the image reconstruction network.

4. The key frame extraction method according to claim 2, characterized in that: After inputting the training image into the initial reconstruction network for training to obtain the predicted image, the method further includes: Obtaining image label data corresponding to the predicted image; Inputting the predicted image into an initial classification network for training to obtain a classification prediction result; A classification loss function is calculated according to the classification prediction result and the image label data; and the gradient of the classification loss function is back-transferred to the initial classification network for iteration to generate the image classification network.

5. The key frame extraction method according to claim 1, characterized in that: The step of generating a key frame extraction result for the video stream to be processed based on the key frame recognition result includes: Traversing the key frame recognition results corresponding to the reconstructed image; If the key frame recognition result indicates that the traversed reconstructed image has key information, then the corresponding video frame image is determined to be a key frame; otherwise, then the video frame image is determined to be a non-key frame image; The key frame recognition result corresponding to the next reconstructed image is detected until all the key frame recognition results corresponding to the reconstructed images are traversed, and the key frame extraction result is generated.

6. The key frame extraction method according to any one of claims 1 to 5, characterized in that: The step of obtaining each video frame image in the video stream to be processed includes: Acquire each initial frame image in the video stream to be processed; Performing bright channel processing on each of the initial frame images to determine a maximum pixel value in a preset color channel; and generating a grayscale image based on the maximum pixel value; Perform maximum value filtering on the grayscale image to obtain the corresponding video frame image.

7. The key frame extraction method according to claim 6, characterized in that: The performing maximum value filtering on the grayscale image to obtain the corresponding video frame image includes: Performing maximum value filtering on the grayscale image to obtain a filtered image; performing normalization processing based on the filtered image to obtain a weight matrix; The cumulative distribution function of the initial frame image is calculated; and according to the weight matrix and the cumulative distribution function, the initial frame image is subjected to weighted equalization processing to obtain the video frame image.

8. A key frame extraction device, characterized in that: The device comprises: An acquisition module, used to acquire a video stream to be processed, and acquire each video frame image in the video stream to be processed; A reconstruction module, used for inputting each of the video frame images into a trained image reconstruction network to generate a reconstructed image; the reconstructed image carries a priori information on the depth generation of the video frame image; A recognition module, used for inputting the reconstructed image into a trained image classification network, and performing recognition processing based on the depth-generated prior information to generate a key frame recognition result; The extraction module is used to generate a key frame extraction result for the video stream to be processed based on the key frame recognition result.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the key frame extraction method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the key frame extraction method according to any one of claims 1 to 7 when running.

Images