Video key frame extraction method and device, computer device and storage medium

Abstract

This invention relates to the field of artificial intelligence technology and discloses a method, apparatus, device, and medium for extracting keyframes from videos. The method includes: extracting features from video frames in a video to be processed using a visual extraction module to obtain visual feature vectors; extracting features from video frames using a temporal extraction module to obtain optical flow motion vectors; measuring the distance between the visual feature vectors and optical flow motion vectors using dynamic weighting coefficients to obtain a hybrid distance metric; clustering the hybrid distance metric and global features using a clustering module to obtain a set of video frames corresponding to each cluster center; and extracting keyframes from all hybrid distance metrics and all video frame sets to obtain a keyframe sequence. This invention can be applied to financial and medical scenarios. By extracting feature vectors and calculating hybrid distance metrics through two different module branches, it achieves clustering of video frame sets and extraction of keyframe sequences, thus improving the accuracy of extraction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and image detection technology, and in particular to a method, apparatus, computer equipment and storage medium for extracting key frames from videos. Background Technology

[0002] Video keyframe extraction aims to select representative frames from videos for efficient content representation. In the financial sector, keyframes are extracted for transaction behavior analysis to capture abnormal operations; in customer service video quality inspection, keyframes of communication between customer service representatives and clients are extracted to analyze service attitude, compliance of scripts, and adequacy of risk warnings. In the medical field, keyframes in image diagnosis are extracted to filter artifacts such as bubbles and blur, helping doctors quickly locate lesions and reduce missed diagnoses; and key operation frames are extracted from surgical procedure recordings to generate surgical step atlases. While deep learning has improved visual representation and temporal modeling capabilities, it still suffers from the following problems: using fixed weights for distance measurement makes it impossible to dynamically balance visual differences and temporal smoothness based on video content; traditional clustering algorithms ignore long-range temporal dependencies, making it difficult to accurately identify semantically related frames across shots. Summary of the Invention

[0003] This invention provides a method, apparatus, computer device, and storage medium for extracting keyframes from videos, in order to solve the technical problems of using fixed weights to calculate distance metrics and traditional clustering algorithms ignoring long-range temporal dependencies in the prior art.

[0004] Firstly, a method for extracting keyframes from video is provided, including: The visual extraction module extracts features from the video frames in the video to be processed, and obtains visual feature vectors corresponding to each video frame. The temporal extraction module extracts features from the video frames in the video to be processed, and obtains the optical flow motion vector corresponding to each video frame. By using dynamic weighting coefficients to measure the distance between the visual feature vector and the optical flow motion vector, a hybrid distance metric corresponding to each video frame is obtained. The hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center. Keyframe extraction is performed on all the hybrid distance metrics and all the video frame sets to obtain the keyframe sequence corresponding to the video to be processed.

[0005] Secondly, a video keyframe extraction device is provided, comprising: The visual feature module is used to extract features from video frames in the video to be processed through the visual extraction module, and obtain visual feature vectors corresponding to each video frame. The optical flow feature module is used to extract features from video frames in the video to be processed through the temporal extraction module, and obtain optical flow motion vectors corresponding to each video frame. The distance measurement module is used to measure the distance between the visual feature vector and the optical flow motion vector through dynamic weighting coefficients, so as to obtain a hybrid distance measurement corresponding to each video frame. The video frame clustering module is used to perform clustering processing on the hybrid distance metric and global features through the clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center; The keyframe extraction module is used to extract keyframes from all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described video keyframe extraction method.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described video keyframe extraction method.

[0008] The aforementioned video keyframe extraction method, apparatus, computer equipment, and storage medium utilize two distinct module branches—a visual extraction module and a temporal extraction module—to extract frame-level visual feature vectors and optical flow motion vectors between adjacent frames in the video to be processed. By calculating the hybrid distance metric using dynamic weight coefficients, automatic weight allocation is achieved, avoiding the use of fixed weights that cannot dynamically balance visual differences and temporal smoothness based on video content. Clustering processing using a clustering module and global features introduces temporal constraints and global features, improving the clustering accuracy of the video frame set. By extracting keyframes from all hybrid distance metrics and all video frame sets, the keyframe sequence is obtained, thereby improving the accuracy of the keyframe sequence. This ensures that the extracted keyframes maintain content coverage while their temporal distribution better conforms to the video's semantic structure. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram of an application environment for a video keyframe extraction method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a video keyframe extraction method according to an embodiment of the present invention; Figure 3 This is a schematic flowchart of a specific implementation of step S101 of the video keyframe extraction method in one embodiment of the present invention; Figure 4 This is a schematic flowchart of a specific implementation of step S102 of the video keyframe extraction method in one embodiment of the present invention; Figure 5 This is a schematic flowchart of a specific implementation of step S103 of the video keyframe extraction method in one embodiment of the present invention; Figure 6 This is a schematic flowchart of a specific implementation of step S104 of the video keyframe extraction method in one embodiment of the present invention; Figure 7 This is a schematic flowchart of a specific implementation of step S105 of the video keyframe extraction method in one embodiment of the present invention; Figure 8 This is a schematic diagram of a video keyframe extraction device according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] The video keyframe extraction method provided in this invention can be applied to, for example... Figure 1In this application environment, the client communicates with the server via a network. The server can extract features from video frames in the video to be processed using a visual extraction module to obtain visual feature vectors corresponding to each video frame; it can also extract features from video frames in the video to be processed using a temporal extraction module to obtain optical flow motion vectors corresponding to each video frame; it can then perform distance measurement on the visual feature vectors and optical flow motion vectors using dynamic weighting coefficients to obtain a hybrid distance measurement corresponding to each video frame; a clustering module can then perform clustering processing on the hybrid distance measurement and global features to obtain at least one cluster center and a set of video frames corresponding to each cluster center; finally, it can extract keyframes from all the hybrid distance measurements and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed. In this invention, the extraction of frame-level visual feature vectors and optical flow motion vectors between adjacent frames in the video to be processed is achieved through two different module branches: the visual extraction module and the temporal extraction module. The calculation of the hybrid distance measurement using dynamic weighting coefficients enables automatic weight allocation, avoiding the use of fixed weights that cannot dynamically balance visual differences and temporal smoothness according to the video content. Clustering is performed using a clustering module and global features, incorporating temporal constraints and global features to improve the clustering accuracy of video frame sets. Keyframe extraction is performed on all mixed distance metrics and all video frame sets to obtain keyframe sequences, thereby improving the accuracy of the keyframe sequences. This ensures that the extracted keyframes maintain content coverage while their temporal distribution better conforms to the video's semantic structure. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0013] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the video keyframe extraction method provided in this embodiment of the invention includes the following steps: S101. The visual extraction module extracts features from the video frames in the video to be processed, and obtains the visual feature vectors corresponding to each video frame.

[0014] In essence, a visual extraction module refers to a pre-defined network structure used to extract visual features, such as ResNet50, ResNet101, or VGG16. A visual feature vector is a one-dimensional numerical array that represents the visual content and semantic information of each video frame (image).

[0015] Specifically, the process involves acquiring the video to be processed and segmenting it into frames to obtain all video frames. Then, a visual extraction module is used, inputting all video frames to extract visual features from each frame. This involves identifying key information within the video frame and extracting abstract information representing its visual content, thus obtaining visual features. Next, the extracted visual features are encoded to obtain a visual feature vector corresponding to each video frame. Here, visual features refer to the abstract information extracted from the video frame that represents its visual content.

[0016] S102. The temporal extraction module extracts features from the video frames in the video to be processed to obtain the optical flow motion vectors corresponding to each video frame.

[0017] Understandably, a temporal extraction module refers to a pre-defined network structure used to extract temporal features from consecutive frames, such as FlowNet2 or RAFT (Recurrent All-Pairs Field Transforms). An optical flow motion vector is a vector calculated for each pixel in a video frame, showing the direction and magnitude of its displacement between two consecutive frames.

[0018] Specifically, the video to be processed is acquired and segmented into frames to obtain all video frames in the video. This can be done by segmenting the video into frames and then inputting them into the visual extraction module and the temporal extraction module, respectively. Then, the temporal extraction module is used, and all video frames are input into it. The temporal extraction module extracts the displacement direction and magnitude of each pixel in consecutive video frames, thus obtaining the optical flow field corresponding to each video frame. This optical flow field is then converted into an inter-frame motion vector through integration, and this inter-frame motion vector is determined as the optical flow motion vector corresponding to each video frame.

[0019] S103. The distance between the visual feature vector and the optical flow motion vector is measured by dynamic weighting coefficients to obtain a hybrid distance metric corresponding to each video frame.

[0020] Understandably, dynamic weighting coefficients refer to those dynamically generated based on the similarity or difference between two consecutive video frames. Hybrid distance metrics refer to the combined distance value obtained by simultaneously using multiple different types of distance calculation methods and fusing them according to weights.

[0021] Specifically, adaptive learning is performed on each video frame to determine the dynamic weight coefficients corresponding to each video frame. Then, the distance between the visual feature vector and the optical flow motion vector is measured using the dynamic weight coefficients. That is, by calculating the product of the dynamic weight coefficients and the visual feature vectors, and subtracting the product of the dynamic weight coefficients and the optical flow motion vectors, and then adding the two products, the hybrid distance metric corresponding to each video frame can be obtained.

[0022] In one embodiment, video frames The mixed distance metric is: ;in, The input is the concatenated vector of features from the i-th and j-th frames, representing the dynamic weight coefficients predicted by the Siamese network. . Visual cosine distance Since the features have been normalized, it is simplified to . : The L2 norm of the optical flow vector, normalized to the interval [0, 1] by maxima and minima.

[0023] S104. The hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center.

[0024] In essence, a clustering module refers to an algorithmic unit that introduces global features and clusters video frames based on a hybrid distance metric. A cluster center is a representative point calculated by the clustering algorithm for each class (cluster), used to summarize the overall characteristics of all samples within the entire cluster. The video frame set refers to the collection of all video frames corresponding to each cluster center. A global feature is a condensed vector obtained after a holistic understanding of all video frames, representing the overall semantics, content, and structure of the video to be processed.

[0025] Specifically, global features and a clustering module are obtained. The clustering module performs clustering processing on the hybrid distance metric and global features. That is, the global features of uniformly sampled video frames are used as the initial cluster centers. Then, all video frames are traversed, and the similarity between each frame and the cluster center is calculated based on the hybrid distance. Each frame is assigned to the nearest cluster, and the cluster center is updated according to the global features of all frames in the cluster. The center position is iterated and optimized repeatedly until the cluster center no longer changes and the clustering result converges. Stable cluster centers and the set of video frames corresponding to each cluster center are output.

[0026] S105. Extract keyframes from all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed.

[0027] Understandably, a keyframe sequence refers to a sequence of frames selected from a continuous video frame set, which are a small number of representative frames with the most information content and arranged in their original chronological order.

[0028] Specifically, keyframe extraction is performed on all hybrid distance metrics and all video frame sets, i.e., temporal constraints are obtained, and the local density of each video frame is calculated based on the hybrid distance metrics and temporal constraints. Then, based on the local density and temporal constraints, at least one candidate keyframe is selected from all video frames, and all candidate keyframes are reordered using the attention weights corresponding to each video frame to obtain a keyframe sorting result. Video keyframes are then selected from the keyframe sorting result and sorted according to time sequence to obtain a keyframe sequence.

[0029] In one embodiment, this keyframe extraction technology can be deeply adapted to multiple scenarios such as financial risk control, operational supervision, and customer service. In bank branch and ATM monitoring scenarios, two different branch modules extract different visual feature sequences and optical flow motion vectors, calculate a hybrid distance metric, and combine it with global features for clustering before extracting video keyframes. This can accurately capture abnormal behaviors such as camera obstruction, password theft, and device tampering, providing security personnel with precise investigation evidence and reducing the risk of illegal activities. In financial bill and voucher video processing, for dynamically captured bill videos, two different branch modules extract different visual feature sequences and optical flow motion vectors, calculate a hybrid distance metric, and combine it with global features for clustering before extracting clear keyframes. This assists OCR in recognizing core information such as bill amount, signature, and date, comparing it with system-reserved data to quickly verify the authenticity of the bill, avoid the risk of bill forgery, and improve bill processing efficiency. This technology improves the accuracy of keyframe extraction by fusing visual and optical flow features.

[0030] In one embodiment, this technology can be widely applied in the medical field to scenarios such as clinical diagnosis and treatment, surgical quality control, rehabilitation assessment, and medical research. In clinical imaging diagnosis, for dynamic video images such as ultrasound, endoscopy, and laparoscopy, different visual feature sequences and optical flow motion vectors are extracted through two different branch modules, and a hybrid distance metric is calculated to combine global features for clustering. Then, keyframes with the clearest lesion morphology and most obvious features are extracted to assist doctors in observing changes in lesion size, location, and morphology, reducing missed diagnoses and misdiagnoses caused by video frame redundancy. For example, it can be used for the accurate diagnosis of early-stage tumors and gastrointestinal diseases. In surgical monitoring and quality control, for videos captured during surgery, different visual feature sequences and optical flow motion vectors are extracted through two different branch modules, and a hybrid distance metric is calculated to combine global features for clustering. Then, key surgical operation frames are extracted for surgical debriefing and training of young physicians.

[0031] The video keyframe extraction method in this embodiment of the invention extracts frame-level visual feature vectors and optical flow motion vectors between adjacent frames from the video to be processed through two different module branches: a visual extraction module and a temporal extraction module. By calculating the hybrid distance metric using dynamic weight coefficients, automatic weight allocation is achieved, avoiding the use of fixed weights that cannot dynamically balance visual differences and temporal smoothness based on video content. Clustering is performed using a clustering module and global features, improving the clustering accuracy of the video frame set and achieving clustering of the video frame set. By extracting keyframes from all hybrid distance metrics and all video frame sets, the keyframe sequence is obtained, thereby improving the accuracy of the keyframe sequence. This ensures that the extracted keyframes maintain content coverage while their temporal distribution better conforms to the video's semantic structure.

[0032] In one embodiment, please refer to Figure 3 As shown, in step S101, the visual extraction module extracts features from the video frames in the video to be processed, obtaining visual feature vectors corresponding to each video frame, including: S201, the initial convolutional layer in the visual extraction module is used to perform convolution processing on all the video frames to obtain convolutional features corresponding to each video frame.

[0033] S202, the maximum pooling layer in the visual extraction module is used to pool all the convolutional features to obtain pooled features corresponding to each convolutional feature.

[0034] S203, using multiple sets of bottleneck residual blocks in the visual extraction module, residual processing is performed on all the pooling features to obtain residual vectors corresponding to each video frame.

[0035] S204, the global pooling layer in the visual extraction module is used to perform average pooling on all the residual vectors to obtain the global pooling vector corresponding to each video frame.

[0036] S205, through the fully connected layer in the visual extraction module, all the global pooling vectors are predicted to obtain the visual feature vectors corresponding to each video frame.

[0037] In essence, convolutional features refer to local feature information extracted after convolution with video frames using a convolution kernel. Pooling features refer to simplified features obtained by downsampling convolutional features. Multiple bottleneck residual blocks refer to bottleneck blocks with different numbers and convolution kernel parameters, connected by residuals. A global pooling vector refers to performing a global pooling operation on the entire feature map, compressing the two-dimensional feature map into a scalar, and finally concatenating all channels to obtain a one-dimensional vector.

[0038] Specifically, all video frames are acquired and preprocessed to standardize them before being input into the visual extraction module. Then, the initial convolutional layer in the visual extraction module performs convolution processing on each video frame, using a kernel with 64 channels, a kernel size of 7, and a stride of 2, to obtain convolutional features corresponding to each video frame. Finally, the max pooling layer in the visual extraction module pools all convolutional features, using a kernel with a kernel size of 3 and a stride of 2, to obtain pooled features corresponding to each convolutional feature.

[0039] Furthermore, through multiple bottleneck residual blocks in the visual extraction module, residual processing is performed on all pooling features. Specifically, this involves three second convolutional layers: three with a kernel of 1, 64 channels, and a stride of 1; three with a kernel of 3, 64 channels, and a stride of 1; and three with a kernel of 1, 256 channels, and a stride of 1. These layers sequentially connect the pooling and convolutional feature residuals and perform convolution processing to obtain the second convolutional features. Finally, four third convolutional layers are used: one with a kernel of 1 and 128 channels. With a stride of 2, a convolutional kernel of 3, 128 channels, and a stride of 1; and a convolutional kernel of 1, 512 channels, and a stride of 1, the second convolutional feature and the convolutional feature residual are concatenated; three convolutional layers are applied sequentially: one with a kernel of 1, 128 channels, and a stride of 1; another with a kernel of 3, 128 channels, and a stride of 1; and a third with a kernel of 1, 512 channels, and a stride of 1; the second convolutional feature is then processed sequentially to obtain the third convolutional feature; similarly, through six fourth convolutional layers, one with a kernel of 1, 128 channels, and a stride of 1, and the third with a stride of 1, the second convolutional feature is processed sequentially to obtain the third convolutional feature; similarly, through six fourth convolutional layers, one with a kernel of 1, 128 channels, and .... The first convolutional feature has 256 channels and a stride of 2. The second and third convolutional features are then concatenated using a kernel of 3, 256 channels, and a stride of 1. The third convolutional feature is then processed sequentially using a kernel of 1, 256 channels, and a stride of 1. Finally, the third convolutional feature is processed through three fifth convolutional layers. First, using convolutional kernel 1, 512 channels, and stride 2; then using convolutional kernel 3, 512 channels, and stride 1; and finally using convolutional kernel 1, 2048 channels, and stride 1, the residuals of the third and fourth convolutional features are concatenated. Then, using two convolutional kernels of 1 each with 512 channels and stride 1, and then using convolutional kernel 3 with 512 channels and stride 1, and finally using convolutional kernel 1 with 2048 channels and stride 1, the fourth convolutional feature is convolved sequentially to obtain the residual vectors corresponding to each video frame. Next, the global pooling layer in the visual extraction module performs average pooling on all residual vectors to obtain the global pooled vectors corresponding to each video frame. Finally, the fully connected layer in the visual extraction module performs prediction processing on all global pooled vectors to obtain the visual feature vectors corresponding to each video frame.

[0040] In this embodiment, depthwise convolution is achieved for each video frame through an initial convolutional layer, a max pooling layer, multiple sets of bottleneck residual blocks, a global pooling layer, and a fully connected layer. This enables the extraction of visual features from each video frame and improves the efficiency of visual feature vector extraction.

[0041] In one embodiment, please refer to Figure 4As shown, in step S102, the temporal extraction module extracts features from the video frames in the video to be processed to obtain the optical flow motion vector corresponding to each video frame, including: S301, optical flow processing is performed on all the video frames through the relevant network layer in the time extraction module to obtain the initial estimated optical flow corresponding to each video frame.

[0042] S302, through multiple simple network layers in the time extraction module, residual estimation is performed on all the initial estimated optical flows to obtain the residual corrected optical flows corresponding to each of the initial estimated optical flows.

[0043] S303, through the small displacement network layer in the time extraction module, displacement estimation is performed on all the video frames to obtain the small displacement optical flow corresponding to each video frame.

[0044] S304, through the optical flow fusion layer in the time extraction module, the residual correction optical flow and the small displacement optical flow corresponding to the same video frame are fused to obtain the target optical flow corresponding to each video frame.

[0045] S305, perform integration processing on all the target optical flows to obtain the optical flow motion vector corresponding to each of the video frames.

[0046] Understandably, the initial estimated optical flow refers to the coarse estimated optical flow output by the relevant network layer. Residual corrected optical flow refers to the fine optical flow field used to correct coarse optical flow errors. Small displacement optical flow refers to the optical flow between two adjacent frames where the pixel motion amplitude is very small. Target optical flow refers to the two-dimensional motion vector field of each pixel between two consecutive video frames.

[0047] Specifically, in step S301, optical flow processing is performed on all video frames through the correlation network layer in the temporal extraction module. Specifically, the two branches of the correlation network layer perform multiple convolutions on two consecutive video frames and downsample them sequentially to obtain high-dimensional feature maps corresponding to each video frame. Then, the similarity between each position in the high-dimensional feature map of the first video frame and each region in the high-dimensional feature map of the second video frame is calculated to obtain the correlation volume. Next, the high-dimensional feature map and correlation volume of the first video frame are input into the encoder. The high-dimensional feature map and correlation volume are concatenated, and the concatenated features are convolutionally processed and downsampled through multiple convolutional layers to obtain deep features. Then, the deep features are upsampled by the decoder, and the features in the downsampling process are skipped until they are upsampled to a preset scale to obtain intermediate optical flow. Finally, bilinear interpolation is performed on the intermediate optical flow to obtain the initial estimated optical flow.

[0048] In step S302, residual estimation is performed on all initially estimated optical flows through multiple simple network layers in the temporal extraction module. Specifically, bilinear interpolation is first applied to the initial estimated optical flows to restore the image to its original size, resulting in a restored image. Then, bilinear interpolation is applied to the restored image and a second video frame to align the second video frame to the restored image according to the coarse optical flow, resulting in a deformed video frame. Next, the brightness error map between the first and deformed video frames is calculated, and the first video frame, the deformed video frame, and the brightness error map are concatenated to obtain a multi-channel map. Then, the multi-channel map is convolved by an encoder to learn the residual optical flow and obtain a correction amount. Finally, based on the correction amount, the restored image is corrected and then processed by another simple network layer to obtain the corrected residual optical flow.

[0049] In step S303, displacement estimation is performed on all video frames using the small displacement network layer in the temporal extraction module. This involves first stitching together two consecutive video frames, then performing convolutional downsampling on the stitched frames using multiple convolutional layers to retain more small displacement details, thus obtaining deep downsampled features. Next, the decoder performs deconvolutional upsampling on the deep downsampled features, and then connects the upsampled and downsampled features through skip connections for further upsampling until a preset scale is reached. This multi-scale optical flow prediction yields the predicted features. An intermediate convolutional layer is added between deconvolutions to smooth the optical flow, reduce noise, and optimize edges. Finally, the predicted features are upsampled to restore the original image size, thus obtaining the small displacement optical flow.

[0050] In steps S304 and S305, the optical flow fusion layer in the temporal extraction module fuses the residual corrected optical flow and small displacement optical flow corresponding to the same video frame. Specifically, the brightness error map, the first video frame, the residual corrected optical flow, and the small displacement optical flow are input into the optical flow fusion layer. First, the brightness error map, the first video frame, the residual corrected optical flow, and the small displacement optical flow are concatenated. Then, a group of convolutional layers performs convolution processing on the concatenated feature map to learn the relationship between the two optical flows and determine the optical flow features dependent on each region. Deconvolution upsampling is then used to restore the original dimensions, thus obtaining multi-scale fusion features. Next, a two-channel convolutional layer performs weighted fusion of the multi-scale fusion features to obtain the target optical flow. Finally, all target optical flows are integrated, i.e., converted into inter-frame motion vectors through integration, thus obtaining the optical flow motion vectors corresponding to each video frame.

[0051] In this embodiment, the optical flow field between two consecutive video frames is determined by the relevant network layer, multiple simple network layers, small displacement network layer and optical flow fusion layer in the temporal extraction module, thereby realizing the acquisition of target optical flow and then realizing the calculation of optical flow motion vector between adjacent frames, which improves the accuracy of subsequent key frame extraction.

[0052] In one embodiment, please refer to Figure 5 As shown, in step S103, a distance metric is performed on the visual feature vector and the optical flow motion vector using dynamic weighting coefficients to obtain a hybrid distance metric corresponding to each video frame, including: S401, the video frames are weighted using the Siamese network in the measurement module to obtain dynamic weight coefficients.

[0053] S402, through the first branch in the measurement module, perform distance processing on all the visual feature vectors to obtain the visual cosine distance corresponding to each visual feature vector.

[0054] S403, through the second branch in the measurement module, perform distance processing on all the optical flow motion vectors to obtain the optical flow motion distance corresponding to each optical flow motion vector.

[0055] S404, determine the hybrid distance metric corresponding to each video frame based on the visual cosine distance, the optical flow motion distance, and the dynamic weighting coefficient corresponding to the same video frame.

[0056] Understandably, dynamic weighting coefficients are dynamically generated based on the similarity or difference between two consecutive video frames. Visual cosine distance refers to the cosine distance between the visual feature vectors of two video frames. Optical flow motion distance refers to the distance used to quantify the correlation between the optical flow motion vectors of two video frames.

[0057] Specifically, the Siamese network in the metric module performs weight processing on video frames. This involves inputting two consecutive video frames into the Siamese network, where two branches sharing weights extract features from each frame, resulting in high-dimensional feature maps. These high-dimensional feature maps are then cross-correlated to obtain feature vectors representing the relationship between the two frames. Finally, dynamic weight coefficients are calculated from these feature vectors. Next, the first branch in the metric module performs distance processing on all visual feature vectors, calculating the similarity between them and converting it to distance, thus obtaining the visual cosine distance for each feature vector. Similarly, the second branch in the metric module performs distance processing on all optical flow motion vectors, calculating the Euclidean distance between the optical flow motion vectors of two video frames, thereby obtaining the optical flow motion distance for each vector. Next, based on the visual cosine distance, optical flow distance, and dynamic weighting coefficient corresponding to the same video frame, the hybrid distance metric corresponding to each video frame is determined. That is, the product of the visual cosine distance and the dynamic weighting coefficient is calculated separately, and the product of the dynamic weighting coefficient and the optical flow distance is subtracted. The two product results corresponding to the same video frame are then added together to obtain the hybrid distance metric.

[0058] In one embodiment, the hybrid distance metric for video frames is .in, For hybrid distance metrics, The visual cosine distance, For dynamic weighting coefficients, This represents the optical flow distance. In another embodiment, It is the L2 norm of the optical flow vector (i.e., the Euclidean distance the pixel moves between two frames), obtained by minimax normalization to the [0, 1] interval.

[0059] In this embodiment, a Siamese network is used to allocate weight coefficients to different video frames. The first and second branches are used to calculate the distance between different video frames, thereby determining the visual cosine distance and optical flow motion distance. This leads to the calculation of the hybrid distance metric, improving the accuracy of subsequent keyframe extraction.

[0060] In one embodiment, please refer to Figure 6 As shown, in step S104, the hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center, including: S501, Construct a spatiotemporal joint kernel function, which is constructed based on a hybrid distance metric, and the kernel bandwidth of the spatiotemporal joint kernel function is dynamically adjusted according to local density and global features.

[0061] S502 uses uniformly sampled video frames as the initial seed points for drift. During the mean drift iteration process, the spatiotemporal joint kernel function causes various sub-points to converge to local maxima along the density gradient direction, and the local maxima are determined as cluster centers.

[0062] S503, match all the video frames with each of the cluster centers respectively, and determine all the video frames corresponding to the same cluster center as the set of video frames corresponding to each of the cluster centers.

[0063] Understandably, the spatiotemporal joint kernel function refers to a function that jointly measures the spatial similarity and temporal correlation of data. The initial seed point for drift refers to a set of representative points selected from the video frame set as the starting position for the mean-shift algorithm iteration. Local density refers to the degree of clustering of other frames within a certain range around a given video frame in the feature space.

[0064] Specifically, a spatiotemporal joint kernel function is constructed based on a hybrid distance metric, and its kernel bandwidth is dynamically adjusted according to local density and global features. The kernel bandwidth employs a dynamic adjustment strategy, using local density as a basis while also considering the frequency of current feature patterns globally. This results in bandwidth reduction in densely populated local regions for finer segmentation, and bandwidth expansion in regions with common global patterns to maintain semantic consistency.

[0065] Then, uniformly sampled video frames are used as initial seed points for drifting. For each seed point, mean drift iteration is performed: the weighted average position of all video frames in the neighborhood of the current point is calculated, where the weights are determined by the spatiotemporal joint kernel function based on the current bandwidth; the point is moved to the weighted average position, and this iteration is repeated until convergence, so that the seed point converges to a local maximum point under the guidance of the local density gradient. The converged local maximum point is determined as the cluster center. In one embodiment, global feature information can be introduced during the iteration process to correct the drift direction, so that the seed point converges to the local maximum point under the dual guidance of the local density gradient and the global structure. Next, all video frames are matched with each cluster center, and all video frames that successfully match the same cluster center are determined as the set of video frames corresponding to each cluster center.

[0066] In this embodiment, a spatiotemporal joint kernel function is constructed by using a hybrid distance metric, and the kernel bandwidth is adjusted based on local density and global features. Using uniformly sampled video frames as initial seed points for drift, cluster centers are determined, thereby determining the set of video frames and improving clustering efficiency.

[0067] In one embodiment, please refer to Figure 7As shown, in step S105, keyframe extraction is performed on all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed, including: S601, based on the hybrid distance metric corresponding to each of the video frames, perform local processing on the set of video frames to obtain the local density corresponding to each of the video frames.

[0068] S602, determine at least one candidate keyframe based on all the said local densities.

[0069] S603, determine the keyframe sequence corresponding to the video to be processed based on the temporal attention weights corresponding to each of the candidate keyframes and all the candidate keyframes; wherein, the temporal attention weights are used to characterize the importance of the candidate keyframes in the temporal distribution.

[0070] Understandably, local density refers to the degree of clustering of other frames within a certain range around a given video frame in the feature space. Candidate keyframes are intermediate sets initially selected from all video frames for further refinement. Temporal attention weights characterize the importance of candidate keyframes in the temporal distribution. Specifically, temporal attention weights can be the attention weights applied by the encoding model when extracting global features from all video frames.

[0071] Specifically, based on the hybrid distance metric corresponding to each video frame, the video frame set is locally processed. That is, density calculation is performed on each video frame in the video frame set using all hybrid distance metrics and a preset bandwidth (used to control the smoothness), thereby obtaining the local density corresponding to each video frame. Then, based on all local densities, at least one candidate keyframe is determined, that is, preset filtering conditions are obtained. The local density of each video frame is judged by the preset filtering conditions to determine whether the local density of the video frame meets the preset filtering conditions. If it meets the preset filtering conditions, the video frame in the video frame set whose local density meets the preset filtering conditions is determined as a candidate keyframe.

[0072] In one embodiment, the formula for calculating local density is: . Let n be the set of video frames, and n be the maximum number of video frames in the set. For hybrid distance metrics, This is the radius of the time window (in frames), used to limit the comparison range of local peaks. The preset filtering criteria are... . Let be the local density of the i-th frame (estimated by Gaussian kernel density, reflecting the degree of clustering of the frame in the feature space and temporal neighborhood). The number of frames in the video. This is a density threshold used to filter out frames with sufficiently high density. Condition 1 is... This is used to ensure that candidate frames have significant density. Condition 2 is... That is, the density of the i-th video frame is the highest within the window of the δ frames before and after it, ensuring that the extracted density peaks are local and avoiding repeated selection of consecutive similar frames.

[0073] Next, based on the temporal attention weights corresponding to each candidate keyframe and all candidate keyframes, the keyframe sequence corresponding to the video to be processed is determined, i.e., the temporal attention weights are obtained. Based on the temporal attention weights and local density corresponding to the same candidate keyframe, the score of each candidate keyframe is calculated, and all candidate keyframes are sorted according to their scores. From all sorted candidate keyframes, video keyframes are extracted according to a preset score threshold. Then, by sorting all video keyframes in chronological order, the keyframe sequence corresponding to the video to be processed is obtained.

[0074] In this embodiment, the local density of video frames is calculated by using a set of video frames and a hybrid distance, thereby enabling the initial screening of candidate keyframes, the extraction of video keyframes, and the determination of the keyframe sequence, thus improving the accuracy and efficiency of keyframe extraction.

[0075] In one embodiment, before step S104, before performing clustering processing on the hybrid distance metric and global features through a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center, the method further includes: The video frame sequence is preprocessed by the input layer in the coding model to obtain the input vector; the video frame sequence is obtained by sorting all video frames by time.

[0076] The input vector is processed by the self-attention layer in the encoding model to obtain the attention vector.

[0077] The attention vector is nonlinearly transformed by the feedforward network layer in the encoding model to obtain global features.

[0078] In essence, a video frame sequence refers to a sequence of all video frames ordered sequentially. The input vector is the vector obtained by standardizing all video frames in the sequence. The attention vector is the vector obtained by weighting and fusing all video frames according to their importance. The global feature is a condensed vector obtained after a holistic understanding of all video frames, representing the overall semantics, content, and structure of the video to be processed.

[0079] Specifically, the video frame sequence is preprocessed through the input layer of the encoding model. This involves resizing all video frames in the sequence to ensure uniformity, followed by normalization. All normalized results are then encoded and dimension-mapped to obtain encoded features. Positional features of each video frame are added to these encoded features to obtain the input vector. Next, the input vector is processed by the self-attention layer of the encoding model. This involves calculating the Q, K, and V vectors within the input vector using multiple attention mechanisms. Specifically, the correlation score between the Q and K vectors is calculated using the dot product method, with each input vector in Q being multiplied by each input vector in K, and the correlation score is normalized. Finally, the softmax function converts the scores between the input vectors into a probability distribution between [0, 1]. This probability distribution is then multiplied by the corresponding Values ​​vector, and the results are summed and normalized to obtain the attention vector. Next, the attention vector is nonlinearly transformed through the feedforward network layer in the encoding model. That is, the attention result is calculated through multiple hidden units with different weights, and the calculation result is subjected to residual summation and normalization to obtain the global feature. When the encoding model stacks multiple encoding layers, the global feature is output sequentially through all encoding layers.

[0080] In this embodiment, the video frame sequence is processed sequentially through the input layer, self-attention layer and feedforward network layer in the coding model, thereby determining the global features of the video frame sequence and adding long-range temporal dependencies in the keyframe extraction process.

[0081] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0082] In one embodiment, a video keyframe extraction device is provided, which corresponds one-to-one with the video keyframe extraction method described in the above embodiments. For example... Figure 8 As shown, the video keyframe extraction device includes a visual feature module 701, an optical flow feature module 702, a distance measurement module 703, a video frame clustering module 704, and a keyframe extraction module 705. Detailed descriptions of each functional module are as follows: The visual feature module 701 is used to extract features from video frames in the video to be processed by the visual extraction module, and obtain visual feature vectors corresponding to each video frame. The optical flow feature module 702 is used to extract features from video frames in the video to be processed through the temporal extraction module to obtain optical flow motion vectors corresponding to each video frame. The distance measurement module 703 is used to measure the distance between the visual feature vector and the optical flow motion vector through dynamic weighting coefficients to obtain a hybrid distance measurement corresponding to each video frame. The video frame clustering module 704 is used to perform clustering processing on the hybrid distance metric and global features through the clustering module to obtain at least one clustering center and a set of video frames corresponding to each clustering center; The keyframe extraction module 705 is used to extract keyframes from all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed.

[0083] In one embodiment, the visual feature module 701 includes: The convolutional feature unit is used to perform convolution processing on all the video frames through the initial convolutional layer in the visual extraction module to obtain convolutional features corresponding to each video frame. The pooling feature unit is used to perform pooling processing on all the convolutional features through the max pooling layer in the visual extraction module to obtain pooled features corresponding to each convolutional feature; The residual vector unit is used to perform residual processing on all the pooling features through multiple sets of bottleneck residual blocks in the visual extraction module to obtain the residual vector corresponding to each video frame. A global pooling vector unit is used to perform average pooling on all the residual vectors through the global pooling layer in the visual extraction module to obtain a global pooling vector corresponding to each video frame. The visual feature vector unit is used to perform fully connected processing on all the global pooling vectors through the fully connected layer in the visual extraction module to obtain the visual feature vectors corresponding to each video frame.

[0084] In one embodiment, the optical flow feature module 702 includes: An initial optical flow unit is used to perform optical flow processing on all the video frames through the relevant network layer in the temporal extraction module to obtain the initial estimated optical flow corresponding to each video frame; The residual correction optical flow unit is used to perform residual estimation on all the initial estimated optical flows through multiple simple network layers in the time extraction module to obtain the residual correction optical flow corresponding to each of the initial estimated optical flows; The small displacement optical flow unit is used to perform displacement estimation on all the video frames through the small displacement network layer in the temporal extraction module to obtain the small displacement optical flow corresponding to each video frame. The optical flow fusion unit is used to fuse the residual correction optical flow and the small displacement optical flow corresponding to the same video frame through the optical flow fusion layer in the temporal extraction module to obtain the target optical flow corresponding to each video frame. An integration processing unit is used to integrate all the target optical flows to obtain the optical flow motion vector corresponding to each video frame.

[0085] In one embodiment, the distance measurement module 703 includes: The dynamic weight coefficient unit is used to perform weight processing on the video frame through the Siamese network in the measurement module to obtain dynamic weight coefficients; The visual cosine distance unit is used to perform distance processing on all the visual feature vectors through the first branch in the metric module to obtain the visual cosine distance corresponding to each visual feature vector. The optical flow motion distance unit is used to perform distance processing on all the optical flow motion vectors through the second branch in the measurement module to obtain the optical flow motion distance corresponding to each optical flow motion vector; The hybrid distance metric unit is used to determine the hybrid distance metric corresponding to each video frame based on the visual cosine distance, the optical flow motion distance and the dynamic weighting coefficient corresponding to the same video frame.

[0086] In one embodiment, the video frame clustering module 704 includes: An adjustment unit is constructed to construct a spatiotemporal joint kernel function, which is constructed based on a hybrid distance metric, and the kernel bandwidth of the spatiotemporal joint kernel function is dynamically adjusted according to local density and global features. The constrained convergence unit is used to take uniformly sampled video frames as the initial seed points for drift. During the mean drift iteration process, it uses the spatiotemporal joint kernel function to make various seed points converge to local maxima along the density gradient direction, and determines the local maxima as the cluster center. The video frame clustering unit is used to match all the video frames with each of the cluster centers, and to determine all video frames corresponding to the same cluster center as the set of video frames corresponding to each of the cluster centers.

[0087] In one embodiment, the keyframe extraction module 705 includes: A local density unit is used to perform local processing on the set of video frames according to the hybrid distance metric corresponding to each of the video frames to obtain the local density corresponding to each of the video frames; Candidate keyframe unit, used to determine at least one candidate keyframe based on all the local densities; A keyframe sequence unit is used to determine a keyframe sequence corresponding to the video to be processed based on the temporal attention weights corresponding to each of the candidate keyframes and all the candidate keyframes; wherein the temporal attention weights are used to characterize the importance of the candidate keyframes in the temporal distribution.

[0088] In one embodiment, the device further includes: The input vector unit is used to preprocess the video frame sequence through the input layer in the coding model to obtain the input vector; the video frame sequence is obtained by sorting all video frames in time. An attention vector unit is used to perform attention processing on the input vector through a self-attention layer in the encoding model to obtain an attention vector. The semantic vector unit is used to perform a nonlinear transformation on the attention vector through a feedforward network layer in the encoding model to obtain global features.

[0089] Specific limitations regarding the video keyframe extraction device can be found in the limitations of the video keyframe extraction method described above, and will not be repeated here. Each module in the aforementioned video keyframe extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0090] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a video keyframe extraction method on the client side. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: The visual extraction module extracts features from the video frames in the video to be processed, and obtains visual feature vectors corresponding to each video frame. The temporal extraction module extracts features from the video frames in the video to be processed, and obtains the optical flow motion vector corresponding to each video frame. By using dynamic weighting coefficients to measure the distance between the visual feature vector and the optical flow motion vector, a hybrid distance metric corresponding to each video frame is obtained. The hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center. Keyframe extraction is performed on all the hybrid distance metrics and all the video frame sets to obtain the keyframe sequence corresponding to the video to be processed.

[0091] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The visual extraction module extracts features from the video frames in the video to be processed, and obtains visual feature vectors corresponding to each video frame. The temporal extraction module extracts features from the video frames in the video to be processed, and obtains the optical flow motion vector corresponding to each video frame. By using dynamic weighting coefficients to measure the distance between the visual feature vector and the optical flow motion vector, a hybrid distance metric corresponding to each video frame is obtained. The hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center. Keyframe extraction is performed on all the hybrid distance metrics and all the video frame sets to obtain the keyframe sequence corresponding to the video to be processed.

[0092] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0093] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0095] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0096] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for extracting keyframes from a video, characterized in that, include: The visual extraction module extracts features from the video frames in the video to be processed, and obtains visual feature vectors corresponding to each video frame. The temporal extraction module extracts features from the video frames in the video to be processed, and obtains the optical flow motion vector corresponding to each video frame. By using dynamic weighting coefficients to measure the distance between the visual feature vector and the optical flow motion vector, a hybrid distance metric corresponding to each video frame is obtained. The hybrid distance metric and global features are clustered using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center. Keyframe extraction is performed on all the hybrid distance metrics and all the video frame sets to obtain the keyframe sequence corresponding to the video to be processed.

2. The video keyframe extraction method as described in claim 1, characterized in that, The step of extracting features from video frames in the video to be processed using a visual extraction module to obtain visual feature vectors corresponding to each video frame includes: The initial convolutional layer in the visual extraction module is used to perform convolution processing on all the video frames to obtain convolutional features corresponding to each video frame. The maximum pooling layer in the visual extraction module is used to pool all the convolutional features to obtain pooled features corresponding to each convolutional feature. The pooling features are processed by multiple sets of bottleneck residual blocks in the visual extraction module to obtain residual vectors corresponding to each video frame. The global pooling layer in the visual extraction module performs average pooling on all the residual vectors to obtain the global pooling vector corresponding to each video frame. The fully connected layer in the visual extraction module performs fully connected processing on all the global pooling vectors to obtain the visual feature vectors corresponding to each video frame.

3. The video keyframe extraction method as described in claim 1, characterized in that, The step of extracting features from video frames in the video to be processed using the temporal extraction module to obtain optical flow motion vectors corresponding to each video frame includes: Optical flow processing is performed on all video frames through the relevant network layer in the temporal extraction module to obtain the initial estimated optical flow corresponding to each video frame. By using multiple simple network layers in the time-series extraction module, residual estimation is performed on all the initial estimated optical flows to obtain the residual corrected optical flows corresponding to each of the initial estimated optical flows. The displacement of all video frames is estimated by the small displacement network layer in the temporal extraction module to obtain the small displacement optical flow corresponding to each video frame. The optical flow fusion layer in the temporal extraction module is used to fuse the residual correction optical flow and the small displacement optical flow corresponding to the same video frame to obtain the target optical flow corresponding to each video frame. Integrating all the target optical flows yields the optical flow motion vectors corresponding to each video frame.

4. The video keyframe extraction method as described in claim 1, characterized in that, The step of performing distance measurement on the visual feature vector and the optical flow motion vector using dynamic weighting coefficients to obtain a hybrid distance measurement corresponding to each video frame includes: The video frames are weighted using a Siamese network in the metric module to obtain dynamic weight coefficients. The first branch in the metric module performs distance processing on all the visual feature vectors to obtain the visual cosine distance corresponding to each visual feature vector. The second branch in the measurement module performs distance processing on all optical flow motion vectors to obtain the optical flow motion distance corresponding to each optical flow motion vector. The hybrid distance metric corresponding to each video frame is determined based on the visual cosine distance, the optical flow motion distance, and the dynamic weighting coefficient corresponding to the same video frame.

5. The video keyframe extraction method as described in claim 1, characterized in that, The step of clustering the hybrid distance metric and global features using a clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center includes: A spatiotemporal joint kernel function is constructed, which is based on a hybrid distance metric, and the kernel bandwidth of the spatiotemporal joint kernel function is dynamically adjusted according to local density and global features; Using uniformly sampled video frames as initial seed points for drift, during the mean drift iteration process, various seed points are made to converge to local maxima along the density gradient direction according to the spatiotemporal joint kernel function, and the local maxima are determined as cluster centers. All video frames are matched with each of the cluster centers, and all video frames corresponding to the same cluster center are determined as the set of video frames corresponding to each cluster center.

6. The video keyframe extraction method as described in claim 1, characterized in that, The step of extracting keyframes from all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed includes: Based on the hybrid distance metric corresponding to each video frame, the set of video frames is locally processed to obtain the local density corresponding to each video frame; Based on all the local densities, at least one candidate keyframe is determined; Based on the temporal attention weights corresponding to each of the candidate keyframes and all the candidate keyframes, a keyframe sequence corresponding to the video to be processed is determined; wherein, the temporal attention weights are used to characterize the importance of the candidate keyframes in the temporal distribution.

7. The video keyframe extraction method as described in claim 1, characterized in that, Before performing clustering processing on the hybrid distance metric and global features through the clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center, the method further includes: The video frame sequence is preprocessed through the input layer in the coding model to obtain the input vector; the video frame sequence is obtained by sorting all video frames in time. The input vector is processed by an attention layer in the encoding model to obtain an attention vector; The attention vector is nonlinearly transformed by the feedforward network layer in the encoding model to obtain global features.

8. A video keyframe extraction device, characterized in that, include: The visual feature module is used to extract features from video frames in the video to be processed through the visual extraction module, and obtain visual feature vectors corresponding to each video frame. The optical flow feature module is used to extract features from video frames in the video to be processed through the temporal extraction module, and obtain optical flow motion vectors corresponding to each video frame. The distance measurement module is used to measure the distance between the visual feature vector and the optical flow motion vector through dynamic weighting coefficients, so as to obtain a hybrid distance measurement corresponding to each video frame. The video frame clustering module is used to perform clustering processing on the hybrid distance metric and global features through the clustering module to obtain at least one cluster center and a set of video frames corresponding to each cluster center; The keyframe extraction module is used to extract keyframes from all the hybrid distance metrics and all the video frame sets to obtain a keyframe sequence corresponding to the video to be processed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the video keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the video keyframe extraction method as described in any one of claims 1 to 7.

Images