A method, device, terminal equipment and computer program product for determining a video key frame

Abstract

The present application is applicable to the field of computer technology, and provides a method, apparatus, terminal device and computer program product for determining a video key frame, wherein the method comprises: obtaining pre-trained features of multiple levels in a target video frame; obtaining multi-angle features of each level according to the pre-trained features of each level; obtaining multi-angle fusion features of each level according to the multi-angle features of each level; obtaining frame-level fusion features of the target video frame according to the multi-angle fusion features of each level; and determining and outputting the video key frame of the target video frame based on the frame-level fusion features. The embodiments of the present application can effectively improve the accuracy of positioning and extracting video key frames, and are conducive to improving the quality and efficiency of subsequent processing of videos based on video key frames.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a method, apparatus, terminal device and computer program product for determining a video key frame. Background Art

[0002] The rapid development of the Internet and multimedia technology has led to an explosive growth in video data, and the processing and analysis of video data has become particularly urgent. Among them, extracting video key frames to summarize the main content of video data plays an extremely important role in application scenarios such as video summarization and video retrieval. However, when extracting video key frames, the accuracy of the extracted video key frames cannot be guaranteed. For example, some important information in the video data is easily missed, or there is a lot of redundant information in the extracted video key frames. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method, apparatus, terminal device and computer program product for determining a video key frame to solve the problem in the prior art that the accuracy of the extracted video key frames cannot be guaranteed when extracting video key frames.

[0004] A first aspect of an embodiment of the present application provides a method for determining a video key frame, comprising:

[0005] Obtain pre-trained features at multiple levels in the target video frame;

[0006] Based on the pre-trained features of each layer, obtain multi-angle features of each layer;

[0007] According to the multi-angle features of each layer, a multi-angle fusion feature of each layer is obtained;

[0008] Obtaining frame-level fusion features of the target video frame according to the multi-angle fusion features of each layer;

[0009] Based on the frame-level fusion features, a video key frame of the target video frame is determined and output.

[0010] In one embodiment, the step of obtaining pre-trained features at multiple levels in the target video frame includes:

[0011] Obtain object features at the object level, scene features at the scene level, character features at the character level, and audio features at the audio level in the target video frame.

[0012] In one embodiment, the step of obtaining multi-angle features of each layer according to the pre-trained features of each layer includes:

[0013] According to the object features at the object level in the target video frame, obtaining a first motion attention feature, a first global attention feature, and a first local attention feature;

[0014] According to the scene features at the scene level in the target video frame, obtaining a second motion attention feature, a second global attention feature, and a second local attention feature;

[0015] According to the character features at the character level in the target video frame, obtaining a third motion attention feature, a third global attention feature, and a third local attention feature;

[0016] According to the audio features at the audio level in the target video frame, a fourth motion attention feature, a fourth global attention feature, and a fourth local attention feature are obtained.

[0017] In one embodiment, obtaining the multi-angle fusion features of each layer according to the multi-angle features of each layer includes:

[0018] Obtaining a first multi-angle fusion feature according to a first motion attention feature, a first global attention feature, and a first local attention feature at an object level in the target video frame;

[0019] Obtaining a second multi-angle fusion feature according to a second motion attention feature, a second global attention feature, and a second local attention feature at a scene level in the target video frame;

[0020] Obtaining a third multi-angle fusion feature according to a third motion attention feature, a third global attention feature, and a third local attention feature at the character level in the target video frame;

[0021] A fourth multi-angle fusion feature is obtained based on the fourth motion attention feature, the fourth global attention feature, and the fourth local attention feature at the audio level in the target video frame.

[0022] In one embodiment, obtaining the frame-level fusion features of the target video frame according to the multi-angle fusion features of each layer includes:

[0023] Perform frame-level feature fusion on the first multi-angle fusion feature, the second multi-angle fusion feature, the third multi-angle fusion feature and the fourth multi-angle fusion feature to obtain the frame-level fusion feature.

[0024] In one embodiment, determining and outputting the video key frame of the target video frame based on the frame-level fusion feature includes:

[0025] The frame-level fusion features are clustered, and cluster center frames and abnormal frames are determined as video key frames and output.

[0026] In one embodiment, the method for determining a video key frame further includes:

[0027] Get the video frame of the target video;

[0028] Preprocessing the video frame to obtain a target video frame;

[0029] According to the number and position number of the target video frame, a hash mapping table is constructed with the number of the target video frame as the key and the position number of the target video frame as the value, so that the video key frame is output according to the corresponding position number after being determined.

[0030] A second aspect of an embodiment of the present application provides a device for determining a video key frame, including:

[0031] A first feature acquisition module, used to acquire pre-trained features at multiple levels in a target video frame;

[0032] The second feature acquisition module is used to acquire multi-angle features of each layer according to the pre-trained features of each layer;

[0033] A third feature acquisition module is used to obtain multi-angle fusion features of each layer according to the multi-angle features of each layer;

[0034] A fourth feature acquisition module, used to obtain a frame-level fusion feature of the target video frame according to the multi-angle fusion feature of each layer;

[0035] The key frame determination module is used to determine the video key frame of the target video frame based on the frame-level fusion feature and output it.

[0036] The third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for determining a video key frame as described in the first aspect of the embodiment of the present application are implemented.

[0037] A fourth aspect of the embodiments of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the method for determining a video key frame as described in the first aspect of the embodiments of the present application are implemented.

[0038] A fifth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method for determining a video key frame as described in the first aspect of the embodiment of the present application are implemented.

[0039] The first aspect of the embodiment of the present application provides a method for determining video key frames, which obtains pre-trained features of multiple levels in the target video frame; obtains multi-angle features of each level based on the pre-trained features of each level; obtains multi-angle fusion features of each level based on the multi-angle features of each level; obtains frame-level fusion features of the target video frame based on the multi-angle fusion features of each level; determines and outputs the video key frames of the target video frame based on the frame-level fusion features; can effectively improve the accuracy of video key frame positioning and extraction, which is conducive to improving the quality and efficiency of subsequent processing of the video based on the video key frames.

[0040] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a first flow chart of a method for determining a video key frame provided in an embodiment of the present application;

[0043] Figure 2 This is a second flow chart of the method for determining a video key frame provided in an embodiment of the present application;

[0044] Figure 3 This is a third flow chart of the method for determining a video key frame provided in an embodiment of the present application;

[0045] Figure 4 It is a structural schematic diagram of a device for determining a video key frame provided in an embodiment of the present application;

[0046] Figure 5 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0048] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0050] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. "Multiple" means "two" or "more than two".

[0051] The rapid development of the Internet and multimedia technology has led to an explosive growth in video data, and the processing and analysis of video data has become particularly urgent. Among them, extracting video key frames to summarize the main content of video data plays an extremely important role in application scenarios such as video summarization and video retrieval. However, when extracting video key frames, the accuracy of the extracted video key frames cannot be guaranteed. For example, some important information in the video data is easily missed, or there is a lot of redundant information in the extracted video key frames.

[0052] Therefore, an embodiment of the present application provides a method for determining video key frames, by obtaining pre-trained features of multiple levels in a target video frame; obtaining multi-angle features of each level based on the pre-trained features of each level; obtaining multi-angle fusion features of each level based on the multi-angle features of each level; obtaining frame-level fusion features of the target video frame based on the multi-angle fusion features of each level; based on the frame-level fusion features, determining and outputting the video key frames of the target video frame; the accuracy of video key frame positioning and extraction can be effectively improved, which is conducive to improving the quality and efficiency of subsequent processing of the video based on the video key frames.

[0053] Embodiment 1

[0054] like Figure 1 As shown, the method for determining a video key frame provided in an embodiment of the present application includes the following steps S1 to S5:

[0055] Step S1, obtain pre-trained features of multiple levels in the target video frame, and proceed to step S2.

[0056] In the application, when obtaining pre-trained features at multiple levels in the target video frame, it may include but is not limited to obtaining pre-trained features at the object level, pre-trained features at the scene level, pre-trained features at the character level, and pre-trained features at the audio level in the target video frame.

[0057] In the application, when obtaining pre-trained features at multiple levels in the target video frame, at least two pre-trained features at any level among the pre-trained features at the object level, the pre-trained features at the scene level, the pre-trained features at the character level, and the pre-trained features at the audio level in the target video frame can also be obtained.

[0058] Specifically, only the pre-trained features at the object level and the pre-trained features at the scene level in the target video frame may be obtained; or, only the pre-trained features at the object level and the pre-trained features at the character level in the target video frame may be obtained; or, only the pre-trained features at the object level and the pre-trained features at the audio level in the target video frame may be obtained; or, only the pre-trained features at the scene level and the pre-trained features at the character level in the target video frame may be obtained; or, only the pre-trained features at the scene level and the pre-trained features at the audio level in the target video frame may be obtained; or, only the pre-trained features at the character level and the pre-trained features at the audio level in the target video frame may be obtained; or, only the pre-trained features at the object level, the pre-trained features at the scene level, and the pre-trained features at the character level in the target video frame may be obtained; or, only the pre-trained features at the object level, the pre-trained features at the scene level, and the pre-trained features at the audio level in the target video frame may be obtained; or, only the pre-trained features at the scene level, the pre-trained features at the character level, the pre-trained features at the audio level, etc. in the target video frame may be obtained.

[0059] It can be understood that when obtaining pre-trained features at multiple levels in the target video frame in step S1, it includes but is not limited to obtaining part or all of the pre-trained features at the object level, the pre-trained features at the scene level, the pre-trained features at the character level, and the pre-trained features at the audio level in the target video frame. In actual applications, pre-trained features at more levels may also be included, which are not listed here one by one.

[0060] In one embodiment, step S1 specifically includes:

[0061] Obtain object features at the object level, scene features at the scene level, character features at the character level, and audio features at the audio level in the target video frame.

[0062] In applications, some deep learning models can be used to obtain pre-trained features at various levels in the target video frame.

[0063] In the application, object features at the object level in the target video frame can be obtained through the pre-trained GoogleNet model on the ImageNet dataset, or through some other pre-trained models, such as the AlexNet model, VGGNet (Visual Geometry Group Network) model, ResNet (Residual Network) model, DenseNet (Densely Connected Convolutional Network) model, MobileNet model, YOLO (YouOnly Look Once) model, SSD (Single Shot MultiBox Detector) model, Mask R-CNN (MaskRegion-based Convolutional Neural Network) model, etc. to extract object features. This is just an example and does not constitute a limitation.

[0064] In the application, the scene features at the scene level in the target video frame can be obtained through the pre-trained ResNet-101 model on the Place365 dataset, or through some other pre-trained models, such as Scene ParsingModels model, DeepLab, FCN (Fully Convolutional Network) model, EfficientNet model, MobileNet model, DenseNet model, AlexNet model, VGGNet model, etc. to extract scene features. This is just an example and does not constitute a limitation.

[0065] In the application, the character features at the character level in the target video frame can be obtained through the pre-trained PP-OCRv4-server model on the ICDAR2019-LSVT dataset, or through some other pre-trained models, such as CNN (Convolutional Neural Network) model, RNN (Recurrent Neural Network) model, CRNN (Convolutional Recurrent Neural Network) model, CTC (Connectionist Temporal Classification) model, Attention Mechanism model, Transformer model, Seq2Seq (Sequence-to-Sequence) model, etc. to extract character features. This is just an example and does not constitute a limitation.

[0066] In the application, the audio features at the audio level in the target video frame can be obtained through the pre-trained Whisper model on the Chinese speech dataset WenetSpeech and the English speech dataset LibriSpeech. The audio features can also be obtained through some other pre-trained models, such as MFCC (Mel Frequency Cepstral Coefficients) model, SpectralFeatures model, Filter Banks model, Delta Features model, Log-Mel Spectrogram model, CNN (Convolutional Neural Network) model, RNN (Recurrent Neural Network) model, etc. to extract speech features. This is just an example and does not constitute a limitation.

[0067] In one embodiment, if Figure 2 As shown, before step S1, the following steps S11 to S13 are included:

[0068] Step S11, obtaining a video frame of a target video, and proceeding to step S12;

[0069] Step S12, preprocessing the video frame to obtain a target video frame, and then proceeding to step S13;

[0070] Step S13: construct a hash mapping table with the target video frame number as the key and the target video frame position number as the value according to the target video frame number and the position number, so that the video key frame is output according to the corresponding position number after being determined.

[0071] In the application, the target video is the video to be analyzed, including but not limited to videos on various social platforms and various surveillance videos.

[0072] In step S11, a video processing library (such as OpenCV, etc.) may be used to read the target video file to obtain basic information of the target video (such as frame rate, resolution, etc.);

[0073] Then, the target video is read frame by frame, and each frame is stored as an image file or processed as image data in a memory, or one frame is extracted from the video every certain number of frames and stored as an image file or processed as image data in a memory, thereby reducing the amount of data to be processed and improving processing efficiency;

[0074] The video frames are then numbered according to the order in which they were acquired. For example, the video frame numbers and position numbers can be expressed as 0001 (1), 0002 (2), 0003 (3), ..., 1001 (1001), ... (the corresponding position numbers are in brackets);

[0075] Performing preprocessing such as image quality assessment on the acquired video frames, including but not limited to image clarity assessment, image distortion detection, image noise detection, image contrast assessment, image blur detection, etc., and removing the unqualified parts of the video frames such as blurry and severely distorted parts according to the corresponding assessment results to obtain the target video frames;

[0076] Since the blurred, severely distorted and other unqualified parts of the video frame are removed, the position number of the target video frame will also change accordingly. For example, if the second and fourth frames are deleted, the number and position sequence of the target video frame will become 0001 (1), 0003 (2), 0005 (3), ..., 1001 (0999), ...;

[0077] According to the changed number and position serial number of the target video frame, a hash mapping table is constructed with the number of the target video frame as the key and the position serial number of the corresponding target video frame as the value, so that the video key frame can be output according to the corresponding serial number or sequence after being determined.

[0078] Step S2: According to the pre-trained features of each layer, obtain the multi-angle features of each layer, and proceed to step S3.

[0079] In the application, three parallel feature extraction branches can be constructed to extract multi-angle features at each level in the target video frame, that is, motion attention features, global attention features and local attention features at each level in the target video frame are extracted.

[0080] The first branch is used to extract motion attention features between adjacent target video frames, specifically including:

[0081] Input the pre-trained features of a certain level in the target video frame obtained from step S1;

[0082] The pre-trained features of adjacent target video frames (e.g., the t-th frame and the t+1-th frame) are input into the multi-head inter-frame attention module; in the multi-head inter-frame attention module, the pre-trained features of each target video frame are converted into the corresponding query matrix (Q), key matrix (K), and value matrix (V);

[0083] Using the multi-head attention architecture, calculate the inter-frame attention between the t-th frame and the t+1-th frame (specifically, when calculating the multi-head inter-frame attention of the t-th frame, use the query matrix of the t-th frame , the key value matrix of the t+1th frame And the value matrix of frame t+1 ; Each attention head calculates the query matrix separately and key matrix The product is then multiplied by the softmax activation function and the dropout layer to obtain the matching weight (used to measure the degree of matching between different frames, where the softmax activation function is used to refer to nonlinear characteristics and the dropout layer is used to help solve the overfitting problem and improve the generalization ability of the model). Finally, the output is compared with the value matrix of the t+1th frame. Multiply them together to get the multi-head inter-frame attention of the t-th frame;

[0084] That is, the attention of the i-th head For the tth frame, it can be expressed as: =dropout(softmax( )) ), the calculation of the multi-head inter-frame attention of the t+1th frame is similar to the calculation of the multi-head inter-frame attention of the tth frame mentioned above, which will not be repeated here;

[0085] Assuming there are H independent attention heads, the multi-head attention module connects the single-head attention heads in series to obtain the inter-frame attention representation of the t-th frame ,Right now Similarly, the inter-frame attention representation of the t+1th frame can also be obtained ,Right now:

[0086] ;

[0087] Then calculate A t and A t+1 The difference between them is transformed through a linear transformation (which can be a fully connected layer or a simple neural network layer with a weight matrix) to derive the motion attention feature of the current time, which can be specifically expressed as: , where linear() represents a linear transformation, which is used to calculate A t and A t+1 The difference between.

[0088] The second branch is used to extract global attention features to capture the long-term relationship between distant target video frames (wherein distant target video frames usually refer to those target video frames that are far apart in time series; for example, in a video with a long duration, the distance between the first few frames and the last few frames can be considered "far away"), including:

[0089] Input the frame sequence X={x t} t=1:N , obtain the initialized query matrix (Q), key matrix (K) and value matrix (V);

[0090] Assume the input is The matrix of is the target video frame sequence length, is the feature dimension of each frame, that is, each target video frame is represented as a dimensional feature vectors, all feature vectors form a matrix, in which each row corresponds to the feature of a target video frame;

[0091] For each attention head, the dot product operation is performed using the transpose of the query matrix (Q) and the key matrix (K), and then the dot product operation results of each attention head are normalized by the softmax function to obtain the attention weight; the normalized attention weight is then used to perform weighted summation on the value matrix (V);

[0092] Assume there are H independent attention heads, and the output of the i-th attention head is ,in is the output dimension of the ith attention head, which is usually the same as the input dimension (hereinafter referred to as replace);

[0093] Then the outputs of all attention heads (weighted value matrices) are concatenated on the feature dimension to form a comprehensive global attention representation vector. That is, the outputs of H attention heads are concatenated on the feature dimension and can be expressed as:

[0094] ;

[0095] Finally, the combined global attention representation vector is processed through a linear layer (which can be a fully connected layer) (the learning weight is , with the bias of b), the final global attention feature is generated, which can be specifically expressed as:

[0096] .

[0097] The third branch is used to extract local attention features, capture local dependencies and finer detail features, including:

[0098] Input the frame sequence X={x t} t=1:N , and divided into M continuous, non-overlapping local segments;

[0099] Each local segment is independently input into the local multi-head attention module, where the global attention calculation process is repeated for each head (only for the data in the local segment);

[0100] For each local segment, the outputs of all attention heads are concatenated in the feature dimension to form the local attention representation of the local segment;

[0101] Similar to the global multi-head attention in the second branch, assuming there are H independent attention heads, the H local attentions of the kth local segment can be concatenated together and expressed as:

[0102] ,in, is the target video frame number of the kth local segment, is the output feature dimension of a single attention head, represents the local attention of the Hth head of the kth local segment.

[0103] The local attention representations of all local segments are concatenated in the sequence dimension to obtain:

[0104] ,(in, ), and mapped through a linear layer (which can be a fully connected layer) to obtain the final local attention feature, which can be specifically expressed as: .

[0105] In the application, the motion clue information between adjacent frames can be obtained through the first branch, and the dependencies of the target video frames can be modeled and analyzed from different granularity levels through the second and third branches. In addition, the second branch can capture the importance of each target video frame to the entire video frame set, which helps to fully understand the video context, while the third branch focuses on the local part of the video frame set and can capture local dependencies and finer detail features; thereby realizing the extraction of video spatiotemporal features in time and space.

[0106] In one embodiment, step S2 specifically includes:

[0107] According to the object features at the object level in the target video frame, obtaining a first motion attention feature, a first global attention feature, and a first local attention feature;

[0108] According to the scene features at the scene level in the target video frame, obtaining a second motion attention feature, a second global attention feature, and a second local attention feature;

[0109] According to the character features at the character level in the target video frame, obtaining a third motion attention feature, a third global attention feature, and a third local attention feature;

[0110] According to the audio features at the audio level in the target video frame, a fourth motion attention feature, a fourth global attention feature, and a fourth local attention feature are obtained.

[0111] In the application, for each level of pre-trained features, the corresponding motion attention features, global attention features, and local attention features can be obtained through the processing flow of the first branch, the second branch, and the third branch. It can be understood that the first, second, third, and fourth here are only used to distinguish the motion attention features, global attention features, and local attention features at different levels, and do not constitute any limitation.

[0112] In the application, corresponding to step S1, at least two pre-trained features of any level among the pre-trained features at the object level, the scene level, the character level, and the audio level in the target video frame can be obtained. In step S2, only the motion attention features, global attention features, and local attention features of the corresponding levels of the pre-trained features obtained in step S1 can be obtained.

[0113] For example, if only the pre-trained features at the object level and the pre-trained features at the scene level in the target video frame are obtained in step S1, then only the first motion attention feature, the first global attention feature, and the first local attention feature at the object level in the target video frame, and the second motion attention feature, the second global attention feature, and the second local attention feature at the scene level in the target video frame are correspondingly obtained in step S2;

[0114] If only the pre-trained features at the object level, the pre-trained features at the scene level, and the pre-trained features at the character level in the target video frame are obtained in step S1, then in step S2, only the first motion attention feature, the first global attention feature, and the first local attention feature at the object level in the target video frame, the second motion attention feature, the second global attention feature, and the second local attention feature at the scene level in the target video frame, and the third motion attention feature, the third global attention feature, and the third local attention feature at the character level in the target video frame are obtained accordingly. It can be understood that the acquisition of multi-angle features at more levels is similar to the above content and will not be repeated here.

[0115] Step S3: According to the multi-angle features of each layer, obtain the multi-angle fusion features of each layer, and proceed to step S4.

[0116] In the application, the multi-angle features of each layer will be fused when output, so that a fused multi-angle fusion feature is finally output.

[0117] Taking the multi-angle features at the object level as an example, the first motion attention feature Y is obtained through the first branch M1 , the first global attention feature Y is obtained through the second branch G1 , the first local attention feature Y is obtained through the third branch L1 , the first motion attention feature Y is obtained before output M1 , the first global attention feature Y G1 , the first local attention feature Y L1 After splicing, we get: , and then mapped through a linear layer (which can be a fully connected layer) to obtain the multi-angle fusion feature at the object level, that is, the first multi-angle fusion feature. It can be understood that the multi-angle feature fusion at other levels is similar to the multi-angle feature fusion at the object level, and will not be repeated here.

[0118] In one embodiment, step S3 specifically includes:

[0119] Obtaining a first multi-angle fusion feature according to a first motion attention feature, a first global attention feature, and a first local attention feature at an object level in the target video frame;

[0120] Obtaining a second multi-angle fusion feature according to a second motion attention feature, a second global attention feature, and a second local attention feature at a scene level in the target video frame;

[0121] Obtaining a third multi-angle fusion feature according to a third motion attention feature, a third global attention feature, and a third local attention feature at the character level in the target video frame;

[0122] A fourth multi-angle fusion feature is obtained based on the fourth motion attention feature, the fourth global attention feature, and the fourth local attention feature at the audio level in the target video frame.

[0123] In the application, the multi-angle features of each level can be fused through this step, and finally a corresponding multi-angle fusion feature is obtained. It can be understood that the first, second, third, and fourth here are only used to distinguish the multi-angle fusion features of different levels, and do not constitute any limitation.

[0124] In the application, corresponding to steps S1 and S2, if only the pre-trained features at the object level and the pre-trained features at the scene level in the target video frame are obtained in step S1, then in step S2, only the first motion attention feature, the first global attention feature, and the first local attention feature at the object level in the target video frame, as well as the second motion attention feature, the second global attention feature, and the second local attention feature at the scene level in the target video frame are obtained accordingly; in step S3, the first multi-angle fusion feature at the object level and the second multi-angle fusion feature at the scene level are also obtained accordingly. It can be understood that the acquisition of multi-angle fusion features at more levels is similar to the above content and will not be repeated here.

[0125] Step S4: Obtain the frame-level fusion features of the target video frame according to the multi-angle fusion features of each layer, and proceed to step S5.

[0126] In the application, the multi-angle fusion features at each level will be fused again in this step, so that a frame-level fusion feature is finally output.

[0127] For example, if the sizes of the multi-angle fusion features output at the object level, scene level, character level, and audio level are all N x P, where N is the number of video frames and P is the feature dimension, then in step S4, what is fused is the fusion of the features of the corresponding video frames at the object level, scene level, character level, and audio level. The specific fusion method can be adaptive frame weighted summation, and the size of the final frame-level fusion feature is also N x P.

[0128] In one embodiment, step S4 specifically includes:

[0129] Perform frame-level feature fusion on the first multi-angle fusion feature, the second multi-angle fusion feature, the third multi-angle fusion feature and the fourth multi-angle fusion feature to obtain the frame-level fusion feature.

[0130] In the application, corresponding to the aforementioned steps, no matter how many multi-angle fusion features are obtained, frame-level feature fusion can be performed in this step, and finally a frame-level fusion feature is obtained. For example, if only the first multi-angle fusion feature and the second multi-angle fusion feature are obtained in the aforementioned steps, then in this step, the first multi-angle fusion feature and the second multi-angle fusion feature are subjected to frame-level feature fusion, and finally a frame-level fusion feature is obtained.

[0131] It can be understood that the frame-level feature fusion of multi-angle fusion features at more levels is similar to the above content and will not be repeated here.

[0132] Step S5: Based on the frame-level fusion features, determine the video key frame of the target video frame and output it.

[0133] In the application, after obtaining the frame-level fusion features, the corresponding clustering algorithm is used to cluster the obtained frame-level fusion features based on the frame-level fusion features, so that the number of key frames can be determined and the corresponding video key frames can be output finally for use in subsequent video processing processes.

[0134] It should be noted that Figure 2 The arrow from step S13 to step S5 indicates that the hash mapping table obtained in step S13 will be applied in step S5, that is, after obtaining the video key frame, step S5 determines the position order of each video key frame based on the hash mapping table, and finally outputs the video key frame according to the corresponding position order.

[0135] In one embodiment, if Figure 3 As shown, step S5 specifically includes the following step S51:

[0136] Step S51: cluster the frame-level fusion features, determine the cluster center frame and abnormal frame as video key frames and output them.

[0137] In the application, based on the frame-level fusion features, the improved density peak clustering algorithm can be used to cluster the obtained frame-level fusion features to obtain each cluster center, and then the target video frame closest to each cluster center is calculated according to the cluster center as the cluster center frame, and the target video frame far away from each cluster center is taken as the abnormal frame. Finally, the sum of the number of all cluster center frames and abnormal frames is the final number of video key frames, that is, all cluster center frames and abnormal frames obtained here are video key frames of the target video.

[0138] Among them, an abnormal frame refers to a target video frame that appears for a very short time (for example, 0.002 seconds or even shorter) but contains important information. When clustering, the abnormal frame itself does not belong to any category or is an independent category; in addition, the cluster center obtained by clustering with the improved density peak clustering algorithm may not have a cluster center that just corresponds to a certain target video frame. Therefore, after obtaining the cluster center, this step uses the target video frame closest to the cluster center as the cluster center frame, and the target video frame that is far from each cluster center as the abnormal frame, that is, the frame that has a large difference from other video frames during clustering is used as the abnormal frame.

[0139] In the application, since the numbers and position numbers of all target video frames are hash mapped in step S13, all cluster center frames and abnormal frames obtained in this step can determine the position order of each frame through hash mapping, and finally output the video key frames according to the corresponding position order, thereby avoiding the problem of disordered order of the output video key frames.

[0140] In application, the method for determining video key frames provided in the embodiment of the present application can not only reduce the redundancy of video key frames, but also avoid the omission of important information. In addition, the present application adopts an improved density peak clustering algorithm, which can automatically determine the number of video key frames, and then automatically locate, identify and output video key frames according to the corresponding order, which effectively improves the accuracy of video key frame positioning and extraction, and is conducive to improving the quality and efficiency of subsequent processing of the target video based on video key frames.

[0141] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0142] Embodiment 2

[0143] The embodiment of the present application also provides a video key frame determination device, which is used to execute the method steps in the above video key frame determination method embodiment. The device can be a virtual appliance in a terminal device, which is run by a processor of the terminal device, or it can be the terminal device itself.

[0144] like Figure 4 As shown, the video key frame determination device 100 provided in the embodiment of the present application includes: a first feature acquisition module 101, a second feature acquisition module 102, a third feature acquisition module 103, a fourth feature acquisition module 104 and a key frame determination module 105.

[0145] A first feature acquisition module 101 is used to acquire pre-trained features at multiple levels in a target video frame;

[0146] The second feature acquisition module 102 is used to acquire multi-angle features of each layer according to the pre-trained features of each layer;

[0147] A third feature acquisition module 103 is used to obtain multi-angle fusion features of each layer according to the multi-angle features of each layer;

[0148] A fourth feature acquisition module 104 is used to obtain a frame-level fusion feature of the target video frame according to the multi-angle fusion feature of each layer;

[0149] The key frame determination module 105 is used to determine and output the video key frame of the target video frame based on the frame-level fusion feature.

[0150] In one embodiment, the first feature acquisition module 101 is further configured to:

[0151] Obtain object features at the object level, scene features at the scene level, character features at the character level, and audio features at the audio level in the target video frame.

[0152] In one embodiment, the second feature acquisition module 102 is further configured to:

[0153] According to the object features at the object level in the target video frame, obtaining a first motion attention feature, a first global attention feature, and a first local attention feature;

[0154] According to the scene features at the scene level in the target video frame, obtaining a second motion attention feature, a second global attention feature, and a second local attention feature;

[0155] According to the character features at the character level in the target video frame, obtaining a third motion attention feature, a third global attention feature, and a third local attention feature;

[0156] According to the audio features at the audio level in the target video frame, a fourth motion attention feature, a fourth global attention feature, and a fourth local attention feature are obtained.

[0157] In one embodiment, the third feature acquisition module 103 is further configured to:

[0158] Obtaining a first multi-angle fusion feature according to a first motion attention feature, a first global attention feature, and a first local attention feature at an object level in the target video frame;

[0159] Obtaining a second multi-angle fusion feature according to a second motion attention feature, a second global attention feature, and a second local attention feature at a scene level in the target video frame;

[0160] Obtaining a third multi-angle fusion feature according to a third motion attention feature, a third global attention feature, and a third local attention feature at the character level in the target video frame;

[0161] A fourth multi-angle fusion feature is obtained based on the fourth motion attention feature, the fourth global attention feature, and the fourth local attention feature at the audio level in the target video frame.

[0162] In one embodiment, the fourth feature acquisition module 104 is further configured to:

[0163] Perform frame-level feature fusion on the first multi-angle fusion feature, the second multi-angle fusion feature, the third multi-angle fusion feature and the fourth multi-angle fusion feature to obtain the frame-level fusion feature.

[0164] In one embodiment, the key frame determination module 105 is further configured to:

[0165] The frame-level fusion features are clustered, and cluster center frames and abnormal frames are determined as video key frames and output.

[0166] In one embodiment, the video key frame determination device 100 further includes:

[0167] The video frame acquisition module 106 is used to acquire the video frame of the target video;

[0168] A target video frame acquisition module 107 is used to pre-process the video frame to obtain a target video frame;

[0169] The mapping module 108 is used to construct a hash mapping table with the target video frame number as the key and the target video frame position number as the value according to the target video frame number and position number, so that the video key frame is output according to the corresponding position number after being determined.

[0170] In application, each unit in the above device may be a software program module, or may be implemented by different logic circuits integrated in a processor or independent physical components connected to a processor, or may be implemented by multiple distributed processors.

[0171] Embodiment 3

[0172] like Figure 5 As shown, the embodiment of the present application further provides a terminal device 200, including: at least one processor 201 ( Figure 5 Only one processor is shown in the figure), memory 202, and a computer program 203 stored in the memory 202 and executable on at least one processor 201. When the processor 201 executes the computer program 203, the steps in the above-mentioned method embodiments are implemented.

[0173] In the application, the terminal device may include, but is not limited to, a processor, a memory, Figure 5 It is only an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as human-computer interaction devices, input and output devices, network access devices, etc. The network access device may include a communication module for the terminal device to communicate with the user terminal.

[0174] In applications, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. For example, the processor may be a timing controller (TCON). A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0175] In applications, the memory may be an internal storage unit of a terminal device in some embodiments, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device. The memory may also include both an internal storage unit and an external storage device of the terminal device. The memory is used to store operating systems, applications, boot loaders, data, and other programs, such as program codes of computer programs. The memory may also be used to temporarily store data that has been output or is to be output.

[0176] In the application, the communication module can be set as any device that can directly or indirectly communicate with the user terminal over a long distance by wire or wireless communication according to actual needs. For example, the communication module can provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to network devices. The communication module can include an antenna, which can have only one array element or an antenna array including multiple array elements. The communication module can receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module can also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0177] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0178] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0179] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0180] An embodiment of the present application provides a computer program product. When the computer program product is executed on a terminal device, the terminal device can implement the steps in the above-mentioned various method embodiments.

[0181] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0182] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0183] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0184] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0185] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for determining a video key frame, characterized in that: include: Obtain pre-trained features at multiple levels in the target video frame; According to the pre-trained features of each level, a multi-angle feature of each level is obtained, wherein the multi-angle feature includes a motion attention feature, a global attention feature, and a local attention feature of each level; According to the multi-angle features of each layer, a multi-angle fusion feature of each layer is obtained; According to the multi-angle fusion features of each layer, a frame-level fusion feature of the target video frame is obtained; Based on the frame-level fusion features, a density peak clustering algorithm is used to cluster the obtained frame-level fusion features, and a video key frame of the target video frame is determined and output; The obtaining of pre-trained features at multiple levels in the target video frame includes: Obtain object features at the object level, scene features at the scene level, character features at the character level, and audio features at the audio level in the target video frame.

2. The method for determining a video key frame according to claim 1, wherein: The method of obtaining multi-angle features of each layer according to the pre-trained features of each layer includes: According to the object features at the object level in the target video frame, obtaining a first motion attention feature, a first global attention feature, and a first local attention feature; According to the scene features at the scene level in the target video frame, obtaining a second motion attention feature, a second global attention feature, and a second local attention feature; According to the character features at the character level in the target video frame, obtaining a third motion attention feature, a third global attention feature, and a third local attention feature; According to the audio features at the audio level in the target video frame, a fourth motion attention feature, a fourth global attention feature, and a fourth local attention feature are obtained.

3. The method for determining a video key frame according to claim 2, wherein: The step of obtaining the multi-angle fusion features of each layer according to the multi-angle features of each layer includes: Obtaining a first multi-angle fusion feature according to a first motion attention feature, a first global attention feature, and a first local attention feature at an object level in the target video frame; Obtaining a second multi-angle fusion feature according to a second motion attention feature, a second global attention feature, and a second local attention feature at a scene level in the target video frame; Obtaining a third multi-angle fusion feature according to a third motion attention feature, a third global attention feature, and a third local attention feature at the character level in the target video frame; A fourth multi-angle fusion feature is obtained based on the fourth motion attention feature, the fourth global attention feature, and the fourth local attention feature at the audio level in the target video frame.

4. The method for determining a video key frame according to claim 3, wherein: The step of obtaining the frame-level fusion features of the target video frame according to the multi-angle fusion features of each layer includes: Perform frame-level feature fusion on the first multi-angle fusion feature, the second multi-angle fusion feature, the third multi-angle fusion feature and the fourth multi-angle fusion feature to obtain the frame-level fusion feature.

5. The method for determining a video key frame according to claim 4, wherein: The determining and outputting the video key frame of the target video frame based on the frame-level fusion feature includes: The frame-level fusion features are clustered, and cluster center frames and abnormal frames are determined as video key frames and output.

6. The method for determining a video key frame according to any one of claims 1 to 5, characterized in that: Also includes: Get the video frame of the target video; Preprocessing the video frame to obtain a target video frame; According to the number and position number of the target video frame, a hash mapping table is constructed with the number of the target video frame as the key and the position number of the target video frame as the value, so that the video key frame is output according to the corresponding position number after being determined.

7. A device for determining a video key frame, characterized in that: include: A first feature acquisition module, used to acquire pre-trained features at multiple levels in a target video frame; A second feature acquisition module is used to acquire multi-angle features of each level according to the pre-trained features of each level, wherein the multi-angle features include motion attention features, global attention features, and local attention features of each level; A third feature acquisition module is used to obtain multi-angle fusion features of each layer according to the multi-angle features of each layer; A fourth feature acquisition module, used to obtain a frame-level fusion feature of the target video frame according to the multi-angle fusion feature of each layer; A key frame determination module, used to cluster the obtained frame-level fusion features based on the frame-level fusion features using a density peak clustering algorithm, determine the video key frame of the target video frame and output it; The obtaining of pre-trained features at multiple levels in the target video frame includes: Obtain object features at the object level, scene features at the scene level, character features at the character level, and audio features at the audio level in the target video frame.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for determining a video key frame according to any one of claims 1 to 6 are implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the method for determining a video key frame according to any one of claims 1 to 6 are implemented.

Images