A key frame extraction method and device based on contrast learning, equipment and medium

Abstract

The application relates to the field of artificial intelligence, and in particular to a key frame extraction method and device based on contrast learning, equipment and a medium. Applied to a medical scene, in the process of iteratively training a feature embedding network, a constraint term of parameter adjustment is determined according to a contrast loss, an update rule of parameters in the initial feature embedding network is determined according to the constraint term, a stability adjustment mechanism based on a nonlinear control theory is introduced, strict convergence guarantee is provided for contrast learning, key frame classification is performed on video frames according to target time sequence saliency and a preset key frame selection strategy, core dynamic changes of video content can be accurately captured, intelligent classification of key frames is realized in combination with the preset selection strategy, the selection of key frames is more in line with the internal logic and time evolution law of video content through quantitative analysis of the target time sequence saliency, and the accuracy of key frame extraction is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for keyframe extraction based on contrastive learning. Background Technology

[0002] In the medical field, keyframe extraction technology is commonly used for the analysis and processing of medical image sequences (such as echocardiography, endoscopic videos, and dynamic CT / MRI sequences). In recent years, self-supervised learning, by utilizing the internal structure of the video (such as temporal continuity and multi-view consistency) to construct supervisory signals, significantly reduces its dependence on labeled data and is frequently used for keyframe extraction. Self-supervised learning, in particular, has become the mainstream paradigm for self-supervised video representation learning by constructing positive and negative sample pairs to learn discriminative features. However, the construction of positive and negative sample pairs often fails to fully incorporate the specialized characteristics of medical image sequences, such as subtle changes in lesion areas and the fixed correlations of anatomical structures. This results in insufficient discriminative power of the learned features, making it difficult to accurately capture crucial medical information and leading to low accuracy in keyframe extraction. Therefore, improving the accuracy of keyframe extraction has become a pressing issue. Summary of the Invention

[0003] In view of this, embodiments of this application provide a keyframe extraction method, apparatus, device, and medium based on contrastive learning to solve the problem of low accuracy in keyframe extraction.

[0004] In a first aspect, embodiments of this application provide a keyframe extraction method based on contrastive learning, the keyframe extraction method comprising: Obtain the initial feature embedding network and training samples, wherein the training samples are multi-frame video frame sequences; For any iteration of training, obtain the positive sample pair and negative sample set corresponding to any sample video frame, and use the initial feature embedding network to extract features from the video frames in the positive sample pair and the negative sample set to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, as well as each negative sample feature in the negative sample set. Based on the first positive sample features, the second positive sample features, and each negative sample feature, a contrast loss is calculated. Based on the contrast loss, a constraint term for parameter adjustment is determined. Based on the constraint term, an update rule for the parameters in the initial feature embedding network is determined. According to the update rule, the parameters of the initial feature embedding network are updated to obtain the updated feature embedding network. One iteration of training ends. When the contrast loss is less than a preset threshold, the iteration training stops to obtain the trained feature embedding network. Obtain the video to be extracted, wherein the video to be extracted is a multi-frame video sequence; For any video frame, the trained feature embedding network is used to extract features from the video frame to obtain the target video frame features; Based on the target video frame features, the target temporal saliency of the video frame is calculated. Based on the target temporal saliency and the preset keyframe selection strategy, the video frame is classified into keyframes to obtain the keyframe classification results. All video frames are traversed to obtain the keyframe extraction results.

[0005] Secondly, embodiments of this application provide a keyframe extraction device based on contrastive learning, the keyframe extraction device comprising: The first acquisition module is used to acquire an initial feature embedding network and training samples, wherein the training samples are multi-frame sample video frame sequences; The first extraction module is used to obtain positive sample pairs and negative sample sets corresponding to any sample video frame for any iteration of training, and use the initial feature embedding network to extract features from the positive sample pairs and the video frames in the negative sample sets to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, and each negative sample feature in the negative sample set. The calculation module is used to calculate the contrast loss based on the first positive sample features, the second positive sample features, and each negative sample feature; determine the constraint terms for parameter adjustment based on the contrast loss; and determine the update rules for the parameters in the initial feature embedding network based on the constraint terms. The update module is used to update the parameters of the initial feature embedding network according to the update rule to obtain the updated feature embedding network, end one iteration of training, and stop the iteration training when the contrast loss is less than a preset threshold to obtain the trained feature embedding network. The second acquisition module is used to acquire the video to be extracted, wherein the video to be extracted is a multi-frame video sequence. The module is used to extract features from any video frame using the trained feature embedding network to obtain the target video frame features. The classification module is used to calculate the target temporal saliency of the video frame based on the target video frame features, classify the video frame into key frames according to the target temporal saliency and a preset key frame selection strategy, obtain the key frame classification result, and traverse all video frames to obtain the key frame extraction result.

[0006] Thirdly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the keyframe extraction method as described above.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the keyframe extraction method as described above.

[0008] The advantages of this application compared to the prior art are: In the iterative training process of the feature embedding network, this application determines the constraint terms for parameter adjustment based on the contrastive loss, and determines the update rules for the parameters in the initial feature embedding network based on the constraint terms. A stability adjustment mechanism based on nonlinear control theory is introduced to provide a strict convergence guarantee for contrastive learning. Based on the target temporal saliency and the preset keyframe selection strategy, video frames are classified into keyframes, which can accurately capture the core dynamic changes of video content. Combined with the preset selection strategy, intelligent classification of keyframes is achieved. Through quantitative analysis of the target temporal saliency, the selection of keyframes is made more in line with the internal logic and temporal evolution of video content, effectively improving the accuracy of keyframe extraction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a schematic diagram illustrating the application environment of a keyframe extraction method based on contrastive learning provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a keyframe extraction method based on contrastive learning provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a keyframe extraction device based on contrastive learning provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0016] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0019] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0020] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0021] To illustrate the technical solution of this application, specific embodiments are described below.

[0022] One embodiment of this application provides a keyframe extraction method based on contrastive learning, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] To illustrate the technical solution of this application, specific embodiments are described below.

[0024] See Figure 2 This is a flowchart illustrating a keyframe extraction method based on contrastive learning provided in an embodiment of this application, as shown below. Figure 2 As shown, the keyframe extraction method based on contrastive learning may include the following steps.

[0025] S201: Obtain the initial feature embedding network and training samples. The training samples are multi-frame video frame sequences.

[0026] In step S201, the feature embedding network is a neural network model used to convert the input video frames into high-dimensional feature vectors. The training samples consist of multiple consecutive video frame sequences, each of which contains several video frames with temporal correlation.

[0027] In this embodiment, an initial feature embedding network is obtained. This initial feature embedding network is a contrastive feature embedding network using the Vision Transformer architecture. This architecture effectively captures local detail features and global semantic information in video frames by segmenting video frames into multiple image patches and adding positional embedding information to each patch. The encoder part of the Vision Transformer consists of a multi-head self-attention mechanism and a feed-forward network. The multi-head self-attention mechanism allows the model to learn the dependencies between regions within a video frame in different representation subspaces, while the feed-forward network performs non-linear transformations and dimensional mappings on the features processed by the attention mechanism, thereby generating a highly discriminative video frame feature representation. By adopting this architecture, the initial feature embedding network can convert the input two-dimensional video frame data into a one-dimensional high-dimensional feature vector containing rich spatiotemporal information, laying the foundation for subsequent contrastive learning tasks.

[0028] Training samples can come from different scenarios and types of video data, such as surveillance videos, movie clips, and short video content, to ensure the diversity and generalization of training samples. Alternatively, video clips containing different scenes, actions, and camera transitions can be selected from publicly available video datasets (such as UCF101, HMDB51, etc.).

[0029] S202: For any iteration of training, obtain the positive sample pair and negative sample set corresponding to any sample video frame, and use the initial feature embedding network to extract features from the video frames in the positive sample pair and negative sample set to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, as well as each negative sample feature in the negative sample set.

[0030] In step S202, iterative training is a single parameter optimization loop in the model parameter update process, that is, using a batch of training samples to complete the entire process of forward propagation to calculate loss, backpropagation to update gradient and adjust model parameters.

[0031] Positive sample pairs are augmented views that differ in visual presentation but have consistent semantic content. Negative sample sets represent the set of video frames that differ significantly from the sample video frames in semantic content or temporal relevance.

[0032] In this embodiment, a positive sample pair and a negative sample set corresponding to any sample video frame are obtained. For each sample video frame, at least two enhanced views are generated using a preset data augmentation strategy. These enhanced views are then combined to form the positive sample pair for that sample video frame. The preset data augmentation strategies include, but are not limited to, random cropping, random flipping, color dithering, brightness adjustment, contrast variation, Gaussian blurring, or adding noise. These strategies introduce visual diversity while maintaining the core semantic content of the video frame. Secondly, when constructing the negative sample set, it can include video frames from different video segments than the sample video frame, non-adjacent video frames within the same video segment that are more than a preset frame number threshold away from the sample video frame, or manually constructed interference video frames with a completely different semantic category from the sample video frame. Introducing such a negative sample set provides a comparative reference for the model during feature learning, enabling the initial feature embedding network to more accurately distinguish between video frame features with similar and different semantics, thereby further optimizing the discriminative power of feature embedding.

[0033] The initial feature embedding network performs independent feature extraction operations on the two positive video frames in the positive sample pair and each negative video frame in the negative sample set, respectively, to obtain the first and second positive features of the positive sample pair, and the features of each negative sample in the negative sample set. Specifically, for each video frame, normalized features are generated by the VisionTransformer encoder during feature extraction, calculated using the following formula: in, The features of the t-th video frame are... For Vision Transformer encoders The generated encoded features It is a 2-norm.

[0034] The first positive sample feature, the second positive sample feature, and each negative sample feature in the negative sample set are normalized features calculated based on the above formula.

[0035] Optionally, obtain the positive sample pairs and negative sample sets corresponding to any sample video frame, including: Enhancement processing is performed on the sample video frames to obtain positive sample pairs of the sample video frames; The initial feature embedding network is used to extract features from all sample video frames to obtain the sample video frame features for each sample video frame. For any sample video frame, the similarity between the target sample video frame features and the sample video frame features of other sample video frames is calculated to obtain the similarity between the sample video frame and other sample video frames. The other sample video frames corresponding to the top K similarity scores are determined as the negative sample set of the sample video frames, where K is an integer greater than zero.

[0036] In this embodiment, for any given training iteration, any sample video frame is augmented to generate a positive sample pair. The augmentation process can employ a combination of various data augmentation strategies, such as random cropping, random horizontal flipping, random color jittering (including random adjustments to parameters such as brightness, contrast, saturation, and hue within preset ranges, Gaussian blur, and random grayscale conversion). By independently applying one or more of these augmentation strategies to the same sample video frame, two augmented views that differ visually but have identical semantic content can be generated. These two augmented views constitute the positive sample pair for that sample video frame. The specific parameter range of each augmentation strategy can be dynamically adjusted according to actual training needs and dataset characteristics to ensure that the generated positive sample pair retains the key semantic information of the original video frame while introducing sufficient diversity to improve the model's generalization ability.

[0037] In this embodiment, an initial feature embedding network is used to extract features from all sample video frames, obtaining sample video frame features for each sample video frame. For the target sample video frame features of any sample video frame, the similarity between the target sample video frame features and the sample video frame features of other sample video frames is calculated, obtaining the similarity score between the sample video frame and other sample video frames. The similarity score can be calculated using common metrics such as cosine similarity and Euclidean distance. For example, if cosine similarity is used, the closer the cosine similarity value between the target sample video frame features and other sample video frame features is to 1, the higher the feature similarity between the two; if Euclidean distance is used, the smaller the distance value, the higher the feature similarity between the two. In practical applications, an appropriate similarity measurement method can be selected based on the specific feature distribution and task requirements. After calculating the similarity scores of all other sample video frames with the target sample video frame, these similarities are sorted in descending order. Then, the top K similarity scores of the other sample video frames are selected, and these video frames together form the negative sample set of the target sample video frame. The K value here is a preset hyperparameter, and its specific size can be adjusted according to the scale of the training data, the number of video frames, and the needs of model training. For example, it can be set to 5, 10, 20, etc., to ensure that the negative sample set contains a sufficient number of negative samples with high similarity, thereby effectively assisting the model in training and improving the model's ability to distinguish key frame features.

[0038] In this embodiment, enhancing any sample video frame yields two similar yet visually distinct video frames, forming a positive sample pair for that sample video frame. This enhancement method effectively simulates various lighting changes, viewpoint shifts, partial occlusion, and noise interference that video frames may encounter in real-world scenarios, helping to improve the robustness and generalization ability of the feature representations learned by the subsequent feature embedding network. Selecting video frames with high similarity as negative samples provides more challenging learning materials for model training. These negative samples and target samples have high visual similarity but are not completely identical keyframes. This subtle difference effectively encourages the model to focus more on the truly discriminative core features in the video frames, rather than relying on some superficial and easily confused non-key features. By allowing the model to continuously learn to distinguish these "difficult-to-distinguish" negative samples from the target samples during training, the model's ability to capture subtle differences in features and its discrimination accuracy can be significantly enhanced. This improves the accuracy and reliability of keyframe extraction in practical applications, avoiding problems such as insufficient training or inadequate generalization ability due to low discrimination of negative samples.

[0039] S203: Calculate the contrast loss based on the features of the first positive sample, the features of the second positive sample, and the features of each negative sample. Determine the constraint terms for parameter adjustment based on the contrast loss. Determine the update rules for the parameters in the initial feature embedding network based on the constraint terms.

[0040] In step S204, the contrastive loss characterizes the similarity between the first positive sample feature and the second positive sample feature of the target sample video frame in the feature space, and the difference between the positive sample features (including the first positive sample feature and the second positive sample feature) of the target sample video frame and each negative sample feature in the negative sample set. The update rule for the parameters in the initial feature embedding network is the size of each update of the parameters in the initial feature embedding network.

[0041] In this embodiment, the contrast loss is calculated based on the features of the first positive sample, the features of the second positive sample, and the features of each negative sample. The calculation formula is as follows: in, To compare the losses, Features of the first positive sample Features of the second positive sample For temperature coefficient, For the negative sample set Negative sample features of video frames. This represents the number of negative samples in the negative sample set.

[0042] Based on the contrast loss, a constraint term for parameter adjustment is determined, and based on this constraint term, the update rule for the parameters in the initial feature embedding network is determined. This constraint term limits the direction and magnitude of parameter updates in the initial feature embedding network, ensuring that the network can more effectively focus on distinguishing positive sample pairs from negative sample features during the learning process. Specifically, by backpropagating the contrast loss to the initial feature embedding network, the network parameters are adjusted in the direction of reducing the contrast loss. When the contrast loss is small, it indicates that the similarity between the first and second positive sample features is high, and the differences between them and each negative sample feature are large. In this case, the effect of the parameter adjustment constraint term is relatively weak, and the network parameters can be fine-tuned. However, when the contrast loss is large, it means that the similarity between positive sample features is insufficient or the distinguishability with negative sample features is not high enough. The constraint term will drive the network parameters to be adjusted more significantly to enhance the discriminative ability of the features, thereby optimizing the feature extraction effect of the initial feature embedding network on video frame features.

[0043] Optionally, based on the contrast loss, constraints for parameter adjustment are determined, including: Based on the contrastive loss, the parameter gradient energy of the initial feature embedding network is calculated; The parameters of the initial feature embedding network are constrained based on the parameter gradient energy, and the constraint terms for parameter adjustment are determined.

[0044] In this embodiment, the parameter gradient energy of the initial feature embedding network is calculated based on the contrastive loss, and the calculation formula is as follows: in, The parameters after the nth iteration of training , Let be the gradient energy of the parameters during the nth iteration of training. To compare the losses, It is the square norm. To compare the gradient of the loss with respect to the parameters.

[0045] The parameters of the initial feature embedding network are constrained based on the parameter gradient energy, and the constraint terms for parameter adjustment are determined.

[0046] The parameters of the initial feature embedding network are constrained based on the parameter gradient energy, thus determining the constraint terms for parameter tuning. The formula for these constraint terms is as follows: in, For constraint term functions, For adjustment coefficients, As the attenuation factor, The gradient energy of the parameters during the (n+1)th training iteration. It can be calculated using the contrastive loss obtained from the (n+1)th iteration of training based on the original stochastic gradient descent method. Let be the gradient energy of the parameters during the nth iteration of training.

[0047] Optionally, based on the constraints, the update rules for the parameters in the initial feature embedding network are determined, including: Determine the target constraint value based on the constraint terms; Based on the target constraint value and the parameter gradient energy, the update rules for the parameters in the initial feature embedding network are determined.

[0048] In this embodiment, the target constraint value is determined based on the constraint terms, i.e. The value of is determined based on the target constraint value and the parameter gradient energy, to establish the update rule for the parameters in the initial feature embedding network. The formula for the update rule is as follows: in, Embed network parameters for features trained in the (n+1)th iteration. Embed network parameters for features trained in the nth iteration. For the target constraint value, The gradient of the target constraint value with respect to the parameters. For adjustment coefficients, For indicator functions, Let be the gradient energy of the parameters during the nth iteration of training. Let be the gradient of the parameters with respect to the gradient energy of the parameters trained in the nth iteration.

[0049] S204: Update the parameters of the initial feature embedding network according to the update rule to obtain the updated feature embedding network, end one iteration of training, and stop iterative training when the contrast loss is less than the preset threshold to obtain the trained feature embedding network.

[0050] In step S204, after the end of one iteration of training, the next iteration of training continues until the contrastive loss is less than a preset threshold, at which point the iteration of training stops and the trained feature embedding network is obtained.

[0051] In this embodiment, the parameters of the initial feature embedding network are updated according to the update rule to obtain the updated feature embedding network. After each iteration of training, the contrastive loss of the updated feature embedding network on the training samples is calculated and compared with a preset threshold. If the contrastive loss is less than the preset threshold, the feature embedding network is considered to have reached the training objective, and the iterative training stops. The feature embedding network obtained at this time is the trained feature embedding network. If the contrastive loss is still greater than or equal to the preset threshold, the next iteration of training continues. This involves obtaining the positive sample pair and negative sample set corresponding to any sample video frame, using the initial feature embedding network to extract features from the video frames in the positive sample pair and negative sample set, obtaining the first positive sample feature and the second positive sample feature of the positive sample pair, and each negative sample feature in the negative sample set. The contrastive loss is calculated based on the first positive sample feature, the second positive sample feature, and each negative sample feature. Based on the contrastive loss, the constraint term for parameter adjustment is determined. Based on the constraint term, the update rule for the parameters in the initial feature embedding network is determined. The parameters of the initial feature embedding network are updated according to the update rule to obtain the updated feature embedding network, and the step of ending one iteration of training is completed.

[0052] S205: Obtain the video to be extracted, which is a multi-frame video sequence.

[0053] In step S205, the video to be extracted is the video from which key frames need to be extracted, and the video to be extracted is a multi-frame video sequence.

[0054] In this embodiment, the video to be extracted is a medical video such as clinical diagnosis, surgical guidance, medical teaching and case review. Medical videos often have the characteristics of long duration, uneven information density and fragmented key content. If medical staff browse frame by frame, it will not only consume a lot of time and energy, but may also miss core diagnosis and treatment details due to visual fatigue.

[0055] In this embodiment, keyframe extraction involves acquiring the video to be extracted. This video can be obtained through methods such as accessing a video storage database, a real-time streaming interface, or uploading a local file from the user. The video storage database can pre-store a large amount of historical video data, allowing the system to quickly locate and retrieve the target video based on video identifier information via a query interface. The real-time streaming interface is suitable for processing dynamic content such as live videos, enabling real-time reception and parsing of video stream data. The user's local file upload method allows users to directly submit video files from their personal devices to the system, meeting personalized video processing needs. During the acquisition process, the system automatically detects and converts the video format to ensure that subsequent processing modules are compatible with video data of different encoding formats. It also verifies the video integrity; if data is missing or corrupted, a re-acquisition mechanism will be triggered or an error message will be sent to the user.

[0056] S206: For any video frame, use the trained feature embedding network to extract features from the video frame to obtain the target video frame features.

[0057] In step S206, the trained feature embedding network is the feature embedding network trained in the above iterations.

[0058] In this embodiment, for any video frame, a trained feature embedding network is used to extract features from the video frame to obtain the target video frame features. Then, the video frame is encoded using a trained Vision Transformer framework encoder to obtain encoded features, which are then normalized to obtain the target video frame features.

[0059] S207: Based on the characteristics of the target video frame, calculate the target temporal saliency of the video frame. Based on the target temporal saliency and the preset keyframe selection strategy, classify the video frame into keyframes to obtain the keyframe classification results. Traverse all video frames to obtain the keyframe extraction results.

[0060] In this embodiment, the target temporal saliency of a video frame is used as a quantitative indicator to measure the degree of content change or importance of that video frame relative to its preceding and following video frames in the video sequence. The keyframe selection strategy is a strategy that compares the target temporal saliency value of a video frame with a preset value. The keyframe classification results include video frames that are keyframes and video frames that are not keyframes.

[0061] In this embodiment, the calculation of target temporal saliency is based on the similarity between the target video frame features and the target video frame features of the preceding and following video frames in time. The cosine similarity is calculated between the target video frame features of the current video frame and the video frame features of the preceding and following video frames, respectively. The average of these two cosine similarities is then used as the temporal similarity measure for the video frame. Finally, subtracting this temporal similarity measure from 1 yields the target temporal saliency of the video frame. A larger value indicates a greater difference in content between the video frame and the preceding and following frames, resulting in higher temporal saliency and a greater likelihood of it becoming a keyframe. This process is repeated for all video frames to obtain the keyframe extraction results.

[0062] An initial saliency threshold is set, which can be dynamically adjusted or manually preset based on the video content type or application scenario requirements. For each video frame traversed, its calculated target temporal saliency value is compared with this threshold. If the target temporal saliency value of the video frame is greater than or equal to the preset threshold, the video frame is determined to be a potential keyframe; otherwise, it is determined to be a non-keyframe. By comparing the target temporal saliency with the above threshold, accurate classification of video frames can be achieved, thereby efficiently extracting keyframes that can represent the core information of the video.

[0063] Optionally, the temporal saliency of the video frame is calculated based on the characteristics of the target video frame, including: The trained feature embedding network is used to extract features from the remaining video frames in the video to be extracted, so as to obtain the remaining video frame features corresponding to each remaining video frame. The differences between the features of the target video frame and the features of each remaining video frame are calculated. Based on the differences, the temporal significance of the video frames is calculated.

[0064] In this embodiment, a trained feature embedding network is used to extract features from the remaining video frames, obtaining the remaining video frame features corresponding to each remaining video frame. Then, the video frames are encoded using a trained Vision Transformer framework encoder to obtain encoded features. These encoded features are then normalized to obtain the target video frame features. The difference between the target video frame features and the features of each remaining video frame is calculated. This difference can be determined based on the distance between the target video frame features and the features of each remaining video frame. Based on this difference, the temporal saliency of the video frames is calculated. The calculation formula is as follows: in, Let be the temporal saliency of the t-th video frame, and let i be the ith remaining video frame. This represents the total number of features in the remaining video frames. Let t be the target video frame feature of the video frame. Let be the feature of the remaining video frame corresponding to the i-th remaining video frame.

[0065] Optionally, based on the differences, the temporal significance of the video frames is calculated, including: Based on the attention mechanism, the degree of correlation between the video frame and each remaining video frame is calculated. Based on the degree of correlation, the differences between the features of the target video frame and the features of each remaining video frame are weighted and summed to obtain the temporal significance of the video frames.

[0066] In this embodiment, based on an attention mechanism, the correlation between a video frame and each remaining video frame is calculated. This correlation can be represented by weight values. The features of the target video frame are concatenated with or interact with the features of all remaining video frames to generate intermediate features for calculating attention weights. Then, a multilayer perceptron (MLP) or scaled dot product model is used to perform a nonlinear transformation on the intermediate features, outputting an initial attention score for each remaining video frame relative to the target video frame. To ensure the attention scores have a probability distribution, all initial attention scores are softmax normalized to obtain normalized attention weights. A larger weight value indicates a stronger temporal correlation between the two frames, and vice versa.

[0067] Based on the degree of correlation, the differences between the features of the target video frame and the features of each remaining video frame are weighted and summed to obtain the temporal significance of the video frames. The calculation formula is as follows: in, Let be the temporal saliency of the t-th video frame, and let i be the ith remaining video frame. This represents the total number of features in the remaining video frames. Let t be the target video frame feature of the video frame. The remaining video frame features corresponding to the i-th remaining video frame. Let represent the degree of correlation between the video frame t and the remaining video frame i.

[0068] Optionally, based on the target temporal saliency and a preset keyframe selection strategy, video frames are classified into keyframes to obtain keyframe classification results, including: Obtain the first and second video frames corresponding to the preset radius of the video frames; Determine whether the first temporal significance of the first video frame is equal to the second temporal significance of the second video frame; When the target temporal significance is greater than the first temporal significance and greater than the second temporal significance, the video frame is classified as a keyframe.

[0069] In this embodiment, a first video frame and a second video frame corresponding to a preset radius are obtained. The preset radius is pre-set, meaning it represents the first video frame with a preceding preset radius and the second video frame with a subsequent preset radius in the current video frame period. The first temporal saliency of the first video frame and the second temporal saliency of the second video frame are determined. This involves calculating the difference between the video features corresponding to the first video frame and the features of each of the remaining video frames, and calculating the temporal saliency of the first video frame based on this difference. Similarly, the difference between the video features corresponding to the second video frame and the features of each of the remaining video frames is calculated, and the temporal saliency of the second video frame is calculated based on this difference. The calculation formula is the same as the formula for calculating temporal saliency described above, and will not be described again in this embodiment.

[0070] Determine whether the target temporal significance is greater than the first temporal significance and the second temporal significance. If the target temporal significance is greater than both the first and second temporal significance, classify the video frame as a keyframe.

[0071] In the iterative training process of the feature embedding network, this application determines the constraint terms for parameter adjustment based on the contrastive loss, and determines the update rules for the parameters in the initial feature embedding network based on the constraint terms. A stability adjustment mechanism based on nonlinear control theory is introduced to provide a strict convergence guarantee for contrastive learning. Based on the target temporal saliency and the preset keyframe selection strategy, video frames are classified into keyframes, which can accurately capture the core dynamic changes of video content. Combined with the preset selection strategy, intelligent classification of keyframes is achieved. Through quantitative analysis of the target temporal saliency, the selection of keyframes is made more in line with the internal logic and temporal evolution of video content, effectively improving the accuracy of keyframe extraction.

[0072] Please see Figure 3 , Figure 3 This is a schematic diagram of a keyframe extraction device based on contrastive learning according to an embodiment of this application. This keyframe extraction device corresponds one-to-one with the keyframe extraction method based on contrastive learning described in the above embodiments. Please refer to [link / reference] for details. Figure 2 as well as Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 3 The keyframe extraction device 30 includes: a first acquisition module 31, a first extraction module 32, a calculation module 33, an update module 34, a second acquisition module 35, an acquisition module 36, and a classification module 37.

[0073] The first acquisition module 31 is used to acquire the initial feature embedding network and training samples, wherein the training samples are multi-frame sample video frame sequences.

[0074] The first extraction module 32 is used to obtain the positive sample pair and negative sample set corresponding to any sample video frame for any iteration of training, and use the initial feature embedding network to extract features from the video frames in the positive sample pair and negative sample set to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, as well as each negative sample feature in the negative sample set.

[0075] The calculation module 33 is used to calculate the contrast loss based on the first positive sample features, the second positive sample features and each negative sample feature, determine the constraint terms for parameter adjustment based on the contrast loss, and determine the update rules for the parameters in the initial feature embedding network based on the constraint terms.

[0076] The update module 34 is used to update the parameters of the initial feature embedding network according to the update rules, obtain the updated feature embedding network, end one iteration of training, and stop iterative training when the contrast loss is less than a preset threshold, thus obtaining the trained feature embedding network.

[0077] The second acquisition module 35 is used to acquire the video to be extracted, which is a multi-frame video sequence.

[0078] Module 36 is obtained, which is used to extract features from any video frame using a trained feature embedding network to obtain the features of the target video frame.

[0079] The classification module 37 is used to calculate the target temporal saliency of the video frame based on the characteristics of the target video frame, classify the video frame into key frames according to the target temporal saliency and the preset key frame selection strategy, obtain the key frame classification result, and traverse all video frames to obtain the key frame extraction result.

[0080] Optionally, the first extraction module 32 includes: The enhancement unit is used to enhance the sample video frames to obtain positive sample pairs of the sample video frames.

[0081] The first extraction unit is used to extract features from all sample video frames using the initial feature embedding network to obtain the sample video frame features for each sample video frame.

[0082] The first calculation unit is used to calculate the similarity between the target sample video frame features and the sample video frame features of other sample video frames for any sample video frame, and obtain the similarity between the sample video frame and other sample video frames.

[0083] The first determining unit is used to determine the other sample video frames corresponding to the top K similarity scores as the negative sample set of the sample video frames, where K is an integer greater than zero.

[0084] Optionally, the above-mentioned calculation module 33 includes: The second computational unit is used to calculate the parameter gradient energy of the initial feature embedding network based on the contrastive loss.

[0085] The constraint unit is used to constrain the parameters of the initial feature embedding network based on the parameter gradient energy, and to determine the constraint terms for parameter adjustment.

[0086] Optionally, the above-mentioned calculation module 33 further includes: The second determining unit is used to determine the target constraint value based on the constraint terms.

[0087] The third determining unit is used to determine the update rules for parameters in the initial feature embedding network based on the target constraint value and parameter gradient energy.

[0088] Optionally, the above classification module 37 includes: The second extraction unit is used to extract features from the remaining video frames in the video to be extracted using a trained feature embedding network, so as to obtain the remaining video frame features corresponding to each remaining video frame.

[0089] The third calculation unit is used to calculate the difference between the features of the target video frame and the features of each remaining video frame, and to calculate the temporal significance of the video frame based on the difference.

[0090] Optionally, the third computing unit mentioned above includes: The computational subunit is used to calculate the degree of correlation between the obtained video frame and each remaining video frame based on the attention mechanism.

[0091] The weighted sub-unit is used to perform a weighted summation of the differences between the features of the target video frame and the features of each remaining video frame according to the degree of correlation, so as to obtain the temporal significance of the video frames.

[0092] Optionally, the classification module 37 mentioned above also includes: The acquisition unit is used to acquire the first video frame and the second video frame corresponding to the preset radius of the video frame.

[0093] The fourth determining unit is used to determine whether the first temporal significance of the first video frame is significant or the second temporal significance of the second video frame.

[0094] The classification unit is used to classify video frames as keyframes when the target temporal significance is greater than the first temporal significance and greater than the second temporal significance.

[0095] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0096] Figure 4This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. For example... Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 The diagram shows only one of the following: a memory and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above embodiments of the keyframe extraction method based on contrastive learning.

[0097] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0098] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0099] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0101] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0102] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0103] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A keyframe extraction method based on contrastive learning, characterized in that, The keyframe extraction method includes: Obtain the initial feature embedding network and training samples, wherein the training samples are multi-frame video frame sequences; For any iteration of training, obtain the positive sample pair and negative sample set corresponding to any sample video frame, and use the initial feature embedding network to extract features from the video frames in the positive sample pair and the negative sample set to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, as well as each negative sample feature in the negative sample set. Based on the first positive sample features, the second positive sample features, and each negative sample feature, a contrast loss is calculated. Based on the contrast loss, a constraint term for parameter adjustment is determined. Based on the constraint term, an update rule for the parameters in the initial feature embedding network is determined. According to the update rule, the parameters of the initial feature embedding network are updated to obtain the updated feature embedding network. One iteration of training ends. When the contrast loss is less than a preset threshold, the iteration training stops to obtain the trained feature embedding network. Obtain the video to be extracted, wherein the video to be extracted is a multi-frame video sequence; For any video frame, the trained feature embedding network is used to extract features from the video frame to obtain the target video frame features; Based on the target video frame features, the target temporal saliency of the video frame is calculated. Based on the target temporal saliency and the preset keyframe selection strategy, the video frame is classified into keyframes to obtain the keyframe classification results. All video frames are traversed to obtain the keyframe extraction results.

2. The keyframe extraction method as described in claim 1, characterized in that, The step of obtaining the positive sample pairs and negative sample sets corresponding to any sample video frame includes: The sample video frames are enhanced to obtain positive sample pairs of the sample video frames; The initial feature embedding network is used to extract features from all sample video frames to obtain the sample video frame features for each sample video frame. For any sample video frame, the similarity between the target sample video frame features and the sample video frame features of other sample video frames is calculated to obtain the similarity between the sample video frame and other sample video frames. The other sample video frames corresponding to the top K similarity scores are determined as the negative sample set of the sample video frames, where K is an integer greater than zero.

3. The keyframe extraction method as described in claim 1, characterized in that, The step of determining the constraint terms for parameter adjustment based on the comparison loss includes: Based on the contrast loss, the parameter gradient energy of the initial feature embedding network is calculated; The parameters of the initial feature embedding network are constrained based on the parameter gradient energy to determine the constraint terms for parameter adjustment.

4. The keyframe extraction method as described in claim 3, characterized in that, The step of determining the update rule for the parameters in the initial feature embedding network based on the constraint terms includes: Based on the aforementioned constraints, determine the target constraint value; Based on the target constraint value and the parameter gradient energy, the update rule for the parameters in the initial feature embedding network is determined.

5. The keyframe extraction method as described in claim 1, characterized in that, The step of calculating the temporal saliency of the video frame based on the target video frame features includes: The trained feature embedding network is used to extract features from the remaining video frames in the video to be extracted, so as to obtain the remaining video frame features corresponding to each remaining video frame. The difference between the features of the target video frame and the features of each remaining video frame is calculated, and the temporal saliency of the video frame is calculated based on the difference.

6. The keyframe extraction method as described in claim 5, characterized in that, The step of calculating the temporal significance of the video frame based on the difference includes: Based on the attention mechanism, the degree of correlation between the video frame and each remaining video frame is calculated. Based on the degree of correlation, the differences between the features of the target video frame and the features of each remaining video frame are weighted and summed to obtain the temporal significance of the video frame.

7. The keyframe extraction method as described in claim 1, characterized in that, The step of classifying the video frames into keyframes based on the target temporal saliency and a preset keyframe selection strategy to obtain keyframe classification results includes: Obtain the first video frame and the second video frame corresponding to the preset radius of the video frame; Determine that the first temporal significance of the first video frame is significantly different from the second temporal significance of the second video frame; When the target temporal significance is greater than the first temporal significance and greater than the second temporal significance, the video frame is classified as a keyframe.

8. A keyframe extraction device based on contrastive learning, characterized in that, The keyframe extraction device includes: The first acquisition module is used to acquire an initial feature embedding network and training samples, wherein the training samples are multi-frame sample video frame sequences; The first extraction module is used to obtain positive sample pairs and negative sample sets corresponding to any sample video frame for any iteration of training, and use the initial feature embedding network to extract features from the positive sample pairs and the video frames in the negative sample sets to obtain the first positive sample feature and the second positive sample feature of the positive sample pair, and each negative sample feature in the negative sample set. The calculation module is used to calculate the contrast loss based on the first positive sample features, the second positive sample features, and each negative sample feature; determine the constraint terms for parameter adjustment based on the contrast loss; and determine the update rules for the parameters in the initial feature embedding network based on the constraint terms. The update module is used to update the parameters of the initial feature embedding network according to the update rule to obtain the updated feature embedding network, end one iteration of training, and stop the iteration training when the contrast loss is less than a preset threshold to obtain the trained feature embedding network. The second acquisition module is used to acquire the video to be extracted, wherein the video to be extracted is a multi-frame video sequence. The module is used to extract features from any video frame using the trained feature embedding network to obtain the target video frame features. The classification module is used to calculate the target temporal saliency of the video frame based on the target video frame features, classify the video frame into key frames according to the target temporal saliency and a preset key frame selection strategy, obtain the key frame classification result, and traverse all video frames to obtain the key frame extraction result.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the keyframe extraction method as described in any one of claims 1 to 7.

Images