Video key frame extraction method and device based on self-supervision, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a video key frame extraction method and device based on self-supervision, equipment and a medium, and the method comprises the steps: obtaining a frame sequence of a target video, extracting the feature vector of each frame in the frame sequence through a pre-training feature extractor, and obtaining a feature vector sequence; determining graph nodes according to the frame sequence, calculating edge weights according to the feature vector sequence, and constructing an initial directed graph; calculating a graph Laplacian matrix, screening positive and negative sample frame pairs from the frame sequence by using a target sampler, and calculating time sequence perception comparison loss; circularly using parameters of the loss updating feature extractor to finely adjust the feature vector, update the edge weight, construct the graph and the like until the loss converges; reconstructing a directed graph, and calculating a Laplacian matrix of the graph to obtain a Laplacian matrix of a target graph; and performing Laplacian matrix spectral clustering on the target image to obtain a target video key frame set. The method can be applied to the business fields of financial science and technology, insurance, medical treatment and the like, and can improve the extraction accuracy of the video key frame.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to online processing business scenarios such as fintech, insurance, and healthcare. In particular, it relates to a self-supervised video keyframe extraction method, apparatus, device, and medium. Background Technology

[0002] Video keyframe extraction is a core supporting technology for video analysis, content understanding, and storage management, and is crucial in fields such as video retrieval and intelligent surveillance. Its core objective is to select representative frames that retain core semantics from highly redundant video sequences, ensuring semantic integrity while reducing storage and computing costs, and providing an efficient data foundation for intelligent applications.

[0003] Traditional keyframe extraction methods rely on manual annotation or heuristic rules, such as uniform sampling and post-sampling after shot boundary detection. These methods lack semantic awareness and select frames based only on surface features or fixed rules. They are difficult to adapt to diverse video scenarios and suffer from problems such as insufficient semantic representativeness of keyframes and redundancy, which cannot meet the needs of high-precision analysis.

[0004] In recent years, self-supervised learning techniques have provided a new path for this field. Related research has enabled the autonomous learning of frame-level representations from unlabeled data, such as learning features based on contrastive learning frameworks like MoCo, or modeling video dynamics by introducing temporal consistency constraints through TimeCycle. However, existing self-supervised methods still have significant limitations: First, sample selection efficiency is low, with random sampling or fixed time windows leading to sample redundancy and insufficient semantic diversity; second, temporal relationship modeling is insufficient, although some methods introduce temporal negative samples, they do not explicitly model complex spatiotemporal dependencies between frames, making it difficult to distinguish the local contextual differences between key frames and non-key frames; third, graph structure utilization is insufficient, with existing graph contrastive learning methods often employing static graph construction, failing to adapt to changes in video content, and lacking efficient node sampling mechanisms, thus limiting key frame extraction performance. Summary of the Invention

[0005] The purpose of this application is to propose a self-supervised video keyframe extraction method, apparatus, computer device, and storage medium to solve the problems of inefficient sample selection and insufficient temporal modeling in existing self-supervised video keyframe extraction methods.

[0006] Firstly, a self-supervised video keyframe extraction method is provided, which adopts the following technical solution: The process involves acquiring a sequence of video frames from the target video, extracting feature vectors for each frame using a pre-trained feature extractor, and obtaining a feature vector sequence. Based on the video frame sequence, multiple graph nodes are identified. Using the feature vector sequence, the edge weights connecting any two graph nodes are calculated. An initial directed graph is constructed based on the multiple graph nodes and edge weights. The graph Laplacian matrix of the initial directed graph is calculated. Based on the graph Laplacian matrix, a target sampler is used to select multiple positive and negative sample frame pairs from the video frame sequence. Finally, based on the positive and negative sample frame pairs… Yes, calculate the temporal-aware contrastive loss; iteratively execute the following operations: update the feature extractor parameters using the temporal-aware contrastive loss to fine-tune the feature vectors, update the edge weights based on the fine-tuned feature vectors, construct a directed graph, calculate the graph Laplacian matrix, filter positive and negative sample frame pairs, and recalculate the temporal-aware contrastive loss, until the temporal-aware contrastive loss converges; after the temporal-aware contrastive loss converges, reconstruct the directed graph, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix; perform spectral clustering on the target graph Laplacian matrix to obtain the keyframe set of the target video.

[0007] Secondly, a self-supervised video keyframe extraction device is provided, which adopts the following technical solution: The acquisition module is used to acquire the video frame sequence of the target video. It extracts the feature vector corresponding to each video frame in the video frame sequence through a pre-trained feature extractor to obtain the feature vector sequence. The construction module is used to determine multiple graph nodes based on a video frame sequence, calculate the edge weights of any two graph nodes connected by a feature vector sequence, and construct an initial directed graph based on the multiple graph nodes and edge weights. The filtering module is used to calculate the graph Laplacian matrix of the initial directed graph. Based on the graph Laplacian matrix, the target sampler filters out multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence. The loss calculation module is used to calculate the time-aware contrastive loss based on positive sample frame pairs and negative sample frame pairs. The loop execution module is used to repeatedly perform the following operations: updating the parameters of the feature extractor using time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive and negative sample frame pairs, and recalculating the time-aware contrastive loss until the time-aware contrastive loss converges. The matrix calculation module is used to reconstruct the directed graph after the time-aware contrastive loss converges, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix. The clustering module is used to perform spectral clustering on the Laplacian matrix of the target graph to obtain a set of keyframes from the target video.

[0008] Thirdly, a computer device is provided, which adopts the following technical solution: The process involves acquiring a sequence of video frames from the target video, extracting feature vectors for each frame using a pre-trained feature extractor, and obtaining a feature vector sequence. Based on the video frame sequence, multiple graph nodes are identified. Using the feature vector sequence, the edge weights connecting any two graph nodes are calculated. An initial directed graph is constructed based on the multiple graph nodes and edge weights. The graph Laplacian matrix of the initial directed graph is calculated. Based on the graph Laplacian matrix, a target sampler is used to select multiple positive and negative sample frame pairs from the video frame sequence. Finally, based on the positive and negative sample frame pairs… Yes, calculate the temporal-aware contrastive loss; iteratively execute the following operations: update the feature extractor parameters using the temporal-aware contrastive loss to fine-tune the feature vectors, update the edge weights based on the fine-tuned feature vectors, construct a directed graph, calculate the graph Laplacian matrix, filter positive and negative sample frame pairs, and recalculate the temporal-aware contrastive loss, until the temporal-aware contrastive loss converges; after the temporal-aware contrastive loss converges, reconstruct the directed graph, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix; perform spectral clustering on the target graph Laplacian matrix to obtain the keyframe set of the target video.

[0009] Fourthly, a computer-readable storage medium is provided, which adopts the following technical solution: The process involves acquiring a sequence of video frames from the target video, extracting feature vectors for each frame using a pre-trained feature extractor, and obtaining a feature vector sequence. Based on the video frame sequence, multiple graph nodes are identified. Using the feature vector sequence, the edge weights connecting any two graph nodes are calculated. An initial directed graph is constructed based on the multiple graph nodes and edge weights. The graph Laplacian matrix of the initial directed graph is calculated. Based on the graph Laplacian matrix, a target sampler is used to select multiple positive and negative sample frame pairs from the video frame sequence. Finally, based on the positive and negative sample frame pairs… Yes, calculate the temporal-aware contrastive loss; iteratively execute the following operations: update the feature extractor parameters using the temporal-aware contrastive loss to fine-tune the feature vectors, update the edge weights based on the fine-tuned feature vectors, construct a directed graph, calculate the graph Laplacian matrix, filter positive and negative sample frame pairs, and recalculate the temporal-aware contrastive loss, until the temporal-aware contrastive loss converges; after the temporal-aware contrastive loss converges, reconstruct the directed graph, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix; perform spectral clustering on the target graph Laplacian matrix to obtain the keyframe set of the target video.

[0010] Compared with existing technologies, the embodiments of this application have the following main advantages: By constructing a dynamic graph structure and an iterative optimization mechanism, a comprehensive improvement in keyframe extraction performance is achieved. A pre-trained feature extractor is used to obtain frame-level feature vectors. An initial directed graph is constructed based on feature similarity. The graph Laplacian matrix is ​​used to characterize inter-frame relationships. Combined with a target sampler to filter positive and negative sample frame pairs, the redundancy problem of random sampling is effectively avoided, improving the semantic diversity and selection efficiency of samples. The feature extractor parameters are iteratively updated through temporal-aware contrastive loss. Combined with dynamic adjustment of edge weights and directed graph reconstruction, complex spatiotemporal dependencies between frames are explicitly modeled, accurately distinguishing the local contextual differences between keyframes and non-keyframes, thus compensating for the shortcomings of insufficient temporal modeling. During the iteration process, the system dynamically adapts to changes in video content, fully leveraging the advantages of graph structures in modeling inter-frame relationships. Finally, keyframes are filtered through spectral clustering of the target graph Laplacian matrix. While ensuring the semantic integrity of the video, storage and computation costs are significantly reduced, and the accuracy and adaptability of keyframe extraction are significantly improved. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 A flowchart of an embodiment of the self-supervised video keyframe extraction method according to this application; Figure 3 This is a schematic diagram of a structure of an embodiment of the self-supervised video keyframe extraction apparatus according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0014] like Figure 1As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0015] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0016] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers.

[0017] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0018] It should be noted that the self-supervised video keyframe extraction method provided in this application is generally executed by a server / terminal device, and correspondingly, the self-supervised video keyframe extraction device is generally set in the server / terminal device.

[0019] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0020] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a self-supervised video keyframe extraction method according to this application. The self-supervised video keyframe extraction method includes the following steps: Step S201: Obtain the video frame sequence of the target video, and extract the feature vector corresponding to each video frame in the video frame sequence through a pre-trained feature extractor to obtain the feature vector sequence.

[0021] The target video refers to various types of video data for which keyframes are to be extracted. These data originate from various sources, including but not limited to intelligent surveillance recordings, short video platform uploads, and film clip extractions, covering different durations, resolutions, and content types. The video frame sequence is an ordered set of images obtained by performing frame-level segmentation on the target video. Each element is a single frame image from the video, and the frame order perfectly matches the playback sequence of the target video.

[0022] Feature extractors, in this context, refer to deep learning models that are pre-trained on large-scale image or video data and possess the ability to extract frame features. These models can include convolutional neural networks, visual Transformers, and more. They can transform pixel-level information from video frames into high-dimensional, abstract feature representations, capturing key information such as intra-frame texture and semantics to provide quantitative basis for modeling inter-frame relationships and selecting keyframes.

[0023] The feature vector is a high-dimensional numerical vector output by the feature extractor after processing a single video frame. Its dimension is set according to the network structure of the feature extractor, and each element in the vector corresponds to a specific semantic or visual feature of the frame. The feature vector sequence is an ordered set of vectors formed by arranging the feature vectors corresponding to each video frame in the original temporal order, corresponding one-to-one with the video frame sequence. This sequence preserves the feature information of individual frames while maintaining the temporal correlation between frames.

[0024] Step S202: Based on the video frame sequence, determine multiple graph nodes; based on the feature vector sequence, calculate the edge weights of any two graph nodes that connect to form an edge; and construct an initial directed graph based on the multiple graph nodes and edge weights.

[0025] In this context, multiple graph nodes refer to a set of nodes formed by mapping each individual video frame in the video frame sequence to an independent node in the graph structure. The number of nodes is the same as the number of frames in the video frame sequence. Each node carries the feature vector information of the corresponding video frame and is the core element for constructing the directed graph. Edge weights are numerical parameters used to quantify the strength of the association between any two graph nodes (corresponding video frames) in the directed graph. They are calculated based on the feature vectors corresponding to the two nodes, such as through cosine similarity or Euclidean distance transformation. The numerical value is positively correlated with the semantic or visual similarity between frames.

[0026] The initial directed graph refers to a graph structure with directional attributes (representing inter-frame temporal or dependency relationships) constructed using multiple graph nodes as vertices and edge weights between any two nodes as connection strengths. This graph structure provides a visual and quantitative representation of the initial inter-frame relationships, intuitively presenting the semantic relationships between frames through the combination of nodes and edges.

[0027] Step S203: Calculate the graph Laplacian matrix of the initial directed graph. Based on the graph Laplacian matrix, select multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence using the target sampler.

[0028] The graph Laplacian matrix is ​​a square matrix calculated based on the degree matrix and weight matrix of a directed graph. Its dimension is consistent with the number of graph nodes, and the matrix elements are derived using a specific formula, such as L=DW (where D is the degree matrix and W is the weight matrix). This matrix contains topological information about the graph structure and the strength of node associations.

[0029] The target sampler is an algorithm module that uses a "disk center-radius" quantization rule to filter samples based on the inter-frame correlation and topological information contained in the graph Laplacian matrix. Its core logic is as follows: First, each node of the initial directed graph is used as the center of a disk, and the corresponding disk radius is calculated using the graph Laplacian matrix. Then, by judging the positional relationship between the disks between graph nodes (the magnitude of the distance and the sum of the radii, and whether they overlap), positive and negative sample frame pairs are accurately filtered. This target sampler does not require random sampling or a fixed time window and can filter samples based on the true inter-frame correlation characteristics, effectively improving the semantic relevance and selection efficiency of the samples.

[0030] The set of positive sample frames consists of several pairs of video frames selected by the target sampler, representing combinations of video frames with strong inter-frame semantic relationships. The selection logic is as follows: the graph Laplacian matrix contains quantified information about inter-frame relationships; based on this matrix, the center position and radius of the disk corresponding to each graph node are calculated. If the distance between the centers of the disks of two graph nodes is greater than the sum of their radii, it indicates a high degree of feature similarity between the video frames corresponding to the two nodes, thus determining their semantic similarity and strong association, and including this pair of frames in the positive sample frame pair set. The set of negative sample frames consists of several pairs of video frames selected by the target sampler, representing combinations of video frames with significant inter-frame semantic differences. The selection logic is as follows: the graph Laplacian matrix contains quantified information about inter-frame relationships; based on this matrix, the center position and radius of the disk corresponding to each graph node are calculated. First, graph nodes whose disks do not overlap with any graph node in the positive sample pair are identified. This non-overlapping relationship indicates low feature similarity between the corresponding video frames, thus determining significant semantic differences. Then, combinations of any graph node in the positive sample pair with these non-overlapping graph nodes are selected, and the corresponding video frame pairs are used as negative sample frame pairs.

[0031] Step S204: Calculate the temporal-aware contrast loss based on positive sample frame pairs and negative sample frame pairs.

[0032] Among them, the temporal-aware contrastive loss refers to a composite loss function that fuses the standard contrastive loss and the temporal regularization term. Its core function is to quantify the semantic similarity of positive sample frame pairs and the temporal differences of negative sample frame pairs, providing a precise basis for optimizing feature extractor parameters. Its computational logic is progressive: first, it calculates the standard contrastive loss based on the semantic similarity of positive sample frame pairs to enhance the semantic discrimination ability between frames; then, it constructs a temporal regularization term using the squared Euclidean distance of the feature vectors of negative sample frame pairs to explicitly constrain the temporal relationship between frames; finally, it weights and fuses the two, ensuring that the loss function both guarantees semantic discrimination and strengthens temporal correlation, perfectly adapting to the dual semantic and temporal requirements of video keyframe extraction.

[0033] Step S205 involves repeatedly performing the following operations: updating the parameters of the feature extractor using time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive and negative sample frame pairs, and recalculating the time-aware contrastive loss, until the time-aware contrastive loss converges.

[0034] In this context, fine-tuning the feature vector refers to the process of re-extracting video frame feature vectors after updating the feature extractor parameters through backpropagation using time-aware contrastive loss during the iteration process. Compared to the initial feature vector, its representational ability is improved through iterative optimization, enabling it to more accurately capture semantic differences and relationships between frames.

[0035] Among them, the convergence of temporal-aware contrastive loss means that the temporal-aware contrastive loss value calculated during the iteration process meets the preset convergence conditions, such as the loss value being less than the preset threshold for multiple consecutive rounds, or the change in the loss value being less than the threshold. This indicates that the feature extractor parameters have been optimized to a better state, and the frame-level feature representation and temporal relationship modeling have reached stable accuracy.

[0036] Step S206: After the time-aware contrastive loss converges, reconstruct the directed graph, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix.

[0037] In this process, after the temporal-aware contrastive loss converges, the reconstructed directed graph recalculates edge weights based on the optimized fine-tuned feature vectors, and then combines these with the original graph nodes to construct a new directed graph. Compared to the initial directed graph, its edge weights more accurately characterize the semantic relationships and temporal dependencies between frames, and the graph structure better reflects the real characteristics of the video content, providing optimized graph structure support for calculating the target graph Laplacian matrix and subsequent spectral clustering.

[0038] The target graph Laplacian matrix is ​​the final graph Laplacian matrix calculated based on the degree matrix and weight matrix of the reconstructed directed graph. It is a precise quantitative representation of the global correlation between video frames and integrates frame-level features and temporal information after iterative optimization.

[0039] Step S207: Perform spectral clustering on the Laplacian matrix of the target image to obtain the set of keyframes of the target video.

[0040] Among them, spectral clustering is a clustering algorithm based on the eigenvalue decomposition of the graph Laplacian matrix. By calculating eigenvalues ​​and eigenvectors of the target graph Laplacian matrix, it maps high-dimensional graph nodes to a low-dimensional space, and then performs clustering based on the low-dimensional features. This algorithm can fully utilize the global association information in the graph structure, clustering semantically similar and temporally related frames into one class, providing core algorithmic support for selecting keyframes from the clustering results.

[0041] The keyframe set is a collection of the most semantically representative video frames selected from each cluster after spectral clustering of the Laplacian matrix of the target graph. The frames in this set not only fully preserve the core semantic information of the target video but also effectively eliminate redundant frames. Their number is far less than the original video frame sequence, thus ensuring semantic integrity while reducing storage and computational costs.

[0042] This application's embodiments achieve a comprehensive improvement in keyframe extraction performance by constructing a dynamic graph structure and an iterative optimization mechanism. A pre-trained feature extractor obtains frame-level feature vectors, and an initial directed graph is constructed based on feature similarity. The graph Laplacian matrix characterizes inter-frame relationships, and a target sampler filters positive and negative sample frame pairs, effectively avoiding the redundancy problem of random sampling and improving sample semantic diversity and selection efficiency. Temporal-aware contrastive loss drives iterative updates of feature extractor parameters, combined with dynamic adjustment of edge weights and directed graph reconstruction, explicitly modeling complex spatiotemporal dependencies between frames. This accurately distinguishes the local contextual differences between keyframes and non-keyframes, compensating for the shortcomings of insufficient temporal modeling. During the iteration process, the system dynamically adapts to changes in video content, fully leveraging the advantages of graph structures in modeling inter-frame relationships. Finally, spectral clustering of the target graph Laplacian matrix filters keyframes, significantly reducing storage and computation costs while ensuring the semantic integrity of the video, and significantly improving the accuracy and adaptability of keyframe extraction.

[0043] In some optional implementations of this embodiment, step 202, based on the feature vector sequence, calculates the edge weights connecting any two graph nodes, and constructs an initial directed graph based on multiple graph nodes and edge weights, specifically including the following steps: Obtain the preset balance coefficient, and determine the temporal Gaussian kernel bandwidth based on the total duration of the target video; based on the feature vector sequence, balance coefficient, and temporal Gaussian kernel bandwidth, use the preset first edge weight algorithm to calculate the edge weight of the edge connecting any two graph nodes; retain the edges whose edge weights are greater than the preset weight threshold, and construct the initial directed graph based on the retained edges and graph nodes.

[0044] The balance coefficient is a pre-defined hyperparameter used to balance the proportions of temporal proximity and semantic similarity in edge weights. Its value ranges from [0,1], with a larger value indicating a higher proportion of temporal proximity in the edge weights, and vice versa. The temporal Gaussian kernel bandwidth is a parameter dynamically determined based on the total duration of the target video, used to quantify the influence of inter-frame temporal distance. A larger temporal Gaussian kernel bandwidth value indicates a weaker constraint of temporal distance on edge weights, while a smaller value indicates a higher proportion of edge weights for frames with similar temporal distances.

[0045] The first edge weight algorithm is a formula for calculating edge weights that integrates temporal proximity and semantic similarity, as shown in the formula. w ij This represents the weight of the edge between the i-th graph node and the j-th graph node in the initial directed graph. For balance coefficient, For the time-series Gaussian kernel bandwidth, t i t j Let f represent the temporal positions (e.g., frame timestamps) of the video frames corresponding to the i-th and j-th graph nodes in the target video, respectively, used to calculate the temporal distance between the two frames. i f j These represent the feature vectors of the video frame corresponding to the i-th and j-th graph nodes, respectively. They are the core data for quantifying the semantic similarity between frames, and are expressed using cosine similarity (cos(f)). i ,f j ) Calculate the semantic association strength.

[0046] The preset weight threshold is a pre-defined numerical standard used to filter valid edges, retaining only edges with weights greater than this threshold. Its function is to eliminate redundant edges with extremely weak connections in the graph structure, thus simplifying the graph's topology.

[0047] In one example, taking the extraction of keyframes from bank counter business monitoring video as a scenario, for the monitoring video of a customer opening an account at a bank counter (target video, total duration 120 seconds), the preset balance coefficient is first obtained. =0.6, the temporal Gaussian kernel bandwidth is determined based on the total video duration. =6 seconds (120 seconds × 5%). Then, the feature vector sequence of each video frame is obtained through a pre-trained feature extractor, and the first-side weight algorithm is used. Calculate the edge weights between graph nodes (corresponding to video frames), where t i t j f is the timestamp of the frame. i f j This is the feature vector. Then, edges with weights greater than 0.4 (a preset weight threshold) are retained, and these edges are combined with graph nodes to construct an initial directed graph.

[0048] This embodiment calculates edge weights between graph nodes by using a preset balance coefficient and temporal Gaussian kernel bandwidth, combined with a first edge weighting algorithm to fuse the temporal proximity and semantic similarity of frames. Edges with weights greater than a threshold are then selected to construct an initial directed graph. In this process, the balance coefficient can flexibly adjust the proportion of temporal and semantic associations, the temporal Gaussian kernel bandwidth adapts to temporal modeling of videos of different durations, the first edge weighting algorithm implements joint encoding of spatiotemporal information, and the weight threshold removes redundant edges to simplify the graph structure. The final constructed directed graph retains the core spatiotemporal associations between frames while avoiding interference from invalid edges.

[0049] In some alternative implementations, step 203, calculating the initial directed graph Laplacian matrix, specifically includes the following steps: Based on the edge weights corresponding to the edges of the initial directed graph, construct the weight matrix of the initial directed graph; based on the weight matrix, construct the degree matrix, which is a diagonal matrix; perform matrix subtraction on the degree matrix and the weight matrix to obtain the graph Laplacian matrix of the initial directed graph.

[0050] The weight matrix is ​​a square matrix constructed based on the edge weights of all edges in the initial directed graph. Its dimension is the same as the number of graph nodes (corresponding to video frames). The element in the i-th row and j-th column of the matrix is ​​the weight of the edge between the i-th and j-th graph nodes (if there is no edge connection between the two nodes, the value is 0). The degree matrix is ​​a diagonal matrix constructed based on the weight matrix. Its dimension is the same as the weight matrix, but only the elements on the diagonal have non-zero values. The element in the i-th row and i-th column is the sum of all elements in the i-th row of the weight matrix (i.e., the sum of all edge weights corresponding to the i-th graph node), and the elements in all other positions are 0.

[0051] In one example, taking keyframe extraction from a surgical procedure video as the scenario, for the monitoring video (target video) of surgery A, an initial directed graph has been constructed (nodes correspond to surgical frames, and edge weights integrate the temporal continuity of the operation actions and the semantic similarity of instrument features). First, based on the edge weights of the initial directed graph, a weight matrix of dimension 1000×1000 is constructed (1000 represents the number of surgical frames). The element in the i-th row and j-th column of the matrix is ​​the edge weight between the i-th and j-th frames (e.g., the weight of continuous frames of action B is 0.8). Then, using the sum of the elements in each row of the weight matrix as the diagonal elements, a degree matrix is ​​constructed (e.g., the diagonal element of the degree matrix corresponding to a certain separated action frame is 5.2, corresponding to the sum of the weights of its associated edges). Finally, by subtracting the weight matrix from the degree matrix, the graph Laplacian matrix of the initial directed graph is obtained.

[0052] This application's embodiments construct a weight matrix based on the edge weights of the initial directed graph, comprehensively recording the association strength between all nodes. Then, using the sum of elements in each row of the weight matrix as the diagonal elements, a degree matrix is ​​constructed to quantify the total node association. Finally, the graph Laplacian matrix is ​​obtained by subtracting the degree matrix from the weight matrix. In this process, the weight matrix retains the topological and weight information of the graph structure, while the degree matrix supplements the global association features of the nodes. The resulting graph Laplacian matrix can simultaneously characterize the local associations and global distribution of nodes, providing accurate inter-frame relationship quantification for the target sampler and solving the problem of insufficient utilization of graph structure information in existing methods.

[0053] In some optional implementations, step S203, based on the graph Laplacian matrix, uses a target sampler to select multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence, specifically including the following steps: Based on the graph Laplacian matrix, with each graph node of the initial directed graph as the center of the disk, the center position and radius of the disk are calculated. Based on the center position and radius of the disk, graph node pairs that satisfy the condition that the distance between the centers of the two graph nodes is greater than the sum of the radii of the two disks are selected, and the video frame pairs corresponding to the graph node pairs are taken as positive sample pairs. Based on the center position and radius of the disk, from the graph nodes whose disks do not overlap with any graph node in the positive sample pairs, combinations of any graph node in the positive sample pairs and their corresponding non-overlapping graph nodes are selected, and the video frame pairs corresponding to the combinations are taken as negative sample pairs.

[0054] The center of the disk corresponds to element c in the graph Laplacian matrix. i =L ii (i.e., the element in the i-th row and i-th column of the graph Laplacian matrix) is the center position parameter that maps each graph node of the initial directed graph to the center of the "disk" in the disk. It quantifies the local features of the corresponding graph node (video frame) in the graph Laplacian matrix and is the core position benchmark for subsequent selection of sample frame pairs. Disk radius , which is the sum of the absolute values ​​of all elements in the i-th row of the graph Laplacian matrix excluding the diagonal elements, is the radius parameter of the "disk" in the disk. It quantifies the association range of the corresponding graph node (video frame) in the graph Laplacian matrix; the larger the radius, the stronger the global association of the node.

[0055] In this context, a graph node pair where the center-to-center distance between the disks corresponding to two graph nodes is greater than the sum of the radii of the two disks refers to a node combination where the center-to-center distance between the disks corresponding to the two graph nodes satisfies the condition "center-to-center distance > sum of radii". These node pairs exhibit maximum difference in the representation space of the graph Laplacian matrix, and their corresponding video frames have strong semantic connections, making them a core criterion for selecting positive sample pairs.

[0056] In this context, non-overlapping graph nodes refer to graph nodes whose corresponding disks do not spatially overlap with those of a node in a positive sample pair; that is, the disk of this node has no intersection with the disk of the positive sample node. These nodes exhibit significant semantic differences from positive sample nodes in the representation space of the graph Laplacian matrix, serving as a source for selecting negative samples. Combinations refer to selecting a node from the non-overlapping graph nodes and forming a node pair with any node in the positive sample pair. The corresponding video frame pairs of this combination exhibit significant semantic differences and represent a specific form of negative sample pairs.

[0057] In one example, taking the extraction of keyframes from a property insurance fire scene video as the scenario, and focusing on the monitoring video (target video) of the fire in warehouse A, an initial directed graph has been constructed and the graph Laplacian matrix L has been obtained. The core step of the target sampler is first executed: based on the graph Laplacian matrix L, the center c of the disk is calculated. i =L ii (i.e., the diagonal element of the i-th row and i-th column of the graph Laplacian matrix L), disk radius (That is, the sum of the absolute values ​​of all elements in the i-th row of the graph Laplace matrix L, excluding the diagonal elements). According to the formula... Filter graph node pairs that meet this condition. The corresponding video frame pairs (such as consecutive frames of flame spread) are used as positive sample pairs. Negative samples are randomly selected from nodes that do not overlap with the positive sample disk, maintaining a ratio of 3:1 between negative and positive samples, and the corresponding video frame pairs (such as empty warehouse scene frames before the fire) are used as negative sample pairs.

[0058] This application's embodiments are based on the graph Laplacian matrix. The center and radius of the disk are calculated using graph nodes as a reference. Positive sample pairs are selected based on the condition that "center spacing > sum of radii," ensuring maximum semantic difference in the representation space. Then, combinations of nodes that do not overlap with the positive sample disks are selected as negative sample pairs. In this process, the disk center and radius quantify the local features and association range of the nodes. The selection rule relies on the topological information of the graph structure to ensure the semantic similarity of positive samples, and strengthens the semantic difference of negative samples through the non-overlapping disk condition. This avoids the redundancy and semantic bias of random sampling, improves the efficiency and accuracy of sample selection, and solves the problem of inefficient sample selection in existing methods.

[0059] In some optional implementations, step S204, based on positive sample frame pairs and negative sample frame pairs, calculates the temporal-aware contrastive loss, specifically including the following steps: Based on the feature vector sequence, the first similarity between each positive sample frame pair is calculated; the first similarity and a preset temperature parameter are normalized exponentially to obtain the similarity index value corresponding to each positive sample frame pair; for each positive sample frame, the second similarity between each positive sample frame and all samples corresponding to each positive sample frame is calculated; the second similarity and the temperature parameter are normalized exponentially and summed to obtain the sum of similarity indices corresponding to each positive sample frame; the standard contrast loss is obtained based on the similarity index value and the sum of the similarity indices of the corresponding positive sample frames; the squared Euclidean distance between the feature vectors of each negative sample frame pair is calculated, and the sum of all squared Euclidean distances is obtained to obtain the global sum of squared Euclidean distances for negative sample frame pairs; the preset regularization term intensity parameter is weighted and multiplied by the global sum of squared Euclidean distances to obtain the temporal regularization term; the standard contrast loss and the temporal regularization term are fused to obtain the temporal-aware contrast loss.

[0060] The first similarity is a quantified value of the matching degree of feature vectors between two frames in a positive sample frame pair. For example, it can be calculated using cosine similarity and directly reflects the semantic correlation between the two frames. For consecutive frames with the same action, the first similarity will be relatively high. The preset temperature parameter is a small constant (e.g., 0.07) set in advance to adjust the discriminative power of the similarity values. The smaller the temperature parameter, the more obvious the numerical difference between similar and dissimilar frames, allowing effective samples to be given more weight in the loss calculation and reducing interference from noisy samples. The normalization exponent operation combines the first or second similarity with the temperature parameter, converting the original similarity into a probabilistic value through exponential calculation. It amplifies the values ​​of high-similarity samples and reduces the values ​​of low-similarity samples, making the results more suitable for the loss calculation logic of contrastive learning. The similarity index value is the result of normalizing the first similarity of a single positive sample frame pair after the exponential operation. The closer the semantic correlation of the positive sample frame pair and the smaller the temperature parameter, the larger this value, directly affecting the weight of the positive sample frame pair in the loss calculation.

[0061] The second similarity is calculated for a single positive sample frame, determining the feature matching degree between it and all its corresponding related samples (including frames within the same positive sample pair and frames associated with the same semantic scene). It covers the global association of the positive sample frame, avoiding bias caused by considering only local factors in the loss calculation. The sum of similarity indices refers to the sum of the normalized exponents of the second similarity between a single positive sample frame and all its corresponding samples. The standard contrastive loss is the loss calculated based on the ratio of the similarity index value to the sum of the similarity indices. The closer the semantic association of the positive sample frame pair, the smaller this loss value; conversely, the weaker the semantic association, the larger the loss value. Its core function is to teach the feature extractor to strengthen the similarity of positive sample frame pairs.

[0062] The squared Euclidean distance is the result of squaring the Euclidean distance between the feature vectors of the two frames in a negative sample frame pair. It reflects the semantic difference between the negative sample frame pairs; the more semantically unrelated the two frames are, the larger this value, making it a key basis for constructing the temporal regularization term. The global sum of squared Euclidean distances refers to the sum of the squared Euclidean distances of all negative sample frame pairs. It reflects the overall semantic difference between all negative sample frame pairs and is the global input data for calculating the temporal regularization term. The regularization term strength parameter is a pre-set constant (e.g., 0.1) used to adjust the proportion of the temporal regularization term in the total loss. The larger this parameter, the stronger the temporal constraint on the negative sample frame pairs, and it can be flexibly adjusted according to the temporal complexity of the video. The temporal regularization term is the result of multiplying the regularization term strength parameter by the global sum of squared Euclidean distances. If the semantic difference between all negative sample frame pairs is greater (the higher the global sum), the smaller the value of this regularization term, and the lighter the "penalty" on the total loss; conversely, the penalty is heavier. The core is to make the negative sample frame pairs more dispersed at the feature level.

[0063] In one example, after the target sampler selects the set of positive sample pairs P and the set of negative sample pairs N based on the graph Laplacian matrix, the time-aware contrastive loss can be expressed by the formula: The calculation is as follows. Specifically, for frame pairs in the set of positive sample pairs P, the first similarity s between the i-th sample and the j-th sample is calculated. ij Combined with preset temperature parameters =0.07, through exp(s ij / Perform normalized exponent calculation to obtain the similarity index value of the positive sample pair; then, for the positive sample frame, calculate its second similarity s with all corresponding samples. ik ,pass Perform the normalized index calculation and summation to obtain the total similarity index. Substitute the above result into the first term of the formula. We obtain the standard contrastive loss to enhance the semantic similarity representation of collision frames. Simultaneously, for the negative sample pair set N, we calculate the squared Euclidean distance between the feature vectors of the i-th sample and the j-th sample. After summing all the results, the intensity parameter of the preset regularization term is combined. =0.1, substitute into the second term of the formula This yields the temporal regularization term. Finally, the standard contrast loss is fused with the temporal regularization term to obtain the temporal-aware contrast loss. .

[0064] This application first calculates the first similarity of positive sample frame pairs based on the feature vector sequence, and then performs a normalized exponential operation using a temperature parameter to obtain a similarity index value. Next, it sums the exponential operations of the second similarity to obtain a global benchmark, thus constructing a standard contrastive loss that accurately quantifies the semantic association strength of positive sample frame pairs. Simultaneously, it calculates and sums the squared Euclidean distances of the feature vectors of negative sample frame pairs, and combines this with a regularization term strength parameter to obtain a temporal regularization term. Finally, it fuses both to form a temporal-aware contrastive loss. This process strengthens the semantic similarity representation of positive samples through the standard contrastive loss and explicitly constrains the temporal differences of negative samples through the temporal regularization term. This allows the loss function to simultaneously consider semantic distinction and temporal association, providing a more accurate basis for the iterative optimization of the feature extractor and effectively compensating for the shortcomings of existing methods in temporal modeling.

[0065] In some optional implementations, step S205 involves iteratively executing the following operations: updating the parameters of the feature extractor using time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive and negative sample frame pairs, and recalculating the time-aware contrastive loss, until the time-aware contrastive loss converges. Specifically, this includes the following steps: Determine whether the temporal-aware contrastive loss meets the preset convergence condition. If not, calculate the gradient of the feature vector of the temporal-aware contrastive loss with respect to each video frame. Based on gradients, the gradient descent method is used to update the parameters of the feature extractor; The feature extractor with updated parameters is used to re-extract the feature vector of each video frame, resulting in a fine-tuned feature vector sequence for each video frame. Based on the fine-tuned feature vector sequence, the third similarity between any two graph nodes is calculated. The third similarity is then adjusted based on preset adjustment parameters to obtain new initial edge weights for connecting any two graph nodes. For the first iteration, the preset initial neighborhood radius is determined as the neighborhood radius of the previous iteration. For iterations other than the first iteration, the neighborhood radius calculated in the previous iteration is determined as the neighborhood radius of the previous iteration of the current iteration. Based on the initial edge weights, the neighborhood radius of the previous iteration, the preset smoothing coefficient, and the total number of edges connecting any two graph nodes, the neighborhood radius of the first iteration is calculated using a preset neighborhood radius time-adaptive adjustment formula. Based on the neighborhood radius of the first iteration, the initial edge weights are sparsified to obtain a set of sparsified edge weights. Based on the edges corresponding to the sparsified edge weight set and multiple graph nodes, a new directed graph is constructed, and the graph Laplacian matrix of the new directed graph is calculated. Based on the new directed graph Laplacian matrix, a sampler is used to filter multiple new positive sample frame pairs and new negative sample frame pairs in the current iteration round. Based on multiple new positive sample frame pairs and new negative sample frame pairs, calculate the new time-aware contrast loss for the current iteration round, return to the execution loss convergence judgment step, until there is a new iteration round whose time-aware contrast loss satisfies the convergence condition.

[0066] The convergence condition is a pre-defined criterion used to determine whether the temporal-aware contrastive loss has reached a stable state, such as the loss value fluctuating less than a preset threshold for multiple consecutive rounds, or the loss value falling below a target threshold. The gradient refers to the partial derivative of the temporal-aware contrastive loss with respect to the feature vector of each video frame, quantifying the direction and magnitude of the loss value's change with the feature vector. It is the core basis for updating the feature extractor parameters; the direction of the gradient corresponds to the direction of the fastest loss descent. Gradient descent is a parameter optimization algorithm that gradually reduces the temporal-aware contrastive loss by updating the feature extractor parameters in the opposite direction of the gradient (the amount of parameter update is related to the gradient magnitude and learning rate).

[0067] The fine-tuned feature vector sequence refers to the set of feature vectors re-extracted from the video frame sequence using a feature extractor with updated parameters. It is the output of the feature extractor after gradient optimization, and compared to the initial feature vectors, it more accurately characterizes the semantics and temporal relationships of the video frames. The initial edge weights are the edge weights between graph nodes calculated based on the fine-tuned feature vector sequence.

[0068] The initial neighborhood radius, used in the first iteration (updated in subsequent iterations based on the previous radius), controls the association range of graph nodes and is a key parameter for sparsification and edge weight adjustment. The neighborhood radius time-adaptive adjustment formula is an algorithm that calculates the current neighborhood radius based on the previous iteration's neighborhood radius, smoothing coefficient, total number of edges, and frame temporal distance. It allows the neighborhood radius to dynamically adjust with the video's temporal rhythm, balancing the temporal coherence of the graph structure with the adaptability to changes in video content. Sparsification is an operation that filters the initial edge weights based on the neighborhood radius. Its purpose is to remove redundant edges, simplify the graph structure, reduce the computational overhead of subsequent graph Laplacian matrix calculations and sample selection, while preserving the core association information of the graph.

[0069] In one example, the scenario is keyframe extraction from a car accident scene video. The target video is a 90-second surveillance video of a highway rear-end collision (5400 frames). After calculating the initial temporal-aware contrastive loss, it is determined that it does not meet the convergence condition of loss fluctuation being less than 0.001 for three consecutive rounds. The gradient of the feature vector with respect to the loss is calculated, and the parameters of the pre-trained feature extractor are updated using gradient descent. The updated feature extractor is then used to re-extract frame features, resulting in a fine-tuned feature vector sequence. Based on formula c... Calculate the third similarity (normalized cosine similarity + 1 to ensure non-negative weights) to obtain the new initial edge weights. Where f i The feature vector representing the i-th graph node (corresponding to the video frame) is output by the feature extractor, f. j Let represent the feature vector of the j-th graph node (corresponding to the video frame). The initial neighborhood radius ρ is taken in the first iteration. t =4.5 seconds, based on the neighborhood radius time-adaptive formula Calculate the radius of the current round's neighborhood. =0.9 is the smoothing coefficient, and |E| is the total number of edges. Let t represent the ij-th edge in the graph, corresponding to the connection between the ith graph node and the j-th graph node. i t represents the temporal position (e.g., timestamp) of the video frame corresponding to the i-th graph node. j This represents the temporal position of the video frame corresponding to the j-th graph node. The top-10 high-weight edges of each node are retained to achieve sparsity, resulting in a sparse edge weight set. A new directed graph is constructed based on the sparse edge weights, the graph Laplacian matrix is ​​calculated, and a target sampler is used to filter new positive sample pairs (e.g., consecutive frames at the moment of collision) and negative sample pairs (e.g., frames of the empty scene before the accident). The temporal-aware contrast loss is recalculated, and the convergence determination step is returned.

[0070] This application's embodiments determine whether iteration should continue based on the loss convergence condition. If convergence fails, the feature extractor parameters are updated using gradient descent based on the loss gradient to obtain a more accurate fine-tuned feature vector. Edge weights are then adjusted using a third similarity (combined with the normalized cosine formula), and the neighborhood radius is dynamically updated using a temporally adaptive formula (fusing smoothing coefficients and the total number of edges). After sparsification, high-weight edges are retained to construct a new graph. In this process, gradient updates ensure the feature vectors align with semantic temporal requirements, dynamic edge weights adapt to feature distribution shifts, neighborhood radius adjustments balance temporal coherence and video rhythm, and sparsification reduces computational overhead. The final graph structure is dynamically optimized with iteration, resulting in more accurate sample selection and solving the problem of existing methods' static graph construction being unable to adapt to content changes.

[0071] In some optional implementations, step 207, performing spectral clustering on the Laplacian matrix of the target graph to obtain a set of keyframes for the target video, specifically includes the following steps: The Laplacian matrix of the target graph is subjected to eigenvalue decomposition to obtain multiple eigenvalues ​​and corresponding eigenvectors. The eigenvectors corresponding to the k smallest eigenvalues ​​are selected, where k is a positive integer. The eigenvectors corresponding to the k smallest eigenvalues ​​are concatenated column-wise to form an eigenvector concatenation matrix. Each row of the eigenvector concatenation matrix is ​​normalized to obtain a normalized eigenvector matrix. A preset clustering algorithm is used to cluster the normalized eigenvector matrix, dividing the multiple graph nodes into k clusters to obtain the clustering results. Based on the clustering results, the central graph node within each cluster is determined, and the video frame corresponding to the central graph node is used as a keyframe to obtain the keyframe set of the target video.

[0072] Here, multiple eigenvalues ​​refer to a set of scalar values ​​obtained after eigenvalue decomposition of the target graph Laplacian matrix. Each eigenvalue corresponds to an eigenvector, reflecting the scaling degree of the graph Laplacian matrix in different dimensions. The corresponding eigenvector is a vector that corresponds one-to-one with the eigenvalue and is one of the output results of the eigenvalue decomposition of the graph Laplacian matrix. The k smallest eigenvalues ​​refer to the k smallest values ​​selected from all eigenvalues ​​(k is a preset positive integer, corresponding to the number of targets in the keyframe). The eigenvector concatenation matrix is ​​a matrix formed by arranging the eigenvectors corresponding to the k smallest eigenvalues ​​column by column, with the number of rows equal to the number of graph nodes and the number of columns equal to k.

[0073] The normalized eigenvector matrix refers to the matrix obtained by normalizing each row of the concatenated eigenvector matrix (corresponding to the k-dimensional distribution coordinates of a single graph node). Clustering algorithms, such as K-Means, are algorithms that partition the normalized eigenvector matrix into clusters. Their core principle is to group graph nodes that are close together in the feature space into the same cluster, achieving semantic grouping of graph nodes. The k clusters refer to the k node groups output by the clustering algorithm. Graph nodes within each cluster are closely distributed in the feature space, corresponding to a set of semantically closely related frames in the video. The clustering result refers to the result obtained after the clustering algorithm completes the cluster partitioning, including the cluster label to which each graph node belongs, clearly defining the semantic grouping relationship of different graph nodes. The cluster center graph node refers to the graph node in each cluster that is closest to other nodes (e.g., the node corresponding to the mean of nodes in the feature space within the cluster; its corresponding video frame is the semantic core of that cluster and is therefore selected as a keyframe).

[0074] In one example, the scenario focuses on extracting keyframes from a video of an outdoor fall injury incident covered by personal accident insurance. The target video is a surveillance video of a tourist falling and getting injured in a scenic area (60 seconds long, 3600 frames). The preset number of keyframes is k=5. Eigenvalue decomposition is performed on the Laplacian matrix of the target graph, yielding 3600 eigenvalues ​​and their corresponding vectors. The eigenvectors corresponding to the 5 smallest eigenvalues ​​are selected. These 5 eigenvectors are then concatenated column-wise to form a eigenvector concatenation matrix (3600×5 dimensions). Normalization is performed on each row of the matrix to obtain a normalized eigenvector matrix. The K-Means clustering algorithm is used to cluster the normalized eigenvector matrix, dividing the 3600 graph nodes into 5 clusters. The mean eigenvalue of each node within each cluster is calculated, and the node closest to the mean is selected as the cluster center node. The corresponding video frame is the keyframe.

[0075] This embodiment first performs eigenvalue decomposition on the Laplacian matrix of the target graph, selecting the eigenvectors corresponding to the k smallest eigenvalues ​​(small eigenvalue vectors better characterize the node cluster distribution), and concatenates them column-wise into an eigenvector concatenation matrix. Then, each row of the matrix is ​​normalized (to eliminate dimensionality differences), and the nodes are divided into k clusters using a clustering algorithm. Finally, the frame corresponding to the center node within each cluster is selected as the keyframe. In this process, small eigenvalue vectors ensure the semantic relevance of the cluster division, normalization improves clustering accuracy, and center node selection ensures the representativeness of the keyframes. This solves the problem of insufficient semantic representativeness in frame selection using traditional methods, allowing the extracted keyframes to cover the core semantics while avoiding redundancy and repetition.

[0076] It should be emphasized that, to further ensure the privacy and security of the aforementioned video frame sequence, feature vector sequence, initial directed graph, graph Laplacian matrix, multiple positive sample frame pairs, multiple negative sample frame pairs, time-aware contrast loss, target graph Laplacian matrix, and keyframe set, these components can also be stored in a blockchain node.

[0077] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0078] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0079] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0080] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a self-supervised video keyframe extraction device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0081] like Figure 3 As shown, the self-supervised video keyframe extraction device 400 of this embodiment includes: an acquisition module 401, a construction module 402, a filtering module 403, a loss calculation module 404, a loop execution module 405, a matrix calculation module 406, and a clustering module 407. Wherein: The acquisition module 401 is used to acquire the video frame sequence of the target video. Through a pre-trained feature extractor, the feature vector corresponding to each video frame in the video frame sequence is extracted to obtain the feature vector sequence. The construction module 402 is used to determine multiple graph nodes based on the video frame sequence, calculate the edge weights of any two graph nodes connected by an edge based on the feature vector sequence, and construct an initial directed graph based on the multiple graph nodes and edge weights. The filtering module 403 is used to calculate the graph Laplacian matrix of the initial directed graph, and based on the graph Laplacian matrix, to filter out multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence through the target sampler. Loss calculation module 404 is used to calculate time-aware contrastive loss based on positive sample frame pairs and negative sample frame pairs; The loop execution module 405 is used to repeatedly execute the following operations: updating the parameters of the feature extractor using time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive sample frame pairs and negative sample frame pairs, and recalculating the time-aware contrastive loss until the time-aware contrastive loss converges. The matrix calculation module 406 is used to reconstruct the directed graph after the time-aware contrastive loss converges, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix. Clustering module 407 is used to perform spectral clustering on the Laplacian matrix of the target graph to obtain the set of keyframes of the target video.

[0082] This application's embodiments achieve a comprehensive improvement in keyframe extraction performance by constructing a dynamic graph structure and an iterative optimization mechanism. A pre-trained feature extractor obtains frame-level feature vectors, and an initial directed graph is constructed based on feature similarity. The graph Laplacian matrix characterizes inter-frame relationships, and a target sampler filters positive and negative sample frame pairs, effectively avoiding the redundancy problem of random sampling and improving sample semantic diversity and selection efficiency. Temporal-aware contrastive loss drives iterative updates of feature extractor parameters, combined with dynamic adjustment of edge weights and directed graph reconstruction, explicitly modeling complex spatiotemporal dependencies between frames. This accurately distinguishes the local contextual differences between keyframes and non-keyframes, compensating for the shortcomings of insufficient temporal modeling. During the iteration process, the system dynamically adapts to changes in video content, fully leveraging the advantages of graph structures in modeling inter-frame relationships. Finally, spectral clustering of the target graph Laplacian matrix filters keyframes, significantly reducing storage and computation costs while ensuring the semantic integrity of the video, and significantly improving the accuracy and adaptability of keyframe extraction.

[0083] In one embodiment, the construction module 402 includes: The acquisition submodule is used to obtain the preset balance coefficient and determine the temporal Gaussian kernel bandwidth based on the total duration of the target video. The weight calculation submodule is used to calculate the edge weight between any two graph nodes based on the feature vector sequence, balance coefficient and temporal Gaussian kernel bandwidth, using a preset first edge weight algorithm. The first construction submodule is used to retain edges whose weights are greater than a preset weight threshold, and to construct an initial directed graph based on the retained edges and graph nodes.

[0084] In one embodiment, the screening module 403 includes: The second construction submodule is used to construct the weight matrix of the initial directed graph based on the edge weights corresponding to the edges of the initial directed graph. The third construction submodule is used to construct the degree matrix based on the weight matrix. The degree matrix is ​​a diagonal matrix. The first operation submodule is used to perform matrix subtraction on the degree matrix and the weight matrix to obtain the graph Laplacian matrix of the initial directed graph.

[0085] In one embodiment, the screening module 403 includes: The computation submodule is used to calculate the center position and radius of the disk based on the graph Laplacian matrix, with each graph node of the initial directed graph as the center of the disk. The first filtering submodule is used to filter graph node pairs based on the center position and radius of the disk, where the distance between the centers of the two graph nodes is greater than the sum of the radii of the two disks, and the video frame pairs corresponding to the graph node pairs are used as positive sample pairs. The selection submodule is used to select a combination of any graph node in the positive sample pair and its corresponding non-overlapping graph node from the graph nodes of the disk that do not overlap with any graph node in the positive sample pair, based on the center position and radius of the disk, and to take the video frame pair corresponding to the combination as the negative sample pair.

[0086] In one embodiment, the loss calculation module 404 includes: The first similarity calculation submodule is used to calculate the first similarity between each pair of positive sample frames based on the feature vector sequence; The second calculation submodule is used to perform normalized index calculation on the first similarity and the preset temperature parameter to obtain the similarity index value corresponding to each positive sample frame pair. The second similarity calculation submodule is used to calculate the second similarity between each positive sample frame and all samples corresponding to each positive sample frame for each positive sample frame; The third operation submodule is used to perform normalized exponent calculation on the second similarity and temperature parameters and sum them to obtain the sum of similarity indices corresponding to each positive sample frame. The resulting submodule is used to obtain the standard contrast loss based on the sum of the similarity index values ​​and the similarity indices of the corresponding positive sample frames; The summation submodule is used to calculate the squared Euclidean distance between the feature vectors of each negative sample frame pair, and sum all the squared Euclidean distances to obtain the global sum of the squared Euclidean distances of the negative sample frame pairs. The weighted submodule is used to perform a weighted multiplication of the preset regularization term strength parameters with the global sum of the squared Euclidean distances to obtain the time-series regularization term; The fusion submodule is used to fuse the standard contrast loss and the temporal regularization term to obtain the temporal-aware contrast loss.

[0087] In one embodiment, the loop execution module 405 includes: The judgment submodule is used to determine whether the temporal-aware contrast loss meets the preset convergence condition. If not, the gradient of the feature vector of the temporal-aware contrast loss with respect to each video frame is calculated. The update submodule is used to update the parameters of the feature extractor based on gradients and using the gradient descent method. The extraction submodule is used to re-extract the feature vector of each video frame using the updated feature extractor, and obtain the fine-tuned feature vector sequence for each video frame. The adjustment submodule is used to calculate the third similarity between any two graph nodes based on the fine-tuned feature vector sequence, and adjust the third similarity based on the preset adjustment parameters to obtain the new initial edge weights for connecting any two graph nodes. The determination submodule is used to determine the preset initial neighborhood radius as the neighborhood radius of the previous iteration for the first iteration round, and to determine the neighborhood radius calculated in the previous iteration round as the neighborhood radius of the previous iteration round for non-first iteration rounds. The neighborhood radius calculation submodule is used to calculate the neighborhood radius of the first iteration based on the initial edge weight, the neighborhood radius of the previous iteration, the preset smoothing coefficient, and the total number of edges connecting any two graph nodes, using a preset neighborhood radius time-adaptive adjustment formula. The sparsification submodule is used to sparsify the initial edge weights based on the neighborhood radius of the first iteration round, so as to obtain the sparsified edge weight set. The fourth construction submodule is used to construct a new directed graph based on the edges corresponding to the sparsified edge weight set and multiple graph nodes, and to calculate the graph Laplacian matrix of the new directed graph. The second filtering submodule is used to filter multiple new positive sample frame pairs and new negative sample frame pairs in the current iteration round based on the graph Laplacian matrix of the new directed graph and a sampler. The loop execution submodule is used to calculate the new time-aware contrast loss for the current iteration based on multiple new positive sample frame pairs and new negative sample frame pairs, and return to the execution loss convergence judgment step until a new iteration time-aware contrast loss satisfies the convergence condition.

[0088] In one embodiment, the clustering module 407 includes: The eigenvalue decomposition module is used to perform eigenvalue decomposition on the Laplacian matrix of the target graph to obtain multiple eigenvalues ​​and corresponding eigenvectors. The eigenvectors corresponding to the k smallest eigenvalues ​​are selected, where k is a positive integer. The concatenation module is used to concatenate the eigenvectors corresponding to the k smallest eigenvalues ​​into an eigenvector concatenation matrix by columns, and normalize each row of the eigenvector concatenation matrix to obtain a normalized eigenvector matrix. The clustering module is used to cluster the normalized feature vector matrix using a preset clustering algorithm, dividing multiple graph nodes into k clusters to obtain the clustering results; The determination module is used to determine the central graph node within each cluster based on the clustering results, and to use the video frame corresponding to the central graph node as the key frame to obtain the key frame set of the target video.

[0089] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0090] Computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0091] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0092] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions based on a self-supervised video keyframe extraction method. In addition, memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0093] In some embodiments, processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or to process data, such as executing computer-readable instructions based on a self-supervised video keyframe extraction method.

[0094] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 6 and other electronic devices.

[0095] This application's embodiments achieve a comprehensive improvement in keyframe extraction performance by constructing a dynamic graph structure and an iterative optimization mechanism. A pre-trained feature extractor obtains frame-level feature vectors, and an initial directed graph is constructed based on feature similarity. The graph Laplacian matrix characterizes inter-frame relationships, and a target sampler filters positive and negative sample frame pairs, effectively avoiding the redundancy problem of random sampling and improving sample semantic diversity and selection efficiency. Temporal-aware contrastive loss drives iterative updates of feature extractor parameters, combined with dynamic adjustment of edge weights and directed graph reconstruction, explicitly modeling complex spatiotemporal dependencies between frames. This accurately distinguishes the local contextual differences between keyframes and non-keyframes, compensating for the shortcomings of insufficient temporal modeling. During the iteration process, the system dynamically adapts to changes in video content, fully leveraging the advantages of graph structures in modeling inter-frame relationships. Finally, spectral clustering of the target graph Laplacian matrix filters keyframes, significantly reducing storage and computation costs while ensuring the semantic integrity of the video, and significantly improving the accuracy and adaptability of keyframe extraction.

[0096] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the self-supervised video keyframe extraction method described above.

[0097] This application's embodiments achieve a comprehensive improvement in keyframe extraction performance by constructing a dynamic graph structure and an iterative optimization mechanism. A pre-trained feature extractor obtains frame-level feature vectors, and an initial directed graph is constructed based on feature similarity. The graph Laplacian matrix characterizes inter-frame relationships, and a target sampler filters positive and negative sample frame pairs, effectively avoiding the redundancy problem of random sampling and improving sample semantic diversity and selection efficiency. Temporal-aware contrastive loss drives iterative updates of feature extractor parameters, combined with dynamic adjustment of edge weights and directed graph reconstruction, explicitly modeling complex spatiotemporal dependencies between frames. This accurately distinguishes the local contextual differences between keyframes and non-keyframes, compensating for the shortcomings of insufficient temporal modeling. During the iteration process, the system dynamically adapts to changes in video content, fully leveraging the advantages of graph structures in modeling inter-frame relationships. Finally, spectral clustering of the target graph Laplacian matrix filters keyframes, significantly reducing storage and computation costs while ensuring the semantic integrity of the video, and significantly improving the accuracy and adaptability of keyframe extraction.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0099] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0100] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A self-supervised video keyframe extraction method, characterized in that, Includes the following steps: The video frame sequence of the target video is obtained, and the feature vector corresponding to each video frame in the video frame sequence is extracted through a pre-trained feature extractor to obtain a feature vector sequence. Based on the video frame sequence, multiple graph nodes are determined. Based on the feature vector sequence, the edge weights connecting any two graph nodes are calculated. Based on the multiple graph nodes and the edge weights, an initial directed graph is constructed. Calculate the graph Laplacian matrix of the initial directed graph, and based on the graph Laplacian matrix, use a target sampler to select multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence; Based on the positive sample frame pair and the negative sample frame pair, calculate the time-aware contrast loss; The process involves iteratively executing the following operations: updating the parameters of the feature extractor using the time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive and negative sample frame pairs, and recalculating the time-aware contrastive loss, until the time-aware contrastive loss converges. After the time-aware contrastive loss converges, the directed graph is reconstructed, and the graph Laplacian matrix of the reconstructed directed graph is calculated to obtain the target graph Laplacian matrix. Spectral clustering is performed on the Laplacian matrix of the target graph to obtain the set of keyframes of the target video.

2. The method according to claim 1, characterized in that, The step of calculating the edge weights of the edges connecting any two graph nodes based on the feature vector sequence, and constructing an initial directed graph based on the plurality of graph nodes and the edge weights, specifically includes: Obtain the preset balance coefficient, and determine the temporal Gaussian kernel bandwidth based on the total duration of the target video; Based on the feature vector sequence, the balance coefficient, and the temporal Gaussian kernel bandwidth, the edge weight of any two graph nodes connected by the preset first edge weight algorithm is calculated. Edges with weights greater than a preset weight threshold are retained, and an initial directed graph is constructed based on the retained edges and the graph nodes.

3. The method according to claim 1, characterized in that, The step of calculating the graph Laplacian matrix of the initial directed graph specifically includes: Based on the edge weights corresponding to the edges of the initial directed graph, construct the weight matrix of the initial directed graph; Based on the weight matrix, a degree matrix is ​​constructed, wherein the degree matrix is ​​a diagonal matrix; The degree matrix and the weight matrix are subtracted to obtain the graph Laplacian matrix of the initial directed graph.

4. The method according to claim 1, characterized in that, The step of selecting multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence based on the graph Laplacian matrix and using a target sampler specifically includes: Based on the graph Laplacian matrix, with each graph node of the initial directed graph as the center of the disk, calculate the center position and radius of the disk; Based on the center position of the disk and the radius of the disk, graph node pairs that satisfy the condition that the center-to-center distance between the two graph nodes is greater than the sum of the radii of the two disks are selected, and the video frame pairs corresponding to the graph node pairs are taken as positive sample pairs. Based on the center position and radius of the disk, from the graph nodes that do not overlap with any graph node in the positive sample pair, a combination of any graph node in the positive sample pair and the corresponding non-overlapping graph node is selected, and the video frame pair corresponding to the combination is taken as the negative sample pair.

5. The method according to claim 1, characterized in that, The step of calculating the temporal-aware contrast loss based on the positive sample frame pair and the negative sample frame pair specifically includes: Based on the feature vector sequence, calculate the first similarity between each pair of positive sample frames; The first similarity and the preset temperature parameter are normalized to obtain the similarity index value corresponding to each positive sample frame pair. For each positive sample frame, calculate the second similarity between each positive sample frame and all samples corresponding to each positive sample frame; The second similarity and the temperature parameter are normalized and summed to obtain the sum of the similarity indices corresponding to each positive sample frame. The standard contrast loss is obtained by summing the similarity index values ​​with the similarity indices of the corresponding positive sample frames. Calculate the squared Euclidean distance between the feature vectors of each negative sample frame pair, sum all the squared Euclidean distances to obtain the global sum of the squared Euclidean distances for negative sample frame pairs; The time-series regularization term is obtained by multiplying the preset regularization term strength parameter with the global sum of the squared Euclidean distances. The standard contrast loss and the temporal regularization term are fused to obtain the temporal-aware contrast loss.

6. The method according to claim 1, characterized in that, The steps of iteratively executing the following operations—updating the parameters of the feature extractor using the time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive and negative sample frame pairs, and recalculating the time-aware contrastive loss—until the time-aware contrastive loss converges, specifically include: Determine whether the temporal-aware contrast loss meets the preset convergence condition; if not, calculate the gradient of the feature vector of the temporal-aware contrast loss with respect to each video frame. Based on the gradient, the parameters of the feature extractor are updated using gradient descent. The feature extractor with updated parameters is used to re-extract the feature vector of each video frame, resulting in a fine-tuned feature vector sequence for each video frame. Based on the fine-tuned feature vector sequence, the third similarity between any two graph nodes is calculated. The third similarity is then adjusted based on preset adjustment parameters to obtain a new initial edge weight for connecting any two graph nodes. For the first iteration, the preset initial neighborhood radius is determined as the neighborhood radius of the previous iteration. For iterations other than the first iteration, the neighborhood radius calculated in the previous iteration is determined as the neighborhood radius of the previous iteration of the current iteration. Based on the initial edge weights, the neighborhood radius of the previous iteration, the preset smoothing coefficient, and the total number of edges connecting any two graph nodes, the neighborhood radius of the first iteration is calculated using a preset neighborhood radius time-adaptive adjustment formula. Based on the neighborhood radius of the first iteration, the initial edge weights are sparsified to obtain a set of sparsified edge weights. Based on the edges corresponding to the sparsified edge weight set and the multiple graph nodes, a new directed graph is constructed, and the graph Laplacian matrix of the new directed graph is calculated. Based on the graph Laplacian matrix of the new directed graph, the sampler is used to filter multiple new positive sample frame pairs and new negative sample frame pairs in the current iteration round. Based on the multiple new positive sample frame pairs and new negative sample frame pairs, calculate the new time-aware contrast loss for the current iteration round, return to the loss convergence judgment step, until there is a new iteration round whose time-aware contrast loss satisfies the convergence condition.

7. The method according to claim 1, characterized in that, The step of performing spectral clustering on the Laplacian matrix of the target image to obtain the keyframe set of the target video specifically includes: The Laplacian matrix of the target graph is subjected to eigenvalue decomposition to obtain multiple eigenvalues ​​and corresponding eigenvectors. The eigenvectors corresponding to the k smallest eigenvalues ​​are selected, where k is a positive integer. The eigenvectors corresponding to the k smallest eigenvalues ​​are concatenated column by column to form an eigenvector concatenation matrix. Each row of the eigenvector concatenation matrix is ​​then normalized to obtain a normalized eigenvector matrix. The normalized feature vector matrix is ​​clustered using a preset clustering algorithm, and the multiple graph nodes are divided into k clusters to obtain the clustering results. Based on the clustering results, a central graph node is determined for each cluster, and the video frame corresponding to the central graph node is used as a key frame to obtain the key frame set of the target video.

8. A self-supervised video keyframe extraction device, characterized in that, include: The acquisition module is used to acquire the video frame sequence of the target video, and to extract the feature vector corresponding to each video frame in the video frame sequence through a pre-trained feature extractor to obtain a feature vector sequence. The construction module is used to determine multiple graph nodes based on the video frame sequence, calculate the edge weights of any two graph nodes connected by the feature vector sequence, and construct an initial directed graph based on the multiple graph nodes and the edge weights. The filtering module is used to calculate the graph Laplacian matrix of the initial directed graph, and based on the graph Laplacian matrix, to filter out multiple positive sample frame pairs and multiple negative sample frame pairs from the video frame sequence through the target sampler. The loss calculation module is used to calculate the time-aware contrast loss based on the positive sample frame pair and the negative sample frame pair; The loop execution module is used to repeatedly execute the following operations: updating the parameters of the feature extractor using the time-aware contrastive loss to fine-tune the feature vector, updating the edge weights based on the fine-tuned feature vector, constructing a directed graph, calculating the graph Laplacian matrix, filtering positive sample frame pairs and negative sample frame pairs, and recalculating the time-aware contrastive loss, until the time-aware contrastive loss converges. The matrix calculation module is used to reconstruct the directed graph after the time-aware contrast loss converges, calculate the graph Laplacian matrix of the reconstructed directed graph, and obtain the target graph Laplacian matrix. The clustering module is used to perform spectral clustering on the Laplacian matrix of the target graph to obtain the set of keyframes of the target video.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the self-supervised video keyframe extraction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the self-supervised video keyframe extraction method as described in any one of claims 1 to 7.

Images