Video key frame extraction method fused with self-supervised deep learning

Abstract

The invention discloses a video key frame extraction method fused with self-supervised deep learning, and the method comprises the following steps: carrying out the standardized sampling of an input video according to a fixed interval, extracting a SuperPoint local key point and a Video MAE global semantic feature, and generating a dense descriptor and a semantic vector; constructing multi-dimensional change indexes such as local matching degree and global similarity; constructing a soft distribution matrix and a matching point set based on the fusion features; dynamically judging the key frame through a self-adaptive multi-threshold rule; and outputting the key frame set. The method fuses local and global spatio-temporal information, has robust feature extraction and key frame discrimination capabilities under a weak supervision condition, and can effectively improve the efficiency and precision of video compression, abstract and event detection.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method for extracting video keyframes by incorporating self-supervised deep learning. Background Technology

[0002] With the ever-increasing scale of video data, keyframe extraction, as a fundamental step in video content understanding, summarization, and efficient indexing, significantly impacts downstream tasks in terms of accuracy and robustness. However, existing keyframe extraction methods generally rely on fixed-interval sampling or simple static feature judgment based on pixel differences and color histograms, lacking deep modeling of temporal semantics and spatial structure changes. This makes it difficult to effectively capture representative frames in situations with drastic scene changes, complex motion, or semantic abrupt changes.

[0003] Some studies have attempted to introduce deep learning methods for keyframe extraction, but several problems remain: First, most methods rely on extensive manual annotation, which limits their scalability in unstructured data scenarios; second, single-type features, such as extracting only local textures or focusing only on global semantics, are insufficient to fully reflect inter-frame changes, leading to redundancy or omissions in frame selection results; third, the lack of structured fusion strategies in feature matching and change index calculation processes affects the accuracy of change measurement; and fourth, decision-making strategies are usually based on fixed thresholds, lacking adaptability to video dynamics and content complexity, which can easily lead to unstable or unbalanced selection results.

[0004] Furthermore, existing methods often fail to provide fine-grained explanations of changes after keyframe selection, lacking interpretable representations of the type, source, and intensity of changes, thus limiting their application value in refined video analysis and automated content understanding scenarios.

[0005] Therefore, how to provide a video keyframe extraction method that integrates self-supervised deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a video keyframe extraction method that integrates self-supervised deep learning. This invention fully combines local feature encoding, global semantic modeling, and multi-dimensional change criterion construction, designing a keyframe extraction process that includes uniform temporal sampling, SuperPoint feature extraction, VideoMAE semantic reconstruction, LightGlue matching optimization, multi-index change detection, and dynamic threshold determination. This achieves accurate identification and stable extraction of keyframes in spatiotemporally complex changing scenarios. This invention possesses advantages such as fine-grained feature fusion, rich dimensions of change criteria, strong robustness, and high degree of automation. It can significantly improve the structural discrimination ability and semantic sensitivity of keyframe extraction algorithms, thereby effectively overcoming the technical bottleneck of traditional methods that rely on only a single feature and are difficult to adapt to complex dynamic scenes.

[0007] A video keyframe extraction method incorporating self-supervised deep learning according to an embodiment of the present invention includes the following steps: S1. Sample the input video at a set time interval. Let the video frame rate be F and the sampling interval be... The actual sampling step size is determined to be F× Frames, generating a normalized sampled frame sequence; S2. Use the SuperPoint network to extract local key points and dense descriptors for each frame in the normalized sampled frame sequence; S3. The VideoMAE self-supervised learning framework is used to extract global semantic vectors from the standardized sampled frame sequence. A high mask rate mask-reconstruction strategy is used to output global semantic vectors under the spatiotemporal joint multi-head attention structure, and the reconstruction loss of the masked region is calculated. S4. Input the dense descriptors of adjacent frames into the LightGlue network through dense descriptors and global semantic vectors to obtain the similarity matrix and matching score, generate the soft assignment matrix and filter out the set of matching point pairs, and output the set of matching point pairs and local feature matching score. S5. Calculate the multidimensional change index between adjacent frames, including the local feature matching degree in the form of matching point ratio, the global semantic vector cosine similarity, the fusion similarity weighted by the local feature matching degree and the global semantic vector cosine similarity, and the change in VideoMAE reconstruction loss between adjacent frames, and output the multidimensional change index. S6. Set multi-threshold decision rules, adjust the local feature matching degree threshold, global semantic vector cosine similarity threshold, fusion similarity threshold and reconstruction loss change threshold according to the video. When any index meets the corresponding condition, the current frame is determined to be a key frame and the key frame set is output.

[0008] Optionally, S1 specifically includes: S11. Read the raw video data, obtain the basic parameters of video frame rate F, total number of frames, frame-by-frame timestamp and resolution, and establish a timeline; S12. Adjust the sampling interval according to the dynamic changes in the video. The sampling step size k is calculated by multiplying the sampling interval by the video frame rate. When the video has a variable frame rate, the sampling step size is accumulated using timestamps. To trigger conditional sampling, when the video is at a constant frame rate, samples are taken at index intervals of k. S13. Determine the sampling start point and perform the first fixed-interval sampling along the time axis. Prioritize retaining the first and last frames. When the last segment is less than one step, merge the last frame into the sampling set. S14. Perform uniformization processing on the sampled frames, align the size to the preset H×W, where H is the height of the input image and W is the width of the input image, unify to a uniform color space and perform numerical normalization to obtain a standardized sampled frame. S15. Rearrange the standardized sampled frames in ascending order of timestamp to generate a standardized sampled frame sequence and a sampled index-timestamp mapping table, and record the sampling step size k and time interval. .

[0009] Optionally, S2 specifically includes: S21. Using the SuperPoint network, read the normalized sampled frame sequence frame by frame in chronological order, linearly map the pixel values ​​to between 0 and 1, and scale them to a preset resolution H×W. After converting the normalized image into a tensor, perform a preset number of convolution operations in sequence. For each group, perform convolution, ReLU nonlinear activation and max pooling. Add batch normalization after convolution. After the iteration is completed, output the middle feature map of the scale. S22. Input the intermediate feature map into the interest point decoder in the SuperPoint network. Map the feature map into a tensor through a convolution operation. Apply a softmax operation to the last dimension of the tensor pixel by pixel to normalize the channel values ​​into a probability distribution and retain the channels other than the last channel as the probability results of the potential interest point categories. S23. Expand the normalized three-dimensional tensor into a two-dimensional grid in row priority order, and perform tensor rearrangement operation to map the probability value of each low-resolution unit back to the corresponding pixel region of the original image, recover the key point probability heatmap with the same size as the input image, and store the corresponding interest point probability value for each pixel position. S24. Input the intermediate feature map into the descriptor decoder in the SuperPoint network, use convolution to linearly map the channels to the set dimensions, apply ReLU nonlinear activation on the channel dimension to obtain a semi-dense descriptor, and perform bicubic interpolation on the semi-dense tensor channel by channel in the spatial dimension. S25. After bicubic interpolation, extract the corresponding tensor for each pixel position, calculate the L2 norm of the tensor, divide the tensor by the L2 norm to complete normalization, and obtain a dense descriptor. Write the dense descriptor of each normalized sampling frame into an array according to the pixel coordinates and frame index and align it with the coordinate system of the key point probability heatmap. S26. The loss function is composed of interest point detection loss and descriptor loss. The interest point detection loss adopts weighted cross-entropy, and the descriptor loss adopts hinge loss. Pixel correspondence is established based on homography transformation to generate pseudo-labels and paired labels, and joint optimization is performed. S27. Thresholding or peak selection is performed based on the key point probability heatmap to form a set of local key points for each frame. Dense descriptors are read at the corresponding image coordinates, and the set of local key points and corresponding dense descriptors for each frame are output.

[0010] Optionally, the loss function composed of the interest point detection loss and the descriptor loss specifically includes: From the standardized sampled frame sequence, select one image I and another image I' as a pair of training samples. If the homography method is used to construct pseudo-labels, then a homography matrix H is randomly generated according to the disclosure or estimated from adjacent frames. I' is obtained by perspective transformation using the homography matrix, and the transformation relationship is saved for the projection of point coordinates. The pixel coordinates are converted into homogeneous coordinates. Multiply by H to obtain the projected coordinates. Normalization If the projected coordinates fall outside the image boundary, they are discarded and not considered as positive samples. Non-maximum suppression is performed on the probability heatmap of interest points in image I, and a set of peak points K is selected according to the threshold. The coordinates of each pixel in set K are projected through the homography matrix H to obtain the corresponding coordinate set K′. Each coordinate in K′ is traversed. If the normalized coordinate falls into the effective pixel area of ​​I′, it is marked as positive on the pseudo-label heatmap Y′ at the corresponding position of I′. Otherwise, it is ignored. The pseudo-label heatmap generated for I itself is denoted as Y. The spatial lattice of the dense descriptor tensor is considered as having a size of... index set For each grid point Perform the following operation to project the precise image coordinates of the corresponding pixels onto the second image via H, obtaining the mapped coordinates. ,Bundle Mapping to the feature grid coordinate system using the same downsampling strategy yields... ; If the mapped grid points exist and the projection error is within tolerance, pair the labels. Set to 1 to indicate positive, otherwise mark as 0. No positive pairing is generated for points that cannot be mapped or fall outside the boundary. To control the amount of computation, a limited number of positive pairs and a negative pair of similar magnitude can be sampled for each image and recorded as a pairing label set S. For each position Using a weighted cross-entropy loss, at each position, the logarithm of the predicted class probability is taken and multiplied by the corresponding class weight coefficient to obtain the single-position loss function. Sum the losses of all grid points and divide by , obtain scalar If processing a pair of images, then calculate for I and I' respectively. and The interest point detection loss scalar is obtained. For all pairings between descriptor grid points of two images, a hinge-type metric learning loss is calculated based on the pairing label S. Positive pairs bring the similarity closer, while negative pairs push the similarity below a set boundary. The average loss is then calculated over all pairings. Add the interest point losses of the two images together, then add the descriptor loss multiplied by the weight coefficient. This forms a joint loss function. .

[0011] Optionally, S3 specifically includes: S31. Using the VideoMAE self-supervised learning framework, read the normalized sampled frame sequence frame by frame, compress the input frame into T frames, and divide it into small blocks of fixed size to form tubelets in the time dimension. Through the spatiotemporal block embedding layer, the pixel blocks are mapped into vector representations one by one, which is equivalent to transforming into a token sequence. The spatial position encoding and temporal position encoding are superimposed on each token to generate a token sequence arranged in frame order. S32. Apply a high masking rate masking-reconstruction strategy to the token sequence, randomly select a set of mask indices, retain only the unmasked visible subset as encoder input, input the visible subset into a spatiotemporal joint multi-head attention structure, and compute the latent representation of the visible tokens by stacking layers. S33, Input the result into the encoder , , By mapping and weighting attention, a spatiotemporal joint multi-head attention is output, which is then combined with behavioral attention. Each group is a single head, and each group learns spatiotemporal dependencies independently, using its own... Calculate single-head attention to obtain the outputs of each group, concatenate the single-head outputs in the feature dimension, and apply a linear mapping to obtain the overall output of multi-head attention; S34. The latent representation of the visible subset and the mask placeholder token are concatenated in the original index order and input into the decoder. The decoder reconstructs the masked visual pixel blocks one by one, outputs a reconstruction marker for each mask position, and keeps the visible representation passed by the encoder for each unmasked position from participating in the calculation of reconstruction loss. After all the mask blocks of a video segment are reconstructed, a set of reconstruction results corresponding one-to-one with the mask index is obtained. S35. Calculate the reconstruction loss in the masked region by taking the difference between the original pixel and the reconstructed pixel for each marker in the masked marker set, squaring the difference, and then averaging the result. Unmasked positions are not included. S36. For each frame, organize the encoder output results by frame index, output the global semantic vector, and use the corresponding mask region reconstruction loss as one of the criteria for global semantic change. All of these are stored by frame index.

[0012] Optionally, S4 specifically includes: S41. Read the SuperPoint dense descriptors of adjacent frames into LightGlue in frame order to form feature sets for image I and image S respectively. In LightGlue, establish a learnable representation for each feature point. S42. Calculate self-attention within the same image. For each point in image I, use the point as the query and other points in I as the key to perform correlation calculation, obtain the attention score within the same image and update the descriptor. Perform the same operation on image S and calculate cross-attention between the two images. Use the points in image I as the query and all points in image S as the key to obtain the attention score from I to S and update the descriptor of I accordingly. Use the points in S as the query and I as the key to obtain the attention score from S to I. S44. Read the dense descriptor set of adjacent frames updated by LightGlue, and then read the first descriptor set of image A. The feature vector and the feature vector of image B Each of the eigenvectors is input into a linear transformation function. Mapped to the same metric space, for each pair Calculate the point-pair similarity score from the mapped vectors, and write the score into the similarity matrix. traverse all elements at that location. By combining the two images, we obtain the point-to-point similarity matrix between them. S45. Predict the matching score for each point. Take the updated representation of the point by LightGlue as input, map it to the 0-1 interval through the Sigmoid activation function, and output the matching score of the point. This process is repeated for each point in image A. For each point in image B, we obtain , as the gate factor of the soft allocation matrix; S46. Based on the similarity matrix and matching scores, pair any two points. The matching scores on both sides are combined with the row or column similarity normalization term to obtain the soft assignment score and generate the soft assignment matrix. S47. Based on the allocation scores of each pair of points in the soft allocation matrix, select the pair of points that meet the following conditions as the matching pair set: the soft allocation score ranks first in the current row or column and is higher than the scores of all other pair of points in the same row or column by at least a set threshold. All the pair of points that meet the conditions are combined into a matching pair set and output. The percentage of valid matching points is calculated as the local feature matching degree and output along with it.

[0013] Optionally, S5 specifically includes: S51. Calculate the proportion of the number of matching points in the number of key points by taking the larger of the number of valid matching points and the number of key points detected in frames 0 and 1 respectively. S52. Take the global semantic vectors extracted by the VideoMAE network from the two frames respectively, calculate the dot product of the global semantic vectors of the two frames, divide by the product of their respective L2 norms, and obtain the cosine similarity of the global semantic vectors. S53. Weighted fusion of local feature matching degree and global semantic similarity is performed, and the influence weight on the fusion similarity is controlled by the weighting factor to generate the fusion similarity. S54. When using VideoMAE for frame reconstruction, calculate the change in reconstruction loss between the current frame and the previous frame to measure the degree of change in spatiotemporal structure between frames. When the value is greater than the threshold, it indicates that there are significant semantic differences between frames.

[0014] Optionally, S6 specifically includes: analyzing the dynamic characteristics between video frames, manually setting a local feature matching degree threshold, a global semantic vector similarity threshold, a fusion similarity threshold, and a reconstruction loss change threshold; when any indicator meets the corresponding condition, determining the current frame as a key frame; adding the frames that meet the conditions to the key frame set; and continuing to repeat the determination process for video frames to output a set of all key frames that meet the key criteria.

[0015] The beneficial effects of this invention are: This invention proposes a video keyframe extraction method integrating self-supervised deep learning. Addressing the problems of existing keyframe extraction methods, such as single feature types, poor robustness, and inability to adapt to complex dynamic changes, this invention introduces SuperPoint and VideoMAE to extract local dense descriptors and global semantic vectors respectively. LightGlue is used for inter-frame feature point matching, and multi-dimensional change indicators such as matching degree, semantic similarity, and reconstruction loss difference are integrated to construct a dynamic threshold determination mechanism for accurate identification of keyframes in videos. This invention possesses advantages such as fine-grained feature modeling, strong sensitivity to multi-scale changes, and high robustness of the determination method. It effectively improves the stability and accuracy of keyframe extraction in multi-dynamic scenarios, significantly reduces semantic information loss and redundant frame interference, and is applicable to various tasks such as video summarization, intelligent editing, and behavior analysis. It solves the technical problems of traditional methods, such as slow response to semantic abrupt changes, weak recognition of structural changes, and unstable frame selection. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of a video keyframe extraction method that integrates self-supervised deep learning, as proposed in this invention. Figure 2This is a flowchart of the self-supervised semantic feature extraction and reconstruction based on SuperPoint and VideoMAE proposed in this invention. Figure 3 This is a schematic diagram of the multi-dimensional feature extraction of video keyframes that combines local and global features, as proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figure 1-3 A video keyframe extraction method integrating self-supervised deep learning includes the following steps: S1. Sample the input video at fixed time intervals. Let the video frame rate be F and the sampling interval be... The actual sampling step size is determined to be F× Frame, take A standardized sampling frame sequence is generated over a period of 0.5 to 2 seconds. S2. Use the SuperPoint network to extract local key points and dense descriptors of each frame in the normalized sampling frame sequence. Transform the normalized sampling frame sequence into an intermediate feature map and input it into the interest point decoder to generate the key point probability distribution. The descriptor decoder obtains the dense descriptor normalized by L2. Then, through the joint optimization of weighted cross-entropy loss and hinge loss, the set of local key points and the corresponding dense descriptor of each frame are output. S3. The VideoMAE self-supervised learning framework is used to extract global semantic vectors from the standardized sampled frame sequence. A high mask rate mask-reconstruction strategy is used to output global semantic vectors under the spatiotemporal joint multi-head attention structure, and the reconstruction loss of the masked region is calculated. S4. Input the dense descriptors of adjacent frames into the LightGlue network through dense descriptors and global semantic vectors, perform self-attention and cross-attention calculations in sequence, obtain the similarity matrix and matching score, generate the soft assignment matrix and filter out the set of matching point pairs, and output the set of matching point pairs and local feature matching degree. S5. Calculate the multidimensional change index between adjacent frames, including the local feature matching degree in the form of matching point ratio, the global semantic vector cosine similarity, the fusion similarity weighted by the local feature matching degree and the global semantic vector cosine similarity, and the change in VideoMAE reconstruction loss between adjacent frames, and output the multidimensional change index. S6. Set multi-threshold decision rules, adjust the local feature matching degree threshold, global semantic vector cosine similarity threshold, fusion similarity threshold and reconstruction loss change threshold according to the video. When any index meets the corresponding condition, the current frame is determined to be a key frame and the key frame set is output.

[0020] This implementation transforms the original video into a standardized sequence of sampled frames through fixed-time sampling, ensuring temporal uniformity of input and comparability of frame sequences. It utilizes a SuperPoint network to extract local keypoints and dense descriptors, obtaining stable and highly discriminative local feature representations. The VideoMAE framework is employed to extract global semantic vectors, and a mask reconstruction strategy is combined to calculate reconstruction loss, thereby acquiring deep semantic features and quantifying content differences. Based on local and global features, the SuperPoint descriptors are input into a LightGlue network, where self-attention and cross-attention are used to obtain a high-quality set of matching point pairs, improving feature matching accuracy. Local feature matching degree, global semantic cosine similarity, fusion similarity, and reconstruction loss difference are calculated to form multi-dimensional variation indicators that comprehensively reflect the similarity and differences between frames. Based on a multi-threshold dynamic decision rule, each threshold is adaptively adjusted; when any indicator meets the condition, the current frame is determined to be a keyframe, and a set of keyframes is output. This implementation effectively integrates local and global features, combining multi-dimensional indicators and dynamic threshold determination to achieve accurate keyframe extraction and improved robustness.

[0021] In this embodiment, S1 specifically includes: S11. Read the raw video data, obtain the basic parameters of video frame rate F, total number of frames, frame-by-frame timestamp and resolution, and establish a timeline; S12. Adjust the sampling interval according to the dynamic changes in the video. The sampling step size k is calculated by multiplying the sampling interval by the video frame rate. When k is less than 1, k is set to 1. When the video has a variable frame rate, the sampling step size is accumulated based on the timestamp. To trigger conditional sampling, when the video is at a constant frame rate, samples are taken at index intervals of k. S13. Determine the sampling start point and perform the first fixed-interval sampling along the time axis. Prioritize retaining the first and last frames. When the last segment is less than one step, merge the last frame into the sampling set to ensure complete coverage of the time period. S14. Perform uniformization processing on the sampled frames, align the size to the preset H×W, where H is the height of the input image and W is the width of the input image, unify to a uniform color space and perform numerical normalization to obtain a standardized sampled frame. S15. Rearrange the standardized sampled frames in ascending order of timestamp to generate a standardized sampled frame sequence and a sampled index-timestamp mapping table, and record the sampling step size k and time interval. .

[0022] In this embodiment, S2 specifically includes: S21. Using the SuperPoint network, read the normalized sampled frame sequence frame by frame in chronological order, linearly map the pixel values ​​to between 0 and 1, and scale them to a preset resolution H×W. After converting the normalized image into a tensor, perform a preset number of convolution operations in sequence. For each group, perform 3×3 convolution, ReLU nonlinear activation and 2×2 max pooling. Add batch normalization after convolution. After the iteration is completed, output the middle feature map of the scale. S22. Input the intermediate feature map into the interest point decoder in the SuperPoint network, and map the feature map to a size of [size missing] through a single 3×3 convolution operation. The tensor is subjected to a Softmax operation pixel by pixel on the last dimension of the tensor, so that the values ​​of the 65 channels are normalized into a probability distribution, and the channels other than the last channel are retained as the probability results of the potential interest point categories. S23. Unfold the normalized 3D tensor into a 2D mesh according to row-major order, and perform tensor rearrangement operation to map the probability value of each low-resolution unit back to the corresponding 8×8 pixel region of the original image, thus restoring an image of the same size as the input image. The key point probability heatmap is generated, and the corresponding interest point probability value is stored for each pixel location, providing input for subsequent threshold selection and key point set generation; S24. Input the intermediate feature map into the descriptor decoder in the SuperPoint network, use a 1×1 convolution layer to linearly map the channels to 256 dimensions, apply ReLU nonlinear activation to the channel dimension to enhance the representation, obtain a semi-dense descriptor, and perform bicubic interpolation on the semi-dense tensor channel by channel in the spatial dimension to interpolate the semi-dense tensor to the same H×W resolution as the original image in the height and width directions. S25. After bicubic interpolation, extract the corresponding tensor for each pixel position, calculate the L2 norm of the tensor, divide the tensor by the L2 norm to complete normalization, and obtain a dense descriptor. Write the dense descriptor of each normalized sampling frame into an array according to the pixel coordinates and frame index and align it with the coordinate system of the key point probability heatmap as the input for subsequent adjacent frame matching. S26. The loss function is composed of interest point detection loss and descriptor loss. The interest point detection loss adopts weighted cross-entropy, and the descriptor loss adopts hinge loss. The pixel correspondence is established based on homography transformation to generate pseudo-labels and paired labels. Joint optimization is performed to stabilize key point probability and descriptor discriminability. S27. Thresholding or peak selection is performed based on the key point probability heatmap to form a set of local key points for each frame. Dense descriptors are read at the corresponding image coordinates, and the set of local key points and corresponding dense descriptors for each frame are output.

[0023] In this implementation, the input raw video is modeled using frame rate parsing and timeline reconstruction to achieve basic parameter modeling. The sampling interval is set based on the video's dynamics, and adaptive sampling generates a standardized sampled frame sequence covering the entire time period. After unified resolution mapping and color normalization, the sampled frames possess temporal and scale consistency. Subsequently, the standardized frame sequence is fed frame-by-frame into the SuperPoint network for local feature encoding. Intermediate feature maps are extracted using a convolution-activation-pooling module, and a keypoint probability heatmap is generated using an interest point decoder to identify potentially salient structural regions. A normalized dense descriptor is then extracted by a descriptor decoder and aligned to the image scale using tensor interpolation to ensure spatial detail accuracy. Stable and high-density local representations are obtained through Softmax normalization, multi-scale interpolation, and L2 norm normalization operations. During the training phase, pseudo-labels and pairing sets are constructed based on homography transformation. A joint loss function is defined to discriminate and optimize interest points and descriptors respectively, outputting the keypoint set and corresponding descriptor for each frame, thus building a feature foundation for subsequent inter-frame matching and keyframe recognition.

[0024] In this embodiment, the loss function composed of the interest point detection loss and the descriptor loss specifically includes: From the standardized sampled frame sequence, select one image I and another image I' as a pair of training samples. If the homography method is used to construct pseudo-labels, then a homography matrix H is randomly generated according to the disclosure or estimated from adjacent frames. I' is obtained by perspective transformation using the homography matrix, and the transformation relationship is saved for the projection of point coordinates. The pixel coordinates are converted into homogeneous coordinates. Multiply by H to obtain the projected coordinates. Normalization If the projected coordinates fall outside the image boundary, they are discarded and not considered as positive samples. Non-maximum suppression is performed on the probability heatmap of interest points in image I, and a set of peak points K is selected according to the threshold. The coordinates of each pixel in set K are projected through the homography matrix H to obtain the corresponding coordinate set K′. Each coordinate in K′ is traversed. If the normalized coordinate falls into the effective pixel area of ​​I′, it is marked as positive on the pseudo-label heatmap Y′ at the corresponding position of I′. Otherwise, it is ignored. The pseudo-label heatmap generated for I itself is denoted as Y. The spatial lattice of the dense descriptor tensor is considered as having a size of... index set For each grid point Perform the following operation to project the precise image coordinates of the corresponding pixels onto the second image via H, obtaining the mapped coordinates. ,Bundle Mapping to the feature grid coordinate system using the same downsampling strategy yields... If the mapped grid points exist and the projection error is within tolerance, then pair the labels. Set to 1 to indicate positive, otherwise mark as 0. No positive pairing is generated for points that cannot be mapped or fall outside the boundary. To control the amount of computation, a limited number of positive pairs and a negative pair of similar magnitude can be sampled for each image and recorded as a pairing label set S. For each position Using a weighted cross-entropy loss, at each position, the logarithm of the predicted class probability is taken and multiplied by the corresponding class weight coefficient to obtain the single-position loss function. Sum the losses of all grid points and divide by , obtain scalar If processing a pair of images, then calculate for I and I' respectively. and The interest point detection loss scalar is obtained. For all pairings between descriptor grid points of two images, a hinge-type metric learning loss is calculated based on the pairing label S. Positive pairs bring the similarity closer, while negative pairs push the similarity below a set boundary. The average loss is then calculated over all pairings. Add the interest point losses of the two images together, then add the descriptor loss multiplied by the weight coefficient. This forms a joint loss function. :

[0025]

[0026]

[0027] in , This represents the height and width of the feature map. Represents the height index position of the feature map. This represents the feature map width index position. Indicates the network's location Output of the predicted probability value of the point of interest. Indicates position pseudo-tags This represents the loss function used for a single location. This indicates the corresponding grid points on the first and second images. and Dense descriptor vectors, Indicates a pairing tag. This represents a single-pair loss term between two descriptors. and Let I and I' represent the interest point losses calculated on images I and I', respectively. This represents the loss metric calculated for descriptor pairings. The weighting hyperparameter of the representation loss in the total loss. The symbols represent feature maps, network responses, descriptors, pseudo-labels, and pairing sets, respectively.

[0028] In this embodiment, S3 specifically includes: S31. Using the VideoMAE self-supervised learning framework, read the normalized sampled frame sequence frame by frame, compress the input frame into T frames, each frame contains H*W*3 pixels, and divides them into small blocks of fixed size, forming tubelets in the time dimension. Through the spatiotemporal block embedding layer, the pixel blocks are mapped one by one into vector representations, which are equivalent to token sequences. Spatial position encoding and temporal position encoding are superimposed on each token to generate a token sequence arranged in frame order. S32. A high masking rate masking-reconstruction strategy is implemented on the token sequence. A masking index set is randomly selected, and only the unmasked visible subset is retained as the encoder input. The visible subset is input into a spatiotemporal joint multi-head attention structure, and the latent representation of the visible token is calculated by stacking layers. This representation is then used to reconstruct the masked token in the subsequent decoding stage. S33, Input the result into the encoder , , By mapping and weighting attention, a spatiotemporal joint multi-head attention is output, which is then combined with behavioral attention. Each group is a single head, and each group learns spatiotemporal dependencies independently, using its own... Calculate single-head attention to obtain the outputs of each group, concatenate the single-head outputs along the feature dimension, and apply a linear mapping to obtain the overall output of multi-head attention, which is used for subsequent layers or decoding reconstruction.

[0029]

[0030]

[0031] in Represents the query vector. Represents the key vector. Represents a value vector. This indicates row-wise normalization. The scaling factor represents the dimension of the key vector. Indicates the transpose symbol. This represents the dot product similarity between the query and the key. This represents the attention calculation function. Indicates the header index. They represent the first The query, key, and value of each size. Indicates the first The output of each attention head, This indicates the output of each end of the channel splicing. This represents the output mapping matrix. This is the final result of the multi-head attention layer; S34. The latent representation of the visible subset and the mask placeholder token are concatenated in the original index order and input into the decoder. The decoder reconstructs the masked visual pixel blocks one by one, outputs a reconstruction marker for each mask position, and keeps the visible representation passed by the encoder for each unmasked position from participating in the calculation of reconstruction loss. After all the mask blocks of a video segment are reconstructed, a set of reconstruction results corresponding one-to-one with the mask index is obtained. S35. Calculate the reconstruction loss in the masked region by taking the difference between the original pixel and the reconstructed pixel for each marker in the masked marker set, squaring the difference, and then averaging the result. Unmasked positions are not included.

[0032] in Indicates the losses incurred during reconstruction. Indicates the tag index, Represents the set of masked labels. This represents the value of the mask marker in the pixel normalization space. This represents the value of the reconstruction marker at the corresponding location in the pixel normalization space; S36. For each frame, organize the encoder output results by frame index, output a global semantic vector to represent global semantic information, and use the corresponding mask region reconstruction loss as one of the criteria for global semantic change. All are stored by frame index for subsequent steps to calculate the inter-frame global semantic similarity and reconstruction loss change and participate in key frame judgment.

[0033] In this implementation, the standardized sampled frame sequence is input into the VideoMAE self-supervised learning framework. Following a process of spatiotemporal embedding encoding, high mask rate reconstruction, joint attention extraction, and reconstruction loss backpropagation, the global semantic vector is modeled. Through tubelet partitioning and spatial-temporal positional encoding, the original frame sequence is mapped to a token sequence. Under a spatiotemporal joint multi-head attention structure, deep dependencies between spatial and temporal dimensions are independently modeled. A random high-masking strategy is employed, retaining only the visible subset for encoder learning. Reconstruction loss is calculated in the masked region, enhancing the model's sensitivity to key structures. The visible tokens output by the encoder are used by the decoder to reconstruct the masked region, ensuring the consistency and reconstructability of semantic features across the entire frame. The output global semantic vector possesses a complete representation of the video's semantic scene, while the reconstruction loss in the masked region effectively characterizes the degree of semantic difference between frames. This provides a robust and quantifiable global semantic criterion for subsequent inter-frame similarity calculation and keyframe selection, significantly improving the global understanding and change perception accuracy of keyframe extraction.

[0034] In this embodiment, S4 specifically includes: S41. Read the SuperPoint dense descriptors of adjacent frames into LightGlue in frame order to form feature sets for image I and image S respectively. In LightGlue, establish a learnable representation for each feature point as input for subsequent attention and matching calculations. S42. Calculate self-attention within the same image. For each point in image I, using the point as the query and other points in I as the key, calculate the relevance to obtain the attention score within the same image and update the descriptor. Perform the same operation on image S.

[0035] in Represents the midpoint of the same image With point Attention score Represents the query vector. Represents the key vector. Indicates the transpose symbol. Rotation code representing relative position , This represents the feature point index within the same image. Cross-attention is calculated between the two images. Using points in image I as the query and all points in image S as the key, the attention score from I to S is obtained, and the descriptor of I is updated accordingly. Using points in S as the query and I as the key, the attention score from S to I is obtained.

[0036] in This represents the attention score from image I to image S. Let represent the key vectors of point i in image I and point j in image S, respectively. Indicates the transpose symbol. Attention score from image S to image I , Indicates the feature point index within the same image, where I and S represent images I to S, respectively; S44. Read the dense descriptor set of adjacent frames updated by LightGlue, and then read the first descriptor set of image A. The feature vector and the feature vector of image B Each of the eigenvectors is input into a linear transformation function. Mapped to the same metric space, for each pair Calculate the point-pair similarity score from the mapped vectors, and write the score into the similarity matrix. traverse all elements at that location. Combining the two images, we obtain the point-to-point similarity matrix between them:

[0037]

[0038] in Represents a linear transformation function. , Represent the first and second halves of the image, respectively. , No. 1 eigenvector Indicates the transpose symbol. This indicates the similarity score between two points. S45. Predict the matching score for each point. Take the updated representation of the point by LightGlue as input, map it to the 0-1 interval through the Sigmoid activation function, and output the matching score of the point. This process is repeated for each point in image A. For each point in image B, we obtain As a gate factor for the soft allocation matrix:

[0039] Where Sigmoid represents the activation function. This represents the matching score for a single feature point. Indicates the first 1 eigenvector The gate factor represents the subsequent soft assignment matrix of image A or B; S46. Based on the similarity matrix and matching scores, pair any two points. The matching scores on both sides are combined with the row or column similarity normalization term to obtain the soft assignment score, generating a soft assignment matrix for subsequent matching filtering.

[0040] in Let represent the matching degree scores of the i-th point in image A and the i-th point in image B, respectively. This indicates row-wise normalization. Let k be all feature points in image A, and calculate the similarity with the j-th point in image B. , To calculate the similarity between image B and the i-th point in image A for all feature points k. , Indicates the soft match score; S47. Based on the allocation scores of each point pair in the soft allocation matrix, select point pairs that meet the following conditions as a set of matching point pairs: the soft allocation score ranks first in the current row or column and is higher than the scores of all other point pairs in the same row or column by at least a set threshold. All point pairs that meet the conditions are combined into a set of matching point pairs and output. The percentage of effective matching points is calculated as the local feature matching degree and output along with it for subsequent calculation of multidimensional change indicators and key frame determination.

[0041] In this implementation, the SuperPoint dense descriptors of adjacent frames are input into the LightGlue lightweight feature matching framework. The local feature matching degree is calculated following a process of feature interaction, self-attention modeling, similarity scoring, and soft matching filtering. First, self-attention operations are performed on the feature dimension to strengthen local semantic expression, and a deep correlation between inter-frame features is established through a cross-attention mechanism. A point-pair similarity matrix is ​​calculated from the attention output, and the matching confidence distribution is obtained using Softmax normalization. This is combined with the matching score matrix to generate a soft assignment matrix. A set of high-confidence matching point pairs is extracted using threshold control and maximum value filtering strategies, automatically filtering out weakly correlated or mismatched points. The output set of matching point pairs exhibits high robustness and structural consistency, effectively characterizing the trend of local structural changes between frames. The local feature matching degree is measured by the proportion of matching points, reflecting the stability and degree of change of local content between frames. This method replaces traditional brute-force matching with a lightweight and efficient attention matching mechanism, improving matching accuracy and real-time performance in complex video scenes, and providing reliable support for the identification of local structural changes in keyframe filtering.

[0042] In this embodiment, S5 specifically includes: S51. Count the number of valid matching points and the number of keypoints detected in frame 0 and frame 1 respectively, select the larger one as the denominator, and calculate the proportion of the number of matching points in the number of keypoints to measure the structural consistency at the local feature level between the two frames:

[0043] in This represents the number of valid matching point pairs obtained by LightGlue matching between two image frames. These represent the number of local keypoints detected in frame 0 and frame 1, respectively. This means that the larger value of the number of key points in two frames is taken as the benchmark for the matching ratio. The output represents the local feature matching degree. S52. Take the global semantic vectors extracted by the VideoMAE network from the two frames respectively, calculate the dot product of the global semantic vectors of the two frames, divide it by the product of their respective L2 norms, and obtain the cosine similarity of the global semantic vectors to measure the degree of similarity between the two frames in the semantic space:

[0044] in They represent the first Frame and the The global semantic vector of the frame is extracted by the VideoMAE network. This represents the vector dot product operation. Let L2 norm of the corresponding vectors be represented respectively. This represents the global semantic similarity output. S53. Weighted fusion of local feature matching degree and global semantic similarity is performed, and the influence weight on the fused similarity is controlled by the weighting factor to generate a fused similarity, resulting in a more robust inter-frame similarity measurement index:

[0045] in Indicates the weighting coefficient. Indicates local feature matching, Represents global semantic vector similarity. This indicates the output of the fusion similarity score; S54. When using VideoMAE for frame reconstruction, calculate the change in reconstruction loss between the current frame and the previous frame to measure the degree of change in spatiotemporal structure between frames. When the value is greater than the threshold, it indicates that there are significant semantic differences between frames.

[0046] in This represents the reconstruction loss of the i-th frame after reconstruction using the VideoMAE mask. This represents the reconstruction loss of the previous frame. This indicates the set threshold for the difference in reconstruction loss. This represents the difference in reconstruction losses.

[0047] In this implementation, based on the extraction of local matching points and global semantic modeling, the degree of change between adjacent frames is calculated according to a multi-index fusion process to form a multi-dimensional change index set. Based on the SuperPoint and LightGlue matching results, the proportion of effective matching points to the total number of keypoints in the two frames is calculated as the local feature matching degree, measuring the consistency of local structure. The VideoMAE global semantic vectors of the two frames are extracted, and cosine similarity is calculated to characterize the correlation of overall semantic information. The local feature matching degree and global semantic similarity are weighted and fused to obtain a fused similarity index, which comprehensively reflects the structural and semantic changes between frames. The VideoMAE mask reconstruction loss values ​​of the two frames are calculated separately, and the difference is taken as the change in reconstruction loss, used to identify potential content mutations. The multi-dimensional change index enables simultaneous perception of local pixel-level changes and global semantic level differences, enhancing the responsiveness of keyframe screening to subtle changes and semantic transitions, and providing a stable and comprehensive quantitative basis for subsequent threshold judgment and decision-making.

[0048] In this embodiment, S6 specifically includes: analyzing the dynamic characteristics between video frames, manually setting a local feature matching threshold, a global semantic vector similarity threshold, a fusion similarity threshold, and a reconstruction loss change threshold, determining the current frame as a key frame when any indicator meets the corresponding condition, adding the frames that meet the conditions to the key frame set, and continuing to repeat the determination process for video frames, outputting a set of all key frames that meet the key criteria.

[0049] Example 1: To verify the practicality and effectiveness of this invention in real-world video analysis tasks, the proposed video keyframe extraction method integrating self-supervised deep learning was deployed on an intelligent traffic monitoring platform of a video surveillance system integration company in a certain city of a province. This platform is primarily used for urban road operation status monitoring and event detection, processing approximately 7.8TB of video data daily, involving over 2400 high-definition camera video sources. Traditional methods based on fixed-interval sampling or simple inter-frame difference detection struggle to address frame redundancy issues in highly dynamic scenes, making it difficult to quickly locate key events and severely limiting the system's retrieval efficiency and data storage optimization capabilities.

[0050] This invention is integrated into the video data preprocessing module of the platform. First, the system reads the raw video data, automatically parses basic parameters such as frame rate, timestamp, and resolution, and constructs a unified timeline. After analyzing the video's dynamic features, it adaptively sets the sampling interval, completing the first round of standardized sampling with a fixed step size k to ensure complete temporal coverage. Next, the system uses a SuperPoint network to extract interest points and generate dense descriptors on the sampled frames, and utilizes heatmaps and L2 normalization to improve the robustness of local keypoint matching.

[0051] Subsequently, the VideoMAE model introduced in this invention extracts global semantic features under unsupervised conditions and enhances representation capabilities through an occlusion prediction and reconstruction strategy. For adjacent frames, the proportion of local matching points, global feature cosine similarity, fusion similarity, and the difference between VideoMAE reconstruction errors are calculated to construct a multi-dimensional change index sequence. A dynamic threshold decision mechanism automatically selects the keyframe set, significantly compressing the number of video frames while retaining key semantic events.

[0052] The system test selected three typical high-dynamic intersection surveillance video segments from August 10th to August 25th, 2025. Each video segment was originally 30 minutes long, with a frame rate of 25 FPS and an image resolution of 1920×1080. The performance after processing by the fusion model is as follows:

[0053] Table 1. Comparison of the method of this invention with other keyframe extraction methods

[0054] Analysis of the data in Table 1 shows that, in terms of the average number of frames extracted, the method of this invention extracts only 728 frames, significantly lower than the 1342 frames extracted by the average interval sampling method, the 987 frames extracted by the inter-frame difference method, and the 864 frames extracted by the YOLO-based detection method. This indicates that the method has stronger frame compression capabilities and can effectively remove redundant frames. Correspondingly, the video compression rate reaches 76.3%, higher than the 55.1% to 70.2% of other methods, greatly saving video storage space and facilitating deployment in edge devices or large-scale storage scenarios.

[0055] While maintaining compression efficiency, this invention demonstrates excellent key event retention rate, reaching 92.8%, which is more than 10 percentage points higher than other methods. This indicates that the method can more accurately capture semantically valuable frames in the scene when extracting keyframes, especially maintaining good semantic integrity in highly dynamic environments. Meanwhile, its average misclassification frame rate is only 4.6%, far lower than the 18.3% of the average interval sampling method and the 15.9% of the inter-frame difference method, significantly reducing the probability of invalid frames being misidentified as keyframes and further improving retrieval quality.

[0056] In terms of average localization time, this invention requires only 1.8 seconds, a significant decrease compared to the YOLO-based method (3.9 seconds) and other traditional methods. This demonstrates the high efficiency of its frame selection and feature extraction mechanisms, facilitating rapid completion of video analysis tasks. Simultaneously, the retrieval recall rate reaches 94.5%, further showcasing the high reliability of this invention in key semantic frame extraction and event recognition.

[0057] Finally, in terms of the reduction rate of reconstruction error, the present invention achieved 24.2%, which is far superior to the performance of other methods of 10% to 13%, indicating that the reconstruction mechanism based on VideoMAE has obvious advantages in information restoration and structure preservation, and provides a solid foundation for subsequent semantic recognition and video generation.

[0058] In summary, the data in the table fully demonstrates that the method of the present invention has multiple advantages in keyframe extraction tasks, including high compression rate, high retention rate, and high robustness, and is suitable for practical application scenarios with high requirements for both video accuracy and efficiency.

[0059] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video keyframe extraction method integrating self-supervised deep learning, characterized in that, Includes the following steps: S1. Sample the input video at a set time interval. Let the video frame rate be F and the sampling interval be... The actual sampling step size is determined to be F× Frames, generating a normalized sampled frame sequence; S2. Use the SuperPoint network to extract local key points and dense descriptors for each frame in the normalized sampled frame sequence; S3. The VideoMAE self-supervised learning framework is used to extract global semantic vectors from the standardized sampled frame sequence. A high mask rate mask-reconstruction strategy is used to output global semantic vectors under the spatiotemporal joint multi-head attention structure, and the reconstruction loss of the masked region is calculated. S4. Input the dense descriptors of adjacent frames into the LightGlue network through dense descriptors and global semantic vectors to obtain the similarity matrix and matching score, generate the soft assignment matrix and filter out the set of matching point pairs, and output the set of matching point pairs and local feature matching score. S5. Calculate the multidimensional change index between adjacent frames, including the local feature matching degree in the form of matching point ratio, the global semantic vector cosine similarity, the fusion similarity weighted by the local feature matching degree and the global semantic vector cosine similarity, and the change in VideoMAE reconstruction loss between adjacent frames, and output the multidimensional change index. S6. Set multi-threshold decision rules, adjust the local feature matching degree threshold, global semantic vector cosine similarity threshold, fusion similarity threshold and reconstruction loss change threshold according to the video. When any index meets the corresponding condition, the current frame is determined to be a key frame and the key frame set is output.

2. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S1 specifically includes: S11. Read the raw video data, obtain the basic parameters of video frame rate F, total number of frames, frame-by-frame timestamp and resolution, and establish a timeline; S12. Adjust the sampling interval according to the dynamic changes in the video. The sampling step size k is calculated by multiplying the sampling interval by the video frame rate. When the video has a variable frame rate, the sampling step size is accumulated using timestamps. To trigger conditional sampling, when the video is at a constant frame rate, samples are taken at index intervals of k. S13. Determine the sampling start point and perform the first fixed-interval sampling along the time axis. Prioritize retaining the first and last frames. When the last segment is less than one step, merge the last frame into the sampling set. S14. Perform uniformization processing on the sampled frames, align the size to the preset H×W, where H is the height of the input image and W is the width of the input image, unify to a uniform color space and perform numerical normalization to obtain a standardized sampled frame. S15. Rearrange the standardized sampled frames in ascending order of timestamp to generate a standardized sampled frame sequence and a sampled index-timestamp mapping table, and record the sampling step size k and time interval. .

3. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S2 specifically includes: S21. Using the SuperPoint network, read the normalized sampled frame sequence frame by frame in chronological order, linearly map the pixel values ​​to between 0 and 1, and scale them to a preset resolution H×W. After converting the normalized image into a tensor, perform a preset number of convolution operations in sequence. For each group, perform convolution, ReLU nonlinear activation and max pooling. Add batch normalization after convolution. After the iteration is completed, output the middle feature map of the scale. S22. Input the intermediate feature map into the interest point decoder in the SuperPoint network. Map the feature map into a tensor through a convolution operation. Apply a softmax operation to the last dimension of the tensor pixel by pixel to normalize the channel values ​​into a probability distribution and retain the channels other than the last channel as the probability results of the potential interest point categories. S23. Expand the normalized three-dimensional tensor into a two-dimensional grid in row priority order, and perform tensor rearrangement operation to map the probability value of each low-resolution unit back to the corresponding pixel region of the original image, recover the key point probability heatmap with the same size as the input image, and store the corresponding interest point probability value for each pixel position. S24. Input the intermediate feature map into the descriptor decoder in the SuperPoint network, use convolution to linearly map the channels to the set dimensions, apply ReLU nonlinear activation on the channel dimension to obtain a semi-dense descriptor, and perform bicubic interpolation on the semi-dense tensor channel by channel in the spatial dimension. S25. After bicubic interpolation, extract the corresponding tensor for each pixel position, calculate the L2 norm of the tensor, divide the tensor by the L2 norm to complete normalization, and obtain a dense descriptor. Write the dense descriptor of each normalized sampling frame into an array according to the pixel coordinates and frame index and align it with the coordinate system of the key point probability heatmap. S26. The loss function is composed of interest point detection loss and descriptor loss. The interest point detection loss adopts weighted cross-entropy, and the descriptor loss adopts hinge loss. Pixel correspondence is established based on homography transformation to generate pseudo-labels and paired labels, and joint optimization is performed. S27. Thresholding or peak selection is performed based on the key point probability heatmap to form a set of local key points for each frame. Dense descriptors are read at the corresponding image coordinates, and the set of local key points and corresponding dense descriptors for each frame are output.

4. The video keyframe extraction method integrating self-supervised deep learning according to claim 3, characterized in that, The loss function, which comprises the interest point detection loss and the descriptor loss, specifically includes: From the standardized sampled frame sequence, select one image I and another image I' as a pair of training samples. If the homography method is used to construct pseudo-labels, then a homography matrix H is randomly generated according to the disclosure or estimated from adjacent frames. I' is obtained by perspective transformation using the homography matrix, and the transformation relationship is saved for the projection of point coordinates. The pixel coordinates are converted into homogeneous coordinates. Multiply by H to obtain the projected coordinates. Normalization If the projected coordinates fall outside the image boundary, they are discarded and not considered as positive samples. Non-maximum suppression is performed on the probability heatmap of interest points in image I, and a set of peak points K is selected according to the threshold. The coordinates of each pixel in set K are projected through the homography matrix H to obtain the corresponding coordinate set K′. Each coordinate in K′ is traversed. If the normalized coordinate falls into the effective pixel area of ​​I′, it is marked as positive on the pseudo-label heatmap Y′ at the corresponding position of I′. Otherwise, it is ignored. The pseudo-label heatmap generated for I itself is denoted as Y. The spatial lattice of the dense descriptor tensor is considered as having a size of... index set For each grid point Perform the following operation to project the precise image coordinates of the corresponding pixels onto the second image via H, obtaining the mapped coordinates. ,Bundle Mapping to the feature grid coordinate system using the same downsampling strategy yields... ; If the mapped grid points exist and the projection error is within tolerance, pair the labels. Set to 1 to indicate positive, otherwise mark as 0. No positive pairing is generated for points that cannot be mapped or fall outside the boundary. To control the amount of computation, a limited number of positive pairs and a negative pair of similar magnitude can be sampled for each image and recorded as a pairing label set S. For each position Using a weighted cross-entropy loss, at each position, the logarithm of the predicted class probability is taken and multiplied by the corresponding class weight coefficient to obtain the single-position loss function. Sum the losses of all grid points and divide by , obtain scalar If processing a pair of images, then calculate for I and I' respectively. and The interest point detection loss scalar is obtained. For all pairings between descriptor grid points of two images, a hinge-type metric learning loss is calculated based on the pairing label S. Positive pairs bring the similarity closer, while negative pairs push the similarity below a set boundary. The average loss is then calculated over all pairings. Add the interest point losses of the two images together, then add the descriptor loss multiplied by the weight coefficient. This forms a joint loss function. .

5. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S3 specifically includes: S31. Using the VideoMAE self-supervised learning framework, read the normalized sampled frame sequence frame by frame, compress the input frame into T frames, and divide it into small blocks of fixed size to form tubelets in the time dimension. Through the spatiotemporal block embedding layer, the pixel blocks are mapped into vector representations one by one, which is equivalent to transforming into a token sequence. The spatial position encoding and temporal position encoding are superimposed on each token to generate a token sequence arranged in frame order. S32. Apply a high masking rate masking-reconstruction strategy to the token sequence, randomly select a set of mask indices, retain only the unmasked visible subset as encoder input, input the visible subset into a spatiotemporal joint multi-head attention structure, and compute the latent representation of the visible tokens by stacking layers. S33, Input the result into the encoder , , By mapping and weighting attention, a spatiotemporal joint multi-head attention is output, which is then combined with behavioral attention. Each group is a single head, and each group learns spatiotemporal dependencies independently, using its own... Calculate single-head attention to obtain the outputs of each group, concatenate the single-head outputs in the feature dimension, and apply a linear mapping to obtain the overall output of multi-head attention; S34. The latent representation of the visible subset and the mask placeholder token are concatenated in the original index order and input into the decoder. The decoder reconstructs the masked visual pixel blocks one by one, outputs a reconstruction marker for each mask position, and keeps the visible representation passed by the encoder for each unmasked position from participating in the calculation of reconstruction loss. After all the mask blocks of a video segment are reconstructed, a set of reconstruction results corresponding one-to-one with the mask index is obtained. S35. Calculate the reconstruction loss in the masked region by taking the difference between the original pixel and the reconstructed pixel for each marker in the masked marker set, squaring the difference, and then averaging the result. Unmasked positions are not included. S36. For each frame, organize the encoder output results by frame index, output the global semantic vector, and use the corresponding mask region reconstruction loss as one of the criteria for global semantic change. All of these are stored by frame index.

6. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S4 specifically includes: S41. Read the SuperPoint dense descriptors of adjacent frames into LightGlue in frame order to form feature sets for image I and image S respectively. In LightGlue, establish a learnable representation for each feature point. S42. Calculate self-attention within the same image. For each point in image I, use the point as the query and other points in I as the key to perform correlation calculation, obtain the attention score within the same image and update the descriptor. Perform the same operation on image S and calculate cross-attention between the two images. Use the points in image I as the query and all points in image S as the key to obtain the attention score from I to S and update the descriptor of I accordingly. Use the points in S as the query and I as the key to obtain the attention score from S to I. S44. Read the dense descriptor set of adjacent frames updated by LightGlue, and then read the first descriptor set of image A. The feature vector and the feature vector of image B Each of the eigenvectors is input into a linear transformation function. Mapped to the same metric space, for each pair Calculate the point-pair similarity score from the mapped vectors, and write the score into the similarity matrix. traverse all elements at that location. By combining the two images, we obtain the point-to-point similarity matrix between them. S45. Predict the matching score for each point. Take the updated representation of the point by LightGlue as input, map it to the 0-1 interval through the Sigmoid activation function, and output the matching score of the point. This process is repeated for each point in image A. For each point in image B, we obtain , as the gate factor of the soft allocation matrix; S46. Based on the similarity matrix and matching scores, pair any two points. The matching scores on both sides are combined with the row or column similarity normalization term to obtain the soft assignment score and generate the soft assignment matrix. S47. Based on the allocation scores of each pair of points in the soft allocation matrix, select the pair of points that meet the following conditions as the matching pair set: the soft allocation score ranks first in the current row or column and is higher than the scores of all other pair of points in the same row or column by at least a set threshold. All the pair of points that meet the conditions are combined into a matching pair set and output. The percentage of valid matching points is calculated as the local feature matching degree and output along with it.

7. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S5 specifically includes: S51. Calculate the proportion of the number of matching points in the number of key points by taking the larger of the number of valid matching points and the number of key points detected in frames 0 and 1 respectively. S52. Take the global semantic vectors extracted by the VideoMAE network from the two frames respectively, calculate the dot product of the global semantic vectors of the two frames, divide by the product of their respective L2 norms, and obtain the cosine similarity of the global semantic vectors. S53. Weighted fusion of local feature matching degree and global semantic similarity is performed, and the influence weight on the fusion similarity is controlled by the weighting factor to generate the fusion similarity. S54. When using VideoMAE for frame reconstruction, calculate the change in reconstruction loss between the current frame and the previous frame to measure the degree of change in spatiotemporal structure between frames. When the value is greater than the threshold, it indicates that there are significant semantic differences between frames.

8. The video keyframe extraction method integrating self-supervised deep learning according to claim 1, characterized in that, S6 specifically includes: analyzing the dynamic characteristics between video frames, manually setting a local feature matching degree threshold, a global semantic vector similarity threshold, a fusion similarity threshold, and a reconstruction loss change threshold, determining the current frame as a key frame when any indicator meets the corresponding condition, adding the frames that meet the conditions to the key frame set, and continuing to repeat the determination process for video frames, outputting a set of all key frames that meet the key criteria.

Images