A large model-based video key frame adaptive extraction method

Abstract

The application relates to the technical field of image extraction, in particular to a video key frame self-adaptive extraction method based on a large model, which comprises the following steps: receiving a to-be-processed video stream and decoding the to-be-processed video stream into continuous sequence frames, performing global semantic analysis on the sequence frames to generate a global semantic text vector, and simultaneously performing pixel feature extraction on each frame image in the sequence frames. The application quantifies the expression contribution of local pictures to a global theme, suppresses the interference of noise frames which are visually clear but irrelevant in semantics, performs a greedy iterative search on this basis, calculates an orthogonal projection component in the remaining candidate set and selects a frame which can maximize the amount of new information as a semantic orthogonal key frame, utilizes an orthogonalization strategy to ensure the independence of the selected frame in a feature space and eliminate the semantic overlap between the key frames, and finally generates a self-adaptive video summary which has high information coverage and low redundancy, thereby improving the accuracy and browsing efficiency of long video content understanding.

Description

Technical Field

[0001] This invention relates to the field of image extraction technology, and in particular to an adaptive extraction method for video keyframes based on a large model. Background Technology

[0002] Image extraction technology identifies and separates the most representative still image frames from a continuous video stream containing a large amount of spatiotemporal redundant information, in order to form a summary sequence that can summarize the main content or significant events of the video.

[0003] Current technologies for video summarization primarily rely on the statistical properties of low-level visual features, such as color histogram differences, optical flow motion vectors, or pixel-level inter-frame differences. Lacking an understanding of the overall thematic content of the video, they determine keyframes solely by detecting drastic changes in visual elements. This leads to misjudgments in scenes rich in visual variation but lacking in semantic information. For example, when filming scenes of leaves swaying violently in the wind or water shimmering, a large number of keyframes are extracted due to significant statistical differences in pixel values, resulting in redundant summarization. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing an adaptive extraction method for video keyframes based on a large model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: an adaptive extraction method for video keyframes based on a large model, comprising the following steps: The system receives the video stream to be processed and decodes it into a continuous sequence of frames. It performs global semantic parsing on the sequence of frames to generate a global semantic text vector. At the same time, it extracts pixel features from each frame in the sequence of frames to construct a video frame feature matrix. The video frame feature matrix is ​​divided into continuous visual units along the time axis. The local probability distribution parameters of each visual unit are statistically analyzed. The divergence score of two adjacent local probability distribution parameters is calculated. Based on the divergence score, the corresponding high information gain video segment is extracted. The local feature vectors corresponding to the segments are indexed from the video frame feature matrix. The semantic signal-to-noise ratio score is calculated based on the local feature vectors. The semantic signal-to-noise ratio score is used as a weighting factor and mapped to the corresponding high information gain video segments to construct a semantically weighted candidate frame set. Based on the semantically weighted candidate frame set, perform iterative search to obtain semantically orthogonal keyframes, repeat the iteration and aggregate all the semantically orthogonal keyframes to generate an adaptive video keyframe summary.

[0006] Preferably, the step of obtaining the video frame feature matrix is ​​as follows: Receive the video stream to be processed, parse the encapsulation header information and encoding parameters, read the bit stream segments in order of start time, decode each segment and output frame images numbered by frame number to form a sequence of frames; Based on the sequence of frames, extract scene object label characters, action description terms, and background context keywords for each frame, merge the time position weights of the same terms and encode them according to a fixed word list order to generate a global semantic text vector. Set the weight configurations for texture statistics, color channel histograms, and spatial distribution raster indexes of pixel feature fields, collect the corresponding fields frame by frame and concatenate them into a single-frame feature vector, then stitch them together in chronological order to generate a video frame feature matrix.

[0007] Preferably, the step of obtaining the local probability distribution parameters is as follows: The video frame feature matrix is ​​divided into several equal-length visual units along the time axis according to the frame number. The value range of each frame feature component in each visual unit is statistically analyzed by binning with fixed binning boundaries. The occurrence frequency of each bin is calculated and normalized with the total number of binnings. The probability components are connected in the binning order to form the probability structure of the visual unit, and local probability distribution parameters are generated.

[0008] Preferably, the step of acquiring the high information gain video segment is as follows: Calculate the divergence score between the current visual unit and the previous visual unit based on the local probability distribution parameters. Based on the divergence scores, the divergence scores corresponding to all visual units are summarized and the average value is calculated as a dynamic baseline. The relationship between each divergence score and the set multiple of the dynamic baseline is compared along the time axis, and the corresponding high information gain video segments are extracted.

[0009] Preferably, the step of obtaining the semantic signal-to-noise ratio score is as follows: Local feature vectors are extracted from the video frame feature matrix in chronological order. Each local feature vector and the global semantic text vector are mapped to a unified semantic embedding space through a pre-trained multimodal projection network. The vector dimensions and distribution range are aligned to generate matching pairs. Based on the matching pairs, a semantic signal-to-noise ratio score is calculated.

[0010] Preferably, the step of obtaining the semantically weighted candidate frame set is as follows: Based on the semantic signal-to-noise ratio (SNR) score, the SNR score of each frame is mapped to the corresponding high information gain video segment. A composite threshold is set, and frames whose SNR scores are higher than both the average segment score and the preset global minimum threshold are selected. These frames are recorded and summarized in time index order to generate a semantically weighted candidate frame set.

[0011] Preferably, the step of obtaining the semantic orthogonal keyframe is as follows: Initialize the selected set to empty, directly select the frame with the highest semantic signal-to-noise ratio score from the semantically weighted candidate frame set as the first key frame and add it to the selected set. Then, calculate the projection component of each frame in the remaining candidate frame set on the feature space spanned by the selected set. After removing the projection component, retain the orthogonal component and time index to generate the orthogonalized remaining candidate set representation. Based on the orthogonalized remaining candidate set representation, the L2 norm of the orthogonal components is calculated frame by frame. The L2 norm of each frame is used as a new information content metric for that frame, independent of the selected set. The L2 norms of all frames are compared, and the frame with the largest L2 norm is selected to generate the semantic orthogonal keyframe for the current round.

[0012] Preferably, the step of obtaining the adaptive video keyframe summary is as follows: The semantic orthogonal keyframes of the current round are added to the selected set and the corresponding frames are removed from the remaining candidate set. It is determined whether the remaining candidate set is empty or whether the maximum L2 norm change is lower than the threshold in two consecutive iterations. If the stopping condition is not met, the iteration continues. If the condition is met, all semantic orthogonal keyframes are summarized in time index order to generate an adaptive video keyframe summary.

[0013] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention receives and decodes a video stream into a continuous sequence of frames, simultaneously performs global semantic parsing and single-frame pixel feature extraction, constructs a video frame feature matrix that integrates visual texture, color, and global latent semantics, establishes a foundation for multimodal feature alignment, divides the feature matrix into continuous visual units along the time axis and statistically analyzes local probability distributions, calculates the divergence score of adjacent distribution parameters to quantify the degree of information fragmentation, locates and extracts high information gain segments, eliminates redundant static images and focuses on areas with significant content changes, and calculates the semantic signal-to-noise ratio score between the extracted segment index local feature vector and the global semantic vector, using this as a weighting factor to construct a semantically weighted candidate set. By quantifying the contribution of local images to the expression of the global theme, interference from visually clear but semantically irrelevant noise frames is suppressed. Based on this, a greedy iterative search is performed, orthogonal projection components are calculated in the remaining candidate set, and frames that maximize new information content are selected as semantically orthogonal keyframes. An orthogonalization strategy is used to ensure the independence of selected frames in the feature space, eliminates semantic overlap between keyframes, and finally generates an adaptive video summary with both high information coverage and low redundancy, improving the accuracy of long video content understanding and browsing efficiency. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0016] Please see Figure 1 This invention provides a technical solution: an adaptive extraction method for video keyframes based on a large model, comprising the following steps: The system receives the video stream to be processed and decodes it into a continuous sequence of frames. It performs global semantic parsing on the sequence of frames to generate a global semantic text vector. At the same time, it extracts pixel features from each frame in the sequence of frames to construct a video frame feature matrix. The video frame feature matrix is ​​divided into continuous visual units along the time axis. The local probability distribution parameters of each visual unit are statistically analyzed. The divergence score between two adjacent local probability distribution parameters is calculated. Based on the divergence score, the corresponding high information gain video segments are extracted. The local feature vectors corresponding to the segments are indexed from the feature matrix of the video frames. The semantic signal-to-noise ratio score is calculated based on the local feature vectors. The semantic signal-to-noise ratio score is used as a weighting factor and mapped to the corresponding high information gain video segments to construct a semantically weighted candidate frame set. Based on the semantically weighted candidate frame set, perform iterative search to obtain semantically orthogonal keyframes, repeat the iteration and aggregate all semantically orthogonal keyframes to generate an adaptive video keyframe summary.

[0017] The steps for obtaining the video frame feature matrix are as follows: Receive the video stream to be processed, parse the encapsulation header information and encoding parameters, read the bit stream segments in order of start time, decode each segment and output frame images numbered by frame number to form a sequence of frames; Based on the sequence of frames, extract scene object label characters, action description terms, and background context keywords for each frame. Merge the temporal position weights of the same terms and encode them according to a fixed vocabulary order to generate a global semantic text vector. Set the weight configurations for texture statistics, color channel histograms, and spatial distribution raster indexes of pixel feature fields, collect the corresponding fields frame by frame and concatenate them into a single-frame feature vector, then stitch them together in chronological order to generate a video frame feature matrix.

[0018] Specifically, the process receives the video stream to be processed, establishes an input buffer to temporarily store the received binary bitstream data, and sets the buffer size to a preset value, such as 4096KB. It reads the file header identifier from the beginning of the bitstream, parses the encapsulation information according to container format specifications such as MP4 or FLV, extracts global metadata including video duration, frame rate, resolution, and encoding format, locates the data atoms or tag positions on the audio and video tracks, separates independent video basic stream data packets, and reads the timestamp information in the header of each data packet, including the decoding timestamp (DTS) and display timestamp (PTS). To address potential out-of-order transmission issues, a sorting queue is constructed, and the read bitstream segments are rearranged according to their DTS values ​​from smallest to largest. For data packets missing DTS, they are padded using a fixed step size increment of the previous frame's DTS, with the step size set to 10 times the reciprocal of the frame rate. At 00 milliseconds, the corresponding decoder instance is started. For example, the context environment is initialized for H.264 encoding format. The sorted bitstream segments are sent to the decoding buffer. Entropy decoding, inverse quantization, and inverse transform operations are performed to reconstruct the original image data in YUV format. Image frames are placed into the output queue according to the order indicated by PTS. Color space conversion is performed on each frame, converting it from YUV420P format to RGB888 format for subsequent pixel-level processing. A frame number counter is set to continuously number each output frame starting from 0. The frame number is appended as metadata to the attribute field of the image matrix. If a corrupted frame with a decoding error is encountered, the time index of the frame is recorded and marked as an empty space in the subsequent sequence or copied and filled using the previous keyframe. The entire video stream is traversed until all segments are decoded, forming a sequence of frames.

[0019] Based on the sequence of frames, a multi-label image recognition neural network model is constructed for feature extraction. This model structure includes a backbone feature extraction network and three parallel classification branches. The backbone network adopts a deep residual network structure, consisting of five stacked convolutional blocks. Each convolutional block contains three 3x3 convolutional layers and a batch normalization layer. The input receives frame images resized to 224x224 pixels, and the output is connected to a global average pooling layer. For the model training process, a labeled dataset containing general scenes, common objects, and action categories is collected. A fixed vocabulary containing 10,000 categories is set. The binary cross-entropy loss function is used to calculate the predicted probability. To differentiate between real labels, a stochastic gradient descent optimizer was used for parameter updates, with an initial learning rate of 0.01. Training was iterated for 100 epochs until the loss function converged. During inference, each frame in the sequence was input into the trained model. The model's three branches output confidence vectors for scene objects, action descriptions, and background context, respectively. A confidence filtering threshold of 0.75 was set, obtained by maximizing the F1-score on the validation set. Only labels, words, and keywords with confidence scores higher than this threshold were retained. The timestamp position of the retained words in the current frame was obtained, and the time position weight was calculated using the following formula: ,in As for time and location weights, This represents the number of seconds in the current frame relative to the start of the video. The base of the natural logarithm is used to assign higher initial weights to the information in the early stages of the video. For the same word that appears repeatedly in multiple frames, the weights of its corresponding time positions are accumulated to obtain the global weight value of the word. According to the pre-set fixed word list order, all extracted non-repeating words and their global weight values ​​are mapped to a sparse vector with a dimension equal to the length of the word list. The element value at the corresponding word index position in the vector is its accumulated global weight value, and the remaining positions are padded with zeros to generate a global semantic text vector.

[0020] The weight configurations for texture statistics, color channel histograms, and spatial distribution raster indexes in the pixel feature field are set, with texture statistics weighted at 0.3, color channel histogram weighted at 0.5, and spatial distribution raster index weighted at 0.2. This weight configuration is based on statistical analysis of recall performance in a large number of test videos during keyframe retrieval tasks. First, for the texture statistics field, the RGB image is converted to grayscale, and the grayscale co-occurrence matrix is ​​calculated. The scanning distance is set to 1 pixel, and the scanning angles are 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Based on the co-occurrence matrix, four statistical measures—energy, contrast, correlation, and entropy—are calculated. These four measures are normalized to the 0-1 interval and then concatenated to form the texture feature vector. For the color channel histogram field, the image is separated into R, G, and B channels. The 0-255 brightness range of each channel is divided into 32 equally wide bins, and the number of pixels falling into each bin is statistically analyzed. The number of pixels is divided by the total number of pixels in the image to obtain the normalized frequency. The frequency vectors of the three channels are concatenated end to end to form the color feature vector. For the spatial distribution raster index field, the image is divided into 16 rectangular grids with 4 rows and 4 columns. The mean and variance of the brightness of the pixels in each grid are calculated to form a spatial feature vector containing 32 values. According to the weight configuration set above, the feature vectors of the three dimensions are multiplied by the corresponding weight coefficients, that is, all elements of the texture feature vector are multiplied by 0.3, all elements of the color feature vector are multiplied by 0.5, and all elements of the spatial feature vector are multiplied by 0.2. The three weighted vectors are concatenated in the order of texture, color, and spatial to obtain a single-frame feature vector of fixed length. According to the time order of the video frames, all single-frame feature vectors are stacked as row vectors to construct a two-dimensional numerical matrix with the number of rows equal to the total number of frames and the number of columns equal to the length of the single-frame feature vector, thus generating the video frame feature matrix.

[0021] The steps to obtain the local probability distribution parameters are as follows: The video frame feature matrix is ​​divided into several equal-length visual units along the time axis according to the frame number. The value range of each frame feature component in each visual unit is statistically analyzed by binning with fixed binning boundaries. The occurrence frequency of each bin is calculated and normalized with the total number of binnings. The probability components are connected in the binning order to form the probability structure of the visual unit, and local probability distribution parameters are generated.

[0022] Specifically, the video frame feature matrix is ​​divided into several equal-length visual units along the time axis according to frame number. The duration parameter of the visual unit is set to a fixed value, which is set with reference to the persistence of human vision and the shortest average cycle of general video shot transitions. For example, a large number of movie and surveillance video samples are selected for statistical analysis, and the average shortest stable shot duration is about 1 second. Combined with a video frame rate of 25 frames per second, each visual unit is set to contain 25 consecutive frames. Starting from the first row of the feature matrix, every 25 rows of data are read as an independent calculation block. If the number of remaining frames at the end is less than 25, it is included in the last visual unit. For each divided visual unit, the 25 single-frame feature vectors contained therein are read. Numerical analysis is performed on each dimension component of the feature vector to determine its value distribution range. In order to unify the quantification standard, feature extraction and analysis are performed on a general video dataset with a duration of one hour in advance to statistically determine the maximum and minimum values ​​of each feature component. For example, the value range of texture feature components is concentrated between 0 and 1. The number of bins is set to a preset value of 20. The design is based on balancing computational load with the preservation of distribution details. By comparing the distribution fitting effects of 10, 20, and 50 bins, it was found that 20 bins can cover 95% of the distribution features with minimal increase in computation time. Therefore, the value range of 0 to 1 is divided into 20 equal intervals, each with a step size of 0.05. All feature component values ​​within the current visual unit (25 frames) are iterated through, and the specific bin index into which each value falls is determined. For example, if the value 0.12 falls into the third bin, the corresponding counter is incremented. After the statistics are completed, the value of each bin is obtained. The absolute frequency of the bins is obtained by taking the total number of samples in the current visual unit, which is the number of frames multiplied by the number of feature dimensions. The statistical frequency of each bin is divided by the total number of samples to obtain the normalized probability value. This ensures that the sum of the probabilities of all bins is 1. The 20 calculated probability values ​​are arranged in ascending order of bin index to construct a probability distribution vector with a dimension of 20. This vector represents the feature distribution state within the visual unit. The above binning statistics and normalization operations are repeated for all visual units segmented in the video stream to generate local probability distribution parameters.

[0023] The steps for obtaining high information gain video clips are as follows: Based on the local probability distribution parameters, the divergence score between the current visual unit and the previous visual unit is calculated using the following formula: ; in, For the divergence score, This is the local probability distribution vector of the previous visual unit. This represents the local probability distribution vector of the current visual unit. As a contextual reference distribution, it is obtained by referring to the continuous sequence preceding the previous visual unit. The average of the local probability distribution vectors of each visual unit is used to represent the historical stable state before the current change. The context deviation factor is used to measure the magnitude of the change in entry, and is defined as follows: ,in For Jensen-Shannon divergence, For context-adaptive mixture probability distribution, it is defined as follows: This represents the weighted mixture distribution among adjacent visual units based on the degree of contextual deviation. This is the Kullback-Leibler divergence, used to calculate the difference between two probability distributions; Based on the divergence scores, the divergence scores corresponding to all visual units are summarized and the average value is calculated as a dynamic baseline. The relationship between each divergence score and the set multiple of the dynamic baseline is compared along the time axis, and the corresponding high information gain video segments are extracted.

[0024] Specifically, the divergence score calculation formula addresses the problem that simply comparing differences between adjacent frames cannot perceive long-term trends in context by measuring the difference between the current distribution and the previous distribution relative to the mixed intermediate state. Its design philosophy lies in using Jensen-Shannon divergence to construct weights. This causes the mixed distribution to deviate significantly from the historical context (contextual reference distribution) when newly emerging visual content deviates from the historical context. It can dynamically adjust the center of gravity according to the degree of deviation, thereby quantifying the information gain at the point of mutation; The local probability distribution vector of the previous visual unit is derived from the first visual unit. The statistical normalization result of each visual unit represents the visual state immediately preceding the current time step. In this example, a simplified two-dimensional vector is used. ; The local probability distribution vector of the current visual unit is derived from the first... The statistical normalization result of each visual unit represents the visual content being evaluated. In this example, we take... This indicates that a drastic change has occurred; As a contextual reference distribution, by referring to the previous The average distribution of visual units is used to characterize the long-term trend of historical background. This example sets... This indicates a relatively balanced historical background, while A certain deviation has occurred; Context deviation factor, a dimensionless coefficient, in the range [0,1], used to adjust the mixing ratio based on the degree of historical deviation; For context-adaptive mixture probability distribution, it is and Weighted mixture; This is the Kullback-Leibler divergence, expressed in bits, used to calculate the distribution difference.

[0025] Calculations based on parameters: calculate and Jensen-Shannon divergence between : intermediate distribution ; ; ; ; calculate and Jensen-Shannon divergence between : intermediate distribution ; ; ; ; Calculate the context deviation factor : ; Computational context adaptive mixture probability distribution : (Keep four decimal places); Calculate divergence score : ; ; ; This result indicates that, in the current visual unit distribution Compared to the previous unit Dramatic changes occurred, and the previous unit relative to historical background Even with only minor deviations, the algorithm calculates a divergence score as high as 1.352 bits. This high score accurately reflects that the current change is a significant content update that is not background noise and will be identified by the system as a potential high information gain keyframe region.

[0026] Based on the divergence scores, a floating-point list is created to store the divergence scores of all visual units corresponding to the entire video length. This list is iterated through, all scores are summed, and then divided by the total number of visual units to obtain the average divergence score of the entire video. The dynamic baseline multiplier is set to 1.5. This multiplier is based on experimental analysis of videos with different dynamic ranges. Ten segments each of news broadcasts, sports events, and landscape documentaries were selected for testing. Key turning points were manually marked, and precision and recall were statistically analyzed at different multipliers. It was found that a multiplier of 1.2 contained too much redundant noise, a multiplier of 2.0 missed some secondary scene transitions, and the F1 score reached its optimum at 1.5. Therefore, the average divergence score was multiplied by 1.5. The dynamic baseline value is calculated, and the divergence score of each visual unit is read sequentially along the time axis. This score is compared with the dynamic baseline. If the current score is greater than the dynamic baseline, the visual unit is marked as a high-information candidate point. Considering the continuity of video content, for visual units that continuously meet the conditions, their start and end times are merged to form a continuous time period. At the same time, a minimum segment length threshold is set. This threshold is determined by statistically analyzing the shortest time required for human vision to understand the semantics of the image, for example, 0.5 seconds. If the length of the extracted segment is less than this threshold, it is extended by the length of one visual unit before and after to complete the segment. Finally, all start and end points that meet the conditions are determined, and the corresponding high-information-gain video segments are extracted.

[0027] The steps to obtain the semantic signal-to-noise ratio score are as follows: Local feature vectors are extracted from the video frame feature matrix in temporal index order. Each local feature vector and the global semantic text vector are mapped to a unified semantic embedding space through a pre-trained multimodal projection network. The vector dimensions and distribution range are aligned to generate matching pairs. Based on the matching pairs, the semantic signal-to-noise ratio score is calculated using the following formula: ; in, Let be the semantic signal-to-noise ratio score of the i-th frame. Let be the local visual embedding vector of the i-th frame. It is a global semantic embedding vector. The dot product of the local visual embedding vector and the global semantic embedding vector represents the correlation strength between the two in the semantic space. The square of the magnitude of the sum of squares of the local visual embedding vector components. The modulus square of the sum of squares of the global semantic embedding vector components. To prevent extremely small positive numbers with a denominator of zero.

[0028] Specifically, local feature vectors are extracted from the video frame feature matrix in temporal index order. A pre-trained multimodal projection network model is then invoked. This model aims to eliminate the heterogeneity between visual and text features. The extracted local feature vectors of each frame are input into the network's visual encoding layer and mapped to a 256-dimensional common semantic space through fully connected matrix multiplication. Simultaneously, global semantic text vectors generated by the natural language processing module are obtained and mapped to the same-dimensional semantic space through the network's text encoding layer. During this process, layer normalization techniques are used to adjust the feature distribution, scaling the activation values ​​of different dimensions to a uniform dynamic range, such as a standard normal distribution with a mean of 0 and a variance of 1, to avoid certain high-amplitude features dominating the calculation results. However, the vector magnitude is not forcibly compressed to 1, thus preserving the intensity information of the features in the semantic space. The processed local visual vectors of the i-th frame are matched one-to-one with the global semantic text vectors to form matching pairs to be calculated. All frames in the video stream are traversed, and spatial projection and alignment operations are performed sequentially to generate matching pairs.

[0029] In the semantic signal-to-noise ratio (SNR) score calculation formula, the semantic SNR is constructed using the orthogonal projection principle. The global semantic vector is used as the reference signal axis, and the energy ratio between the projection component (signal) of the local visual vector on the axis and the orthogonal component (noise) perpendicular to the axis is calculated. Compared with cosine similarity, this formula greatly stretches the score difference between high-relevance and low-relevance frames through nonlinear mapping, making the key frame distinction more significant. This is the local visual embedding vector for the i-th frame, derived from the output of the projection network, containing 256-dimensional feature components. In this example, to simplify the calculation, a truncated vector representing the significant feature dimension is selected. , which represents the specific coordinates of the current frame in the semantic space; This is a global semantic embedding vector, derived from the output of the text encoding branch. In this example, the corresponding vector is defined. , representing the direction of the target semantics in space; Let be the dot product of the local visual embedding vector and the global semantic embedding vector, and let be a scalar representing the correlation strength. The square of the magnitude of the sum of squares of the local visual embedding vector components represents the total energy of the visual vector; The modulus square of the sum of squares of the global semantic embedding vector components represents the total energy of the semantic vector; For a very small positive number, set to . It is used for numerical stability protection.

[0030] Calculations based on parameters: calculate The square of the modulus: ; calculate The square of the modulus: ; Calculate the vector dot product : ; Calculate the square of the dot product (signal energy term): ; Calculate the total energy product: ; Calculate the denominator (noise energy term): ; Calculate the final score: ; The results indicate that the signal energy directly related to "global semantics" in the visual content of the current frame is 3.76 times that of irrelevant background noise energy. This value is significantly higher than the level of random noise, meaning that the frame contains highly matched semantic information.

[0031] The steps for obtaining the semantically weighted candidate frame set are as follows: Based on the semantic signal-to-noise ratio (SNR) score, the SNR score of each frame is mapped to the corresponding high information gain video segment. A composite threshold is set, and frames whose SNR scores are higher than both the average segment score and the preset global minimum threshold are selected. These frames are recorded and summarized in chronological order to generate a semantically weighted candidate frame set.

[0032] Specifically, based on the semantic signal-to-noise ratio (SNR) score, a dual threshold screening mechanism is initiated to construct a high-quality candidate set. First, the local average score within each high information gain video segment is calculated. The SNR values ​​of all frames within the segment are traversed, summed, and divided by the number of frames to obtain the local mean. For example, the mean of a segment is 3.0. This mean reflects the basic semantic relevance level of the current scene. At the same time, a preset global minimum threshold is set. This threshold is set based on the statistical analysis of the SNR distribution of a large number of negative samples (i.e., completely irrelevant video frames). By calculating the 99th percentile of the negative sample scores, a value of 1.2 is obtained. This is used as the global baseline to eliminate absolutely irrelevant noise frames. Frame-by-frame judgment logic is executed. The score of the i-th frame (e.g., 3.7647) is read and compared with the mean of the segment (3.0) and the global threshold (1.2). Only when the score is greater than both of these reference values ​​is the frame determined as a valid candidate. Frames that meet the conditions are stored in a dynamic array in timestamp index order, and their original scores are retained as a reference for subsequent sorting weights. All the frame data that pass the screening are summarized to generate a semantically weighted candidate frame set.

[0033] The steps for obtaining semantic orthogonal keyframes are as follows: The selected set is initialized to be empty. The frame with the highest semantic signal-to-noise ratio score is directly selected from the semantically weighted candidate frame set as the first key frame and added to the selected set. Then, the projection component is calculated on the feature space spanned by the selected set for each frame in the remaining candidate frame set. After removing the projection component, the orthogonal component and time index are retained to generate the orthogonalized remaining candidate set representation. Based on the orthogonalized remaining candidate set representation, the L2 norm of the orthogonal components is calculated frame by frame. The L2 norm of each frame is used as a new information content metric for that frame, independent of the selected set. The L2 norms of all frames are compared, and the frame with the largest L2 norm is selected to generate the semantic orthogonal keyframe for the current round.

[0034] Specifically, the selected set is initialized to empty. All candidate frames are traversed from the semantically weighted candidate frame set, and the semantic signal-to-noise ratio score of each frame is read. The frame with the highest score is found through a sorting algorithm, marked as the first seed keyframe, and added to the selected set. This step establishes the most representative initial visual benchmark. Subsequently, for each remaining frame in the set, a feature projection operation based on the Schmidt orthogonalization principle is performed. First, the feature vectors of all keyframes in the current selected set are obtained, and a set of orthogonal basis vectors is constructed. For each remaining candidate frame feature vector to be processed, its projection components in each orthogonal basis direction are calculated. The calculation formula is that the projection component is equal to the dot product of the candidate frame vector and the basis vector divided by the square of the modulus of the basis vector and then multiplied by the basis vector. The original candidate frame vector is subtracted from the sum of the projection components in all basis directions to obtain the orthogonal component perpendicular to the selected space. This orthogonal component represents the novel information in the candidate frame that is completely independent of the selected keyframes. The numerical vector of this orthogonal component and its corresponding original time index are recorded. The data structure of all remaining frames after processing is reorganized to generate the orthogonalized remaining candidate set representation.

[0035] Based on the orthogonalized residual candidate set representation, each element in the set is traversed, and its corresponding orthogonal component vector is extracted. The L2 norm of the vector is calculated, which is the square root of the sum of the squares of all components of the vector. The L2 norm value directly quantifies the degree of difference or information gain of the frame image relative to the selected keyframe set. The larger the norm value, the more new information the frame contains. All the calculated L2 norms are stored in a temporary list, and the maximum value search process is started. The values ​​in the list are compared, and the frame index corresponding to the L2 norm with the largest value is located. This frame is the preferred frame that can provide the maximum marginal information gain in the current step. It is marked as selected, and the semantic orthogonal keyframe of the current round is generated.

[0036] The steps for obtaining adaptive video keyframe summaries are as follows: The semantic orthogonal keyframes of the current round are added to the selected set and the corresponding frames are removed from the remaining candidate set. It is determined whether the remaining candidate set is empty or whether the maximum L2 norm change is lower than the threshold in two consecutive iterations. If the stopping condition is not met, the iteration continues. If the condition is met, all semantic orthogonal keyframes are summarized in time index order to generate an adaptive video keyframe summary.

[0037] Specifically, the semantically orthogonal keyframes of the current round are added to the selected set, and the corresponding frames are removed from the remaining candidate set. The feature space basis of the selected set is updated, and then the iteration termination determination stage begins. First, it checks whether the remaining candidate set is empty; if it is, the process stops directly. Second, it calculates the difference between the L2 norm of the orthogonal component of the keyframe selected in the current round and the L2 norm of the keyframe selected in the previous round. The threshold for the change in the L2 norm is set to 0.15. This threshold is based on keyframe redundancy analysis of a large number of test videos, observing the norm... The inflection point of the decline curve reveals that when the amount of new information is less than 15% of the initial maximum amount of information, the newly added keyframes are mostly repetitive or slightly modified frames, and no longer have summarizing value. If the current change is still higher than the threshold, it means that there is still high-value information to be extracted from the remaining frames. The process continues to return to the previous stage for the next round of orthogonalization calculation and selection. If any stopping condition is met, the iteration loop ends, and all keyframes in the selected set are rearranged in ascending order of their original timestamps. The corresponding image data and metadata are extracted to generate an adaptive video keyframe summary.

[0038] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An adaptive keyframe extraction method for video based on a large model, characterized in that, Includes the following steps: The system receives the video stream to be processed and decodes it into a continuous sequence of frames. It performs global semantic parsing on the sequence of frames to generate a global semantic text vector. At the same time, it extracts pixel features from each frame in the sequence of frames to construct a video frame feature matrix. The video frame feature matrix is ​​divided into continuous visual units along the time axis. The local probability distribution parameters of each visual unit are statistically analyzed. The divergence score of two adjacent local probability distribution parameters is calculated. Based on the divergence score, the corresponding high information gain video segment is extracted. The local feature vectors corresponding to the segments are indexed from the video frame feature matrix. The semantic signal-to-noise ratio score is calculated based on the local feature vectors. The semantic signal-to-noise ratio score is used as a weighting factor and mapped to the corresponding high information gain video segments to construct a semantically weighted candidate frame set. Based on the semantically weighted candidate frame set, perform iterative search to obtain semantically orthogonal keyframes, repeat the iteration and aggregate all the semantically orthogonal keyframes to generate an adaptive video keyframe summary.

2. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the video frame feature matrix are as follows: Receive the video stream to be processed, parse the encapsulation header information and encoding parameters, read the bit stream segments in order of start time, decode each segment and output frame images numbered by frame number to form a sequence of frames; Based on the sequence of frames, extract scene object label characters, action description terms, and background context keywords for each frame, merge the time position weights of the same terms and encode them according to a fixed word list order to generate a global semantic text vector. Set the weight configurations for texture statistics, color channel histograms, and spatial distribution raster indexes of pixel feature fields, collect the corresponding fields frame by frame and concatenate them into a single-frame feature vector, then stitch them together in chronological order to generate a video frame feature matrix.

3. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the local probability distribution parameters are as follows: The video frame feature matrix is ​​divided into several equal-length visual units along the time axis according to the frame number. The value range of each frame feature component in each visual unit is statistically analyzed by binning with fixed binning boundaries. The occurrence frequency of each bin is calculated and normalized with the total number of binnings. The probability components are connected in the binning order to form the probability structure of the visual unit, and local probability distribution parameters are generated.

4. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the high information gain video clip are as follows: Calculate the divergence score between the current visual unit and the previous visual unit based on the local probability distribution parameters. Based on the divergence scores, the divergence scores corresponding to all visual units are summarized and the average value is calculated as a dynamic baseline. The relationship between each divergence score and the set multiple of the dynamic baseline is compared along the time axis, and the corresponding high information gain video segments are extracted.

5. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the semantic signal-to-noise ratio score are as follows: Local feature vectors are extracted from the video frame feature matrix in chronological order. Each local feature vector and the global semantic text vector are mapped to a unified semantic embedding space through a pre-trained multimodal projection network. The vector dimensions and distribution range are aligned to generate matching pairs. Based on the matching pairs, a semantic signal-to-noise ratio score is calculated.

6. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the semantically weighted candidate frame set are as follows: Based on the semantic signal-to-noise ratio (SNR) score, the SNR score of each frame is mapped to the corresponding high information gain video segment. A composite threshold is set, and frames whose SNR scores are higher than both the average segment score and the preset global minimum threshold are selected. These frames are recorded and summarized in time index order to generate a semantically weighted candidate frame set.

7. The adaptive extraction method for video keyframes based on a large model according to claim 1, characterized in that, The steps for obtaining the semantic orthogonal keyframes are as follows: Initialize the selected set to empty, directly select the frame with the highest semantic signal-to-noise ratio score from the semantically weighted candidate frame set as the first key frame and add it to the selected set. Then, calculate the projection component of each frame in the remaining candidate frame set on the feature space spanned by the selected set. After removing the projection component, retain the orthogonal component and time index to generate the orthogonalized remaining candidate set representation. Based on the orthogonalized remaining candidate set representation, the L2 norm of the orthogonal components is calculated frame by frame. The L2 norm of each frame is used as a new information content metric for that frame, independent of the selected set. The L2 norms of all frames are compared, and the frame with the largest L2 norm is selected to generate the semantic orthogonal keyframe for the current round.

8. The adaptive extraction method for video keyframes based on a large model according to claim 7, characterized in that, The steps for obtaining the adaptive video keyframe summary are as follows: The semantic orthogonal keyframes of the current round are added to the selected set and the corresponding frames are removed from the remaining candidate set. It is determined whether the remaining candidate set is empty or whether the maximum L2 norm change is lower than the threshold in two consecutive iterations. If the stopping condition is not met, the iteration continues. If the condition is met, all semantic orthogonal keyframes are summarized in time index order to generate an adaptive video keyframe summary.

Images