Keyframe determination methods, electronic devices, storage media, and computer program products

Abstract

This application provides a keyframe determination method, electronic device, storage medium, and computer program product. The method includes: fusing features of each video frame and the audio of a video stream to obtain a multimodal feature map of each video frame; performing attention interaction between a first text and the multimodal feature map of each video frame to obtain a first feature value of each video frame in the video stream, wherein the first text indicates a business requirement for the video stream; and determining a candidate video frame set from all video frames in the video stream based on the first feature value of each video frame, and determining one or more keyframes from the video frames in the candidate video frame set.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a keyframe determination method, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, keyframe extraction methods include extracting keyframes through low-level image features or frame intervals, and extracting keyframes through features of multiple modalities. These two keyframe extraction methods are usually used to extract representative keyframes in video streams. However, when used to extract specific keyframes according to specific business needs, the matching accuracy between the extracted keyframes and business needs is not high, resulting in inaccurate keyframe extraction. Summary of the Invention

[0003] To address the related technical issues, embodiments of this application provide a keyframe determination method, an electronic device, a storage medium, and a computer program product.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a keyframe determination method, the method comprising: The features of each video frame and the audio of the video stream are fused to obtain a multimodal feature map of each video frame; The first text is interacted with the multimodal feature map of each video frame through attention interaction to obtain the first feature value of each video frame in the video stream, and the first text indicates the business requirements for the video stream; Based on the first feature value of each video frame in the video stream, a set of candidate video frames is determined from all video frames in the video stream, and one or more keyframes are determined from the video frames in the set of candidate video frames.

[0005] In the above scheme, the first feature value includes attention similarity and attention-weighted feature value; the step of determining a candidate video frame set from all video frames in the video stream based on the first feature value of each video frame in the video stream includes: Based on the attention-weighted feature value of each video frame and the first text, a jump sensitivity score is calculated for each video frame. The jump sensitivity score is used to measure the degree of alignment between the semantic jump direction of the corresponding video frame and the business requirements. Based on the attention similarity of each video frame, the information entropy contribution rate of each video frame is calculated. The information entropy contribution rate is used to evaluate the importance of the corresponding video frame in the global semantic distribution of the video stream. The total score of each video frame is determined based on the jump sensitivity score and the information entropy contribution rate of each video frame. Based on the total score of each video frame, the candidate video frame set is determined from all video frames in the video stream.

[0006] In the above scheme, determining one or more keyframes from the video frames within the candidate video frame set includes: The candidate video frame set is divided into multiple non-overlapping candidate video frame subsets; the number of video frames in each of the multiple non-overlapping candidate video frame subsets is equal. Based on the first ratio corresponding to each candidate video frame subset, keyframe sampling quotas are allocated to each candidate video frame subset; the first ratio represents the proportion of the total score of all video frames in the corresponding candidate video frame subset to the total score of all video frames in the candidate video frame set. Based on the total score of each video frame in each candidate video frame subset and the corresponding number of keyframe samples, one or more keyframes are determined.

[0007] In the above scheme, calculating the jump sensitivity score for each video frame based on the attention-weighted feature value of each video frame and the first text includes: For each video frame, based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius of the video frame, the attention-weighted feature value of the video frame is smoothed to obtain a smoothed value of the attention-weighted feature value of the video frame. The second-order difference of the smoothed value of the attention-weighted feature value of the video frame is calculated by performing second-order difference calculation to obtain the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame. The jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed value of the attention-weighted feature value and the first text.

[0008] In the above scheme, calculating the information entropy contribution rate of each video frame based on the attention similarity of each video frame includes: The attention similarity of each video frame is normalized to obtain the attention weight of each video frame; Based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the removal entropy represents the overall information entropy calculated after removing the corresponding video frame from the video stream. The difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

[0009] In the above scheme, the step of fusing features of each video frame and the audio of the video stream to obtain a multimodal feature map of each video frame includes: The audio features of the video stream are time-aligned with the visual features of each video frame to obtain aligned audio features. The visual features of each video frame and the aligned audio features are fused together to obtain the fused features of each video frame. The fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame.

[0010] In the above scheme, determining the total score of each video frame based on the jump sensitivity score and the information entropy contribution rate of each video frame includes: The total score for each video frame is obtained by weighting and summing the jump sensitivity score and information entropy contribution rate of each video frame based on the set weights.

[0011] This application also provides an electronic device, including a processor and a memory for storing a computer program that can run on the processor, wherein the processor, when running the computer program, performs the steps of any of the methods described above.

[0012] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the above methods.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0014] The keyframe determination method, electronic device, storage medium, and computer program product provided in this application embodiment fuse the features of each video frame and the audio of the video stream to obtain a multimodal feature map of each video frame; an attention interaction is performed between a first text and the multimodal feature map of each video frame to obtain a first feature value of each video frame in the video stream, wherein the first text indicates a business requirement for the video stream; based on the first feature value of each video frame in the video stream, a candidate video frame set is determined from all video frames in the video stream, and one or more keyframes are determined from the video frames in the candidate video frame set. This scheme, by fusing video and audio features, can improve the accuracy of semantic understanding of video frames. By calculating the first feature value of each video frame through a first text-guided attention mechanism, the semantic relevance of the video frame to the business requirement and the importance of the video frame to the business requirement can be evaluated. This allows for the extraction of keyframes with high semantic relevance to the business requirement, improving the accuracy and flexibility of keyframe extraction and making this keyframe extraction method applicable to various business scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a keyframe determination method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a keyframe determination method according to an application embodiment of this application; Figure 3 This is a schematic diagram of a keyframe determination device according to an embodiment of this application; Figure 4 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0016] Currently, keyframe extraction methods can be categorized into traditional methods and multimodal-based methods. Traditional keyframe extraction methods typically determine keyframes based on low-level image features or inter-frame statistical differences. These low-level features mainly include color histogram variations, edge contour differences, pixel differences, and optical flow. This approach lacks semantic understanding and a global perspective, is susceptible to redundant frames, and cannot adapt to complex event representations, thus failing to model structural changes. Multimodal-based keyframe extraction methods incorporate multimodal information such as audio, subtitles, and tag text. By fusing multimodal information through multimodal encoders or attention mechanisms, keyframe extraction is performed based on this multimodal information, improving accuracy. This approach achieves superior performance compared to traditional methods, but it also suffers from several drawbacks, including insufficient intermodal temporal alignment, coarse semantic guidance granularity, lack of structural understanding, a simplistic scoring mechanism, lack of inter-frame structural relationship modeling, weak redundancy control, and a missing semantic mutation detection mechanism.

[0017] Based on this, in various embodiments of this application, feature fusion is performed on each video frame and the audio of the video stream to obtain a multimodal feature map of each video frame; attention interaction is performed between a first text and the multimodal feature map of each video frame to obtain a first feature value of each video frame in the video stream, wherein the first text indicates the business requirements for the video stream; based on the first feature value of each video frame in the video stream, a candidate video frame set is determined from all video frames in the video stream, and one or more keyframes are determined from the video frames in the candidate video frame set. The above scheme, by fusing video and audio features, can improve the accuracy of semantic understanding of video frames. By calculating the first feature value of each video frame through a first text-guided attention mechanism, the semantic relevance of the video frame to the business requirements and the importance of the video frame to the business requirements can be evaluated. This allows for the extraction of keyframes with high semantic relevance to the business requirements, improving the accuracy and flexibility of keyframe extraction and making this keyframe extraction method applicable to various business scenarios.

[0018] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0019] This application provides a keyframe determination method, applied to electronic devices. For example... Figure 1 As shown, the method includes: Step 101: Perform feature fusion on each video frame and the audio of the video stream to obtain a multimodal feature map of each video frame.

[0020] Here, the system receives a first text, along with a video stream and its audio. The video stream and audio can be obtained based on specific business requirements. The first text, video stream, and audio can be input from business processes. The method for obtaining the first text, video stream, and audio can be determined based on the specific business scenario. The first text can be pre-defined; that is, it can also be a query text corresponding to a specific business scenario or requirement. The first text can be understood as query text or query semantic text. For example, in a smart commentary scenario for a football match, business requirements might include identifying key football events that need to be highlighted, and obtaining the video stream and audio from the football match playback platform.

[0021] After receiving the video stream and its audio, corresponding modal encoders, including image encoders and audio encoders, are used to extract visual features from the video frames and audio features from the audio stream. Both visual and audio features are temporal features, meaning they are multidimensional features containing a time dimension. For example, the sequence of video frames in the received video stream is... The audio signal segment sequence is Each video frame in the video stream is input to an image encoder for characterization, and the visual features of each video frame are output. The values ​​of visual features are within the set of real numbers in a matrix, which has the same dimension as the visual features. The dimension of the visual features can be M×N×T. This process can also be understood as video frames being converted into visual features by an image encoder. Each audio signal segment is input into an audio encoder for representation, and audio features are output. The range of values ​​for audio features is the set of real numbers in a matrix that is the same as the dimension of the audio features. The dimension of the audio features includes the time dimension T. This process can also be understood as audio signal segments being converted into audio features by an audio encoder.

[0022] Having obtained the visual features and audio features of each video frame, feature fusion is performed on the visual features and audio features of each video frame to obtain a multimodal feature map for each video frame. Specifically, the visual features of each video frame and the audio features corresponding to the time point of the video frame can be fused through concatenation or an attention mechanism. Alternatively, concatenation and attention mechanisms can be combined to achieve feature fusion, resulting in a multimodal feature map for each video frame. For example, the multimodal feature map of each video frame can be represented as follows: .

[0023] By fusing features from video frames and audio, a multimodal feature map is obtained. This multimodal feature map can more comprehensively reflect the semantic information of the video content, thereby enabling more accurate capture of event occurrence points related to the query semantics. For example, in a live sports event scenario, video frames may contain visual information such as player movements and ball trajectories, while audio may contain sound information such as referee whistles and audience cheers. By fusing the visual information of video frames and the sound information of audio, the events occurring at corresponding time points can be more accurately represented, improving the accuracy of keyframe extraction.

[0024] To avoid inaccurate keyframe recognition due to insufficient multimodal time alignment, in one embodiment, the step of fusing features of each video frame and the audio of the video stream to obtain a multimodal feature map of each video frame includes: The audio features of the video stream are time-aligned with the visual features of each video frame to obtain aligned audio features. The visual features of each video frame and the aligned audio features are fused together to obtain the fused features of each video frame. The fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame.

[0025] Here, having obtained the visual features and audio features of each video frame, the audio features of the video stream are aligned with the visual features of each video frame based on the audio features of the video stream's audio and the visual features of each video frame. Specifically, temporal alignment can be achieved through a cross-modal local attention mechanism. Within a local time window before and after the time point of each video frame, the audio segment with the strongest semantic relevance to that video frame is searched, and the audio features corresponding to that audio segment are aligned to the time point of that video frame. This corrects the temporal misalignment problem caused by natural delays between audio and video (e.g., in sports events, the referee's whistle usually lags slightly behind the occurrence of the on-screen event), ensuring semantic consistency between the audio signal and the video frame.

[0026] For example, for each video frame, the visual features of that video frame can be used as the query vector, and the audio features within the local time window of that video frame can be used as the key and value; the aligned audio features at each time point can be calculated using the following formula:

[0027] in, The local attention weights represent the audio features at the j-th time point within a local time window centered at time point t, where the local time window includes all time points within a set alignment window radius. and The projection matrix representing the trainable model can be determined in advance through end-to-end supervised training. The alignment window radius can be set by technicians based on experience, or it can be determined based on specific events and video frame rates. For example, the window size can range from 8 to 16 frames, or the window radius can range from 4 to 8 frames, to ensure sufficient modeling of the differences in audio and video modal response latency, while ensuring the contextual consistency and computational efficiency of local attention. Characterizes the aligned audio features at time point t.

[0028] Having obtained the aligned audio features at each time point, each video frame is fused with the aligned audio features at the corresponding time point to obtain the fused features for each video frame; for example, each video frame is concatenated with the aligned audio features at the corresponding time point, that is, the features at each time point are... and The video frames are stitched together to obtain the fused features of each video frame; each video frame can be understood as a video frame at each point in time.

[0029] Having obtained the fused features, the fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame. Specifically, through fully connected layers and trained linear mapping parameters, the fused features of each video frame are linearly mapped to a unified representation space. This projects the fused high-dimensional features into a low-dimensional and compact feature space, generating a multimodal feature map for each video frame. This not only reduces the computational complexity of subsequent processing but also enhances the semantic correlation between different modalities. For example, this can be achieved through formulas... The multimodal features at each time point are calculated, thus obtaining the multimodal feature map for each video frame. Characterizing the multimodal features at time t, and The parameters representing the linear mapping can be obtained in advance through end-to-end supervised training.

[0030] In this embodiment, a time alignment mechanism is introduced. By calculating local cross-modal attention within a sliding window, the natural delay differences between audio and video are corrected, solving the problem of inconsistent audio and video timing. This enables audio information to accurately match the corresponding video content, improving the accuracy of multimodal fusion. Through feature fusion, information from the two modalities is integrated to construct a richer semantic representation, increasing semantic integrity. By using linear mapping, the fused features are compressed to a dimension suitable for downstream tasks, constructing a multimodal feature map, improving the efficiency and quality of video content understanding, and enhancing the accuracy of keyframe extraction.

[0031] Step 102: Perform attention interaction between the first text and the multimodal feature map of each video frame to obtain the first feature value of each video frame in the video stream.

[0032] The first text indicates the service requirements for the video stream.

[0033] Here, upon receiving the first text, a text encoder or pre-trained language model can be used to convert the first text into a semantic vector, which can also be understood as a query vector. The semantic vector is then interacted with the multimodal feature map of each video frame through attention, allowing for element-wise matching between the multimodal feature map and the semantic vector. This can be understood as guiding the semantic focusing process of the multimodal feature map through the semantic vector, resulting in the first feature value for each video frame. The first text can be a natural language description input by the user for a specific business requirement, used to guide attention to video content related to the business needs. For example, given a video and audio of a football match, if the business is to query the football events that occurred in the video, the first text could be "What football events occurred in this video?", where football events include goals, referee whistles, etc.

[0034] The first feature value can include attention-weighted feature values, or it can include attention similarity and attention-weighted feature values. For example, the first feature value can be calculated using the following formula:

[0035] in, Attention similarity is used to represent the relevance weight of a corresponding video frame to the first text. The attention similarity of all video frames can be expressed as follows: ; The attention-weighted feature values ​​represent the structural and semantic orientation information of video frames. The attention-weighted feature values ​​for all video frames can be represented as follows: ; Characterizing the multimodal feature map at time point t, Semantic vectors are represented by language models, which can be obtained by processing the first text.

[0036] Step 103: Based on the first feature value of each video frame in the video stream, determine a set of candidate video frames from all video frames in the video stream, and determine one or more keyframes from the video frames in the set of candidate video frames.

[0037] Here, having obtained the first feature value for each video frame, a candidate video frame set can be directly determined from all video frames in the video stream based on the first feature value. The first feature value can include an attention-weighted feature value, which is used to measure the degree of relevance between each video frame and the first text semantics. The higher the first feature value, the more the video frame meets the business requirements. The video frames can be sorted based on the first feature value of all video frames in the video stream, and the first set number of video frames can be selected and placed into the candidate video frame set. Alternatively, further analysis can be performed based on the first feature value, and then the candidate video frame set can be determined from all video frames in the video stream.

[0038] Given a candidate video frame set, one or more keyframes can be identified from the candidate video frame set based on the first feature value of all video frames within the set. Specifically, one or more keyframes can be selected from the candidate video frame set according to the distribution of the first feature values ​​of the video frames. Before or after keyframes are identified, redundant frames can be removed based on pairwise similarity between keyframes to improve keyframe representation efficiency and semantic diversity. Specifically, a similarity can be calculated between every two keyframes, and if the similarity is greater than a similarity threshold, the keyframe with the lower first feature value among the two keyframes corresponding to that similarity can be removed. The similarity can be an embedded cosine similarity.

[0039] It should be noted that, when one or more keyframes are identified, these keyframes are extracted from the video stream and used as representative frames of the business requirements indicated by the first text, for efficient processing of the business in the subsequent stages.

[0040] To accurately locate keyframes and improve the semantic integrity and query matching accuracy of keyframes, in one embodiment, the first feature value includes attention similarity and attention-weighted feature value; determining a candidate video frame set from all video frames in the video stream based on the first feature value of each video frame includes: Based on the attention-weighted feature value of each video frame and the first text, a jump sensitivity score is calculated for each video frame. The jump sensitivity score is used to measure the degree of alignment between the semantic jump direction of the corresponding video frame and the business requirements. Based on the attention similarity of each video frame, the information entropy contribution rate of each video frame is calculated. The information entropy contribution rate is used to evaluate the importance of the corresponding video frame in the global semantic distribution of the video stream. The total score of each video frame is determined based on the jump sensitivity score and the information entropy contribution rate of each video frame. Based on the total score of each video frame, the candidate video frame set is determined from all video frames in the video stream.

[0041] Here, the first feature value can include attention similarity and attention-weighted feature value. Attention similarity is used to measure the degree of relevance between each video frame and the semantics of the first text. The higher the value, the more the video frame meets the query requirements. Attention-weighted feature value preserves the spatial and temporal structural information of the video frame, which is beneficial for further analysis.

[0042] Given the attention similarity and attention weighted feature values ​​of each video frame, the jump sensitivity score of each video frame can be calculated based on the attention weighted feature values ​​and the semantic vector of the first text. That is, calculating the jump sensitivity score can be understood as local mutation detection based on the semantic jump direction projection, which is used to identify structural changes related to business needs and capture video frames with drastic semantic jumps and correct jump directions. Specifically, the second-order difference algorithm can be used to capture mutations in the semantic trajectory and calculate the jump sensitivity score.

[0043] Furthermore, the information entropy contribution rate of each video frame can be calculated based on the attention similarity of each frame. This is achieved by calculating the information entropy contribution rate based on the global attention distribution, which measures the global impact of a single video frame on the overall semantic focus structure. Specifically, the information entropy contribution rate of a video frame can be obtained by calculating the change in the overall information entropy of the video stream after each video frame is removed. The information entropy contribution rate of a video frame can also be understood as the degree of impact on the overall semantics after removing that video frame, ensuring that high-confidence frames are not missed. A dual-channel scoring strategy is formed by combining the jump sensitivity score and the information entropy contribution rate. For each video frame, the total score of that video frame is determined by combining its jump sensitivity score and information entropy contribution rate, thus obtaining the total score for each video frame.

[0044] Given the total score for each video frame, sort each video frame in descending order of total score, and select the first set number of video frames to add to the candidate keyframe set; for example, select the first 2n video frames with the highest total scores to add to the candidate video frame set.

[0045] In this embodiment, by introducing two complementary importance evaluation parameters, jump sensitivity and information entropy contribution rate, the total score of each video frame is calculated. This not only captures semantic abrupt changes in the video stream but also measures the importance of the video frame in the overall semantic structure, overcoming the shortcomings of a single scoring method that cannot identify structural jumps. Based on the total score of each video frame, video frames with high semantic sensitivity and global importance can be selected from the video frames in the video stream, improving the accuracy and interpretability of keyframe extraction.

[0046] In one embodiment, determining one or more keyframes from the video frames within the candidate video frame set includes: The candidate video frame set is divided into multiple non-overlapping candidate video frame subsets; the number of video frames in each of the multiple non-overlapping candidate video frame subsets is equal. Based on the first ratio corresponding to each candidate video frame subset, keyframe sampling quotas are allocated to each candidate video frame subset; the first ratio represents the proportion of the total score of all video frames in the corresponding candidate video frame subset to the total score of all video frames in the candidate video frame set. Based on the total score of each video frame in each candidate video frame subset and the corresponding number of keyframe samples, one or more keyframes are determined.

[0047] Here, based on the total score of each video frame, a candidate video frame set is determined from all video frames in the video stream. Then, based on the total score of each video frame, one or more keyframes are determined from the video frames within the candidate video frame set. First, the candidate video frame set is divided into multiple non-overlapping subsets. Specifically, this can be done by dividing the video frames equally, for example, arranging all video frames in the candidate video frame set chronologically to obtain a candidate video frame sequence, dividing this sequence into multiple parts, and placing the candidate video frames of each part into the corresponding candidate video frame subset. In this case, the number of video frames in each candidate video frame subset is equal. Alternatively, the candidate video frame subsets can be divided along a timeline, for example, dividing the timeline into multiple equal-length, continuous, non-overlapping windows, each window corresponding to a candidate video frame subset. The candidate video frames within each window are the video frames within the corresponding candidate video frame subset. In this case, the number of video frames in each candidate video frame subset can be the same or different. The number of candidate video frame subsets determines the fineness of the dynamic sampling and can be set according to the actual application.

[0048] Given multiple non-overlapping candidate video frame subsets, calculate the sum of the total scores of all video frames within each candidate video frame subset to obtain the total score of all video frames within the corresponding candidate video frame subset, and calculate the total score of all video frames within the candidate video frame set; calculate the proportion of the total score of all video frames within each candidate video frame subset to the total score of all video frames within the candidate video frame set, obtaining the first proportion corresponding to each candidate video frame subset; allocate keyframe sampling slots to each candidate video frame subset according to the first proportion corresponding to each candidate video frame subset. The keyframe sampling slots can be understood as the number of keyframes that can be selected in the corresponding candidate video frame subset. Specifically, the first proportion corresponding to each candidate video frame subset can be multiplied by the set number of keyframe outputs to obtain the keyframe sampling slots for the corresponding candidate video frame subset. The set number of keyframe outputs can be understood as the maximum number of output frames. For example, if the number of keyframe outputs is set to m, and the first proportion of a certain candidate video frame subset is 1 / 2, then the number of keyframe samples for that candidate video frame subset is m / 2. The number of keyframe outputs can be set according to the actual application.

[0049] With keyframe sampling quotas allocated to each candidate video frame subset, for each candidate video frame subset, based on the total score of all video frames within that subset, a number of video frames equal to the allocated keyframe sampling quota are selected as keyframes. This process is repeated for each candidate video frame subset, resulting in one or more keyframes. Specifically, within the candidate video frame subset, the video frame with the highest total score and the corresponding number of keyframe sampling quotas can be selected as the keyframe.

[0050] It should be noted that if the number of keyframe sampling slots allocated to a candidate video frame subset is greater than the total number of video frames in that candidate video frame subset, the excess keyframe sampling slots of that candidate video frame subset can be allocated to one or more candidate video frame subsets with the highest total score, based on the sum of the scores of all video frames in each candidate video frame subset.

[0051] In this embodiment, after obtaining the candidate video frame set, the sampling positions can be dynamically scheduled by comprehensively considering information such as the importance distribution of video frames, temporal sparsity, and semantic concentration. Specifically, a first ratio is used to allocate keyframe sampling slots to each candidate video frame subset, thereby enabling the extraction of more keyframes in semantically dense regions. This achieves adaptive distribution of keyframes on the timeline, effectively improving the expressive power and summarization effect of the keyframe set and enhancing the accuracy of keyframe extraction. Furthermore, the denser the semantic information, the more keyframe sampling slots are allocated to the candidate video frame subset. Even if the actual total score of some frames is insufficient to be selected, the overall semantic bonus of the candidate video frame subset can still make it a keyframe, avoiding the omission of some important video frames.

[0052] To consider boundary changes of local events and perceive semantic abrupt changes during keyframe extraction, thereby increasing the accuracy of keyframe extraction, in one embodiment, the step of calculating the jump sensitivity score of each video frame based on the attention-weighted feature value of each video frame and the first text includes: For each video frame, based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius of the video frame, the attention-weighted feature value of the video frame is smoothed to obtain a smoothed value of the attention-weighted feature value of the video frame. The second-order difference of the smoothed value of the attention-weighted feature value of the video frame is calculated by performing second-order difference calculation to obtain the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame. The jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed value of the attention-weighted feature value and the first text.

[0053] Here, having obtained the first feature value of each video frame, for each video frame, based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius, the attention-weighted feature value of that video frame is smoothed to eliminate potential jitter or noise in the semantic trajectory composed of the original attention-weighted feature values, enhancing semantic coherence and stability, thus obtaining a smoothed value of the attention-weighted feature value of that video frame. Specifically, the attention-weighted feature value of each video frame can be smoothed using a weighted sliding window convolution. For example, it can be obtained by weighted summation based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius. The sliding step size can be set according to the actual application; for example, the sliding step size can be set to 1. The set window radius is used to determine the length of the time window formed by several adjacent frames selected before and after the current video frame, which is used to extract local time context information. For example, if the time point of the current video frame is 5, i.e., t=5, and the window radius is 2, then the local time window of the current video frame includes the current video frame as well as the two frames before and the two frames after the current video frame. The multiple video frames within the set window radius of the current video frame include the video frames at time points 3, 4, 5, 6, and 7.

[0054] For example, through formula The attention-weighted feature values ​​of each video frame are smoothed; where, The smoothed value representing the attention-weighted feature value of the video frame at time t; k The characteristic window radius can be set according to the actual application. Characterize the elements in the convolution kernel, and set the convolution kernel as... Set the convolution kernel to satisfy The convolution kernel can be set to a fixed Gaussian kernel; the corresponding video frames v t The set window radius range, i.e., the sliding window is .

[0055] After smoothing the attention-weighted feature values ​​of the video frame to obtain smoothed values, a second-order difference is calculated on these smoothed values ​​to capture semantic abrupt changes, resulting in a second-order residual vector of the smoothed attention-weighted feature values. Specifically, the smoothed value of the attention-weighted feature values ​​of the following video frame is subtracted from the smoothed value of the current video frame to obtain a first difference; the smoothed value of the attention-weighted feature values ​​of the following video frame is subtracted from the smoothed value of the attention-weighted feature values ​​of the preceding video frame to obtain a second difference; and the first difference is subtracted from the second difference to obtain the second-order residual vector of the smoothed attention-weighted feature values ​​of the current video frame.

[0056] For example, through formula The second-order residual vector of the smoothed values ​​of the attention-weighted feature values ​​of the video frame is calculated, where, The second-order residual vector representing the smoothed value of the attention-weighted feature value of the video frame (the video frame at time t) can also be understood as the semantic acceleration at time t, indicating the direction and magnitude of the fastest change in the context. It is used to determine whether the current semantic mutation is consistent with the direction of the semantic vector of the first text. Characterizing the first difference, Characterizes the second difference.

[0057] Given the second-order residual vector of the smoothed values ​​of the attention-weighted feature values ​​of the video frame, the jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed values ​​of the attention-weighted feature values ​​of the video frame and the first text. Specifically, the jump sensitivity score of the video frame can be calculated by projecting the second-order residual vector of the smoothed values ​​of the attention-weighted feature values ​​of the video frame onto the normalized semantic vector of the first text.

[0058] For example, through formula The jump sensitivity score of the video frame is calculated; where, The jump sensitivity score characterizes the video frame and is used to measure the alignment between the semantic jump direction of the video frame and the business requirements indicated by the first text, or to capture video frames with drastic semantic jumps and correct jump directions. The semantic vector of the normalized first text represents the semantic direction, which can be expressed by the formula... Calculated.

[0059] To improve the accuracy of keyframe extraction by considering the importance of video frames in the global semantic distribution, one embodiment involves calculating the information entropy contribution rate of each video frame based on the attention similarity of each video frame, including: The attention similarity of each video frame is normalized to obtain the attention weight of each video frame; Based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the removal entropy represents the overall information entropy calculated after removing the corresponding video frame from the video stream. The difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

[0060] Here, after obtaining the first feature value of each video frame, the attention similarity of each video frame is normalized to obtain the attention weight of each video frame, so as to avoid an excessively large attention similarity dominating the overall information entropy distribution. Specifically, the attention weight of each video frame can be calculated based on the ratio of the attention similarity of each video frame to the sum of the attention similarities of all video frames. For example, it can be calculated using the formula... The attention weights for each video frame are calculated; where, The attention weights represent the video frames at time t.

[0061] Having obtained the attention weights for each video frame, the overall information entropy of the video stream and the removal entropy for each video frame are calculated based on these attention weights. For example, this can be achieved using the formula... The overall information entropy of the video stream is calculated, where, The overall information entropy of a video stream can be represented by the formula... The removal entropy of each video frame is calculated, where, The overall information entropy of the video stream after removing the video frame at time point t is calculated.

[0062] Given the overall information entropy of the video stream and the removal entropy of each video frame, the difference between the overall information entropy and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame. The information entropy contribution rate of a video frame can be understood as the change in the overall information entropy of the video stream after removing that video frame. The formula for calculating the information entropy contribution rate of a video frame can be expressed as follows: .

[0063] It should be noted that, after obtaining the information entropy contribution rate of each video frame, an attention density adjustment factor can be introduced to emphasize the global importance of high-confidence frames, and the final information entropy contribution rate of each video frame can be calculated; specifically, this can be achieved through the formula... Calculate the final information entropy contribution rate for each video frame.

[0064] To jointly consider global semantic entropy contribution and local semantic jump direction response during keyframe extraction, avoiding the problem of single scoring and inability to identify structural jumps, thereby improving the semantic integrity and business matching accuracy of keyframes, in one embodiment, determining the total score of each video frame based on the jump sensitivity score and the information entropy contribution rate of each video frame includes: The total score for each video frame is obtained by weighting and summing the jump sensitivity score and information entropy contribution rate of each video frame based on the set weights.

[0065] Here, given the jump sensitivity score and information entropy contribution rate for each video frame, a weighted sum of these scores is calculated based on predefined weights to obtain the total score for each video frame. The predefined weights include jump sensitivity weights and information entropy contribution rate weights, used to balance the contributions of global structure and jump sensitivity. The sum of the jump sensitivity weights and information entropy contribution rate weights is 1, and their specific values ​​are determined based on the actual situation; for example, the jump sensitivity weights and information entropy contribution rate weights can both be 0.5 and 0.5.

[0066] For example, through formula Calculate the total score of the video frames, where, The total score of the video frames at time point t represents the total score of the video frames. The weight representing the information entropy contribution rate in the weighting is defined. The jump sensitivity weight in the weighting is represented, and .

[0067] The first set number of video frames with the highest total score are selected as candidate video frames and added to the candidate video frame set. The number of candidate video frames is greater than or equal to the maximum output frame number, which can also be understood as the set number of keyframes to output. For example, if the maximum output frame number is set to n, the number of candidate video frames can be 2n.

[0068] It should be noted that, based on set weights, the jump sensitivity score and the final information entropy contribution rate of each video frame can also be weighted and summed to obtain the total score of each video frame; for example, it can be obtained through the formula... Calculate the total score for the video frames.

[0069] In this embodiment, by using a weighted summation method, the two scoring methods of jump sensitivity and information entropy are combined to obtain a total score that can reflect the global information structure and local semantic mutations of the video frame, thereby improving the accuracy of key frame extraction.

[0070] The following section provides a more detailed description of this application with reference to application examples.

[0071] like Figure 2 As shown, the keyframe determination method includes the following steps: Step 201: Feature extraction.

[0072] Here, the video stream and the first text are acquired; the video frames of the video stream are input into an image encoder for feature extraction to obtain the visual features of the video frames; the audio of the video stream is input into an audio encoder for feature extraction to obtain the audio features of the video stream; the first text is input into a text encoder for feature extraction to obtain the semantic vector of the first text; alternatively, the first text can be input into a language model for feature extraction to obtain the semantic vector of the first text.

[0073] Step 202: Time-align the audio features of the video stream with the visual features of each video frame to obtain the aligned audio features.

[0074] The specific implementation methods for steps 202-204 are described above and will not be repeated here.

[0075] Step 203: Fuse the visual features and aligned audio features of each video frame to obtain the fused features of each video frame.

[0076] Step 204: Perform linear mapping on the fused features of each video frame to obtain the multimodal feature map of each video frame.

[0077] Step 205: Perform attention interaction between the first text and the multimodal feature map of each video frame to obtain the first feature value of each video frame in the video stream.

[0078] Here, the first text is interacted with the multimodal feature map of each video frame through cross-modal attention to obtain the first feature value of each video frame in the video stream. The first feature value may include attention similarity and attention weighted feature value. The specific implementation method is described above and will not be repeated here.

[0079] Step 206: Calculate the jump sensitivity score for each video frame based on the attention-weighted feature value and the first text of each video frame.

[0080] Here, based on the attention-weighted feature value in the first feature value of each video frame obtained in step 205 and the semantic vector of the first text obtained in step 201, the jump sensitivity score of each video frame is calculated. Specifically, for each video frame, based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius of the video frame, the attention-weighted feature value of the video frame is smoothed to obtain a smoothed value of the attention-weighted feature value of the video frame; a second-order difference is calculated on the smoothed value of the attention-weighted feature value of the video frame to obtain a second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame; based on the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame and the first text, the jump sensitivity score of the video frame is calculated, thereby calculating the jump sensitivity score of each video frame.

[0081] Step 207: Calculate the information entropy contribution rate of each video frame based on the attention similarity of each video frame.

[0082] Here, based on the attention similarity in the first feature value of each video frame obtained in step 205, the information entropy contribution rate of each video frame is calculated. Specifically, the attention similarity of each video frame can be normalized to obtain the attention weight of each video frame; based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

[0083] It should be noted that steps 206 and 207 can be executed simultaneously or sequentially.

[0084] Step 208: Determine the total score for each video frame based on the jump sensitivity score and the information entropy contribution rate of each video frame.

[0085] Here, the total score for each video frame can be obtained by weighting and summing the jump sensitivity score and information entropy contribution rate of each video frame based on the set weights.

[0086] Step 209: Based on the total score of each video frame, determine the set of candidate video frames from all video frames in the video stream.

[0087] Here, the specific implementation method of step 209 is described in the above-mentioned relevant description, and will not be repeated here.

[0088] Step 210: Identify one or more keyframes from the video frames in the candidate video frame set.

[0089] Here, the candidate video frame set is divided into multiple non-overlapping candidate video frame subsets. Based on a first proportion corresponding to each candidate video frame subset, keyframe sampling slots are allocated to each candidate video frame subset. Based on the total score of each video frame within each candidate video frame subset and the corresponding keyframe sampling slot, one or more keyframes are determined. Before determining the division of candidate video frame subsets, or after determining one or more keyframes based on the total score of each video frame within each candidate video frame subset and the corresponding keyframe sampling slot, redundant frames can also be removed based on the pairwise similarity between video frames.

[0090] After identifying one or more keyframes, one or more keyframes can be extracted from the original video stream as representative frames of the business requirements indicated by the first text; a keyframe index set can also be established based on the identified one or more keyframes, and one or more keyframes can be extracted from the original video stream based on the keyframe index set.

[0091] It should be noted that before using the keyframe extraction method, end-to-end training can be performed to optimize various adjustable parameters. These adjustable parameters include a trainable projection matrix that may be used when temporally aligning the audio features of the video stream with the visual features of each video frame. and Linear mapping parameters that may be used when linearly mapping the fused features of each video frame. and The training dataset can be a video understanding task dataset that takes audio, video and text as input and can extract keyframes as ground truth, such as the Ego4D dataset.

[0092] The keyframe determination method provided in this application is a query-guided multimodal adaptive keyframe sampling method, which can achieve accurate scoring and adaptive selection of keyframes, improve the semantic integrity and query matching accuracy of keyframes, and improve the accuracy and interpretability of keyframe sampling. It solves the problems of insufficient multimodal time alignment, inaccurate semantic guidance, and inability to perceive structural abrupt changes in current keyframe extraction methods. Specifically, time alignment is achieved through local cross-modal attention computation using a sliding window, improving the fusion accuracy of multimodal information and solving the problem of modal response delay misalignment. The semantic vector of the first text is used as the query vector to extract semantically relevant frame features from the visual-audio joint representation. This involves cross-modal attention interaction between the first text and the multimodal feature map, overcoming the current coarse-grained screening method that relies on static label or timestamp matching. Combining jump sensitivity and information entropy contribution rate as scoring methods captures semantic mutation points in the video stream and measures the importance of video frames in the overall semantic structure, resulting in more accurate scoring. By dividing candidate video frames into subsets and dynamically sampling based on the total score of video frames within each subset, the extracted keyframes are evenly distributed on the timeline and semantically rich, improving the expressive power and summarization effect of the keyframes.

[0093] The keyframe determination method provided in this application is applicable to structured event recognition in sports scenarios, such as identifying football goals, basketball fouls, offside disputes, etc. It has the advantages of accurate semantic focus and strong interpretability of frame selection. This is only an example of an application scenario and does not impose specific limitations on the application scenario.

[0094] To implement the method of the embodiments of this application, the embodiments of this application also provide a keyframe determination device, such as... Figure 3 As shown, the device includes: The fusion unit 301 is used to fuse the features of each video frame in the video stream and the audio of the video stream to obtain a multimodal feature map of each video frame; Attention unit 302 is used to perform attention interaction between the first text and the multimodal feature map of each video frame to obtain the first feature value of each video frame in the video stream, wherein the first text indicates the business requirements for the video stream; The determining unit 303 is configured to determine a set of candidate video frames from all video frames in the video stream based on a first feature value of each video frame in the video stream, and to determine one or more keyframes from the video frames in the set of candidate video frames.

[0095] In one embodiment, the first feature value includes attention similarity and attention weighted feature value; the determining unit 303 is specifically used to calculate the jump sensitivity score of each video frame based on the attention weighted feature value of each video frame and the first text, wherein the jump sensitivity score is used to measure the degree of alignment between the semantic jump direction of the corresponding video frame and the business requirement. Based on the attention similarity of each video frame, the information entropy contribution rate of each video frame is calculated. The information entropy contribution rate is used to evaluate the importance of the corresponding video frame in the global semantic distribution of the video stream. The total score of each video frame is determined based on the jump sensitivity score and the information entropy contribution rate of each video frame. Based on the total score of each video frame, the candidate video frame set is determined from all video frames in the video stream.

[0096] In one embodiment, the determining unit 303 is specifically used to divide the candidate video frame set into multiple non-overlapping candidate video frame subsets; Based on the first ratio corresponding to each candidate video frame subset, keyframe sampling quotas are allocated to each candidate video frame subset; the first ratio represents the proportion of the total score of all video frames in the corresponding candidate video frame subset to the total score of all video frames in the candidate video frame set. Based on the total score of each video frame in each candidate video frame subset and the corresponding number of keyframe samples, one or more keyframes are determined.

[0097] In one embodiment, the determining unit 303 is specifically used to smooth the attention weighted feature value of each video frame based on a set convolution kernel and the attention weighted feature values ​​of multiple video frames within a set window radius of the video frame, to obtain a smoothed value of the attention weighted feature value of the video frame. The second-order difference of the smoothed value of the attention-weighted feature value of the video frame is calculated by performing second-order difference calculation to obtain the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame. The jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed value of the attention-weighted feature value and the first text.

[0098] In one embodiment, the determining unit 303 is specifically used to normalize the attention similarity of each video frame to obtain the attention weight of each video frame; Based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the removal entropy represents the overall information entropy calculated after removing the corresponding video frame from the video stream. The difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

[0099] In one embodiment, the fusion unit 301 is specifically used to temporally align the audio features of the video stream with the visual features of each video frame to obtain aligned audio features. The visual features of each video frame and the aligned audio features are fused together to obtain the fused features of each video frame. The fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame.

[0100] In one embodiment, the determining unit 303 is specifically used to perform a weighted summation based on a set weight, the jump sensitivity score of each video frame, and the information entropy contribution rate of each video frame to obtain the total score of each video frame.

[0101] In practical applications, the fusion unit 301, the attention unit 302, and the determination unit 503 can be implemented by the processor in the keyframe determination device.

[0102] It should be noted that the keyframe determination device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the keyframe determination device and the keyframe determination method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0103] Based on the hardware implementation of the above program modules, embodiments of this application also provide an electronic device, such as... Figure 4 As shown, the electronic device 400 includes: Communication interface 401 allows for information exchange with other devices or network nodes.

[0104] The processor 402 is connected to the communication interface 401 to enable information interaction with other devices or network nodes, and to execute the methods provided by one or more technical solutions in the above embodiments when running computer programs.

[0105] Memory 403 is used to store computer programs that can run on processor 402.

[0106] Specifically, the processor 402 is used to perform feature fusion on each video frame in the video stream and the audio of the video stream to obtain a multimodal feature map of each video frame; The first text is interacted with the multimodal feature map of each video frame through attention interaction to obtain the first feature value of each video frame in the video stream, and the first text indicates the business requirements for the video stream; Based on the first feature value of each video frame in the video stream, a set of candidate video frames is determined from all video frames in the video stream, and one or more keyframes are determined from the video frames in the set of candidate video frames.

[0107] In one embodiment, the first feature value includes attention similarity and attention weighted feature value; the processor 402 is specifically used to calculate the jump sensitivity score of each video frame based on the attention weighted feature value of each video frame and the first text, wherein the jump sensitivity score is used to measure the degree of alignment between the semantic jump direction of the corresponding video frame and the business requirement. Based on the attention similarity of each video frame, the information entropy contribution rate of each video frame is calculated. The information entropy contribution rate is used to evaluate the importance of the corresponding video frame in the global semantic distribution of the video stream. The total score of each video frame is determined based on the jump sensitivity score and the information entropy contribution rate of each video frame. Based on the total score of each video frame, the candidate video frame set is determined from all video frames in the video stream.

[0108] In one embodiment, the processor 402 is specifically configured to divide the candidate video frame set into multiple non-overlapping candidate video frame subsets; Based on the first ratio corresponding to each candidate video frame subset, keyframe sampling quotas are allocated to each candidate video frame subset; the first ratio represents the proportion of the total score of all video frames in the corresponding candidate video frame subset to the total score of all video frames in the candidate video frame set. Based on the total score of each video frame in each candidate video frame subset and the corresponding number of keyframe samples, one or more keyframes are determined.

[0109] In one embodiment, the processor 402 is specifically configured to, for each video frame, smooth the attention-weighted feature value of the video frame based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius of the video frame, to obtain a smoothed value of the attention-weighted feature value of the video frame. The second-order difference of the smoothed value of the attention-weighted feature value of the video frame is calculated by performing second-order difference calculation to obtain the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame. The jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed value of the attention-weighted feature value and the first text.

[0110] In one embodiment, the processor 402 is specifically configured to normalize the attention similarity of each video frame to obtain the attention weight of each video frame; Based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the removal entropy represents the overall information entropy calculated after removing the corresponding video frame from the video stream. The difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

[0111] In one embodiment, the processor 402 is specifically configured to temporally align the audio features of the audio of the video stream with the visual features of each video frame to obtain aligned audio features. The visual features of each video frame and the aligned audio features are fused together to obtain the fused features of each video frame. The fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame.

[0112] In one embodiment, the processor 402 is specifically configured to perform a weighted summation based on a set weight, the jump sensitivity score of each video frame, and the information entropy contribution rate of each video frame to obtain the total score of each video frame.

[0113] It should be noted that the specific processing procedure of processor 402 can be understood by referring to the above method.

[0114] Of course, in practical applications, the various components in electronic device 400 are coupled together through bus system 404. It can be understood that bus system 404 is used to realize the connection and communication between these components. In addition to a data bus, bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 4The general designated all buses as Bus System 404.

[0115] The memory 403 in this embodiment is used to store various types of data to support the operation of the electronic device 400. Examples of such data include any computer program used to operate on the electronic device 400.

[0116] The methods disclosed in the embodiments of this application can be applied to the processor 402, or implemented by the processor 402. The processor 402 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 402 or by instructions in the form of software. The processor 402 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 402 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the memory 403. The processor 402 reads the information in the memory 403 and completes the steps of the aforementioned method in combination with its hardware.

[0117] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0118] It is understood that the memory (memory 403) in this embodiment of the application can be volatile memory or non-volatile memory, or it can include both volatile memory and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0119] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 403 storing a computer program, which can be executed by the processor 402 of the electronic device 400 to complete the steps described in any of the aforementioned methods. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0120] For example, embodiments of this application also provide a computer program product, including a computer program that can be executed by a processor 402 of an electronic device 400 to perform the steps described in any of the foregoing methods.

[0121] It should be noted that terms such as "first" and "second" are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. The term "and / or" in this document merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The terms "one or more" and "one or more" in this document are exemplary expressions and can be replaced with any possible expressions, such as one or more, at least one, or at least one. Furthermore, the technical solutions described in the embodiments of this application can be arbitrarily combined without conflict. The above descriptions are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. A keyframe determination method, characterized in that, The method includes: The features of each video frame and the audio of the video stream are fused to obtain a multimodal feature map of each video frame; The first text is interacted with the multimodal feature map of each video frame through attention interaction to obtain the first feature value of each video frame in the video stream. The first text indicates the business requirements for the video stream. The first feature value includes attention similarity and attention weighted feature value. Based on the attention-weighted feature value of each video frame and the first text, a jump sensitivity score is calculated for each video frame. The jump sensitivity score is used to measure the degree of alignment between the semantic jump direction of the corresponding video frame and the business requirements. Based on the attention similarity of each video frame, the information entropy contribution rate of each video frame is calculated. The information entropy contribution rate is used to evaluate the importance of the corresponding video frame in the global semantic distribution of the video stream. The total score of each video frame is determined based on the jump sensitivity score and the information entropy contribution rate of each video frame. Based on the total score of each video frame, a set of candidate video frames is determined from all video frames in the video stream, and one or more keyframes are determined from the video frames in the set of candidate video frames.

2. The method according to claim 1, characterized in that, Determining one or more keyframes from the video frames within the candidate video frame set includes: The candidate video frame set is divided into multiple non-overlapping candidate video frame subsets; Based on a first ratio corresponding to each candidate video frame subset, a keyframe sampling quota is allocated to each candidate video frame subset; the first ratio represents the proportion of the total score of all video frames in the corresponding candidate video frame subset to the total score of all video frames in the candidate video frame set. Based on the total score of each video frame in each subset of candidate video frames and the corresponding number of keyframe samples, one or more keyframes are determined.

3. The method according to claim 1, characterized in that, The calculation of the jump sensitivity score for each video frame based on the attention-weighted feature value of each video frame and the first text includes: For each video frame, based on a set convolution kernel and the attention-weighted feature values ​​of multiple video frames within a set window radius of the video frame, the attention-weighted feature value of the video frame is smoothed to obtain a smoothed value of the attention-weighted feature value of the video frame. The second-order difference of the smoothed value of the attention-weighted feature value of the video frame is calculated by performing second-order difference calculation to obtain the second-order residual vector of the smoothed value of the attention-weighted feature value of the video frame. The jump sensitivity score of the video frame is calculated based on the second-order residual vector of the smoothed value of the attention-weighted feature value and the first text.

4. The method according to claim 1, characterized in that, The step of calculating the information entropy contribution rate of each video frame based on the attention similarity of each video frame includes: The attention similarity of each video frame is normalized to obtain the attention weight of each video frame; Based on the attention weight of each video frame, the overall information entropy of the video stream and the removal entropy of each video frame are calculated; the removal entropy represents the overall information entropy calculated after removing the corresponding video frame from the video stream. The difference between the overall information entropy of the video stream and the removal entropy of each video frame is calculated to obtain the information entropy contribution rate of each video frame.

5. The method according to claim 1, characterized in that, The step of fusing features from each video frame and the audio of the video stream to obtain a multimodal feature map for each video frame includes: The audio features of the video stream are time-aligned with the visual features of each video frame to obtain aligned audio features. The visual features of each video frame and the aligned audio features are fused together to obtain the fused features of each video frame. The fused features of each video frame are linearly mapped to obtain a multimodal feature map for each video frame.

6. The method according to claim 1, characterized in that, The determination of the total score for each video frame based on the jump sensitivity score and the information entropy contribution rate of each video frame includes: The total score for each video frame is obtained by weighting and summing the jump sensitivity score and information entropy contribution rate of each video frame based on the set weights.

7. An electronic device, characterized in that, The method includes a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 6.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images