Lightweight video summary generation method, device, equipment and medium

Abstract

The present application belongs to the technical field of video data processing and artificial intelligence, and provides a kind of lightweight video abstract generation method, device, equipment and medium, including: obtaining the two-dimensional visual feature matrix and log-mel spectrum graph of original video data, and using adaptive step and double-path parallel pooling mechanism to down-sample in time dimension;The audio feature matrix and two-dimensional visual feature matrix obtained are extracted by two convolution pipelines with the same structure for double-flow isomorphic convolution feature, and modulation parameters are generated, and the visual features are dynamically corrected and dynamically gated and fused by modulation parameters;The fusion features are subjected to time sequence smoothing processing, important degree scores are obtained, and video clips are selected and combined under the time constraint according to the important degree scores and global optimization objective function, to obtain video abstract generation results, the technical scheme of the present application improves the coherence and diversity of video abstract, and improves the extraction efficiency and reduces the extraction consumption.

Description

Technical Field

[0001] This invention relates to the fields of video data processing and artificial intelligence technology, and in particular to a lightweight video summarization method, apparatus, device and medium. Background Technology

[0002] With the rapid development of mobile internet technology and the widespread adoption of smart terminal devices, video data has grown exponentially, leading to a serious "information overload" problem. Video summarization technology, as an important branch of multimedia content analysis, aims to automatically analyze video content through algorithms, eliminate redundant segments, and select the most representative and semantically rich keyframes or video shots to generate concise and informative summary videos.

[0003] Against this backdrop, existing technologies for video summarization generation include: Spatiotemporal attention schemes based on Convolutional Neural Networks (CNNs) (such as the CSTA model), which stack video frame features into a two-dimensional feature map and process it using the sliding window mechanism of 2D CNNs; Existing multimodal video summarization methods typically employ a "heterogeneous stacking" strategy. That is, the visual branch uses CNNs to extract spatial features, the audio branch uses RNNs (such as LSTM) or Transformers to extract temporal features, and finally, they are concatenated in a fully connected layer; Large multimodal models based on the Transformer architecture are applied to video summarization tasks. These methods typically utilize large models pre-trained on large-scale datasets (such as CLIP variants or Video-LLM) to extract features or directly generate summaries, capturing complex semantic relationships through massive parameters; Fusion techniques based on fixed rules or simple weighting, traditional lightweight solutions typically employ simple feature concatenation or rule-based weighting (such as fixed audiovisual weights of 0.5:0.5).

[0004] The aforementioned existing technologies have the following drawbacks: When facing the real-time processing needs of massive video data, the inference speed of large models is extremely slow and the deployment cost is high, making it difficult to meet the needs of high-throughput business scenarios; visual features are usually represented as high-dimensional spatial tensors, while audio features are represented as one-dimensional temporal signals, making it difficult to bridge the physical dimension gap between audiovisual features and achieve alignment, and making it difficult to solve the problem of audiovisual asynchrony; existing simple fusion strategies (such as splicing) are prone to introducing environmental noise and lack adaptive anti-interference mechanisms, resulting in a decrease in summarization quality; and it is impossible to balance quality and efficiency. Summary of the Invention

[0005] In order to at least solve one of the technical problems existing in the prior art, the present invention provides a lightweight video summarization method, apparatus, device and medium.

[0006] One aspect of the present invention provides a lightweight video summarization method, comprising: The original video stream is acquired, and sparse sampling and short-time Fourier transform are applied to the original video to obtain a two-dimensional visual feature matrix and a log-Mel spectrogram. The log-Mel spectrogram is downsampled in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain the audio feature matrix. The audio feature matrix and the two-dimensional visual feature matrix are extracted by two structurally isomorphic convolution pipelines through dual-stream isomorphic convolution feature extraction to obtain visual features and audio features. Based on audio features, a feature-layer linear modulation mechanism and auditory saliency are used to generate modulation parameters. The visual features are then dynamically corrected and dynamically gated fused using the modulation parameters to obtain fused features. The fusion features are subjected to temporal smoothing to obtain an importance score. Based on the importance score and the global optimization objective function, video segments are selected and combined under the time constraint to obtain the video summary generation result. The global optimization objective function includes semantic accumulation benefits and content redundancy penalties.

[0007] According to the lightweight video summarization method described above, the original video stream is obtained, and the original video is processed by sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrum, including: Call the decoder to obtain the original video stream, and separate the original video stream into a YUV format video stream and a PCM format audio stream; The video stream is sparsely sampled at equal intervals according to the preset sampling frequency parameters, and the obtained keyframes are scaled to the standard resolution. Feature vectors of the keyframes are extracted through a lightweight pre-trained network, and the feature vectors are stacked according to the time sequence of the video stream to obtain a two-dimensional visual feature matrix that includes the semantic continuity of the time dimension. The audio stream is processed synchronously, and the audio stream is resampled according to the audio sampling frequency parameters and a short-time Fourier transform is performed, which is then mapped to the Mel scale to obtain a log-Mel spectrogram.

[0008] According to the lightweight video summarization method described above, the log-Mel spectrum is downsampled in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain an audio feature matrix, including: Based on the time dimension of the two-dimensional visual feature matrix With the time dimension of the log-Mel spectrogram Calculate the downsampling step size of the original video stream ; Based on the downsampling step size, max pooling and average pooling operations are performed in parallel on the time axis of the log-Mel spectrum. The two pooling results are then concatenated along the channel dimension and passed through... The convolutional layers are fused point by point to obtain the audio feature matrix.

[0009] According to the lightweight video summarization method described above, the method further includes: Learnable downsampling is achieved using multi-scale pooling or depth-separable one-dimensional convolution along the time axis.

[0010] According to the lightweight video summarization method described above, the audio feature matrix and the two-dimensional visual feature matrix are extracted through two structurally isomorphic convolutional pipelines using dual-stream isomorphic convolutional features to obtain visual features and audio features, including: The visual feature matrix is ​​formed by multiple stacked 2D convolutional blocks to form a visual convolution branch. Convolution operation, batch normalization and nonlinear activation are performed sequentially to obtain visual features. A depthwise asymmetric convolution strategy is employed to perform independent orthogonal scanning of each frequency band channel of the audio feature matrix without performing cross-channel interaction; The horizontal convolutional kernel slides along the frequency axis to obtain a horizontal spectral structure feature map; through The vertical convolutional kernel slides along the time axis to obtain the vertical energy mutation feature map; the horizontal spectral structure feature map is then compared with the vertical energy mutation feature map. Point convolutional layers are used to perform depthwise separable convolutions to obtain audio features.

[0011] According to the lightweight video summarization method described above, based on audio features, a feature-layer linear modulation mechanism and auditory saliency are used to generate modulation parameters. These modulation parameters are then used to dynamically correct and dynamically gate the visual features to obtain fused features, including: Visual features are compressed and mapped using a lightweight convolutional layer, and then processed by a sigmoid activation function to obtain a visual confidence mask, which is used to characterize the weights of visual context in determining the relevance of sound signals. The audio features are weighted and suppressed according to the visual confidence mask to obtain the verified audio features. Based on the verified audio features, modulation parameters are generated, including scale modulation parameters and bias modulation parameters. Modulation fusion is performed on visual features by applying linear modulation at the feature layer based on modulation parameters to obtain fused features. for: ; Among them, fusion features This is the fused multimodal feature matrix; Visual features; Represents element-wise multiplication; These are scale modulation parameters. Bias modulation parameters.

[0012] According to the lightweight video summarization method described above, the fused features are subjected to temporal smoothing to obtain an importance score. Based on the importance score and a global optimization objective function, video segments are selected and combined under a duration constraint to obtain the video summarization result, including: The fusion features are smoothed by a one-dimensional convolutional smoothing layer of a lightweight regression network. The smoothing kernel slides on the time axis to perform a weighted average or total variation constraint on the scores in the local neighborhood, thus obtaining the importance score. Based on the feature differences between adjacent frames with fused features, the scores of each frame are aggregated within the shot to obtain the shot-level score of the original video stream and a set of candidate video segments. A global optimization objective function is used to combine the shot-level scores and the candidate segment set to obtain the video summary generation result. The global optimization objective function... for: ; in, This represents the cumulative semantic importance score of the selected segment. This represents the total number of fragments in the candidate fragment set; The index number of the candidate segment; Let be a binary decision variable, representing the result when the th When a segment is selected to enter the summary ,otherwise ; Represents the pre-calculated first... The semantic importance score of each segment; Used to characterize the cost of redundancy penalties for content duplication. is the diversity adjustment coefficient, and is a preset hyperparameter used to adjust the model's penalty weight for duplicate content; double summation symbol. Used to iterate through all unique candidate fragment pairs; Representing fragments With fragments Similarity values ​​in feature space; product terms Constitutes a logical activation condition, indicating that only if the fragment With fragments When both are selected, the product is 1, thus activating the similarity assessment of the segment. The penalty is calculated; otherwise, the product is 0.

[0013] Another aspect of the present invention provides a lightweight video summarization generation apparatus, comprising: The first module is used to acquire the original video stream. The original video is processed by sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrum. The second module is used to downsample the log-Mel spectrogram in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain the audio feature matrix. The third module is used to extract visual and audio features by performing dual-stream isomorphic convolutional features from the audio feature matrix and the two-dimensional visual feature matrix through two structurally isomorphic convolutional pipelines. The fourth module is used to generate modulation parameters based on audio features using a feature-layer linear modulation mechanism and auditory saliency. The modulation parameters are then used to dynamically correct and dynamically gated fuse the visual features to obtain fused features. The fifth module performs temporal smoothing on the fused features to obtain an importance score. Based on the importance score and a global optimization objective function, video segments are selected and combined under a duration constraint to generate a video summary. The global optimization objective function includes semantic accumulation gains and content redundancy penalties. Another aspect of this invention provides an electronic device, including a processor and a memory. The memory is used to store programs; The processor executes the program to implement the method as described above.

[0014] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the methods described above.

[0015] The beneficial effects of this invention are as follows: Based on an adaptive step-size dual-path parallel pooling fusion mechanism, heterogeneous audio streams are transformed into two-dimensional Mel-spectrum feature maps that are completely consistent with the mathematical structure of video frame sequences, achieving lightweight soft alignment and improving robustness to noise peaks and slight audiovisual asynchrony; the idea of ​​image convolution is transferred to the audio domain, constructing two structurally isomorphic 2D CNN pipelines, and using depthwise separable convolution technology to extract audiovisual textures in parallel, reducing the computational power required for extraction; modulation parameters are generated based on audio saliency to dynamically correct visual features, avoiding complex matrix multiplication, and significantly improving the interactive expressiveness of cross-modal features while maintaining the friendliness of convolution operators and extremely low computational overhead, and optional bidirectional gating can be added to achieve adaptive noise resistance; by first outputting frame-level importance scores and performing temporal smoothing, and then performing lightweight shot segmentation and converging into shot-level scores, a redundant penalty term is introduced for segment selection under time constraints, taking into account both high scores and coverage, and improving the coherence and diversity of the summary. Attached Figure Description

[0016] Figure 1This is a schematic diagram of a lightweight video summarization generation process according to an embodiment of the present invention.

[0017] Figure 2 This is a flowchart of video stream processing and feature stacking according to an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of a selected summary generation process according to an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of a lightweight video summarization generation device according to an embodiment of the present invention. Detailed Implementation

[0020] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings. Throughout the description, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions. In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, "module," "part," or "unit" can be used interchangeably. Terms such as "first," "second," etc., are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the sequential relationship of the indicated technical features. In the following description, the consecutive reference numerals for method steps are for ease of review and understanding. Adjusting the implementation order of steps, in conjunction with the overall technical solution of the present invention and the logical relationship between the various steps, will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] refer to Figure 1 , Figure 1 This invention provides a lightweight video summarization method, which includes, but is not limited to, steps S100-S500: S100: Obtain the original video stream, and process the original video using sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrum.

[0022] In some embodiments, a decoder is invoked to obtain the original video stream, which is then separated into a YUV format video stream and a PCM format audio stream. The video stream is sparsely sampled at equal intervals according to a preset sampling frequency parameter, and the resulting keyframes are scaled to a standard resolution. Feature vectors of the keyframes are extracted using a lightweight pre-trained network, and these feature vectors are stacked according to the time sequence of the video stream to obtain a two-dimensional visual feature matrix that includes temporal semantic continuity. Processing is performed synchronously on the audio stream, and the audio stream is resampled according to the audio sampling frequency parameter and subjected to a short-time Fourier transform, which is then mapped to a Mel scale to obtain a log-Mel spectrogram.

[0023] In some embodiments, reference Figure 2 The flowchart shown is for video stream processing and feature stacking. Figure 2 The illustrated embodiment receives the original video file (including audiovisual dual-track data) and transforms the physically inconsistent original audio / visual data into a two-dimensional feature map with a consistent mathematical structure, laying the foundation for subsequent isomorphic computation. This step uses the original video file stored on the mobile device as input, as detailed below: (1) The system starts in the background, calls the decoder to read the original video file, and separates it into a YUV format video stream and a PCM format audio stream.

[0024] (2) The system receives the video stream. In order to reduce the computational load and preserve the temporal semantics, the system performs sparse sampling of the video stream at equal intervals according to the preset sampling frequency parameters. The sampled keyframes are uniformly scaled to the standard resolution and then input into the selected lightweight pre-trained network. For example, the lightweight pre-trained network adopts MobileNetV3.

[0025] The system extracts feature vectors only from specific layers of the network (such as the Pool layer). These vectors are then stacked strictly in temporal order, compressing the 3D video volume into a 2D matrix that maintains the semantic continuity of the temporal dimension. The output of this sub-step is a 2D matrix with dimension 1. Two-dimensional visual feature matrix (in Represents the time dimension after sampling. (Represents semantic feature dimension).

[0026] (3) The system performs synchronous processing on the audio stream. To address the issue of aligning the audio signal, which is a one-dimensional time-series signal, with two-dimensional visual features, the system resamples the data based on the audio sampling rate parameter and then performs a short-time Fourier transform (STFT). To simulate the characteristics of human auditory perception, the system maps the power spectrum to the Mel scale to generate a log-Mel spectrogram. Its dimensions are ,in The time step of the audio. The frequency dimension is set.

[0027] S200 employs an adaptive step size and dual-path parallel pooling mechanism to perform log-Melogram spectrogram analysis. Downsampling is performed in the time dimension to obtain the audio feature matrix.

[0028] In some embodiments, based on the time dimension of the two-dimensional visual feature matrix With the time dimension of the log-Mel spectrogram Calculate the downsampling step size of the original video stream Based on the downsampling step size, max pooling and average pooling operations are performed in parallel on the time axis of the log-Mel spectrum. The two pooling results are then concatenated along the channel dimension and passed through... The convolutional layers are fused point by point to obtain the audio feature matrix.

[0029] It is understandable that, using a two-dimensional visual feature matrix And logarithmic Mel spectrum As input, it addresses the inconsistency in the physical dimension of audiovisual modalities, specifically the audio time step. Much greater than the number of video frames To address the asynchronous issue, a lightweight soft alignment is achieved through an adaptive algorithm, ensuring that audiovisual features correspond one-to-one in subsequent convolutions, as detailed below: (1) Obtain the time dimension of the visual matrix and the time dimension of the audio spectrogram It automatically calculates the downsampling step size to suit the current video characteristics. Based on this step size, max pooling and average pooling operations are performed in parallel on the time axis of the spectrogram.

[0030] (2) The above two pooling sampling results are spliced ​​in the channel dimension and then connected through a... The convolutional layers are fused point-by-point, outputting an audio feature matrix whose physical dimension is strictly aligned with the visual matrix. This achieves complementarity between peak information and energy trend information, providing standardized input data for subsequent isomorphic convolution calculations.

[0031] Optionally, learnable downsampling can be achieved by using multi-scale pooling (parallel pooling with different kernel sizes) or depth-separable one-dimensional convolution along the time axis to improve alignment robustness in scenarios with slight audiovisual asynchrony and noise peaks.

[0032] S300 extracts audio and two-dimensional visual features by performing dual-stream isomorphic convolutional feature extraction through two structurally isomorphic convolutional pipelines, thus obtaining visual and audio features.

[0033] In some embodiments, the two-dimensional visual feature matrix is ​​constructed by multi-layered stacked 2D convolutional blocks forming visual convolution branches, and convolution operations, batch normalization, and non-linear activation are performed sequentially to obtain visual features; a depth-asymmetric convolution strategy is used to perform independent orthogonal scanning of each frequency band channel of the audio feature matrix without performing cross-channel interaction; through The horizontal convolutional kernel slides along the frequency axis to obtain a horizontal spectral structure feature map; through The vertical convolutional kernel slides along the time axis to obtain the vertical energy mutation feature map; the horizontal spectral structure feature map is then compared with the vertical energy mutation feature map. Point convolutional layers are used to perform depthwise separable convolutions to obtain audio features.

[0034] refer to Figure 3 The diagram shown illustrates the selected summary generation process. In this embodiment of the invention, a visual feature matrix is ​​output in step S100 using a dual-stream isomorphic lightweight CNN network architecture. The aligned audio feature matrix output from step S200 As input, two structurally isomorphic 2D CNN pipelines are started to extract audiovisual textures (high-dimensional visual features) in parallel with extremely low computing power.

[0035] For example, the visual convolutional branch consists of multiple stacked 2D convolutional blocks, with each convolutional operation followed by a batch normalization layer and a non-linear activation function. This deep structure aims to perform non-linear transformations, gradually mapping shallow pixel-level or object-level features into high-dimensional abstract feature maps with stronger semantic expressive power. At the same time, the length of the feature map in the time dimension T remains unchanged.

[0036] In some embodiments, during the convolution calculation process, the present invention uses a standard 3×3 convolution kernel to perform sliding calculations on the visual matrix.

[0037] In some embodiments, in order to adapt to the audio spectrogram To address the starkly different physical meanings along the "time axis" and "frequency axis," and to meet the demand for extreme lightweight design, this invention employs a depthwise asymmetric convolution strategy. Each frequency band channel is independently scanned in two steps orthogonally, and no cross-channel interaction occurs during this stage.

[0038] S400 uses a feature-layer linear modulation mechanism and auditory saliency to generate modulation parameters based on audio characteristics. The modulation parameters are then used to dynamically correct and dynamically gated fuse visual features to obtain fused features.

[0039] In some embodiments, visual features are subjected to channel compression and feature mapping through lightweight convolutional layers and processed by a sigmoid activation function to obtain a visual confidence mask, wherein the visual confidence mask is used to characterize the relevance judgment weight of visual context to sound signals.

[0040] In a preferred embodiment, the system is configured with a two-way gating verification mechanism for generating the visual confidence mask. And pre-filtering of audio features. Gating. The generation process is as follows: the system acquires the high-dimensional features of the visual branch output. Channel compression and feature mapping are performed through lightweight convolutional layers (such as 1×1 convolutional kernels), and then processed by the Sigmoid activation function to generate a visual confidence mask with values ​​ranging from [0,1]. The mask Used to characterize the relevance of visual context to the current sound signal. (Generation) Then, the system analyzes the audio features. Perform weighted suppression operation (i.e.) ), and utilize the verified audio features Generate the modulation parameters and .

[0041] In some embodiments, the audio features are weighted and suppressed according to the visual confidence mask to obtain the verified audio features, and modulation parameters are generated based on the verified audio features, wherein the modulation parameters include scale modulation parameters and bias modulation parameters.

[0042] For example, based on the feature map The audio feature map is reduced in dimensionality and mapped using a 1×1 convolutional layer to calculate the scale modulation parameter γ and the bias modulation parameter β. These parameters are then constrained to a stable range using activation functions such as Sigmoid / Tanh, forming FiLM modulation (Scale+Bias). The generation process of the scale modulation parameter γ and the bias modulation parameter β specifically includes the following steps: First, the audio feature map... A convolution operation is applied. In a preferred embodiment, this convolution operation is implemented using a 1x1 convolution kernel, the dimension of which is configured to the dimensions of the high-dimensional visual features. The high-dimensional feature map generated by the aforementioned convolutional operation is then equally segmented along the channel dimension to obtain two independent feature subsets corresponding to the scale modulation parameter γ and the bias modulation parameter β, respectively. Finally, to ensure numerical stability and achieve effective modulation, the two feature subsets are subjected to nonlinear activation: the feature subset for the scale modulation parameter γ is activated by the Sigmoid activation function, constraining its value range to the [0,1] interval to achieve dynamic scaling control of visual features; the feature subset for the bias modulation parameter β is activated by the Tanh activation function, constraining its value range to the [-1,1] interval to achieve baseline translation of visual features. For example, during the period when "laughter" or "intense commentary" is detected, the generated γ parameter will exhibit a higher response value, while it will approach 0 during the meaningless noise period.

[0043] In some embodiments, modulation fusion is performed on visual features by linear modulation of the feature layer according to modulation parameters to obtain fused features. for: ; Among them, fusion features This is the fused multimodal feature matrix; Visual features; Represents element-wise multiplication; These are scale modulation parameters. Bias modulation parameters.

[0044] For example, in an embodiment of the present invention, performing FiLM (Feature Layer Linear Modulation) modulation fusion includes: receiving generated modulation parameters, and processing visual features. Perform linear modulation operation. In specific implementation, follow the formula... Calculate the final multimodal feature representation. Among them, This represents the fused multimodal feature matrix. Represents the original visual feature map. This represents element-wise multiplication. In this process, the system multiplies each feature channel element of the visual feature map by... Dynamic amplitude scaling is performed, followed by feature baseline shifting by superimposing β. Through the joint modulation of γ and β, the system can inject auditory semantics into the visual feature space in a nonlinear manner while maintaining the computational efficiency and low latency of the convolution operator.

[0045] The feature-layer linear modulation mechanism implemented in this invention constructs a dynamic affine transformation based on feature channels, possessing adaptive adjustment characteristics: when audio features contain highly saliency semantics, the modulation parameters significantly enhance visual features; when audio is determined to be background noise, the modulation parameters automatically approach zero, allowing the model to smoothly revert to the visually dominant mode. When β=0 and When Ga is a soft-gated mask generated based on audio feature mapping, this formula is compatible with traditional soft-gated amplification modes. This sub-step ultimately outputs the multimodal feature representation fused via the aforementioned bidirectional verification mechanism. .

[0046] S500 performs temporal smoothing on the fused features to obtain an importance score. Based on the importance score and the global optimization objective function, video segments are selected and combined under the time constraint to obtain the video summary generation result. The global optimization objective function includes semantic accumulation benefits and content redundancy penalties.

[0047] The closed-loop feedback and summary output of this invention adopt a global optimization summary generation strategy, using the fusion features output by S400. Based on the input and user-defined time constraints, the final summary is generated, and a global optimization strategy is introduced to balance the quality and diversity of the content.

[0048] In some embodiments, the fused features are smoothed by a one-dimensional convolutional smoothing layer of a lightweight regression network, and the smoothing kernel slides along the time axis to perform a weighted average or total variation constraint on the scores in the local neighborhood to obtain an importance score.

[0049] For example, fusion features Input a lightweight regression network (FC+Sigmoid), output a sequence of frame-level importance scores. ,in To prevent severe jitter caused by frame-by-frame scoring, a one-dimensional convolutional smoothing layer or total variation constraint is introduced on the time axis.

[0050] Specifically, a one-dimensional convolutional smoothing layer uses a pre-defined smoothing kernel to slide along the time axis to perform a weighted average of the scores in the local neighborhood, physically filtering out high-frequency noise from abrupt changes; or it uses a total variation constraint, that is, adding a penalty term for the "sum of the absolute values ​​of the differences between scores of adjacent frames" to the loss function during training, and assigning it a specific weight coefficient.

[0051] In some embodiments, based on the feature differences between adjacent frames of the fused features, the scores of each frame are aggregated within the shot to obtain the shot-level score and candidate video segment set of the original video stream. In subsequent processing, the frame-level scores are aggregated within the shot (e.g., by taking the average or maximum value) to form the shot-level score and candidate segment set. This step transforms fine-grained frame-level information into semantically complete narrative units.

[0052] In some embodiments, a global optimization objective function is used to combine the shot-level score and the candidate segment set to obtain the video summary generation result, wherein the global optimization objective function... for: ; in, This represents the cumulative semantic importance score of the selected segment. This represents the total number of fragments in the candidate fragment set; The index number of the candidate segment; Let be a binary decision variable, representing the result when the th When a segment is selected to enter the summary ,otherwise ; Represents the pre-calculated first... The semantic importance score of each segment; Used to characterize the cost of redundancy penalties for content duplication. is the diversity adjustment coefficient, and is a preset hyperparameter used to adjust the model's penalty weight for duplicate content; double summation symbol. Used to iterate through all unique candidate fragment pairs; Representing fragments With fragments Similarity values ​​in the feature space (e.g., cosine similarity); product terms Constitutes a logical activation condition, indicating that only if the fragment With fragments When both are selected, the product is 1, thus activating the similarity assessment of the segment. The penalty is calculated; otherwise, the product is 0.

[0053] By deducting the penalty item from the total score, the system not only selects high-scoring segments but also applies negative constraints to combinations of segments with highly similar features, thereby generating video summaries that combine semantic importance and content diversity.

[0054] Furthermore, in this embodiment of the invention, based on the solved optimal combination of segments, the video processing engine is invoked to perform rendering and generation. All segments marked as selected are strictly reassembled according to their timestamp order in the original video to maintain the coherence of the narrative logic. At the splicing points of adjacent segments, crossfade-in / fade-out or audio envelope smoothing techniques are applied to eliminate the abruptness and audio popping sounds caused by frame skipping. Finally, the spliced ​​data stream is re-encoded to generate an independent playable video file as the final selected summary.

[0055] Figure 4 This is a schematic diagram of a lightweight video summarization generation device according to an embodiment of the present invention. The device includes a first module 410, a second module 420, a third module 430, a fourth module 440, and a fifth module 450.

[0056] The system comprises five modules: First, the original video stream is acquired and processed using sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrogram. Second, the log-Mel spectrogram is downsampled temporally using adaptive stride and dual-path parallel pooling to obtain an audio feature matrix. Third, the audio and visual feature matrices are extracted using a dual-stream isomorphic convolutional pipeline with two structurally isomorphic convolutional pipelines to obtain visual and audio features. Fourth, based on the audio features, modulation parameters are generated using a feature-layer linear modulation mechanism and auditory saliency. These modulation parameters are then used to dynamically correct and dynamically gate the visual features to obtain fused features. Fifth, the fused features are temporally smoothed to obtain an importance score. Based on the importance score and a global optimization objective function, video segments are selected and combined under time constraints to generate a video summary. The global optimization objective function includes semantic accumulation gains and content redundancy penalties.

[0057] For example, with the cooperation of the first, second, third, fourth, and fifth modules in the device, the embodiment device can implement any of the aforementioned lightweight video summarization generation methods, namely, acquiring the original video stream, processing the original video using sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrogram; downsampling the log-Mel spectrogram in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain an audio feature matrix; extracting visual features and audio features from the audio feature matrix and the two-dimensional visual feature matrix through two structurally isomorphic convolution pipelines using dual-stream isomorphic convolutional features; generating modulation parameters based on the audio features using a feature layer linear modulation mechanism and auditory saliency; dynamically correcting and dynamically gating the visual features using the modulation parameters to obtain fused features; performing temporal smoothing processing on the fused features to obtain an importance score; and selecting and combining video segments under a duration constraint based on the importance score and a global optimization objective function to obtain the video summarization generation result, wherein the global optimization objective function includes semantic accumulation gains and content redundancy penalties. The beneficial effects of this invention are as follows: Based on an adaptive step-size dual-path parallel pooling fusion mechanism, heterogeneous audio streams are transformed into two-dimensional Mel-spectrum feature maps that are completely consistent with the mathematical structure of video frame sequences, achieving lightweight soft alignment and improving robustness to noise peaks and slight audiovisual asynchrony; the idea of ​​image convolution is transferred to the audio domain, constructing two structurally isomorphic 2D CNN pipelines, and using depthwise separable convolution technology to extract audiovisual textures in parallel, reducing the computational power required for extraction; modulation parameters are generated based on audio saliency to dynamically correct visual features, avoiding complex matrix multiplication, and significantly improving the interactive expressiveness of cross-modal features while maintaining the friendliness of convolution operators and extremely low computational overhead, and optional bidirectional gating can be added to achieve adaptive noise resistance; by first outputting frame-level importance scores and performing temporal smoothing, and then performing lightweight shot segmentation and converging into shot-level scores, a redundant penalty term is introduced for segment selection under time constraints, taking into account both high scores and coverage, and improving the coherence and diversity of the summary.

[0058] This invention also provides an electronic device, which includes a processor and a memory; The memory stores the program; The processor executes a program to perform the aforementioned lightweight video summarization method; the electronic device has the function of carrying and running the lightweight video summarization software system provided in the embodiments of the present invention, such as a personal computer, minicomputer, mainframe, workstation, network or distributed computing environment, standalone or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0059] This invention also provides a computer-readable storage medium storing a program that is executed by a processor to implement the lightweight video summarization method described above.

[0060] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented in the embodiments of this invention. Alternative embodiments are contemplated, in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0061] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned lightweight video summarization method.

[0062] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, considering the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed in the embodiments of the invention, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can include, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0065] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0066] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0067] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0069] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A lightweight video summarization method, characterized in that, include: The original video stream is acquired, and sparse sampling and short-time Fourier transform are applied to the original video to obtain a two-dimensional visual feature matrix and a log-Mel spectrum. The log-Mel spectrogram is downsampled in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain the audio feature matrix; The audio feature matrix and the two-dimensional visual feature matrix are extracted by two structurally isomorphic convolution pipelines through dual-stream isomorphic convolution feature extraction to obtain visual features and audio features. Based on audio features, a feature-layer linear modulation mechanism and auditory saliency are used to generate modulation parameters. The visual features are then dynamically corrected and dynamically gated fused using the modulation parameters to obtain fused features. The fusion features are processed by temporal smoothing to obtain an importance score. Based on the importance score and the global optimization objective function, video segments are selected and combined under the time constraint to obtain the video summary generation result. The global optimization objective function includes semantic accumulation benefits and content redundancy penalties. The process of extracting visual and audio features by performing dual-stream isomorphic convolutional features from the audio feature matrix and the two-dimensional visual feature matrix through two structurally isomorphic convolutional pipelines includes: The visual feature matrix is ​​formed by multiple stacked 2D convolutional blocks to form a visual convolution branch. Convolution operation, batch normalization and nonlinear activation are performed sequentially to obtain visual features. A depthwise asymmetric convolution strategy is employed to perform independent orthogonal scanning of each frequency band channel of the audio feature matrix without performing cross-channel interaction; The horizontal convolutional kernel slides along the frequency axis to obtain a horizontal spectral structure feature map; through The vertical convolutional kernel slides along the time axis to obtain the vertical energy mutation feature map; the horizontal spectral structure feature map is then compared with the vertical energy mutation feature map. Point convolutional layers are used to perform depthwise separable convolutions to obtain audio features; Based on audio features, a feature-layer linear modulation mechanism and auditory saliency are used to generate modulation parameters. These modulation parameters are then used to dynamically correct and dynamically gate the visual features to obtain fused features, including: Visual features are compressed and mapped using a lightweight convolutional layer, and then processed by a sigmoid activation function to obtain a visual confidence mask, which is used to characterize the weights of visual context in determining the relevance of sound signals. The audio features are weighted and suppressed according to the visual confidence mask to obtain the verified audio features. Based on the verified audio features, modulation parameters are generated, including scale modulation parameters and bias modulation parameters. Modulation fusion is performed on visual features by applying linear modulation at the feature layer based on modulation parameters to obtain fused features. for: Among them, fusion features This is the fused multimodal feature matrix; Visual features; Represents element-wise multiplication; These are scale modulation parameters. These are the bias modulation parameters.

2. The lightweight video summarization method according to claim 1, characterized in that, The process of acquiring the original video stream involves sparse sampling and short-time Fourier transform processing to obtain a two-dimensional visual feature matrix and a log-Mel spectrum, including: Call the decoder to obtain the original video stream, and separate the original video stream into a YUV format video stream and a PCM format audio stream; The video stream is sparsely sampled at equal intervals according to the preset sampling frequency parameters, and the obtained keyframes are scaled to the standard resolution. Feature vectors of the keyframes are extracted through a lightweight pre-trained network, and the feature vectors are stacked according to the time sequence of the video stream to obtain a two-dimensional visual feature matrix that includes the semantic continuity of the time dimension. The audio stream is processed synchronously, and the audio stream is resampled according to the audio sampling frequency parameters and a short-time Fourier transform is performed, which is then mapped to the Mel scale to obtain a log-Mel spectrogram.

3. The lightweight video summarization method according to claim 1, characterized in that, The process involves downsampling the log-Mel spectrum in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain the audio feature matrix, including: Based on the time dimension of the two-dimensional visual feature matrix With the time dimension of the log-Mel spectrogram Calculate the downsampling step size of the original video stream ; Based on the downsampling step size, max pooling and average pooling operations are performed in parallel on the time axis of the log-Mel spectrum. The two pooling results are then concatenated along the channel dimension and passed through... The convolutional layers are fused point by point to obtain the audio feature matrix.

4. The lightweight video summarization method according to claim 3, characterized in that, The method further includes: Learnable downsampling is achieved using multi-scale pooling or depth-separable one-dimensional convolution along the time axis.

5. The lightweight video summarization method according to claim 1, characterized in that, The fusion features are subjected to temporal smoothing to obtain an importance score. Based on the importance score and the global optimization objective function, video segments are selected and combined under a duration constraint to obtain the video summary generation result, including: The fusion features are smoothed by a one-dimensional convolutional smoothing layer of a lightweight regression network. The smoothing kernel slides on the time axis to perform a weighted average or total variation constraint on the scores in the local neighborhood, thus obtaining the importance score. Based on the feature differences between adjacent frames with fused features, the scores of each frame are aggregated within the shot to obtain the shot-level score of the original video stream and a set of candidate video segments. A global optimization objective function is used to combine the shot-level scores and the candidate segment set to obtain the video summary generation result. The global optimization objective function... for: in, This represents the cumulative semantic importance score of the selected segment. This represents the total number of fragments in the candidate fragment set; The index number of the candidate segment; Let be a binary decision variable, representing the result when the th When a segment is selected to enter the summary ,otherwise ; Represents the pre-calculated first... The semantic importance score of each segment; Used to characterize the cost of redundancy penalties for content duplication. is the diversity adjustment coefficient, and is a preset hyperparameter used to adjust the model's penalty weight for duplicate content; double summation symbol. Used to iterate through all unique candidate fragment pairs; Representing fragments With fragments Similarity values ​​in feature space; product terms Constitutes a logical activation condition, indicating that only if the fragment With fragments When both are selected, the product is 1, thus activating the similarity assessment of the segment. The penalty is calculated; otherwise, the product is 0.

6. A lightweight video summarization generation device, characterized in that, include: The first module is used to acquire the original video stream. The original video is processed by sparse sampling and short-time Fourier transform to obtain a two-dimensional visual feature matrix and a log-Mel spectrum. The second module is used to downsample the log-Mel spectrogram in the time dimension using an adaptive step size and a dual-path parallel pooling mechanism to obtain the audio feature matrix. The third module is used to extract visual and audio features by performing dual-stream isomorphic convolutional features from the audio feature matrix and the two-dimensional visual feature matrix through two structurally isomorphic convolutional pipelines. The fourth module is used to generate modulation parameters based on audio features using a feature-layer linear modulation mechanism and auditory saliency. The modulation parameters are then used to dynamically correct and dynamically gated fuse the visual features to obtain fused features. The fifth module is used to perform temporal smoothing on the fused features to obtain an importance score. Based on the importance score and the global optimization objective function, video segments are selected and combined under the time constraint to obtain the video summary generation result. The global optimization objective function includes semantic accumulation benefits and content redundancy penalties. The process of extracting visual and audio features by performing dual-stream isomorphic convolutional features from the audio feature matrix and the two-dimensional visual feature matrix through two structurally isomorphic convolutional pipelines includes: The visual feature matrix is ​​formed by multiple stacked 2D convolutional blocks to form a visual convolution branch. Convolution operation, batch normalization and nonlinear activation are performed sequentially to obtain visual features. A depthwise asymmetric convolution strategy is employed to perform independent orthogonal scanning of each frequency band channel of the audio feature matrix without performing cross-channel interaction; The horizontal convolutional kernel slides along the frequency axis to obtain a horizontal spectral structure feature map; through The vertical convolutional kernel slides along the time axis to obtain the vertical energy mutation feature map; the horizontal spectral structure feature map is then compared with the vertical energy mutation feature map. Point convolutional layers are used to perform depthwise separable convolutions to obtain audio features; Based on audio features, a feature-layer linear modulation mechanism and auditory saliency are used to generate modulation parameters. These modulation parameters are then used to dynamically correct and dynamically gate the visual features to obtain fused features, including: Visual features are compressed and mapped using a lightweight convolutional layer, and then processed by a sigmoid activation function to obtain a visual confidence mask, which is used to characterize the weights of visual context in determining the relevance of sound signals. The audio features are weighted and suppressed according to the visual confidence mask to obtain the verified audio features. Based on the verified audio features, modulation parameters are generated, including scale modulation parameters and bias modulation parameters. Modulation fusion is performed on visual features by applying linear modulation at the feature layer based on modulation parameters to obtain fused features. for: Among them, fusion features This is the fused multimodal feature matrix; Visual features; Represents element-wise multiplication; These are scale modulation parameters. These are the bias modulation parameters.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the lightweight video summarization generation method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the lightweight video summarization method as described in any one of claims 1-5.

Images