Live long video clipping method and device based on memory perception collaboration

Abstract

The application provides a live long video clip method and device based on memory perception cooperation, relates to the technical field of image processing and video clip, and the application links and analyzes voice, subtitles and picture action on a unified time grid, realizes low complexity representation of the long video through an efficient perception module, deposits core characters, events and semantic clues across segments through entity-centered long-term memory, and enhances consistency and robustness through memory retrieval feedback to current perception; On the basis, the application generates a three-module fusion wonderful degree score, executes interval optimization and boundary refinement with time length budget constraints, and finally outputs an editing decision list and quickly splices into a high-quality refined video; The application solves the problems of large long-time live material quantity, low efficiency and easy-to-miss wonderful segment of manual review and editing, and significantly improves the post-production efficiency and finished film quality of live content.

Description

Technical Field

[0001] This invention relates to the fields of image processing and video editing technology, and in particular to a method and apparatus for editing long live videos based on memory-perception collaboration. Background Technology

[0002] With the booming development of the live streaming industry, the amount of live streaming material has exploded, and massive amounts of video data have surged in like a tidal wave. In this context, traditional manual review and editing methods have proven inadequate. Not only are they extremely inefficient, requiring a lot of manpower and time, leading to increased post-production costs, but the tedious review process is also prone to oversights, causing many wonderful clips to be buried and unable to be fully showcased.

[0003] Furthermore, existing automation methods are also proving inadequate: some methods lack scalability when processing long videos lasting hours, making it difficult to meet actual needs; while others lack a consistent understanding across video segments, failing to effectively maintain the continuity of characters and topics, resulting in inconsistent quality and poor stability of the final edited content.

[0004] Therefore, there is an urgent need for a live long video editing method and device based on memory perception collaboration to solve the above problems. Summary of the Invention

[0005] To address the aforementioned technical problems in related technologies, this invention proposes a method and apparatus for live long video editing based on memory perception collaboration.

[0006] In a first aspect, the present invention provides a method for editing long live videos based on memory-aware collaboration, comprising the following steps:

[0007] S1. Convert the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks N is the total number of video frames; K is the total number of global words.

[0008] S2. Map the three-modal alignment blocks to a unified spatial dimension, and add fixed absolute position encoding for each modality to obtain the video embedding sequence. Audio embedding sequence and subtitle embedding sequence ;

[0009] S3. Linearly update the video embedding sequence using Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ;

[0010] S4. Update the video representation. With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory ;

[0011] S5, via memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ;

[0012] S6. Based on enhanced cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. Among them, the upper threshold is a real number greater than 0 and less than 1;

[0013] S7. Based on the candidate fragment set Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ;

[0014] S8. Compile selected clips Convert to an editing decision list, and then stitch together the edited footage and the editing decision list file sequentially on the timeline based on the editing decision list.

[0015] Specifically, step S1 includes the following steps:

[0016] S11. Convert the original video stream to a constant frame rate to obtain a video frame sequence. ,in, For a video frame, the set of frame indices corresponding to second t is: and Where H is the video frame height; W is the video frame width; and 3 represents the RGB three channels. The video has a fixed frame rate; n is the frame index; F is the frame rate. Indicates the number of elements in the set;

[0017] S12. Resample the original audio stream to obtain a discrete audio sequence a[m]. The set of audio sample indices corresponding to the t-th second is... , where m is the sample index; Represents the set of integers; Sampling rate;

[0018] S13. Convert the discrete audio sequence a[m] into a time-stamped word sequence using speech recognition technology. ;in, For the k-th recognized word, The time stamp for the identified word is in milliseconds; ASR is the speech recognition function.

[0019] S14. Process the video frame sequence within a time bucket per second. Discrete audio sequence a[m] and word sequence Aggregate to obtain video blocks Audio blocks and subtitle blocks .

[0020] Specifically, step S14 includes the following steps:

[0021] S141, regarding the first A representative frame is obtained by temporal mean fusion of video frames within a second. Then, representing the frame The original subgraph is obtained by uniformly dividing it into M rectangular grid blocks. For each original subgraph Video blocks are obtained by bilinear scaling to uniform perceptual resolution. ;in G x G is the number of grid rows. y is the number of grid columns, and p is the index of the rectangular grid block; each rectangular grid block is a video patch; To achieve a unified perception resolution height, To ensure uniform perception resolution width;

[0022] S142, for the discrete audio sequence at second t. Calculate the STFT amplitude spectrum and map it to the Mel frequency band to obtain the audio block. Where A is the first... The number of STFT time frames per second; F is the number of Mel frequency bands;

[0023] S143, Word Sequence middle The words are collected in order as subtitle blocks ;in Indicates the first Within seconds The word symbol of each word; S is the word index for the second interval; S is the word index for the second interval. Word count within seconds.

[0024] Specifically, step S2 includes the following steps:

[0025] S21, Regarding video blocks The video embedding sequence is obtained by performing patch-by-patch linear projection and absolute position encoding. :

[0026] For the video block at second t The pth video patch The video patch is embedded by linearly projecting it onto the spatial dimension d and then performing absolute position encoding according to the following formula. :

[0027] ,

[0028] Then embed the video patch Stacking yields video embedding sequences ;

[0029] Where p = 1, 2, ..., M, M is the number of fixed video patches per second; To prioritize the subsequent movement within the channel; It is a learnable linear projection matrix; Encode the absolute position of learnable patches;

[0030] S22, Audio Block The audio embedding sequence is obtained by performing time-series frame-by-frame linear projection and absolute position coding. :

[0031] For the first Second audio block The Log-Mel frames The audio frame embedding is obtained by linearly projecting it onto the spatial dimension d and performing absolute position encoding according to the following formula. :

[0032] ,

[0033] Then embed the audio frame Stacking yields audio embedding sequences ;

[0034] in, It is a learnable linear projection matrix; Encode the absolute position of learnable audio frames;

[0035] S23, Subtitle Blocks The subtitle embedding sequence is obtained by performing word vector lookup and absolute position encoding. :

[0036] For the first Second Subtitle Block The first in chronological order Word The word order embedding is obtained by performing a word vector lookup table and absolute position encoding according to the following formula. :

[0037] ;

[0038] Then embed the subtitle word order Stacking yields the subtitle embedding sequence ;

[0039] Among them, Vocab is a vocabulary list; For learnable word embedding matrix; For indexing functions, return words Index in the vocabulary; This is a learnable absolute word order encoding; S represents the number of words in the subtitle block at second t; |Vocab| represents the length of the vocabulary Vocab.

[0040] Specifically, step S3 includes the following steps:

[0041] S31. Video embedding sequence via Mamba-2 Perform linear updates to obtain the video update representation. ;

[0042] S32. Embedding text into a sequence With audio embedding sequence The text audio sequence is obtained by concatenating along the sequence dimensions. Text audio sequence Text-audio attention is obtained through single-head self-attention function processing. ;

[0043] S33, Text-audio sequence With video update representation The text-audio-video cross-attention function is used to obtain the cross-attention of text and audio / video. ;

[0044] S34, Attention to Text and Audio Cross-attention with this audio and video Cross-modal semantic representations are obtained through scalar gating fusion. .

[0045] Specifically, step S4 includes the following steps:

[0046] S41. At second t, update the video representation. With cross-modal semantic representation Aggregate characterization is obtained through mean aggregation and splicing. As shown in the following formula:

[0047] ,

[0048] in, This represents the arithmetic mean of a sequence of length L over the sequence dimension;

[0049] S42, Characterization of polymerization Candidate keys are obtained by performing linear mapping. With candidate values The two constitute candidate key-value pairs The linear mapping formula is as follows:

[0050]

[0051] in, These are the linear mapping of the memory keys and the offset of the memory keys, respectively; These are the linear mapping of the memory value header and the bias of the memory value header, respectively;

[0052] S43, from memory bank Find candidate keys The most similar memory entry with the highest similarity, if the similarity is less than a preset similarity threshold or the memory bank... When empty, select candidate key-value pairs. New memory entries are created and written to the memory bank. Otherwise, select the candidate key-value pairs. Merge the closest memory entries into the memory bank to obtain the updated memory bank. .

[0053] Specifically, step S5 includes the following steps:

[0054] S51, Cross-modal semantic representation After performing sequence mean pooling, a linear transformation is applied to construct the query vector. ;

[0055] S52, Utilizing query vectors In memory bank The memory vector is obtained by retrieval. ;

[0056] S53, Calculate the memory vector scalar gating :

[0057] ,

[0058] in For learnable weight matrix, For bias terms, ; ; Indicates transpose;

[0059] S54, Memory Vector Copying to cross-modal semantic representation After the number of rows, pass through scalar gating With cross-modal semantic representation Aligned and summed, we obtain memory-enhanced cross-modal semantics. .

[0060] Specifically, step S6 includes the following steps:

[0061] S61. Enhancing cross-modal semantics through single-layer linear mapping and Sigmoid function computation. Local saliency of each semantic item Take one of them Maximum value as an enhancement of cross-modal semantics semantic channel score ;

[0062] S62. Calculate the root mean square energy of the discrete audio sequence a[m] per second, and then perform minimum-maximum normalization of the root mean square energy over the total audio duration to obtain the audio channel score. ;

[0063] S63. Update representation based on video. The video update representation difference between adjacent seconds is calculated to measure the intensity of motion change. Then, the minimum-maximum normalization of motion change intensity is performed over the total video duration to obtain the visual channel score. ;

[0064] S64. Based on semantic channel scoring Audio channel score and visual channel score A weighted fusion is performed to obtain the brilliance score. Score based on the level of excitement Building a sequence of wonderful moments every second ;

[0065] S65, Based on the upper threshold From the exciting sequence every second Filter out the first part of the collection The first part will be collected. After decomposing the fragment into several consecutive integer intervals, a cumulative score is calculated for each interval to obtain a set of candidate fragments. ;in Candidate segments; Let be the starting integer of the i-th largest consecutive integer interval, representing the starting boundary of the interval; Let be the ending integer of the i-th largest continuous integer interval, representing the boundary of the interval; is the cumulative score over the interval; Nc is the number of the largest consecutive integer intervals.

[0066] Specifically, step S7 includes the following steps:

[0067] S71, Calculate candidate fragments interval length With front-wheel drive ;

[0068] S72, Based on candidate fragments and its interval length With front-wheel drive Calculate the maximum cumulative integral of candidate segments to construct the state matrix. Then to The selected interval set is obtained by backtracking and filtering the selected intervals. ;

[0069] S73, Select the set of intervals Each selected interval Symmetric window refinement is performed on the start and end boundaries and the boundary cost function is applied. Calculate the single-point cost at the start and end boundaries of each selected interval, and construct the refined interval by taking the minimum single-point cost at the start and end boundaries. ;

[0070] S74, Refining the interval Sort by time in ascending order and remove overlapping intervals to obtain a selected collection of segments. .

[0071] In a second aspect, the present invention provides a live long video editing device based on memory-aware collaboration, and a live long video editing method based on memory-aware collaboration as described in the first aspect above, comprising the following units:

[0072] Live data time alignment unit, used to convert the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks N is the total number of video frames; K is the total number of global words.

[0073] Multimodal embedding building blocks are used to map three-modal aligned blocks to a unified spatial dimension and add fixed absolute position encoding for each modality to obtain a video embedding sequence. Audio embedding sequence and subtitle embedding sequence ;

[0074] Highly efficient perceptual coding unit for linearly updating video embedding sequences via Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ;

[0075] Entity-centric long-term memory building blocks are used to update video representations. With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory ;

[0076] Memory-perception collaborative building unit, used to utilize memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ;

[0077] Candidate fragment generation unit, used to enhance cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. ;

[0078] Selected segment generation unit, used to generate segments based on the candidate segment set Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ;

[0079] The finished product and decision document export unit is used to export selected fragments. Convert to an editing decision list, and then stitch together the edited footage and the editing decision list file sequentially on the timeline based on the editing decision list.

[0080] This invention provides a method and apparatus for editing long live videos based on memory-aware collaboration. It files the live stream using a unified 1Hz time grid and constructs lightweight embeddings for audio, subtitles, and video motion. Based on this, it achieves efficient perception using "video Mamba-2 linear updates + text / audio self-attention and cross-attention to the video," obtaining a robust semantic representation of the current moment. Simultaneously, it constructs a long-term memory bank centered on people and events, and feeds the memory information back to the current representation through a single retrieval and scalar gating to maintain consistency across segments. Based on this, the invention generates and merges the highlights per second according to the text / audio / visual three channels, selects segments using interval optimization with duration budget constraints, refines the boundaries using a simple cost function of audio energy and shot changes, and finally generates an edit decision list (EDL) for direct cutting and splicing, quickly producing high-quality, broadcastable, finely edited videos.

[0081] Furthermore, this invention links and analyzes speech, subtitles, and on-screen actions on a unified time grid. It achieves low-complexity representation of long videos with an efficient perception module, while leveraging the long-term memory of entity centers to store key characters, events, and semantic cues across segments. Memory retrieval is then used to feed back into current perception, enhancing consistency and robustness. Based on this, the invention generates a three-mode fusion performance score, performs interval optimization and boundary refinement with time budget constraints, and finally outputs an editing decision list that is quickly stitched together into a high-quality, finely edited video. This significantly improves the post-production efficiency and final quality of live streaming content, effectively solving the problems of large amounts of long-duration live streaming material, inefficient manual review and editing, and the risk of missing exciting segments. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0083] Figure 1 A schematic diagram of a live long video editing method based on memory perception collaboration provided in an embodiment of the present invention;

[0084] Figure 2 This is a schematic diagram of a live long video editing device based on memory perception collaboration provided in an embodiment of the present invention. Detailed Implementation

[0085] The present invention will be explained in detail through the following embodiments. The purpose of this invention is to protect all technical improvements within its scope. In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0086] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0087] Example 1

[0088] refer to Figure 1 This embodiment provides a method for editing long live videos based on memory-aware collaboration, including the following steps:

[0089] S1. Live Stream Data File Conversion and Time Alignment: Converting the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks ;

[0090] The live media stream is converted into three types of standardized blocks on a unified time bucket (1Hz grid) with one block per second: video blocks Audio blocks Subtitle Blocks All subsequent encoding is performed on this aligned grid.

[0091] A time bucket is a common concept in data processing or analysis, used to divide continuous time into discrete intervals or "buckets" to aggregate, analyze, or manipulate time series data.

[0092] In this embodiment, the time bucket is 1Hz; in this embodiment, the live stream is aligned according to "one bucket per second", and all subsequent encoding and scoring are performed on this grid. The time bucket can also be changed according to requirements.

[0093] The input live media stream includes:

[0094] The original video stream is decoded by a container to obtain a video frame sequence. Each video frame (Height × Width × RGB).

[0095] raw audio stream , is a continuous audio waveform. Time is measured in seconds.

[0096] S11. Convert the original video stream to a constant frame rate to obtain a video frame sequence. ,in, For a video frame, the set of frame indices corresponding to second t is: and Where N is the total number of video frames, H is the height of the video frame, W is the width of the video frame, 3 represents the RGB three channels, n is the frame index, and F is the frame rate;

[0097] Constant frame rate conversion: Converting the original video stream to a video with a constant frame rate. (frames / second) to obtain the video frame sequence At this time, the set of frame indices corresponding to second t is: and If the last second is insufficient, it can be filled to zero. ; For a set, Indicates the number of elements in the set;

[0098] Fixed frame rate video The frame rate is represented by F, which corresponds to a frame rate (fps). F can be selected according to actual needs, such as 24, 30, or 60. The total number of video frames N = Tf v T represents the total duration of the original video stream, in seconds; t = 1, 2, ..., T;

[0099] The original audio stream and the original video stream have the same total duration, both being T;

[0100] It is understandable that a video stream is essentially composed of a series of consecutive images (frames) that are played in a specific time sequence to form dynamic video content.

[0101] After the original video stream is converted to a constant frame rate, a corresponding video frame sequence will be generated every second. ;

[0102] A constant frame rate (CFR) means that each frame in a video is displayed at fixed time intervals, providing a stable foundation for video synchronization, editing, and playback. For example, in video editing software, a constant frame rate ensures the accuracy and consistency of operations such as trimming and adding effects; during video transmission, a constant frame rate helps maintain the stability of the video stream, reducing stuttering and latency; and during video playback, a constant frame rate provides a smooth visual experience.

[0103] Constant frame rate normalization is the process of normalizing a video to have a fixed frame rate (frames per second). Many existing video coding standards (such as H.264, H.265, etc.) and video processing tools support the setting and processing of constant frame rates. Normalizing a video to a constant frame rate is also a common operation step in video transcoding, format conversion, etc., which is existing technology and will not be elaborated here.

[0104] S12. Resample the original audio stream to obtain a discrete audio sequence a[m]. The set of audio sample indices corresponding to the t-th second is... , where m is the sample index; Represents the set of integers;

[0105] Audio resampling: Resamples the original audio stream to an audio sampling rate f. a We obtain a discrete audio sequence a[m], and the set of audio sample indices corresponding to the t-th second. In this embodiment, the audio sampling rate f a The value is 16000Hz; where m is an integer sample index. This represents the set of integers, whose values ​​are determined by the interval specified in the above formula.

[0106] Discrete audio sequences Indicates the sampling rate (This embodiment uses 16 kHz) for the original audio stream The resampled first Given discrete sample values, the relationship between them is: ;

[0107] The discrete audio sequence a[m] represents the audio sequence at a sampling rate of Below the original audio stream The discrete sequences obtained after resampling / anti-aliasing filtering are related as follows: ;

[0108] It is the set of sample indices within the bucket at second t, therefore the relationship between the two is: The bucket should contain the following information: The index to be used.

[0109] S13. Convert the discrete audio sequence a[m] into a time-stamped word sequence using speech recognition technology. ;in, For the kth recognition word (string), The timestamp for the identified word is in milliseconds. It can be either the time stamp of the word's ending or the time stamp of the word's center; either can be chosen, but in this embodiment, the word ending time is used. ASR is the speech recognition function; K is the global word count, which is the ASR output word sequence. The length is obtained directly from the recognition results.

[0110] Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims to convert the lexical content of human speech into computer-readable input, such as keystrokes, binary codes, or character sequences.

[0111] S14. Process the video frame sequence within a time bucket per second. Discrete audio sequence a[m] and word sequence Aggregate to obtain video blocks Audio blocks and subtitle blocks ;

[0112] Specifically, the following steps are included:

[0113] S141, Video Block Construction: For the first A representative frame is obtained by temporal mean fusion of video frames within a second. Then, representing the frame The original subgraph is obtained by uniformly dividing it into M rectangular grid blocks. For each original subgraph Video blocks are obtained by bilinear scaling to uniform perceptual resolution. ;in G x G is the number of grid rows. y is the number of grid columns, and p is the index of the rectangular grid block;

[0114] For the first Video frames within seconds The representative frame is obtained by performing temporal mean fusion, as shown in the following formula:

[0115] ,

[0116] Video frame sequence The Frame is a tensor , It is the pixel value of this tensor, representing the first pixel. Frame in spatial coordinates The value of the c-th color channel at position c; where , , (Corresponds to RGB).

[0117] Will The grid is uniformly divided into M non-overlapping rectangular grid blocks. G x G is the number of grid rows. y For the number of grid columns, the first... A set of rectangular grid blocks (p=1, 2, M, sorted in a fixed order of row-first-column or column-first-row) is denoted as the original subgraph. For each Perform bilinear scaling to uniform perceptual resolution Get video blocks As shown in the following formula: ;

[0118] in To achieve a unified perception resolution height, To ensure uniform perceived resolution width; Resize() is a bilinear scaling function; SamplePool() represents the video block constructor;

[0119] Furthermore, let M be the fixed number of video tokens at second t; each rectangular grid block is considered a video token.

[0120] Time-domain mean fusion is a method of data fusion that calculates the average value of multiple signals or data in the time domain. Its core is to use the statistical properties of the mean to reduce the impact of random interference and improve the signal-to-noise ratio and stability of the data.

[0121] Bilinear scaling is an interpolation method used in image or signal processing to generate smoother, more natural transitions during scaling (enlarging or shrinking). It combines the principles of linear interpolation, performing interpolation calculations in both the horizontal and vertical directions to obtain the color or brightness values ​​of the target pixel. This is existing technology and will not be elaborated upon here.

[0122] S142. Audio block construction: For the discrete audio sequence at second t... Calculate the STFT amplitude spectrum and map it to the Mel frequency band to obtain the audio block. As shown in the following formula:

[0123]

[0124] in, The amplitude spectrum of STFT; It is a short-time Fourier transform. Use a fixed window length With moving (Sample points); For fixed Mel filter banks; Let A be the numerical stability constant. Indicates the first The number of STFT time frames per second is determined by the window length. With moving (Unit: sample points) Determined: K is the number of FFT points. ; For time frame indexing, For FFT frequency bin index, This represents the Mel frequency band index; F represents the number of Mel frequency bands.

[0125] S143. Subtitle Block Construction (Bottling by Time Scale): Combining word sequences middle The words are collected in order as subtitle blocks ;in Indicates the first Within seconds Word symbols (arranged chronologically), It is the timestamp of the word (in milliseconds, satisfying...) ); s=1,...,S; S is the number of digits. The number of words per second; their relationship to the global word sequence. The relationship is: Take all pairs of strings that satisfy the condition within the interval [tl, t). The entries, by Ascending order yields the set of word indices for the second interval. , For the second-range word index, then define and Therefore, subtitle blocks It is the ordered subsequence of the global word sequence at second t; the value of S can be 0, representing that the number of words in second t is 0, at which point the subtitle block... If it is empty, the corresponding index is also empty.

[0126] Understandably, the word level from ASR in S13 is typically down to the millisecond level. This step uses a 1Hz grid for alignment / bucketing: collecting all words that satisfy the condition within the interval [tl, t). The identified words may be 0, 1, or multiple (e.g., 12131ms, 12470ms, and 12900ms are all assigned to the subtitle block at t=13 (s)). Therefore, this step involves forming an ordered subsequence of several words within the same second, which is then used to align with the audio / video features of the same second. The conversion and comparison between milliseconds and seconds is a standard existing technique and will not be elaborated here.

[0127] For each second (t=1, 2, ..., T) of the live media stream, video blocks, audio blocks, and subtitle blocks are constructed. Each second, a video block is generated. audio block and subtitle blocks The three-mode alignment blocks are constructed, and then the three-mode alignment block sequence is obtained with an alignment accuracy of 1Hz;

[0128] S2. Multimodal Embedding Construction: The aligned blocks of the three modalities are mapped to a unified spatial dimension, and a fixed absolute position code is added to each modality to obtain the video embedding sequence. Audio embedding sequence and subtitle embedding sequence ;

[0129] This step involves inputting video blocks. audio block and subtitle blocks ;

[0130] S21, Regarding video blocks The video embedding sequence is obtained by performing patch-by-patch linear projection and absolute position encoding. ;

[0131] For the video block at second t The pth video patch The video patch is embedded by linearly projecting it onto the spatial dimension d and then performing absolute position encoding according to the following formula. :

[0132] ;

[0133] Then embed the video patch Stacking yields video embedding sequences ;

[0134] Where p = 1, 2, M, and M is the number of fixed video patches per second; To prioritize the subsequent movement within the channel; It is a learnable linear projection matrix; Encoding the absolute position of learnable patches (and) One-to-one correspondence, no sharing across seconds);

[0135] S22, Audio Block The audio embedding sequence is obtained by performing time-series frame-by-frame linear projection and absolute position coding. ;

[0136] For the first Second audio block The Log-Mel frames The audio frame embedding is obtained by linearly projecting it onto the spatial dimension d and performing absolute position encoding according to the following formula. :

[0137]

[0138] Then embed the audio frame Stacking yields audio embedding sequences ;

[0139] in, It is a learnable linear projection matrix; Encoding the absolute position of learnable audio frames (and) correspond);

[0140] S23, Subtitle Blocks The subtitle embedding sequence is obtained by performing word vector lookup and absolute position encoding. ;

[0141] For the first Second Subtitle Block The first in chronological order Word The word order embedding is obtained by performing a word vector lookup table and absolute position encoding according to the following formula. :

[0142] ,

[0143] Then embed the subtitle word order Stacking yields the subtitle embedding sequence ;

[0144] Among them, Vocabulary is a vocabulary list. For learnable word embedding matrix, For indexing functions, return words Index in the vocabulary, The absolute positional encoding of the learnable word order (based on j within that second); S represents the number of words in the subtitle block at second t; |Vocab| represents the length of the vocabulary Vocab;

[0145] The vocabulary Vocab described is a dictionary that converts text into numbers. It is trained using collected live-stream corpora and open-source word segmentation tools. This vocabulary can also be interchanged with common open-source vocabulary (such as the Chinese WordPiece / BPE table) without affecting the algorithm flow described in this paper.

[0146] The final output is a three-modal embedding sequence with a unified dimension:

[0147]

[0148] These tensors will directly enter the subsequent "efficient perceptual encoding (Mamba-2 video update + text / audio self-attention + text-to-video cross-attention)" and "memory writing / retrieval" stages.

[0149] S3. Efficient perceptual coding: Within the t-th second, the video embedding sequence is linearly updated using Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ;

[0150] The input for this step is a three-modal embedding. ;

[0151] Let M be the video embedding sequence at second t; M is the fixed number of video patches (tokens) at second t.

[0152] Let A be the audio embedding sequence at second t; and let A be the acoustic frame number at second t.

[0153] Let S be the subtitle embedding sequence at second t; S is the number of words at second t.

[0154] Dimensional constants: To unify the dimensions of the latent space (consistent with S2);

[0155] In this embodiment, the fixed time bucket is 1 Hz, that is, each time bucket is 1 second; therefore, the expression "within the time bucket t per second" is equivalent to "at the t-th second".

[0156] S31. Video embedding sequence via Mamba-2 Perform linear updates to obtain the video update representation. ;

[0157] Mamba-2 is a deep learning model developed by a team led by Albert Gu and Tri Dao, designed to overcome the limitations of existing Transformer models in processing long sequence data. Mamba-2 combines the attention mechanism in Transformer with the State-Space Model (SSM) through the Structured State-Space Duality (SSD) theoretical framework, achieving a significant improvement in the ability to process long sequence data while maintaining high efficiency.

[0158] A state-space model (SSM) is a mathematical model used to describe the evolution of the state of a dynamic system over time. An SSM uses a set of matrices and state variables to describe how the system progresses with time steps. This model typically includes state equations and output equations and can be computed in continuous or discrete time.

[0159] Define linear functions based on Mamba-2 As shown below:

[0160]

[0161] Given sequence (in fixed grid order) (Permutation), M2 scan and update according to the selective state-space model (SSM), where, The Sigmoid function is used for channel-by-channel gating. This is element-wise multiplication; For learnable projection matrices, These are learnable bias parameters; This is the SSM parameter matrix; ; and Representing the gating coefficient and the first The video patch is hidden after the update.

[0162] Learnable projection matrix With SSM parameter matrix The projected matrix is ​​learned through backpropagation during end-to-end training; Initialization uses standard Xavier random initialization, and the values ​​after training are determined adaptively by the data, without manually specifying constants.

[0163] The Sigmoid function (also known as the sigmoid function or logistic function) is a mathematical function whose output value is compressed between 0 and 1. It is often used to map any real number to a probability range and is widely used in machine learning, deep learning, especially binary classification tasks and neural networks.

[0164] The output is the same length as the input. Tongwei The time / memory complexity of this function is . .

[0165] Embed video into sequence The video update representation is obtained by inputting the M2 function. ;

[0166] ;

[0167] By using Mamba-2 to perform linear state-space updates on the video embedding sequence, computational complexity is reduced.

[0168] S32. Embedding text into a sequence With audio embedding sequence The text audio sequence is obtained by concatenating along the sequence dimensions. Text audio sequence Text-audio attention is obtained through single-head self-attention function processing. ;

[0169] The concatenation here is performed along the sequence dimension (token dimension), not along the column (feature) dimension; therefore The feature dimension d remains unchanged, only the sequence length is increased.

[0170] The single-head self-attention function SA is as follows:

[0171]

[0172] in Represents the input sequence; To learn the linear projection matrix for single-head self-attention, respectively... The mapping is into query, key, and value; Normalize at the row (Query) level; Indicates transpose;

[0173] Text audio sequence Input the SA function to obtain text audio attention ;

[0174] S33, Text-audio sequence With video update representation The text-audio-video cross-attention function is used to obtain the cross-attention of text and audio / video. ;

[0175] The single-head cross-attention function CA is as follows:

[0176] ,in ; where L q To query the number of rows in matrix Q, L k Let K be the number of rows in the key matrix and V be the number of columns in the value matrix;

[0177] In this embodiment,

[0178] in is a single-head cross-attention learnable matrix, and is the training parameters. During training, is randomly initialized using Xavier and obtained through end-to-end backpropagation, without manual specification.

[0179] Therefore, text, audio, and video cross-attention ;

[0180] S34, Attention to Text and Audio Cross-attention with this audio and video Cross-modal semantic representations are obtained through scalar gating fusion. ;

[0181] scalar (Learnable) Fusing two attention outputs to obtain cross-modal semantic representations ;

[0182] in Let be a single real number and be a scalar. Scalar-gated fusion is performed on the self-attention results of text / audio and their cross-attention results on video to ensure the fusion is as simple as possible and interpretable.

[0183] S4. Building Long-Term Memory for Entities: Updating Video Representations With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory The step of selecting candidate key-value pairs Write to memory The strategy follows the rule of merging similar items and adding different ones;

[0184] S41. At second t, update the video representation. With cross-modal semantic representation Aggregate characterization is obtained through mean aggregation and splicing. As shown in the following formula:

[0185]

[0186] in, This represents the arithmetic mean of a sequence of length L over the sequence dimension, and the output is... ;

[0187] S42, Characterization of polymerization Candidate keys are obtained by performing linear mapping. With candidate values The two constitute candidate key-value pairs The linear mapping formula is as follows:

[0188]

[0189] in, These are the linear mapping of the memory keys and the offset of the memory keys, respectively; These are the linear mapping of the memory value header and the bias of the memory value header, respectively;

[0190] , During training, Xavier random initialization is used, and updates occur during end-to-end backpropagation across the entire network; no manual constants or priors are set, and during the inference phase, the current data is updated according to the formula described above. Calculated in real time;

[0191] The "video mean + semantic mean → linear head" approach is the minimum feasible entity / event digester; ultimately, each 1 Hz time bucket generates only one candidate key-value pair. .

[0192] The memory bank Each memory entry is represented by a key vector. Value vector With time coverage Composition, in which The integer second interval covered by this memory entry; Indicates the start time of this memory entry. Indicates the end time of this memory entry; This is the current memory size;

[0193] S43, from memory bank Find candidate keys The most similar memory entry with the highest similarity, if the similarity is less than a preset similarity threshold or the memory bank... When empty, select candidate key-value pairs. New memory entries are created and written to the memory bank. Otherwise, select the candidate key-value pairs. Merge and write the closest memory entry;

[0194] Specifically, it includes:

[0195] Constructing a memory bank The key matrix, by calculating candidate keys AND key matrix Find the similarity between all keys in the set and the candidate key. Index of most similar keys At the same time, record the maximum similarity value. Understandably, finding candidate keys... Index of most similar keys This also found the candidate key. The most similar memory entry with the highest similarity ;

[0196]

[0197] Where sim() is the similarity function, and the formula for calculating the similarity function in this embodiment is: Where x and y are two vectors used to calculate similarity; Indicates transpose;

[0198] The preset similarity threshold ;

[0199] like Then the candidate key-value pairs New memory entries are created and written to the memory bank. Perform the following new operation: Numbered : ;

[0200] Otherwise, select candidate key-value pairs. Most similar memory entry After performing exponential moving average merging, update the closest memory entries. As shown below:

[0201] ;

[0202] in During merging, only the end time is updated to the current second t; coefficients ;

[0203] In addition, if (If the memory is empty), also select candidate key-value pairs. New memory entries are created and written to the memory bank. ;

[0204] Finally, the updated memory bank was obtained. ,in or Depending on whether it's a merge or a new entry, each memory entry is represented by a key vector. Value vector With time coverage composition.

[0205] S5, Memory-Perception Collaboration: Through the memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ;

[0206] The purpose of this step is to: at second t, through the memory bank The search results are relevant to the current cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. This is to reduce "cross-segment forgetting" and "entity / topic drift".

[0207] S51. Query Vector Construction (Query): Cross-modal semantic representation After performing sequence mean pooling, a linear transformation is applied to construct the query vector. ;

[0208] Let L be the cross-modal semantic representation at second t, where L = S + A, A is the number of acoustic frames at second t, and S is the number of words at second t. For implicit space dimension; denoted as , .

[0209] First, cross-modal semantic representation Perform sequence mean pooling to obtain the global semantic summary for that second. Then, the global semantic summary is linearly transformed to generate a query vector. ;

[0210] in , These are learnable parameters; For length of The arithmetic mean of the sequences is calculated over the sequence dimension. ;

[0211] S52. Memory Retrieval (Scaled Dot-Product Attention): Utilizing query vectors In memory bank The memory vector is obtained by retrieval. ;

[0212] memory bank ,in , Let the key matrix Value matrix ;

[0213] By query vector AND key matrix Calculate attention score With probability Then based on probability AND-value matrix The weighted sum is used to obtain the memory vector. ;memory vector This refers to the search results;

[0214] in, Here is the normalization function along the vector dimension; The natural exponential function (base) );

[0215] S53. Gating Strength Estimation (Scalar Gating): Calculate the memory vector. scalar gating ;

[0216] Compute a scalar gate To control the strength of the impact of memory on the current moment;

[0217]

[0218] in For learnable weight matrix, For bias terms, ; ; Indicates transpose;

[0219] S54, Memory Feedback and Broadcast Add: Add memory vectors Copying to cross-modal semantic representation After the number of rows, pass through scalar gating With cross-modal semantic representation Aligned and summed, we obtain memory-enhanced cross-modal semantics. ;

[0220] memory vector Copy to line L and combine with cross-modal semantic representation After aligning, add them together as shown below:

[0221]

[0222] in, Enhanced cross-modal semantics for memory enhancement at second t; It is a vector consisting entirely of 1s; This is a row broadcast copy function used to copy... Copy as matrix;

[0223] S6. Based on enhanced cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. ;

[0224] S61. Semantic Channel Score: Enhanced cross-modal semantics calculated using a single-layer linear mapping and the Sigmoid function. Local saliency of each semantic item Take one of them Maximum value as an enhancement of cross-modal semantics semantic channel score ; Let L be the cross-modal semantic representation at second t, where L = S + A, A is the number of acoustic frames at second t; S is the number of words at second t; and d is the dimension of the latent space. , This represents the j-th enhanced cross-modal semantic term at second t;

[0225] For each enhanced cross-modal semantic item Local significance is calculated using a single-layer linear mapping and the Sigmoid function. :

[0226]

[0227] Take the local significance of that second. The maximum value is used as the (text) semantic channel score. ,in For the Sigmoid function; , Learnable weights and biases for a single-layer linear mapping, used to enhance cross-modal semantic terms. This is mapped to the significance logarithm of the video update representation patch;

[0228] S62. Audio Channel Score: Calculate the root mean square energy of the discrete audio sequence a[m] per second, and then perform minimum-maximum normalization of the root mean square energy over the total audio duration to obtain the audio channel score. ;

[0229] a[m] is a discrete audio sequence, the first... Second corresponding index set ; This refers to the audio sampling rate.

[0230] Calculate the root mean square energy (RMS) of the discrete audio sequence a[m] at second t. ,

[0231] ;

[0232] The min-max normalization over the total audio duration specifically includes: the root mean square energy of the discrete audio sequence a[m] per second over the total audio duration T. Find the maximum root mean square energy With minimum root mean square energy Then globally The audio channel scores are obtained by performing min-max normalization. : ;

[0233] in ; , is the numerical stability constant; T is the total duration of the original video stream / original audio stream, in seconds;

[0234] S63, Visual Channel Score: Representation updated based on video. The video update representation difference between adjacent seconds is calculated to measure the intensity of motion change. Then, the minimum-maximum normalization of motion change intensity is performed over the total video duration to obtain the visual channel score. ;

[0235] The intensity of motion change is measured by the difference in video updates between adjacent seconds using the following formula. :

[0236]

[0237] in Update the representation of the video at second t. The One video update characterization patch, of which Update the representation of the video at second t-1. The One video update characterization patch;

[0238] Calculate the difference in video update characteristics between adjacent seconds over the total video duration T to measure the intensity of motion (shot) changes and identify the intensity of the maximum motion change. With minimum intensity of motion change Then, the audio channel scores are obtained by performing min-max normalization over the total global video duration T. ;

[0239] S64. Based on semantic channel scoring Audio channel score and visual channel score A weighted fusion is performed to obtain the brilliance score. Score based on the level of excitement Building a sequence of wonderful moments every second ;

[0240] The weighted fusion formula is as follows:

[0241]

[0242] Fusion weights ,satisfy ;

[0243] in, For semantic channel weights, For audio channel weights, These are the visual channel weights; each weight value can be selected according to requirements. Preferably, in this embodiment, =1 / 3; =1 / 3, =1 / 3;

[0244] S65, Based on the upper threshold From the exciting sequence every second Filter out the first part of the collection The first part will be collected. After decomposing the fragment into several consecutive integer intervals, a cumulative score is calculated for each interval to obtain a set of candidate fragments. ;

[0245] First, define the upper threshold set. ;

[0246] in The upper threshold is... Its value is selected according to the actual situation; in this embodiment, it is taken as 0.6.

[0247] The first part will be collected. Decomposed into several intervals of the largest consecutive integers:

[0248] The largest continuous integer interval and Each pair is neither adjacent nor overlapping; Let be the starting integer of the i-th largest consecutive integer interval, representing the starting boundary of the interval; Let be the ending integer of the i-th largest continuous integer interval, representing the boundary of the interval;

[0249] Nc is the number of the largest consecutive integer intervals; it refers to the longest consecutive interval of seconds that cannot be extended further to the left or right within the upper threshold set. (Endpoints are integers in seconds). A single linear scan (or run-length encoding) is sufficient to... Decompose it into several such intervals. This refers to the number of intervals obtained; the start and end times of each interval are denoted as follows: Its cumulative score over the interval .

[0250] Calculate the cumulative score for each interval Candidate segments are constructed based on the largest consecutive integer interval and its corresponding cumulative score. Obtain a set of candidate fragments ;

[0251] This step constructs a per-second highlight score across T buckets of the total video length. and with a single threshold Transform it into the set of the largest continuous intervals Each interval includes a cumulative score. .

[0252] S7. Segment Selection Optimization and Boundary Refinement: Based on the candidate segment set... Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ;

[0253] S71, Calculate candidate fragments interval length With front-wheel drive ;

[0254] The interval length The unit is seconds; and the candidate fragments Sort by end time: Without loss of generality, let .

[0255] The predecessor ; On the timeline and candidate segments Non-overlapping and closest candidate fragment The candidate fragment index (0 if it does not exist);

[0256] S72, Based on candidate fragments and its interval length With front-wheel drive Calculate the maximum cumulative integral of candidate segments to construct the state matrix. Then to The selected interval set is obtained by backtracking and filtering the selected intervals. ;

[0257] Specifically, it includes:

[0258] Let the state matrix be... For in front The interval length of each candidate segment does not exceed The maximum cumulative score in seconds, of which , For a given total length ;

[0259] initialization Then, transfer the value of i. ,right Transfer, :

[0260]

[0261] Then to The selected interval set is obtained by backtracking and filtering the selected intervals. Specifically, it includes:

[0262] Create selection set ,right from Working backwards, if Then select the candidate segment , and Constructing intervals Add to the selected range set and order Otherwise Finally, a set of selected intervals arranged in chronological order is obtained. ;

[0263] The computational complexity is: time The space can be scrolled to .

[0264] S73, Select the set of intervals Each selected interval Symmetric window refinement is performed on the start and end boundaries and the boundary cost function is applied. Calculate the single-point cost at the start and end boundaries of each selected interval, and construct the refined interval by taking the minimum single-point cost at the start and end boundaries. ;

[0265] Define the boundary cost function (In seconds, the smaller the value, the more suitable it is as an in / out point):

[0266]

[0267] in, Score the audio channel; Score the visual channel; For audio cost weights, For visual cost weights; where and In this embodiment (Audio silence was the dominant signal during the live stream), which can be verified later. Grid fine-tuning.

[0268] This function favors weaker audio frequencies. Small) and the differences between adjacent images are large ( Cut in / out at the position with the largest value, because the cost function value may be smaller at this point.

[0269] It is understandable that the start and end boundaries include the starting boundary of the interval and the ending boundary of the interval;

[0270] Each selected interval The starting boundary of the interval Perform symmetric window refinement and apply the boundary cost function. Calculate the single-point cost for each point in the symmetric window, and take the minimum single-point cost among them. :

[0271]

[0272] Each selected interval The endpoint boundary of the interval Perform symmetric window refinement and apply the boundary cost function. Calculate the single-point cost for each point in the symmetric window, and take the minimum single-point cost among them. :

[0273]

[0274] Wherein, δ is the preset window radius. The window expands to both sides with the original boundary point as the center to form a symmetrical search range, that is, a symmetrical window; δ is generally taken as 1-2; in this embodiment, δ is given as 1 under 1 Hz grid, which can be finely adjusted according to the actual situation;

[0275] S74, Refining the interval Sort by time in ascending order and remove overlapping intervals to obtain a selected collection of segments. ;

[0276] For refined intervals Process in ascending order of time, ensuring no overlap and preserving the original chronological order:

[0277] make ,right : ;

[0278] in Represents the set of selected intervals The number of selected intervals;

[0279] like If so, delete that segment (this rarely happens, usually). (Taking the smaller value can avoid this), resulting in a final set of selected segments with no overlapping intervals. The unit is seconds; This represents the total number of segments in the selected segment collection.

[0280] S8. Generating and exporting from rough cut to fine cut: This involves creating a collection of selected clips. Convert to an editing decision list, and then stitch together the edited footage sequentially on the timeline based on the editing decision list to output the final cut and the editing decision list file.

[0281] The input for this step includes:

[0282] A collection of selected clips: , Unit: seconds.

[0283] Fixed frame rate video (frames per second); audio sampling rate Hz.

[0284] Video frame sequence Discrete audio sequence a[m];

[0285] S81. Establish a closed interval second boundary to discrete index mapping for selected segments, including video frame start and end index mapping and audio sample start and end index mapping;

[0286] Selected Excerpts cover The range of the discrete index mapping for the closed interval of the video frame corresponding to an integer second is... (Including endpoints), referred to as the video frame start and end index mapping, corresponds to the closed interval discrete audio sequence. The range of discrete index mapping is (Including endpoints), this is called the audio sample start-end index mapping;

[0287]

[0288] Create an index mapping for video frames; Map the video frame index; Create an index mapping for the audio samples; For audio sample index mapping; To round down, ensure consistency between the mapping time and the index;

[0289] S82. Generate a set of selected clips in chronological order based on the video frame start-end index mapping and the audio sample start-end index mapping. The edit decision list uses a unified three-coordinate system based on second-frame samples. The minimum field of the edit decision list is shown in EDL:

[0290]

[0291] in, The boundary is in seconds; For video frame indexing; For audio sample indexing; The segments are numbered sequentially (starting from 1). In this embodiment, the time order is based on... Ascending order;

[0292] S83. Concatenate the video frames and discrete audio in the timeline according to the editing decision list to obtain the selected video frame sequence. and selected audio sample sequences ;

[0293] Define the connection operator Seamlessly stitch together several frames / sample sequences in sequence (without introducing interpolation or transition):

[0294]

[0295] in, Indicates the first The video frame subsequence corresponding to the selected clip; This represents the audio sample subsequence corresponding to the i-th selected segment;

[0296] Output a sequence of selected video frames sequentially according to the given input sequence. and selected audio sample sequences The output remains a video frame sequence / audio sample sequence;

[0297] S84, Selected video frame sequence and selected audio sample sequences The final edited video is obtained by reusing and encapsulating it in a container format, and the editing decision list is written to the editing decision list file.

[0298] Selected video frame sequences and selected audio sample sequences The final edited video is obtained by reusing and encapsulating it in a container format;

[0299] The container format includes video formats such as MP4 and MKV, and the specific container format is selected according to the actual situation.

[0300] Simultaneously, the edit decision list will be written to the edit decision list file in JSON or CSV format;

[0301] The final edited video's total length ;

[0302] An Edit Decision List (EDL) is a file format widely used in film and television post-production to record all editing decisions made during the video editing process, including the location of cut points, the selection of source material, timecode information, etc.

[0303] The multiplexing encapsulation adopts standard container multiplexing: the spliced ​​video frame sequence and audio sample sequence are written into the MP4 / MKV container according to the timestamp, and the stream copy is prioritized to avoid re-encoding. For example, it can be achieved by calling the FFmpeg API, which is existing technology and will not be described in detail here.

[0304] This embodiment uses a unified 1Hz time grid to file the live stream, constructing lightweight embeddings for audio, subtitles, and video motion. Based on this, it achieves efficient perception using "video Mamba-2 linear updates + text / audio self-attention and cross-attention to the video," obtaining a robust semantic representation of the current moment. Simultaneously, it constructs a long-term memory bank centered on people and events, and feeds back the memory information to the current representation through a single retrieval and scalar gating to maintain consistency across segments. Based on this, the invention generates and merges the highlights per second according to the text / audio / visual three channels, uses interval optimization with duration budget constraints to select segments, refines the boundaries with a simple cost function of audio energy and shot changes, and finally generates an Edit Decision List (EDL) for direct cutting and splicing, quickly producing high-quality, broadcastable, finely edited videos.

[0305] Furthermore, this embodiment links and analyzes audio, subtitles, and video actions on a unified time grid. It achieves low-complexity representation of long videos using an efficient perception module, while leveraging the long-term memory of entity centers to preserve key characters, events, and semantic cues across segments. Memory retrieval is then used to feed back into current perception, enhancing consistency and robustness. Based on this, the invention generates a three-mode fusion performance score, performs interval optimization and boundary refinement with time budget constraints, and finally outputs an editing decision list, quickly stitching it together into a high-quality, finely edited video. This significantly improves the post-production efficiency and final quality of live stream content, effectively solving the problems of large amounts of long-duration live stream material, inefficient manual review and editing, and the risk of missing exciting segments.

[0306] Example 2

[0307] refer to Figure 2 This embodiment provides a live long video editing device based on memory-aware collaboration, which is based on the live long video editing method based on memory-aware collaboration described in Embodiment 1, and includes the following units:

[0308] Live data time alignment unit, used to convert the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks N is the total number of video frames; K is the total number of global words.

[0309] Multimodal embedding building blocks are used to map three-modal aligned blocks to a unified spatial dimension and add fixed absolute position encoding for each modality to obtain a video embedding sequence. Audio embedding sequence and subtitle embedding sequence ;

[0310] Highly efficient perceptual coding unit for linearly updating video embedding sequences via Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ;

[0311] Entity-centric long-term memory building blocks are used to update video representations. With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory ;

[0312] Memory-perception collaborative building unit, used to utilize memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ;

[0313] Candidate fragment generation unit, used to enhance cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. ;

[0314] Selected segment generation unit, used to generate segments based on the candidate segment set Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ;

[0315] The finished product and decision document export unit is used to export selected fragments. Convert to an editing decision list, and then stitch together the edited footage and the editing decision list file sequentially on the timeline based on the editing decision list.

[0316] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 A process, multiple processes, and / or boxes Figure 1 Devices that specify the functions in one or more boxes.

[0317] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction device, which is implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0318] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0319] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A method for editing long live videos based on memory-perception collaboration, characterized in that, Includes the following steps: S1. Convert the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks N is the total number of video frames; K is the total number of global words. For the k-th recognized word, For identifying the time stamp of words; S2. Map the three-modal alignment blocks to a unified spatial dimension, and add fixed absolute position encoding for each modality to obtain the video embedding sequence. Audio embedding sequence and subtitle embedding sequence ; S3. Linearly update the video embedding sequence using Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ; S4. Update the video representation. With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory ;in, Candidate key; Candidate values; S5, via memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ; S6. Based on enhanced cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. Among them, the upper threshold is a real number greater than 0 and less than 1; S7. Based on the candidate fragment set Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ; S8. Compile selected clips Convert to an editing decision list, and then stitch together the edited footage and the editing decision list file sequentially on the timeline based on the editing decision list.

2. The method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11. Convert the original video stream to a constant frame rate to obtain a video frame sequence. ,in, For a video frame, the set of frame indices corresponding to second t is: and Where H is the video frame height; W is the video frame width; and 3 represents the RGB three channels. The video has a fixed frame rate; n is the frame index; F is the frame rate. Indicates the number of elements in the set; S12. Resample the original audio stream to obtain a discrete audio sequence a[m]. The set of audio sample indices corresponding to the t-th second is... , where m is the sample index; Represents the set of integers; Sampling rate; S13. Convert the discrete audio sequence a[m] into a time-stamped word sequence using speech recognition technology. ;in, For the k-th recognized word, The time stamp for the identified word is in milliseconds; ASR is the speech recognition function. S14. Process the video frame sequence within a time bucket per second. Discrete audio sequence a[m] and word sequence Aggregate to obtain video blocks Audio blocks and subtitle blocks .

3. The method according to claim 2, characterized in that, Step S14 specifically includes the following steps: S141, regarding the first A representative frame is obtained by temporal mean fusion of video frames within a second. Then, representing the frame The original subgraph is obtained by uniformly dividing it into M rectangular grid blocks. For each original subgraph Video blocks are obtained by bilinear scaling to uniform perceptual resolution. ;in G x G is the number of grid rows. y is the number of grid columns, and p is the index of the rectangular grid block; each rectangular grid block is a video patch; To achieve a unified perception resolution height, To ensure uniform perception resolution width; S142, for the discrete audio sequence at second t. Calculate the STFT amplitude spectrum and map it to the Mel frequency band to obtain the audio block. Where A is the first... The number of STFT time frames per second; F is the number of Mel frequency bands; S143, Word Sequence middle The words are collected in order as subtitle blocks ;in Indicates the first Within seconds The word symbol of each word; S is the word index for the second interval; S is the word index for the second interval. Word count within seconds.

4. The method according to claim 3, characterized in that, Step S2 specifically includes the following steps: S21, Regarding video blocks The video embedding sequence is obtained by performing patch-by-patch linear projection and absolute position encoding. : For the video block at second t The pth video patch The video patch is embedded by linearly projecting it onto the spatial dimension d and then performing absolute position encoding according to the following formula. : ； Then embed the video patch Stacking yields video embedding sequences ; Where p = 1, 2, ..., M, M is the number of fixed video patches per second; To prioritize the subsequent movement within the channel; It is a learnable linear projection matrix; Encode the absolute position of learnable patches; S22, Audio Block The audio embedding sequence is obtained by performing time-series frame-by-frame linear projection and absolute position coding. : For the first Second audio block The Log-Mel frames The audio frame embedding is obtained by linearly projecting it onto the spatial dimension d and performing absolute position encoding according to the following formula. : ， Then embed the audio frame Stacking yields audio embedding sequences ; in, It is a learnable linear projection matrix; Encode the absolute position of learnable audio frames; S23, Subtitle Blocks The subtitle embedding sequence is obtained by performing word vector lookup and absolute position encoding. : For the first Second Subtitle Block The first in chronological order Word The word order embedding is obtained by performing a word vector lookup table and absolute position encoding according to the following formula. : ， Then embed the subtitle word order Stacking yields the subtitle embedding sequence ; Among them, Vocab is a vocabulary list; For learnable word embedding matrix; For indexing functions, return words Index in the vocabulary; This is a learnable absolute word order encoding; S represents the number of words in the subtitle block at second t; |Vocab| represents the length of the vocabulary Vocab.

5. The method according to claim 4, characterized in that, Step S3 specifically includes the following steps: S31. Video embedding sequence via Mamba-2 Perform linear updates to obtain the video update representation. ; S32. Embedding text into a sequence With audio embedding sequence The text audio sequence is obtained by concatenating along the sequence dimensions. Text audio sequence Text-audio attention is obtained through single-head self-attention function processing. ; S33, Text-audio sequence With video update representation The text-audio-video cross-attention function is used to obtain the cross-attention of text and audio / video. ; S34, Attention to Text and Audio Cross-attention with this audio and video Cross-modal semantic representations are obtained through scalar gating fusion. .

6. The method according to claim 5, characterized in that, Step S4 specifically includes the following steps: S41. At second t, update the video representation. With cross-modal semantic representation Aggregate characterization is obtained through mean aggregation and splicing. As shown in the following formula: ， in, This represents the arithmetic mean of a sequence of length L over the sequence dimension; This means replacing M. L in the video update representation Calculate the arithmetic mean; This means replacing S+A. L-pair cross-modal semantic representation Calculate the arithmetic mean; S42, Characterization of polymerization Candidate keys are obtained by performing linear mapping. With candidate values The two constitute candidate key-value pairs The linear mapping formula is as follows: ， in, These represent the linear mapping of the memory keys and the offset of the memory keys, respectively. These are the linear mapping of the memory value header and the bias of the memory value header, respectively; S43, from memory bank Find candidate keys The most similar memory entry with the highest similarity, if the similarity is less than a preset similarity threshold or the memory bank When empty, select candidate key-value pairs. New memory entries are created and written to the memory bank. Otherwise, select the candidate key-value pairs. The most similar memory entries are merged and written to obtain the updated memory bank. .

7. The method according to claim 6, characterized in that, Step S5 specifically includes the following steps: S51, Cross-modal semantic representation After performing sequence mean pooling, a linear transformation is applied to construct the query vector. ; S52, Utilizing query vectors In memory bank The memory vector is obtained by retrieval. ; S53, Calculate the memory vector scalar gating : ， in For learnable weight matrix, For bias terms; ; Indicates transpose; S54, Memory Vector Copying to cross-modal semantic representation After the number of rows, pass through scalar gating With cross-modal semantic representation Aligned and summed, we obtain memory-enhanced cross-modal semantics. .

8. The method according to claim 7, characterized in that, Step S6 specifically includes the following steps: S61. Enhancing cross-modal semantics through single-layer linear mapping and Sigmoid function computation. Local saliency of each semantic item Take one of them Maximum value as an enhancement of cross-modal semantics semantic channel score ; S62. Calculate the root mean square energy of the discrete audio sequence a[m] per second, and then perform minimum-maximum normalization of the root mean square energy over the total audio duration to obtain the audio channel score. ; S63. Update representation based on video. The video update representation difference between adjacent seconds is calculated to measure the intensity of motion change. Then, the minimum-maximum normalization of motion change intensity is performed over the total video duration to obtain the visual channel score. ; S64. Based on semantic channel scoring Audio channel score and visual channel score A weighted fusion is performed to obtain the brilliance score. Score based on the level of excitement Building a sequence of wonderful moments every second ; S65, Based on the upper threshold From the exciting sequence every second Filter out the first part of the collection The first part will be collected. After decomposing the fragment into several consecutive integer intervals, a cumulative score is calculated for each interval to obtain a set of candidate fragments. ;in Candidate segments; Let be the starting integer of the i-th largest consecutive integer interval, representing the starting boundary of the interval; Let be the ending integer of the i-th largest continuous integer interval, representing the boundary of the interval; is the cumulative score over the interval; Nc is the number of the largest consecutive integer intervals.

9. The method according to claim 8, characterized in that, Step S7 specifically includes the following steps: S71, Calculate candidate fragments interval length With front-wheel drive ; S72, Based on candidate fragments and its interval length With front-wheel drive Calculate the maximum cumulative integral of candidate segments to construct the state matrix. Then to The selected interval set is obtained by backtracking and filtering the selected intervals. ; S73, Select the set of intervals Each selected interval Symmetric window refinement is performed on the start and end boundaries and the boundary cost function is applied. Calculate the single-point cost at the start and end boundaries of each selected interval, and construct the refined interval by taking the minimum single-point cost at the start and end boundaries. ; S74, Refining the interval Sort by time in ascending order and remove overlapping intervals to obtain a selected collection of segments. .

10. A live-stream long video editing device based on memory-aware collaboration, based on the live-stream long video editing method based on memory-aware collaboration as described in any one of claims 1-9, characterized in that, include: Live data time alignment unit, used to convert the original video stream to a constant frame rate to obtain a video frame sequence. The original audio stream is resampled to construct a discrete audio sequence a[m] and a word sequence. The three are then aggregated within a time bucket per second to obtain a trimodal aligned block sequence; the trimodal aligned block sequence includes video blocks. Audio blocks and subtitle blocks N is the total number of video frames; K is the total number of global words. Multimodal embedding building blocks are used to map three-modal aligned blocks to a unified spatial dimension and add fixed absolute position encoding for each modality to obtain a video embedding sequence. Audio embedding sequence and subtitle embedding sequence ; Highly efficient perceptual coding unit for linearly updating video embedding sequences via Mamba-2. Obtain video update representation It combines single-head self-attention and cross-attention mechanisms to process audio embedding sequences. and subtitle embedding sequence Cross-modal semantic representations are generated through scalar gating fusion. ; Entity-centric long-term memory building blocks are used to update video representations. With cross-modal semantic representation Candidate key-value pairs are obtained by mean aggregation and concatenation followed by linear mapping. and candidate key-value pairs Write to memory ; Memory-perception collaborative building unit, used to utilize memory bank The search results are helpful for cross-modal semantic representation. Consistency enhancement is performed to obtain enhanced cross-modal semantics. ; Candidate fragment generation unit, used to enhance cross-modal semantics With video update representation Calculate semantic channel score Audio channel score and visual channel score The system calculates the brilliance score sequence based on the weighted fusion of the three factors, and then decomposes and calculates the candidate segment set after filtering by the upper threshold. ; Selected segment generation unit, used to generate segments based on the candidate segment set Calculate the interval length and predecessor, and construct a state matrix for backtracking filtering to obtain the selected interval set. Then, for the selected interval set After refining the selected intervals using symmetrical windows at their start and end boundaries, the single-point cost is calculated to construct the refined intervals. Finally, the refined intervals are sorted in ascending time order and overlaps are removed to obtain a set of selected segments. ; The finished product and decision document export unit is used to export selected fragments. Convert to an editing decision list, and then stitch together the edited footage and the editing decision list file sequentially on the timeline based on the editing decision list.

Images