An intelligent video summarization system based on cross-modal alignment

Abstract

The application discloses an intelligent video summary system based on cross-modal alignment, relates to the technical fields of cross-modal alignment and video processing, and aims to solve the technical problems that in the prior art, a fine cross-modal alignment mechanism is lacked during multi-modal summary generation, the time sequence offset among frames, phonemes and words cannot be solved, and global consistency cannot be realized at the event level and the semantic paragraph level, specifically, an improved hierarchical cross-modal alignment algorithm and a dynamic weight distribution mechanism are introduced to solve the problems of multi-modal feature time sequence misplacement, semantic inconsistency and rigid weight distribution in the video summary, fine alignment of frames, phonemes and words is realized through soft dynamic time warping and sparse optimal transmission, and event level and semantic paragraph level representations are generated in combination with time sequence change detection and hierarchical attention, so that the accuracy of cross-modal fusion is improved.

Description

Technical Field

[0001] This invention relates to the field of cross-modal alignment and video processing technology, specifically to an intelligent video summarization system based on cross-modal alignment. Background Technology

[0002] With the widespread application of video in fields such as security monitoring, education and training, news reporting and social media, how to quickly generate summaries that reflect core events and key information from massive amounts of video has become a research hotspot.

[0003] Currently, most methods utilize deep learning to achieve multimodal summarization, simply concatenating or weighting video, audio, and text features to generate a unified representation. However, these methods lack sophisticated cross-modal alignment mechanisms, fail to address temporal offsets between frames, phonemes, and words, and fail to achieve global consistency at the event and semantic paragraph levels. This results in limited feature fusion performance and an inability to effectively eliminate temporal misalignment and semantic inconsistency between modalities.

[0004] Furthermore, conventional weighting mechanisms mostly use fixed weights or static factors, making it difficult to dynamically adjust based on the signal-to-noise ratio, text completeness, or video frame clarity of different modalities. When the signal of a certain modality degrades, it will still have a negative impact on the results, reducing the robustness and accuracy of the summary. In addition, although summary generation methods based on attention mechanisms or reinforcement learning improve the degree of automation, they generally lack the ability to customize summaries by combining user input intent, and cannot guarantee that the output results are highly matched with the user's search needs.

[0005] To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0006] The purpose of this invention is to solve the problems mentioned above by proposing an intelligent video summarization system based on cross-modal alignment.

[0007] The objective of this invention can be achieved through the following technical solution: an intelligent video summarization system based on cross-modal alignment, comprising: The multimodal acquisition and preprocessing module is used to acquire video frame sequences, audio signals and text data, and perform frame extraction, feature extraction, speech recognition transcription and timestamp alignment. The hierarchical cross-modal alignment coding module is used to encode video, audio and text features, and achieve local alignment at the frame level, phoneme level and word level, and global alignment at the event level and semantic paragraph level, to obtain multi-level aligned feature representations; The dynamic weight allocation module is used to perform modal reliability evaluation on the feature representation and adaptively adjust the modal weights according to the signal-to-noise ratio, text integrity, and video frame clarity to generate a dynamically weighted feature representation. The intent-conditional summary selection module maps user input to a shared semantic space, calculates its relevance to dynamically weighted feature representations, and selects a set of target fragments by combining event boundaries and diversity constraints. The summary generation module is used to generate video summaries and timeline-style key points based on the target segment set; The results output module is used to output the video summary and key points.

[0008] Furthermore, the process of the multimodal acquisition and preprocessing module is as follows: S11. Acquire and decode the video stream to obtain the original video frame sequence with timestamps; S12. Perform frame rate standardization, duplicate frame removal, color space unification, size normalization, brightness normalization, and noise reduction on the original video frame sequence. S13. Perform frame extraction on the preprocessed video frame sequence and output the video keyframe sequence according to the preset frame rate. S14. Separate the audio signal from the video stream, extract features from the audio signal, and obtain an audio feature vector sequence; S15. Perform speech recognition and transcription on the audio signal to generate a text transcription sequence with timestamps; S16. Obtain the text transcription sequence or video stream subtitle file, perform character encoding unification, symbol cleaning, sentence segmentation and word segmentation on the text content, assign start and end timestamps to the word segmentation units, and output the text sequence with time tags. S17. Align the video keyframe sequence, audio feature vector sequence, and time-stamped text sequence on the time axis to obtain synchronized multimodal input.

[0009] Furthermore, the process of hierarchically aligning the coding modules is as follows: S21. Input the video keyframe sequence from the synchronized multimodal input into the visual feature encoder to extract frame-level video feature vectors; S22. Input the audio feature vector sequence in the synchronized multimodal input into the audio feature encoder to extract phoneme-level audio feature vectors; S23. Input the time-stamped text sequence from the synchronized multimodal input into the text feature encoder to extract word-level text feature vectors; S24. Map the frame-level video feature vector, phoneme-level audio feature vector, and word-level text feature vector to a shared semantic space of a unified dimension; S25. Form an initial multimodal feature representation in the shared semantic space.

[0010] Furthermore, the process of the dynamic weight allocation module is as follows: S41. Perform reliability detection on the video feature subset in the hierarchically aligned multimodal feature representation, calculate frame sharpness, edge sharpness and motion blur, and generate visual reliability index. S42. Perform reliability detection on the audio feature subset in the hierarchically aligned multimodal feature representation, calculate the signal-to-noise ratio, speech segment integrity and silence ratio, and generate speech reliability index. S43. Perform reliability detection on the text feature subset in the hierarchically aligned multimodal feature representation, calculate text integrity, transcription timestamp integrity and missing word rate, and generate text reliability index. S44. Input the visual reliability index, voice reliability index and text reliability index into the modality weight generator to generate three types of modality basic weights. The modal weight generator normalizes the visual reliability index, speech reliability index, and text reliability index to make their value ranges the same; and maps the normalized indexes to weight coefficients through linear transformation or nonlinear function mapping. S45. Apply the three types of modal base weights to the hierarchically aligned multimodal feature representation, weight the multimodal feature representation, and output a dynamically weighted multimodal feature representation.

[0011] Furthermore, the process of the intent-conditional summary selection module is as follows: S51. Encode the prompts or query content entered by the user to generate a user intent vector; S52. Map the user intent vector to the same shared semantic space as the dynamically weighted multimodal feature representation to form an intent representation; S53. Perform stepwise similarity calculation on the time axis between the intent representation and the dynamically weighted multimodal feature representation to generate a temporal correlation score sequence; S54. Perform normalization processing on the time-series correlation score sequence to ensure that the scores are within a preset range. S55. Output the normalized time-series correlation score sequence.

[0012] Furthermore, the event boundary detection process is as follows: S61. Perform abrupt change detection on the time-series correlation score sequence and mark the time positions where the correlation score changes significantly as candidate boundary points; S62. Filter the candidate boundary points according to the minimum event duration threshold and the adjacent boundary distance constraint to generate an event boundary set; S63. Divide the temporal correlation score sequence into several event intervals according to the event boundary set; S64. Extract the corresponding video frame sequence, audio feature sequence and text feature sequence within each event interval to construct a candidate segment set; S65. Calculate the relevance score between the candidate fragment and the user intent representation, and generate a candidate fragment relevance vector; S66. Calculate the similarity between candidate segments and construct a candidate segment similarity matrix; S67. Based on the candidate segment relevance vector and similarity matrix, diversity constraints are applied to obtain a preliminary set of segments; The diversity constraint establishes a coverage term based on the relevance vector and a redundancy penalty term based on the similarity matrix to form a weighted combination selection objective function. Sub-modulus optimization is performed on the objective function to output a preliminary selection fragment set with high relevance and low redundancy. S68. Filter the initial fragment set according to the preset relevance threshold and the upper limit of the summary duration, and output the target fragment set.

[0013] Furthermore, the process of the abstract generation module is as follows: S71. Perform inter-frame difference detection on each segment in the target segment set, select key frames based on the maximum visual difference and representative score, and generate a key frame set. S72. Arrange the keyframe set in chronological order to form a video summary frame sequence; S73. Extract the audio and text segments corresponding to the video summary frame sequence from the target segment set, and perform boundary cropping and temporal alignment to generate a candidate segment sequence; S74. The candidate segment sequence is spliced ​​together to maintain the consistency of the time order and output continuous video summary segments. S75. Perform keyword extraction and semantic compression on the text content in the candidate segment sequence to generate a set of key sentences; S76. Sort the set of key points according to the timestamps of the segments to form a timeline-style key point description; S77. Encapsulate the video summary frame sequence, continuous video summary segments, and timeline-style key points, and output the final summary result.

[0014] Compared with the prior art, the beneficial effects of the present invention are: First, this invention addresses the problems of temporal misalignment, semantic inconsistency, and rigid weight allocation of multimodal features in video summarization by introducing an improved hierarchical cross-modal alignment algorithm and a dynamic weight allocation mechanism. It also achieves fine alignment of frames, phonemes, and words through soft dynamic time warping and sparse optimal transmission, and improves the accuracy of cross-modal fusion by combining temporal change detection and hierarchical attention to generate event-level and semantic paragraph-level representations.

[0015] In the modal fusion stage, this invention also achieves dynamic weighting based on signal-to-noise ratio, text integrity, and video frame clarity through modal reliability detection and modal weight generator, thereby enhancing the system's adaptability to different input qualities.

[0016] Ultimately, by combining the relevance calculation of user intent representation with the selection of diversity constraints, this invention generates personalized, low-redundancy, and highly relevant video summaries and timeline-style key points, significantly improving the accuracy, robustness, and practicality of video summaries. Attached Figure Description

[0017] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0018] Fig. 1 This is an overall flowchart of an intelligent video summarization system based on cross-modal alignment proposed in this invention; Fig. 2 This is a schematic diagram of an improved hierarchical cross-modal alignment processing structure for an intelligent video summarization system based on cross-modal alignment proposed in this invention. Fig. 3 This is a flowchart of the target segment selection process for an intelligent video summarization system based on cross-modal alignment proposed in this invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] Please see Figs. 1-3 As shown, an intelligent video summarization system based on cross-modal alignment includes: a multimodal acquisition and preprocessing module, a hierarchical cross-modal alignment coding module, a dynamic weight allocation module, an intent-conditional summary selection module, a summary generation module, and a result output module; The multimodal acquisition and preprocessing module will acquire video frame sequences, audio signals, and text data, and preprocess them to obtain synchronized multimodal input; The hierarchical cross-modal aligned coding module encodes features for each multimodal input and maps the encoding results to a shared semantic space to form an initial multimodal feature representation. An improved hierarchical cross-modal alignment algorithm is used to achieve local alignment at the frame, phoneme, and word levels, and global alignment at the event and semantic paragraph levels for the initial multimodal feature representation, resulting in a hierarchically aligned multimodal feature representation. The dynamic weight allocation module performs modal reliability assessment on the hierarchically aligned multimodal feature representation and calculates the weight of each modality to obtain the dynamically weighted feature representation; The intent-conditional summary selection module maps user input to a shared semantic space to obtain intent representation, and performs relevance calculation with dynamic weighted feature representation to generate a temporal relevance score sequence. Event boundary detection is performed based on temporal correlation score sequences. A candidate segment set is constructed within the detected event interval, and a target segment set is selected based on diversity constraints and correlation thresholds. The summary generation module extracts keyframes and splices segments based on the target segment set to generate a video summary and corresponding timeline-style key points descriptions, and outputs the final result. The process of the multimodal acquisition and preprocessing module is as follows: S11. Acquire and decode the video stream to obtain the original video frame sequence with timestamps; S12. Perform frame rate standardization, duplicate frame removal, color space unification, size normalization, brightness normalization, and noise reduction on the original video frame sequence. S13. Perform frame extraction on the preprocessed video frame sequence and output the video keyframe sequence according to the preset frame rate. S14. Separate the audio signal from the video stream, extract features from the audio signal, and obtain an audio feature vector sequence; S15. Perform speech recognition and transcription on the audio signal to generate a text transcription sequence with timestamps; S16. Obtain the text transcription sequence or video stream subtitle file, perform character encoding unification, symbol cleaning, sentence segmentation and word segmentation on the text content, assign start and end timestamps to the word segmentation units, and output the text sequence with time tags. S17. Align the video keyframe sequence, audio feature vector sequence, and time-stamped text sequence along the timeline to obtain synchronized multimodal input.

[0022] In the multimodal acquisition and preprocessing stage, this invention decodes the video stream, standardizes the frame rate, and extracts frames to generate keyframe sequences. It then extracts features from the audio signal and performs speech recognition and transcription. Combined with subtitle text, it completes character cleaning, sentence segmentation, word segmentation, and timestamp annotation. Ultimately, it achieves timeline alignment of the video, audio, and text sequences, forming a synchronized multimodal input. This method ensures the consistency and temporal matching of data from different modalities, providing a reliable data foundation for cross-modal feature encoding and alignment.

[0023] The process of hierarchically aligning the coding modules across modalities is as follows: S21. Input the video keyframe sequence from the synchronized multimodal input into the visual feature encoder to extract frame-level video feature vectors; S22. Input the audio feature vector sequence from the synchronized multimodal input into the audio feature encoder to extract phoneme-level audio feature vectors; S23. Input the time-labeled text sequence from the synchronized multimodal input into the text feature encoder to extract word-level text feature vectors; S24. Map frame-level video feature vectors, phoneme-level audio feature vectors, and word-level text feature vectors to a shared semantic space of a unified dimension; S25. Form initial multimodal feature representations in the shared semantic space.

[0024] This invention feeds synchronous multimodal inputs into a visual feature encoder, an audio feature encoder, and a text feature encoder, respectively, to extract frame-level video features, phoneme-level audio features, and word-level text features. These features are then uniformly mapped to a shared semantic space to form an initial multimodal feature representation. This method achieves unified modeling and semantic alignment of multimodal features, providing high-quality feature inputs for subsequent hierarchical cross-modal alignment and improving the accuracy and robustness of cross-modal information fusion. The improved hierarchical cross-modal alignment algorithm specifically includes: At the local alignment level, a soft dynamic time warping algorithm with monotonicity constraints is used to generate a temporal mapping index between the frame level and the phoneme level. A cross-attention mechanism is used between the frame level and the word level, and the attention key value is biased to generate matching weights through an intent conditional adapter. A sparse optimal transmission algorithm is further used between frames, phonemes and words to perform sparse matching on the similarity matrix to generate soft alignment weights and discrete mappings. The soft dynamic time warping algorithm: Frame-level vector sequences and phoneme-level vector sequences, after feature encoding, are input to the soft dynamic time warping module; a cost matrix is ​​constructed using the pairwise similarity between frame vectors and phoneme vectors, and this cost matrix serves as the input for soft dynamic time warping; through differentiable dynamic time warping operations, a global optimal path search is performed on the cost matrix, outputting a soft alignment path between frames and phonemes; a temporal monotonicity constraint is applied during the path search process to ensure that the alignment path increases sequentially in time, without regression or skipping; the alignment path is output in the form of a probability distribution, with each frame corresponding to several phoneme alignment weight indices, generating a temporal mapping index; Sparse optimal transmission algorithm: It establishes sparse matching relationships between frame-level features, phoneme-level features and word-level features. It constructs a transmission cost matrix from the cross-modal similarity matrix and solves the sparse transmission plan. It outputs a weighted mapping index to generate sparse alignment results between frame-phoneme, frame-word and phoneme-word. The results are then embedded as local alignment into the input fusion unit, and confidence information is output at the same time. At the global alignment level, mutation analysis and boundary screening are performed on the difference measure of local alignment feature sequences to form event boundaries. Event-level feature representations are generated through temporal pooling within each event interval. Several adjacent event-level features are input into a hierarchical attention aggregation network to generate semantic paragraph-level feature representations. Round-trip mapping is performed between the event level and the paragraph level to form cyclic consistency constraints.

[0025] The process of the dynamic weight allocation module is as follows: S41. Perform reliability detection on the video feature subset in the hierarchically aligned multimodal feature representation, calculate frame sharpness, edge sharpness and motion blur, and generate visual reliability index. S42. Perform reliability detection on the audio feature subset in the hierarchically aligned multimodal feature representation, calculate the signal-to-noise ratio, speech segment integrity and silence ratio, and generate speech reliability index. S43. Perform reliability detection on the text feature subset in the hierarchically aligned multimodal feature representation, calculate text completeness, transcription timestamp completeness and word missing rate, and generate text reliability index. S44. Input the visual reliability index, voice reliability index and text reliability index into the modality weight generator to generate the three types of modality basic weights. The modal weight generator normalizes the visual reliability index, speech reliability index, and text reliability index to make their value ranges the same; and maps the normalized indexes to weight coefficients through linear transformation or nonlinear function mapping. S45. Apply the three types of modal base weights to the hierarchically aligned multimodal feature representation, weight the multimodal feature representation, and output the dynamically weighted multimodal feature representation.

[0026] In the dynamic weight allocation stage, this invention performs modal reliability detection on the hierarchically aligned multimodal feature representations. It calculates video sharpness and motion blur, audio signal-to-noise ratio and speech integrity, and text coverage and transcription accuracy, generating reliability indices for visual, audio, and text data. These indices are then input into a modal weight generator for normalization and mapping to obtain dynamic weight coefficients. Finally, these weights are applied to the multimodal feature representations to form a dynamically weighted result, achieving adaptive fusion of inputs of different quality and improving the stability and robustness of the summary.

[0027] The process of selecting the intent-conditional summary module is as follows: S51. Encode the prompts or query content entered by the user to generate a user intent vector; S52. Map the user intent vector to the same shared semantic space as the dynamically weighted multimodal feature representation to form an intent representation; S53. Perform stepwise similarity calculation on the time axis between the intent representation and the dynamically weighted multimodal feature representation to generate a time-series correlation score sequence; S54. Perform normalization processing on the time-series correlation score sequence to ensure that the scores are within the preset value range; S55. Output the normalized time-series correlation score sequence; The event boundary detection process is as follows: S61. Perform abrupt change detection on the temporal correlation score sequence and mark the time locations where the correlation score changes significantly as candidate boundary points; S62. Filter the candidate boundary points according to the minimum event duration threshold and the adjacent boundary distance constraint to generate an event boundary set; S63. Divide the temporal correlation score sequence into several event intervals based on the event boundary set; S64. Extract the corresponding video frame sequence, audio feature sequence and text feature sequence within each event interval to construct a candidate segment set; S65. Calculate the relevance score between the candidate fragment and the user intent representation, and generate a candidate fragment relevance vector; S66. Calculate the similarity between candidate segments and construct a candidate segment similarity matrix; S67. Based on the candidate segment relevance vector and similarity matrix, apply diversity constraints to obtain the initial segment set; The diversity constraint establishes a coverage term based on the relevance vector and a redundancy penalty term based on the similarity matrix, forming a weighted combination selection objective function. Sub-modulus optimization is then performed on the objective function to output a set of initial selected segments with high relevance and low redundancy. S68. Filter the initial selection fragment set according to the preset relevance threshold and the upper limit of the summary duration, and output the target fragment set; This invention detects abrupt changes in temporal relevance score sequences and filters event boundaries using duration thresholds, generating event intervals based on these boundaries. Within each interval, video, audio, and text segments are extracted to construct a candidate set, and their relevance vectors to user intent and the similarity matrix between segments are calculated. Based on this, diversity constraints are introduced, and an objective function is constructed using coverage and redundancy penalty terms. Submodular optimization is then performed, outputting a preliminary set of highly relevant and low-redundancy segments. This preliminary set is then filtered using relevance thresholds and duration constraints to obtain the target segment set, achieving precise selection of summary segments.

[0028] The process of the abstract generation module is as follows: S71. Perform inter-frame difference detection on each segment in the target segment set, select key frames based on the maximum visual difference and representative score, and generate a key frame set. S72. Arrange the keyframe set in chronological order to form a video summary frame sequence; S73. Extract the audio and text segments corresponding to the video summary frame sequence from the target segment set, and perform boundary clipping and temporal alignment to generate a candidate segment sequence; S74. Perform splicing on the candidate segment sequence to maintain temporal consistency and output continuous video summary segments; S75. Perform keyword extraction and semantic compression on the text content in the candidate segment sequence to generate a set of key sentences; S76. Sort the set of key points according to the timestamps of the segments to form a timeline-style explanation of key points; S77. Encapsulate the video summary frame sequence, continuous video summary segments, and timeline-style key points, and output the final summary result.

[0029] This invention performs inter-frame difference detection on a target segment set, extracts keyframes based on visual differences and representativeness scores, and generates a summary frame sequence. Simultaneously, it extracts corresponding audio and text segments, performs boundary clipping and temporal alignment, and splices them to output continuous summary segments. Furthermore, it performs keyword extraction and semantic compression on the segment text to generate a set of key sentences, and sorts them by timestamp to form a timeline-style key point description. Finally, it encapsulates the video summary frame sequence, spliced ​​segments, and key point descriptions to output personalized and low-redundancy video summary results, ensuring the integrity and readability of the summary.

[0030] This invention encodes user-input prompts or query content and maps them to a shared semantic space to generate an intent representation. Then, the intent representation and the dynamically weighted multimodal feature representation are progressively similar along the time axis to obtain a temporal relevance score sequence. Normalization is then used to ensure that the numerical range is consistent. The final output relevance score sequence provides a reliable basis for subsequent event boundary detection and fragment selection, achieving accurate matching between user intent and multimodal features, and improving the relevance and personalization of the summarization results.

[0031] Thresholds, preset values, preset ranges, etc. are set for result comparison and analysis to determine whether they are good or bad. The value of these thresholds is determined by a combination of large-scale model analysis of sample data and human experience. They can also be adjusted appropriately based on seasonal or common-sense influences. The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An intelligent video summarization system based on cross-modal alignment, characterized in that, include: The multimodal acquisition and preprocessing module is used to acquire video frame sequences, audio signals and text data, and perform frame extraction, feature extraction, speech recognition transcription and timestamp alignment. The hierarchical cross-modal alignment coding module is used to encode video, audio and text features, and achieve local alignment at the frame level, phoneme level and word level, and global alignment at the event level and semantic paragraph level, to obtain multi-level aligned feature representations; The dynamic weight allocation module is used to perform modal reliability evaluation on the feature representation and adaptively adjust the modal weights according to the signal-to-noise ratio, text integrity, and video frame clarity to generate a dynamically weighted feature representation. The intent-conditional summary selection module maps user input to a shared semantic space, calculates its relevance to dynamically weighted feature representations, and selects a set of target fragments by combining event boundaries and diversity constraints. The summary generation module is used to generate video summaries and timeline-style key points based on the target segment set; The results output module is used to output the video summary and key points.

2. The intelligent video summarizing system based on cross-modal alignment according to claim 1, characterized in that, The process of the multimodal acquisition and preprocessing module is as follows: S11. Acquire and decode the video stream to obtain the original video frame sequence with timestamps; S12. Perform frame rate standardization, duplicate frame removal, color space unification, size normalization, brightness normalization, and noise reduction on the original video frame sequence. S13. Perform frame extraction on the preprocessed video frame sequence and output the video keyframe sequence according to the preset frame rate. S14. Separate the audio signal from the video stream, extract features from the audio signal, and obtain an audio feature vector sequence; S15. Perform speech recognition and transcription on the audio signal to generate a text transcription sequence with timestamps; S16. Obtain the text transcription sequence or video stream subtitle file, perform character encoding unification, symbol cleaning, sentence segmentation and word segmentation on the text content, assign start and end timestamps to the word segmentation units, and output the text sequence with time tags. S17. Align the video keyframe sequence, audio feature vector sequence, and time-stamped text sequence on the time axis to obtain synchronized multimodal input.

3. The intelligent video summarizing system based on cross-modal alignment according to claim 2, characterized in that, The process of hierarchically aligning the coding modules across modalities is as follows: S21. Input the video keyframe sequence from the synchronized multimodal input into the visual feature encoder to extract frame-level video feature vectors; S22. Input the audio feature vector sequence in the synchronized multimodal input into the audio feature encoder to extract phoneme-level audio feature vectors; S23. Input the time-stamped text sequence from the synchronized multimodal input into the text feature encoder to extract word-level text feature vectors; S24. Map the frame-level video feature vector, phoneme-level audio feature vector, and word-level text feature vector to a shared semantic space of a unified dimension; S25. Form an initial multimodal feature representation in the shared semantic space.

4. The intelligent video summarizing system based on cross-modal alignment according to claim 3, characterized in that, The process of the dynamic weight allocation module is as follows: S41. Perform reliability detection on the video feature subset in the hierarchically aligned multimodal feature representation, calculate frame sharpness, edge sharpness and motion blur, and generate visual reliability index. S42. Perform reliability detection on the audio feature subset in the hierarchically aligned multimodal feature representation, calculate the signal-to-noise ratio, speech segment integrity and silence ratio, and generate speech reliability index. S43. Perform reliability detection on the text feature subset in the hierarchically aligned multimodal feature representation, calculate text integrity, transcription timestamp integrity and missing word rate, and generate text reliability index. S44. Input the visual reliability index, voice reliability index and text reliability index into the modality weight generator to generate three types of modality basic weights. The modal weight generator normalizes the visual reliability index, speech reliability index, and text reliability index to make their value ranges the same; and maps the normalized indexes to weight coefficients through linear transformation or nonlinear function mapping. S45. Apply the three types of modal base weights to the hierarchically aligned multimodal feature representation, weight the multimodal feature representation, and output a dynamically weighted multimodal feature representation.

5. The intelligent video summarizing system based on cross-modal alignment according to claim 4, characterized in that, The process of selecting the intent-conditional summary module is as follows: S51. Encode the prompts or query content entered by the user to generate a user intent vector; S52. Map the user intent vector to the same shared semantic space as the dynamically weighted multimodal feature representation to form an intent representation; S53. Perform stepwise similarity calculation on the time axis between the intent representation and the dynamically weighted multimodal feature representation to generate a temporal correlation score sequence; S54. Perform normalization processing on the time-series correlation score sequence to ensure that the scores are within a preset range. S55. Output the normalized time-series correlation score sequence.

6. The intelligent video summarizing system based on cross-modal alignment according to claim 5, characterized in that, The event boundary detection process is as follows: S61. Perform abrupt change detection on the time-series correlation score sequence and mark the time positions where the correlation score changes significantly as candidate boundary points; S62. Filter the candidate boundary points according to the minimum event duration threshold and the adjacent boundary distance constraint to generate an event boundary set; S63. Divide the temporal correlation score sequence into several event intervals according to the event boundary set; S64. Extract the corresponding video frame sequence, audio feature sequence and text feature sequence within each event interval to construct a candidate segment set; S65. Calculate the relevance score between the candidate fragment and the user intent representation, and generate a candidate fragment relevance vector; S66. Calculate the similarity between candidate segments and construct a candidate segment similarity matrix; S67. Based on the candidate segment relevance vector and similarity matrix, diversity constraints are applied to obtain a preliminary set of segments; The diversity constraint establishes a coverage term based on the relevance vector and a redundancy penalty term based on the similarity matrix to form a weighted combination selection objective function. Sub-modulus optimization is performed on the objective function to output a preliminary selection fragment set with high relevance and low redundancy. S68. Filter the initial fragment set according to the preset relevance threshold and the upper limit of the summary duration, and output the target fragment set.

7. The intelligent video summarization system based on cross-modal alignment according to claim 6, characterized in that, The process of the abstract generation module is as follows: S71. Perform inter-frame difference detection on each segment in the target segment set, select key frames based on the maximum visual difference and representative score, and generate a key frame set. S72. Arrange the keyframe set in chronological order to form a video summary frame sequence; S73. Extract the audio and text segments corresponding to the video summary frame sequence from the target segment set, and perform boundary cropping and temporal alignment to generate a candidate segment sequence; S74. The candidate segment sequence is spliced ​​together to maintain the consistency of the time order and output continuous video summary segments. S75. Perform keyword extraction and semantic compression on the text content in the candidate segment sequence to generate a set of key sentences; S76. Sort the set of key points according to the timestamps of the segments to form a timeline-style key point description; S77. Encapsulate the video summary frame sequence, continuous video summary segments, and timeline-style key points, and output the final summary result.

Images