A multi-modal artificial intelligence-based intelligent video caption automatic generation and synchronization method and system

Abstract

The application discloses a kind of intelligent video caption automatic generation and synchronization method and system based on multi-modal artificial intelligence.The method obtains video file or live stream and separates audio stream and video frame sequence, performs speech recognition on audio stream to obtain recognized text with initial timestamp, performs semantic boundary recognition and caption segment reorganization on recognized text, performs visual element detection and picture feature analysis on video frame sequence, and based on structured caption segment, picture importance distribution and video rhythm characteristics, the display time, display position and display style of caption segment are jointly optimized, and finally generate caption file, embedded caption video or real-time caption stream.The application can improve caption punctuation, reduce blocking key picture elements and improve caption readability in complex background.

Description

Technical Field

[0001] This invention relates to the fields of video processing, speech recognition, computer vision, and artificial intelligence, and in particular to a method and system for automatic generation and synchronization of intelligent video subtitles based on multimodal artificial intelligence. Background Technology

[0002] In scenarios such as short videos, live streaming, online education, and corporate promotion, subtitle creation typically involves multiple steps, including speech recognition, text proofreading, sentence segmentation, timeline adjustment, screen occlusion, and style design. Existing subtitle tools are mostly based on audio recognition and fixed templates, making it difficult to simultaneously handle semantic integrity, screen occlusion, readability, and real-time performance.

[0003] Common problems with existing solutions include: semantic fragmentation caused by punctuation based solely on pauses or fixed word counts; fixed display of subtitles at the bottom, which can easily obscure faces, logos, or key text; the need for manual adjustment of style parameters, resulting in unstable readability in complex backgrounds; numerous recognition errors in professional terminology, multilingual, and dialect scenarios; and difficulty in balancing low latency and subtitle quality in live streaming scenarios. Summary of the Invention

[0004] This invention provides a multimodal subtitle generation scheme that inputs audio recognition results, text semantic information, video image information, and user constraints into a joint optimization module to uniformly determine the content, time, position, and style of subtitles, thereby improving the quality of subtitle generation and reducing the workload of manual proofreading and typesetting.

[0005] In one embodiment, the system first performs speech recognition on the audio stream and corrects errors by combining a professional glossary, reference text, or user confirmation results; then it performs semantic boundary recognition on the recognized text to form structured subtitle segments; at the same time, it performs face, logo, on-screen text, and moving target detection on the video frames to generate the importance distribution of the screen; finally, it performs joint optimization based on semantic integrity, reading speed, video rhythm, avoidance constraints, and style readability.

[0006] Compared to subtitle solutions based solely on audio or fixed templates, this invention reduces the probability of subtitles obscuring key visual elements, improves readability in complex backgrounds, and provides a unified technical path for offline videos and live streaming without altering the user's creation process. The recognition model, visual detection model, and language model mentioned herein can all be replaced by similar models, and the specific model names do not constitute a limitation on the scope of protection. Attached Figure Description

[0007] Figure 1 This is a system architecture diagram of one embodiment of the present invention.

[0008] Figure 2This is a flowchart illustrating the subtitle generation process according to an embodiment of the present invention.

[0009] Figure 3 This is a flowchart of an intelligent alignment algorithm according to an embodiment of the present invention.

[0010] Figure 4 This is a process for generating a heatmap of image importance according to an embodiment of the present invention.

[0011] Figure 5 This is a schematic diagram illustrating subtitle avoidance and positioning according to an embodiment of the present invention.

[0012] Figure 6 This is a flowchart illustrating the style adaptation process of one embodiment of the present invention. Detailed Implementation

[0013] like Figure 1 As shown, the system comprises a user interaction layer, a core processing layer, and an output rendering layer. The user interaction layer receives video files, live streams, reference subtitle text, and user style constraints. The core processing layer includes modules for audio / video separation, speech recognition, semantic analysis, video analysis, intelligent alignment, and style adaptation. The output rendering layer outputs standard subtitle files, videos with embedded subtitles, or real-time subtitle streams.

[0014] like Figure 2 As shown, during offline video processing, the system reads the video file to be processed, extracts the audio, and unifies it into an audio format suitable for recognition; the speech recognition module generates an initial subtitle segment containing start time, end time, and text content; the semantic analysis module reorganizes the subtitle segment based on context, punctuation, pauses, and syntactic integrity; the video analysis module extracts keyframes, shot transitions, motion amplitude, and visual element positions; and the joint optimization module outputs the final subtitle timeline and style parameters.

[0015] The speech recognition module can load a professional vocabulary list corresponding to the video content domain and perform hot word enhancement on the candidate recognition results. For mixed Chinese and English or dialect segments, the system can first perform language or dialect recognition and then select the corresponding recognition configuration. If the user provides reference text, the system uses the reference text as the basis for proofreading and generates error correction suggestions; low-confidence suggestions only prompt the user for confirmation and do not automatically replace them.

[0016] like Figure 3 As shown, the intelligent alignment module identifies text, semantic boundaries, keyframes, motion amplitude, and reading speed constraints as inputs to the timeline optimization process. This process avoids splitting complete semantic units into discontinuous subtitles and ensures that the subtitle display duration meets the user's reading speed and the video's rhythm. For excessively short segments, the system can merge adjacent segments; for excessively long segments, the system can split and allocate adjacent time periods based on semantic boundaries.

[0017] like Figure 4 and Figure 5 As shown, the video analysis module divides the screen into multiple candidate regions and detects faces, logos, text, and moving targets. The system calculates the importance weight of each region based on visual element confidence, region location, motion amplitude, and user position constraints, and selects regions with low weights and meeting safety spacing requirements to display subtitles. When a moving target is detected, the system can predict its trajectory based on its historical location and dynamically adjust the subtitle position.

[0018] like Figure 6 As shown, the style adaptive module determines the subtitle color, stroke width, shadow intensity, and transparency based on the screen's brightness, contrast, hue, and texture complexity. For bright backgrounds, dark text and light strokes are preferred; for dark backgrounds, light text and dark strokes are preferred; for high-motion or fast-paced clips, animation complexity can be reduced to minimize visual interference.

[0019] In one software embodiment, the system provides audio and video uploading, subtitle generation, reference text proofreading, bilingual subtitles, punctuation removal, sentence merging, progress display, video frame extraction brightness analysis, ASS style recommendation, SRT / VTT / ASS multi-format download, and JSON processing report functions in the form of a web application. After receiving the uploaded file, the server transcodes the audio to a uniform sampling rate and mono format and calls the voice-to-subtitle interface to obtain sentence segmentation results. Simultaneously, it extracts representative video frames to calculate the average brightness, contrast, dark area ratio, and bright area ratio. Then, based on user options, it performs proofreading, translation, merging, formatting, and style recommendation, ultimately returning a subtitle file or processing report. This embodiment is one implementation of the present invention and does not limit the present invention to using only a specific cloud service or a specific model.

[0020] For live streaming, the system divides the audio stream into continuous segments and uses streaming recognition, while performing low-complexity or frame-by-frame analysis on the video frames. The system outputs subtitle segments incrementally and makes minor corrections to segments that have not yet been finalized as subsequent context arrives, in order to strike a balance between low latency and subtitle quality. Beneficial effects

[0021] This invention utilizes multimodal joint optimization to enable subtitle content, timing, location, and style to simultaneously adapt to audio, semantic, and visual information. Its beneficial effects include: reducing the likelihood of subtitles obscuring key visual elements; improving subtitle readability in complex backgrounds; enhancing the naturalness of sentence breaks and the continuity of the timeline; reducing manual proofreading work in scenarios involving technical terminology and reference text; and supporting both offline video and live streaming applications.

[0022] The above effects can be verified through indicators such as occlusion rate, manual proofreading time, consistency of subtitle phrasing, readability score, and end-to-end latency. Before submitting the application, if the applicant provides real test records, the relevant data can be used to supplement the effects of the example; if no real test records are provided, this document will not present unverified numerical values ​​as established test conclusions.

Claims

1. A method for automatic generation and synchronization of intelligent video subtitles based on multimodal artificial intelligence, characterized in that, include: Acquire the video file or live stream to be processed and separate it into audio stream and video frame sequence; Speech recognition is performed on the audio stream to obtain recognized text with an initial timestamp; Semantic boundary recognition and subtitle fragment reorganization are performed on the identified text to obtain structured subtitle fragments; visual element detection and image feature analysis are performed on the video frame sequence to obtain image importance distribution; based on the structured subtitle fragments, image importance distribution and video rhythm features, the display time, display position and display style of the subtitle fragments are jointly optimized; and subtitle files, embedded subtitle videos or real-time subtitle streams are generated according to the optimization results.

2. The method according to claim 1, characterized in that, The speech recognition includes: loading a professional vocabulary list corresponding to the video content domain; performing hot word enhancement and context correction on candidate words in the recognition results; and selecting the corresponding recognition configuration based on the language or dialect recognition results when multilingual or dialect speech segments are detected.

3. The method according to claim 1, characterized in that, The semantic boundary recognition includes: determining candidate punctuation breakpoints by combining punctuation prediction, pause information, syntactic integrity, and contextual semantic coherence; when candidate punctuation breakpoints cause subtitle segments to be too short or semantically incomplete, merging adjacent segments or reallocating display time.

4. The method according to claim 1, characterized in that, The visual element detection includes detecting faces, logos, text on the screen, and moving targets, and dividing the video screen into multiple candidate display areas; calculating the visual importance weight for each candidate display area, the weight being determined at least by the visual element confidence, area location, motion amplitude, and user position constraints.

5. The method according to claim 4, characterized in that, The joint optimization of the display position includes: excluding areas that overlap with high-weight visual elements; predicting the trajectory of moving targets; and selecting the target display area from candidate areas that meet the constraints of subtitle safety spacing, subtitle size, and non-overlapping multiple subtitle tracks.

6. The method according to claim 1, characterized in that, The joint optimization of display style includes: determining the subtitle color, stroke width, shadow intensity, and transparency based on screen brightness, contrast, hue, and texture complexity; and adjusting the subtitle animation intensity or canceling the animation based on video rhythm characteristics.

7. The method according to claim 1, characterized in that, It also includes correcting the identified text based on reference text or human confirmation results; the correction results retain the original identified text, suggested correction text, and human confirmation text, and are used to update user preference configurations.

8. The method according to claim 1, characterized in that, In the real-time live streaming processing mode, the audio stream is divided into continuous audio segments. The system performs streaming recognition, incremental sentence segmentation, low-complexity visual analysis, and incremental subtitle rendering on the continuous audio segments to reduce end-to-end latency.

9. A smart video subtitle automatic generation and synchronization system based on multimodal artificial intelligence, characterized in that, include: The system comprises an audio / video access module, a speech recognition module, a semantic analysis module, a video analysis module, a joint optimization module, and a rendering output module. The audio / video access module receives video files or live streams. The speech recognition module generates recognized text with an initial timestamp. The semantic analysis module generates structured subtitle segments. The video analysis module generates image importance distribution and image features. The joint optimization module determines the display time, position, and style of the subtitle segments. The rendering output module outputs subtitle files, embedded subtitle videos, or real-time subtitle streams.

10. The system according to claim 9, characterized in that, The system also includes a user interaction module, which is used to receive reference subtitle text, display error correction suggestions, configure subtitle style parameters, select whether to generate bilingual subtitles, and export subtitle files.

Images