Automatic movie and television video script extraction method based on multi-modal large model

Abstract

The invention relates to an automatic movie and television video script extraction method based on a multi-modal large model, and belongs to the field of artificial intelligence and video content analysis. Aiming at the problem that an existing automatic speech recognition tool cannot generate a structured script, the method comprises the following steps: firstly, carrying out multi-mode decomposition on an input video, extracting frames by adopting a scene self-adaptive strategy, and establishing sound and picture timestamp alignment mapping; then extracting features through a CLIP-ViT visual feature encoder and a Whisper audio feature encoder, and associating semantics by using a cross-modal attention mechanism; role identity recognition, scene type recognition and action character description are achieved, and finally a script file conforming to the standardization specification is generated and comprises a scene title, a time code mark and a special narrative mark. Compared with a mode of manually dictating marks and an automatic voice recognition tool, the method effectively improves the accuracy and efficiency of movie and television video script extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and video content analysis, and particularly relates to a method for automatically extracting scripts of film and television videos based on a multi-modal large model. BACKGROUND

[0002] Currently, short video dramas are emerging in an endless stream, and how to effectively extract and analyze the scripts of a large number of short video dramas has become a problem that practitioners in the film and television industry are concerned about. The existing extraction of scripts of film and television videos relies on the manual transcription and marking method, which is low in labeling efficiency and high in cost. The automatic speech recognition technology (ASR) can only convert the original film and television videos into corresponding text streams according to the audio information, and lacks the script structuring capability, such as scene segmentation, role labeling, action description and picture text analysis, etc. In addition, the automatic speech recognition technology usually produces recognition errors due to the sound quality of short video dramas, actor pronunciation, etc. The optical character recognition technology (OCR) can effectively extract video dialogue text, but cannot be applied to a large number of videos without subtitles, and still lacks the script structuring capability. SUMMARY

[0003] (I) Technical problem to be solved

[0004] The technical problem to be solved by the present application is how to provide a method for automatically extracting scripts of film and television videos based on a multi-modal large model, so as to solve the problem that the existing automatic speech recognition tools cannot generate structured scripts.

[0005] (II) Technical scheme

[0006] In order to solve the above technical problem, the present application provides a method for automatically extracting scripts of film and television videos based on a multi-modal large model, which comprises the following steps:

[0007] S1, multi-modal decomposition is performed on an input video to obtain an audio stream and a video frame sequence;

[0008] S2, a pre-trained multi-modal large model is used to jointly analyze the audio semantics and visual scene features;

[0009] S3, script elements in the video are identified, including: role dialogue, scene switching mark, action description text;

[0010] S4, the identified elements are reorganized according to the script format specification, and a structured script file is output.

[0011] (III) Beneficial effects

[0012] The application provides a multi-modal large model-based automatic extraction method for film and television video scripts. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 A flowchart of the multi-modal large model-based automatic extraction method for film and television video scripts. DETAILED DESCRIPTION

[0014] To make the purpose, content and advantages of the application clearer, the specific embodiments of the application are described in further detail below in combination with the drawings and examples.

[0015] The application designs a multi-modal large model-based automatic extraction method for film and television video scripts, which synchronously analyzes video audio and video picture content and realizes cross-modal alignment. Moreover, the method provides script element recognition capability, binds character identity according to voiceprint and face recognition results, and provides scene segmentation and character action description. Finally, the method converts unstructured information into a standardized script format.

[0016] To solve the problem that existing automatic speech recognition tools cannot generate structured scripts, the method first performs multi-modal decomposition on the input video, adopts a scene adaptive strategy to frame and establish audio-visual timestamp alignment mapping; then extracts features through a CLIP-ViT visual feature encoder and a Whisper audio feature encoder, and correlates semantics using a cross-modal attention mechanism; realizes character identity recognition, scene type recognition and action text description, and finally generates a script file conforming to standardized specifications, including scene title, time code mark and special narrative mark. Compared with the manual transcription mark and automatic speech recognition tool method, the application effectively improves the accuracy and efficiency of film and television video script extraction.

[0017] To achieve the above purpose, the application adopts the following technical solutions:

[0018] A multi-modal large model-based automatic extraction method for film and television video scripts, comprising the following steps:

[0019] S1, multi-modal decomposition is performed on the input video to obtain an audio stream and a video frame sequence;

[0020] S2, the pre-trained multi-modal large model is used to jointly analyze the audio semantics and visual scene features;

[0021] S3, script elements in the video are identified, including character dialogue, scene switching markers, and action description text;

[0022] S4, the identified elements are reorganized according to the script format specification, and a structured script file is output.

[0023] Further, the step S2 comprises:

[0024] The multi-modal large model adopts a cross-modal attention mechanism to realize deep fusion of audio and visual information: first, a mel-frequency spectrum audio feature of the S1 audio stream input voice is extracted based on Whisper through an audio feature encoder, capturing its expression information in the time-frequency domain; at the same time, high-level semantic features of the S1 video frame sequence image are extracted based on a CLIP-ViT neural network using a visual feature encoder; then, the features of the two modalities are input into a cross-attention layer in a Transformer structure, the cross-modal attention mechanism is used to model the correlation between audio and vision, information complementation and semantic alignment are realized, and the expression ability of multi-modal representation and the performance of downstream tasks are improved.

[0025] Further, the step S3 comprises:

[0026] Voiceprint clustering and face recognition technology are used to realize the binding and unified identification of character identity; scene boundaries in the video are accurately positioned through lens edge detection and semantic segmentation methods, and different space and context units are divided; and further, a visual Transformer (ViT) model is used to analyze the selected key frames, and corresponding action description text is generated, thereby realizing the extraction of script core elements driven by multi-modal information.

[0027] Further, the step S4 comprises:

[0028] The voiceprint, scene segmentation, action description, and other information output in steps S1-S3 are arranged into a standard format, and the script format output adopts the Fountain or FDX industry standard format, which includes clear scene titles, character names, and dialogue blocks, follows a three-part structure, and ensures that the content of each scene is clearly hierarchical, facilitating script writing, editing, and typesetting. At the same time, this format is compatible with mainstream script creation and production tools, ensuring seamless integration with industry workflows.

[0029] Embodiment 1:

[0030] AsFigure 1 As shown, a flowchart of the multi-modal large model-based automatic extraction method for film and television scripts provided by the embodiments of the present application is shown.

[0031] In step S1, the original video is first acquired, and the audio data and video frame sequence data of the original video are extracted.

[0032] In step S2, the data decomposed in step S1 is input into the pre-trained multi-modal large model, wherein:

[0033] The visual feature encoder adopts the CLIP-ViT architecture, and outputs a 4096-dimensional visual feature vector; the audio feature encoder adopts the Whisper architecture, and extracts a 128-dimensional Mel spectrum audio feature. The features are fused through the cross-modal attention layer, and the attention weight calculation satisfies the formula:

[0034]

[0035] wherein is the visual feature, is the audio feature, is the feature dimension, and by default, it is 4096, and the audio feature is supplemented to 4096.

[0036] In step S3, the YOLOv9 is used to detect the shot switching screen in the video to realize the labeling of the scene switching label information, and the 5 frames of pictures before and after the switching shot are taken as the representative frames of the current scene. For identity recognition in multi-role dialogue, the method combines voiceprint feature extraction and face recognition to realize cross-modal identity binding. The method first performs frame processing on the audio stream, extracts a 256-dimensional voiceprint embedding vector using the Resemblyzer deep learning sound analysis tool; in the visual channel, the system runs the Deepface model in the key frame for face recognition, extracts a face embedding vector, and tracks the face trajectory through an overlapping area matching algorithm, and sets the matching threshold to IoU≥0.7. On this basis, the system establishes a cross-modal binding relationship when the two conditions of “the face appearance time interval containing the speech segment” and “the Euclidean distance between the voiceprint embedding and the face embedding is less than 0.35” are met.

[0037] For the recognition of off-screen sound, the method comprehensively analyzes the multi-modal features of the audio and visual channels: in the audio dimension, the stable sound source signal with a sound source direction angle θ greater than 45° and a volume fluctuation ΔdB less than 2 per unit time is detected; in the visual dimension, the LipNet model is used to analyze the motion of the lip region of the detectable person in the picture, and if no synchronous lip movement is detected, it is considered as non-speech.

[0038] For the identification of nonlinear narrative fragments such as dreams and memories, the system uses a statistical analysis model of HSV color space changes for detection. When the mutation characteristics of the saturation drop rate ΔS greater than or equal to 30% and the hue change ΔH greater than or equal to 15° appear in the continuous frames of the picture, the system automatically judges that the region is a dream or a non-real sequence.

[0039] To realize the structured script output of non-dialogue action scenes, the method automatically identifies continuous action fragments from the video through shot cut detection and time series clustering algorithm, and extracts representative key frames based on motion intensity changes and frame visual information change rate; then, a pre-trained ViT model is used to perform high-dimensional semantic encoding on the key frame images to identify core elements such as main characters, action categories, action directions, and interactive objects in the picture.

[0040] In step S4, the identified elements are converted into a structured script file using a pre-trained natural speech large model. Scene titles are generated according to the same rule, with the format "[scene type. location - time period]". Character dialogues follow the three-part specification.

[0041] Embodiment 2:

[0042] A multi-modal large model-based automatic extraction method for film and television video scripts, comprising the following steps:

[0043] S1, multi-modal decomposition of the input video is performed to obtain audio stream and visual frame sequence;

[0044] S2, pre-trained multi-modal large model is used to jointly analyze audio semantics and visual scene features;

[0045] S3, script elements in the video are identified, including character dialogues, scene transition markers, and action description texts;

[0046] S4, the identified elements are reorganized according to the script format specification, and a structured script file is output.

[0047] The shot cut detector in step S2 is implemented based on the YOLOv9 algorithm, with a detection response time ≤40ms.

[0048] The multi-modal large model in step S2 includes: the visual feature encoder uses the CLIP-ViT architecture, outputting a 4096-dimensional feature vector; the audio feature encoder uses the Whisper architecture, extracting a 128-dimensional Mel spectrum feature.

[0049] The character identity recognition in step S3 includes: the audio stream is processed by frame and the voiceprint feature is extracted, the video key frame is recognized by face, and the voiceprint feature and face recognition result information are matched for correlation degree.

[0050] The action description text in the step S3: based on shot switching detection and timing clustering to automatically extract key frames, and using a pre-trained ViT model to obtain high-dimensional semantic encoding, to identify main characters, action categories, directions, and interactive objects, and other elements.

[0051] The video script dream scene identification in the step S3 is realized based on HSV color space mutation detection.

[0052] An electronic device comprising a memory, a processor and a computer program stored on the memory, the processor implementing the steps of the method when executing the program.

[0053] Via the above technical solution, compared with the prior art, the present application has the following beneficial effects:

[0054] The present application synchronously analyzes video pictures and audio semantics through cross-modal attention mechanism, effectively solves the problem of mutual fragmentation of character dialogue and action scenes in traditional methods, can accurately associate character behavior and corresponding lines, and improves the overall accuracy of script element recognition and matching. At the same time, the system supports automatic conversion of film and television language into standard script format, generates three-section structure output compatible with screenwriting software, covers scene title, character name and dialogue block, and significantly reduces manual sorting and typesetting cost. It also has good recognition and understanding ability for complex narrative structures such as voice-over, dream and flashback, and can generate description text conforming to script grammar specifications for action scenes without lines, improving the efficiency of script structuring while providing efficient and automated script generation support for film and television production.

[0055] The above is only the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A multi-modal large model-based automatic extraction method for film and television scripts, characterized in that, The method comprises the following steps: S1, multi-modal decomposition is performed on the input video to obtain an audio stream and a video frame sequence; S2, the audio semantics and visual scene features are jointly analyzed by using a pre-trained multi-modal large model; S3, script elements in the video are identified, including: role dialogue, scene switching mark, and action description text; S4, the identified elements are reorganized according to the script format specification, and a structured script file is output.

2. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 1, wherein, In S2, the multi-modal large model adopts a cross-modal attention mechanism to realize deep fusion of audio and visual information.

3. The multi-modal large model-based film and television script automatic extraction method of claim 2, wherein, In S2, specifically, first, a mel-frequency spectrum audio feature of the input voice of the audio stream is extracted based on Whisper by using an audio feature encoder to capture expression information in the time-frequency domain; meanwhile, high-level semantic features of images of the video frame sequence are extracted based on a CLIP-ViT neural network by using a visual feature encoder; then, the features of the two modalities are input into a cross-attention layer in a Transformer structure to model the association between the audio and the visual through a cross-modal attention mechanism, so as to realize information complementation and semantic alignment, thereby improving the expression ability of the multi-modal representation and the performance of the downstream task.

4. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 3, characterized in that, The visual feature encoder adopts a CLIP-ViT architecture to output a 4096-dimensional visual feature vector; The audio feature encoder adopts a Whisper architecture to extract a 128-dimensional mel-frequency spectrum audio feature; the features are fused through the cross-modal attention layer, and the attention weight calculation satisfies the formula: wherein is a visual feature, is an audio feature, is a feature dimension, is 4096, and the audio feature makes up the 4096.

5. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 1, wherein, S3 includes: the role identity is bound and uniformly identified based on voiceprint clustering and face recognition technology; the scene boundaries in the video are accurately positioned by using a shot edge detection and semantic segmentation method, and different space and context units are divided; and further, the visual Transformer model is used to analyze the selected key frames to generate corresponding action description texts, so as to realize extraction of script core elements under the driving of multi-modal information.

6. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 5, characterized in that, The YOLOv9 is used to detect the shot switching pictures in the video to realize labeling of the scene switching mark information, and the pictures of 5 frames before and after the switching shot are taken as the representative frames of the current scene; For identity recognition in multi-role dialogue, the cross-modal identity binding is realized by combining voiceprint feature extraction and face recognition; first, the audio stream is frame-processed, and a 256-dimensional voiceprint embedding vector is extracted by using a Resemblyzer deep learning sound analysis tool; the Deepface model is run in the key frames on the visual channel to perform face recognition and extract a face embedding vector, and the face trajectory is tracked by using an overlapping area matching algorithm, and the matching threshold is set to IoU>=0.7; on this basis, when the two conditions of "the face appearance time interval containing the voice segment" and "the Euclidean distance between the voiceprint embedding and the face embedding being less than 0.35" are met, the cross-modal binding relationship is established.

7. The multi-modal large model-based film and television script automatic extraction method of claim 6, wherein, For the recognition of off-screen voice, the multi-modal features of audio and visual channels are comprehensively analyzed: in the audio dimension, stable sound source signals with a sound source direction angle θ greater than 45° and a volume fluctuation ΔdB less than 2 per unit time are detected; in the visual dimension, the LipNet model is used to analyze the motion of the lip region of the detectable person in the picture, and if no synchronous lip movement is detected, it is considered as non-spoken language.

8. The multi-modal large model-based film and television script automatic extraction method of claim 6, wherein, For the recognition of dream and nonlinear narrative segments, a statistical analysis model of HSV color space variation is used for detection; when the saturation drop rate ΔS is greater than or equal to 30% and the hue change ΔH is greater than or equal to 15° in the continuous frames of the picture, it is automatically determined that the area is a dream or a non-real sequence.

9. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 6, characterized in that, To realize the structured script output of the action scene without dialogue, the continuous action segments are automatically identified from the video through lens switching detection and time series clustering algorithm, and the representative key frames are extracted based on the motion intensity change and the frame visual information change rate; Then, the pre-trained ViT model is used to perform high-dimensional semantic encoding on the key frame images to identify the main characters, action categories, action directions, and interactive objects in the picture.

10. The multi-modal large model-based automatic extraction method for film and television scripts according to claim 1, wherein, The S4 includes: organizing the voiceprint, scene segmentation, and action description information output in steps S1-S3 into a standard format, and the output script format adopts the Fountain or FDX industry standard format, which contains clear scene titles, character names, and dialogue blocks, follows a three-part structure, and ensures that the content of each scene is clearly hierarchical, facilitating script writing, editing, and typesetting; at the same time, this format is compatible with mainstream script creation and production tools, ensuring seamless integration with industry workflows.

Images