A method and device for extracting a script of a movie or TV series, a storage medium and a computer device

Abstract

The film and television script extraction method and device, the storage medium and the computer device provided by the application, after the audio and video file of the film and television is split into a video file and an audio file, the video file is subjected to feature recognition to obtain a subtitle text, a speaker face information and a video understanding text; and the audio file is subjected to speech understanding to obtain a speech transcription text and a speech understanding text; wherein the speech transcription text can be corrected into a standard transcription text with higher accuracy through the subtitle text. Therefore, based on the speaker face information, the standard transcription text and the dialogue segment of each speaker in the audio and video file are aligned, the speaker information with accurate segmentation and coherent semantics can be obtained, then the script information is constructed by combining the character profile text generated by the profile analysis of the speaker based on the video, the speech understanding text and the speaker information, the script information can cover the related feature description of the character on the basis of the dialogue content, thereby enriching the content and depth of the script.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and computer equipment for extracting film and television scripts. Background Technology

[0002] With the rapid development of the internet, online films and television dramas are widely disseminated globally, and cross-cultural exchanges are becoming increasingly frequent. Simultaneously, the booming development of self-media has provided a broad platform for the secondary creation and dissemination of film and television content. Against this backdrop, the need for script extraction from films and television dramas is becoming increasingly urgent. Whether translating films and television dramas into other languages ​​to expand into overseas markets or creating secondary works based on the original scripts to enrich cultural products, accurate and efficient script extraction is a crucial prerequisite.

[0003] Currently, script extraction for film and television dramas mainly relies on audio transcription using ASR or video subtitle extraction using OCR. Each extracted line is then labeled with a speaker tag to complete the script extraction. However, ASR may lead to transcription errors due to unclear speech or accent differences, while OCR can be affected by subtitle font and background interference, resulting in poor accuracy in script extraction. Furthermore, speaker segmentation is often ineffective, making it difficult to accurately distinguish between different speakers' lines, leading to a lack of semantic coherence in the extracted results and an inability to fully present the logical flow of the plot. In summary, existing script extraction methods for film and television dramas suffer from poor accuracy and speaker segmentation, resulting in a lack of semantic coherence and failing to meet the needs of extracting long, continuous scripts. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the above-mentioned technical defects, in particular the technical defects of existing script extraction methods, which have poor accuracy in script extraction and poor speaker segmentation, resulting in a lack of semantic coherence in the extraction results.

[0005] This application provides a method for extracting film and television scripts, the method comprising:

[0006] The audio and video files of the film and television drama are obtained, and the audio and video files are split into video files and audio files. In addition, the human voice track in the audio file is separated.

[0007] The video file is subjected to feature recognition to obtain subtitle text carrying timestamp information and speaker facial information. Based on the speaker facial information, the speaker's state is understood to generate video understanding text.

[0008] The audio file is subjected to speech understanding to obtain speech transcribed text and speech understood text carrying timestamp information. The speech transcribed text is then corrected according to the subtitle text to obtain standard transcribed text.

[0009] Based on the human voice track and the speaker's facial information, the audio and video file is segmented into dialogue segments to obtain a dialogue segment sequence carrying timestamp information. Based on the timestamp information, the standard transcribed text and the dialogue segment sequence are aligned to obtain speaker information.

[0010] Based on the video-understood text, the speech-understood text, and the speaker information, a speaker profiling analysis is performed to generate a character profile text. Then, based on the speaker information and the character profile text, the script information of the film or television drama is generated.

[0011] Optionally, the step of performing feature recognition on the video file to obtain subtitle text carrying timestamp information and speaker facial information includes:

[0012] The video file is subjected to scene detection to obtain the detection results, and video semantic segments are generated based on the detection results;

[0013] The video semantic segment is subjected to character recognition to obtain subtitle text carrying timestamp information, and the video semantic segment is subjected to facial lip-sync recognition to obtain speaker facial information carrying timestamp information.

[0014] Optionally, the step of performing speech understanding on the audio file to obtain speech-transcribed text and speech-understood text carrying timestamp information includes:

[0015] The audio file is subjected to silence detection to obtain the detection result, and a speech activity segment is generated based on the detection result;

[0016] The speech activity segment is converted into speech-to-text to obtain speech-transcribed text, and the speech activity segment is subjected to emotion understanding to generate speech-understood text.

[0017] Optionally, the step of segmenting the audio / video file into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments carrying timestamp information includes:

[0018] Based on the speaker's facial information, determine the audio and video segments from the audio and video file where the speaker's facial information is not missing, and determine the audio segments from the audio file where the speaker's facial information is missing based on the speaker's voice track and the audio and video segments;

[0019] Based on the human voice track and the speaker's facial information, a multimodal segmentation model is used to segment the audio and video segments into speakers, resulting in a first dialogue segment carrying timestamp information.

[0020] Based on the human voice track, a speech segmentation model is used to segment the audio segment into speakers to obtain a second dialogue segment carrying timestamp information.

[0021] The first and second dialogue segments are merged based on the timestamp information to generate a dialogue segment sequence.

[0022] Optionally, aligning the standard transcribed text and the dialogue fragment sequence based on the timestamp information to obtain speaker information includes:

[0023] The standard transcribed text is split into minimum semantic units to obtain a semantic unit sequence, and the dialogue fragment sequence is split into minimum dialogue units to obtain a dialogue unit sequence.

[0024] The semantic unit sequence and the dialogue unit sequence are aligned one by one according to the timestamp information to generate speaker information.

[0025] Optionally, the step of performing speaker profiling analysis based on the video-understood text, the speech-understood text, and the speaker information to generate a person profile text includes:

[0026] Determine the character profile model;

[0027] The video understanding text, the speech understanding text, and the speaker information are input into the character profiling model so that the character profiling model can perform profile analysis on the speaker and output the character profile text.

[0028] Optionally, generating the script information of the film / television drama based on the speaker information and the character profiling text includes:

[0029] Based on the speaker information, determine the speaker tag and timestamp of each smallest semantic unit in the standard transcribed text, and determine the profile content corresponding to each timestamp from the character profile text;

[0030] The speaker tags, smallest semantic units, and profile content corresponding to each timestamp are sequentially merged in a structured manner to generate the script information of the film and television drama.

[0031] This application also provides a device for extracting film and television scripts, including:

[0032] The file splitting module is used to obtain audio and video files of movies and TV series, split the audio and video files into video files and audio files, and separate the human voice track from the audio file.

[0033] The video recognition module is used to perform feature recognition on the video file, obtain subtitle text carrying timestamp information and speaker facial information, and perform state understanding of the speaker based on the speaker facial information to generate video understanding text;

[0034] An audio understanding module is used to perform speech understanding on the audio file to obtain speech transcribed text and speech understood text carrying timestamp information, and to perform text correction on the speech transcribed text based on the subtitle text to obtain standard transcribed text;

[0035] The information alignment module is used to segment the audio and video file into dialogue segments based on the human voice track and the speaker's facial information to obtain a dialogue segment sequence carrying timestamp information, and to align the standard transcribed text and the dialogue segment sequence based on the timestamp information to obtain speaker information;

[0036] The script generation module is used to perform profiling analysis on the speaker based on the video understanding text, the speech understanding text, and the speaker information, generate character profile text, and generate script information for the film and television drama based on the speaker information and the character profile text.

[0037] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the film and television script extraction method as described in any of the above embodiments.

[0038] This application also provides a computer device, including: one or more processors, and memory;

[0039] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the film and television script extraction method as described in any of the above embodiments.

[0040] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0041] The method, apparatus, storage medium, and computer equipment for extracting film and television scripts provided in this application can, after obtaining the audio and video files of a film or television drama, split the audio and video files into video files and audio files, and separate the human voice tracks in the audio files, providing clear and independent data sources for subsequent feature recognition and understanding of the video and audio in different dimensions. Among them, the subtitle text and speaker facial information obtained by feature recognition of the video file can achieve precise correspondence between text, time, and speaker, ensuring the semantic coherence of the dialogue. In addition, the video understanding text generated by understanding the speaker's state based on the speaker's facial information can further enrich the semantic content. The speech transcription text and speech understanding text obtained by speech understanding of the audio file can more comprehensively capture the semantic information in the speech. Subsequently, the speech transcription text can be corrected based on the subtitle text to obtain the standard transcription text, thereby improving the accuracy of text extraction. After obtaining the relevant feature recognition results of the audio and video files, this application can segment the audio and video files into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments with timestamp information, thereby reducing the amount of subsequent data processing. Then, based on the timestamp information, the standard transcribed text and dialogue segments can be aligned to accurately associate each line of dialogue with the corresponding speaker, resulting in accurately segmented and semantically coherent speaker information. Finally, based on video understanding text, speech understanding text, and speaker information, speaker profiling analysis can be performed to deeply explore the speaker's personality, emotions, behavioral characteristics, and other information, thereby generating character profile text. Based on this, script information for film and television dramas can be generated according to the speaker information and character profile text, which can include relevant character feature descriptions in addition to dialogue content, thereby enriching the content and depth of the script and providing more comprehensive and detailed information for the secondary creation and translation of film and television dramas. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a method for extracting film and television scripts, provided as an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a dialogue segmentation process provided in an embodiment of this application;

[0045] Figure 3A schematic flowchart of a film and television script extraction device provided in this application embodiment;

[0046] Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0048] Currently, script extraction for film and television dramas mainly relies on audio transcription using ASR or video subtitle extraction using OCR. Each extracted line is then labeled with a speaker tag to complete the script extraction. However, ASR may lead to transcription errors due to unclear speech or accent differences, while OCR can be affected by subtitle font and background interference, resulting in poor accuracy in script extraction. Furthermore, speaker segmentation is often ineffective, making it difficult to accurately distinguish between different speakers' lines, leading to a lack of semantic coherence in the extracted results and an inability to fully present the logical flow of the plot. In summary, existing script extraction methods for film and television dramas suffer from poor accuracy and speaker segmentation, resulting in a lack of semantic coherence and failing to meet the needs of extracting long, continuous scripts.

[0049] Based on this, this application proposes the following technical solution, as detailed below:

[0050] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides a flowchart illustrating a method for extracting film and television scripts, which includes the following specific steps:

[0051] S110: Obtain the audio and video files of the TV series, split the audio and video files into video files and audio files, and separate the human voice track from the audio files.

[0052] In this step, after obtaining the audio and video files of the film and television drama, the computer device can split the audio and video files into video files and audio files, and separate the human voice track in the audio file, providing a clear and independent data source for subsequent feature recognition and understanding of the video and audio in different dimensions.

[0053] Specifically, when splitting audio and video files, computer devices can parse the multimedia format of the audio and video files, identify the video encoding stream and audio encoding stream, and then demultiplex the format based on multimedia parsing algorithms, thereby splitting the audio and video files into independent video files and audio files, so that subsequent parallel and accurate video picture feature analysis and audio speech feature analysis can be carried out.

[0054] Furthermore, after splitting the audio file, the computer device can use a sound source separation algorithm or a speech separation model to perform acoustic signal separation processing on the audio file, removing the background audio track containing environmental noise, background music and other non-speech components, and extracting the human voice track containing the dialogue content. This allows subsequent speech recognition and processing to more accurately focus on the human voice content, thereby improving the accuracy of speech transcription.

[0055] S120: Perform feature recognition on the video file to obtain subtitle text with timestamp information and speaker facial information, and perform state understanding on the speaker based on the speaker facial information to generate video understanding text.

[0056] In this step, after the video file is split in step S110, the computer device can perform feature recognition on the video file to obtain subtitle text with timestamp information and speaker facial information, so as to achieve accurate correspondence between text, time and speaker, and ensure the semantic coherence of the dialogue. In addition, the video understanding text generated by understanding the speaker's state based on the speaker's facial information can further enrich the semantic content of the video dimension.

[0057] Among them, subtitle text refers to text information presented in the form of text in the video, mainly used to help viewers better understand the dialogue, narration and other information in the video; speaker facial information refers to the visual feature data related to the speaker in the video, including face position, facial expression, and mouth movement features, mainly used to determine who is speaking at the current moment; and video understanding text refers to the text content generated after a comprehensive analysis and understanding of the speaker's behavioral intentions, emotional changes and other states, mainly used to express the speaker's speaking state at the current moment.

[0058] Specifically, the computer device can perform multi-level feature recognition processing on video files. For example, it can acquire consecutive frames in a video file through image frame decoding technology, then extract the subtitles marked on the screen and their corresponding timestamps to generate subtitle text. Simultaneously, the computer device can also perform facial recognition and lip-syncing on the characters in the screen based on optical flow analysis and object detection algorithms to determine the speaker's facial information for each time period. Therefore, through subtitle text and speaker facial information, this application can achieve a precise correspondence between text, time, and speaker, ensuring the continuity and logical consistency of the dialogue semantics along the timeline and avoiding semantic fragmentation caused by speaker recognition errors. Furthermore, the computer device can also perform facial expression analysis, emotion recognition, and posture judgment on the speaker based on the speaker's facial information, combining scene changes and background context to generate higher-level video understanding text, thereby constructing a semantically richer and emotionally nuanced descriptive text for the speaker at the video dimension.

[0059] S130: Perform speech understanding on the audio file to obtain speech transcribed text and speech understanding text carrying timestamp information, and perform text correction on the speech transcribed text based on the subtitle text to obtain standard transcribed text.

[0060] In this step, after the audio file is split in step S110, the computer device can perform speech understanding on the audio file to obtain speech transcribed text and speech understanding text carrying timestamp information, thereby capturing the semantic information in the speech more comprehensively. Subsequently, the computer device can perform text correction on the speech transcribed text based on the subtitle text to obtain standard transcribed text, thereby improving the accuracy of text extraction.

[0061] Among them, speech-transcribed text refers to the text data obtained after transcribing the speech content in the audio signal into text. It is the basic result of converting audio content into readable text. Speech-understanding text refers to the text content generated after performing operations such as semantic analysis, intonation recognition, emotion recognition and context understanding on the audio content. It is mainly used to express the implicit information such as the speaker's tone, emotion and intention at the current moment.

[0062] Specifically, computer devices can perform multi-level speech understanding processing on audio files. For example, by using speech recognition models to perform acoustic and language modeling analysis on audio files, continuous speech waveforms can be converted into readable speech-transcribed text, and precise timestamps can be added to each sentence or word to achieve accurate time-tracking of the text content. Building on this, computer devices can also combine semantic understanding models to perform intonation feature analysis and contextual semantic reasoning, gaining a deeper understanding of the audio content and generating speech-understood text to more comprehensively capture the semantic features and emotional expressions in speech.

[0063] Understandably, in actual audio-visual processing of films and television dramas, the extracted speech-transcribed text and subtitle text often differ, and both can lead to recognition errors due to noise interference, accent differences, font obstruction, missing subtitles, etc. Therefore, relying solely on either text will result in semantic deviations or errors in proper noun recognition. To improve the overall accuracy of text recognition, computer equipment can use the subtitle text extracted from the video file as a reference to perform text correction on the speech-transcribed text. That is, through the semantic reasoning and contextual understanding capabilities of a large language model, the computer equipment can automatically correct erroneous words or sentence structures in the speech-transcribed text based on semantic consistency, syntactic logic, and background knowledge during the comparison of the two types of text. In particular, it can effectively maintain the consistency and correctness of proper nouns such as names of people, places, and organizations that frequently appear in films and television dramas, thereby generating standard transcribed text.

[0064] S140: Based on the human voice track and the speaker's facial information, the audio and video files are segmented into dialogue segments to obtain a dialogue segment sequence carrying timestamp information. The standard transcribed text and the dialogue segment sequence are then aligned based on the timestamp information to obtain the speaker information.

[0065] In this step, after processing the audio and video files through steps S120 and S130, the computer device can segment the audio and video files into dialogue segments based on the processed human voice track and speaker's facial information, obtaining a dialogue segment sequence carrying timestamp information to reduce the amount of subsequent data processing. Then, based on the timestamp information, the standard transcribed text and the dialogue segment sequence can be aligned to accurately associate each line of dialogue with the corresponding speaker, forming speaker information that is accurately segmented and semantically coherent.

[0066] The dialogue segment sequence refers to the sequence of segments in an audio or video file where characters speak, usually corresponding to the semantically complete dialogue content of each character in a film or television drama; while speaker information can be used to determine the text content of each line in the dialogue segment sequence and its corresponding speaker, which realizes the accurate association between the speech content and the visual character.

[0067] Specifically, computer equipment can segment a sequence of dialogue fragments from a complete audio-visual file by analyzing the start and end positions of speech in the voice track and the speaker's speaking state in the speaker information. This reduces the amount of data processing required for subsequent semantic recognition and sentiment analysis at the source, significantly improving data processing efficiency. Subsequently, based on timestamp information, the computer equipment can automatically align the previously generated standard transcribed text with the dialogue fragment sequence, achieving high-precision matching between audio-visual content, text content, and the speaker. This accurately associates each line of dialogue with its corresponding person, generating precisely segmented and semantically coherent speaker information.

[0068] S150: Based on video understanding text, speech understanding text and speaker information, perform speaker profiling analysis, generate character profile text, and generate script information for film and television dramas based on speaker information and character profile text.

[0069] In this step, after the speaker information is generated in step S140, the computer device can perform a profile analysis of the speaker based on the video understanding text, the speech understanding text, and the speaker information, and deeply explore information such as the speaker's personality, emotions, and behavioral characteristics, thereby generating a character profile text. Therefore, generating script information for film and television dramas based on speaker information and character profile text can cover the relevant characteristic descriptions of the characters on the basis of including the dialogue content, thereby enriching the content and depth of the script.

[0070] Specifically, based on speaker information, computer equipment can combine visual semantic features and behavioral characteristics from video-understanding text with tone and emotional intensity and language expression from speech-understanding text to perform deep semantic reasoning and emotion modeling. This allows for in-depth analysis of the speaker's personality traits, emotional state, psychological activities, and behavioral characteristics, thereby generating a character profile text with rich semantic layers. This character profile text can reflect the speaker's emotional expression and language style in a specific scenario.

[0071] Therefore, the script information of film and television dramas constructed by computer equipment based on speaker information and character profiling text can automatically supplement the original dialogue content with the character's tone, emotions, actions and psychological descriptions. This makes the generated script information not only accurately reproduce the dialogue content, but also present the character's inner qualities and emotional depth.

[0072] In the above embodiments, after obtaining the audio and video files of the film and television drama, the audio and video files can be split into video files and audio files, and the human voice track in the audio file can be separated, providing a clear and independent data source for subsequent feature recognition and understanding of the video and audio in different dimensions. Among them, the subtitle text with timestamp information and the speaker's facial information obtained by feature recognition of the video file can achieve a precise correspondence between text, time and speaker, ensuring the semantic coherence of the dialogue. In addition, the video understanding text generated by understanding the speaker's state based on the speaker's facial information can further enrich the semantic content. The speech transcription text with timestamp information and the speech understanding text obtained by speech understanding of the audio file can more comprehensively capture the semantic information in the speech. Subsequently, the speech transcription text can be corrected based on the subtitle text to obtain the standard transcription text, thereby improving the accuracy of text extraction. After obtaining the relevant feature recognition results of the audio and video files, this application can segment the audio and video files into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments with timestamp information, thereby reducing the amount of subsequent data processing. Then, based on the timestamp information, the standard transcribed text and dialogue segments can be aligned to accurately associate each line of dialogue with the corresponding speaker, resulting in accurately segmented and semantically coherent speaker information. Finally, based on video understanding text, speech understanding text, and speaker information, speaker profiling analysis can be performed to deeply explore the speaker's personality, emotions, behavioral characteristics, and other information, thereby generating character profile text. Based on this, script information for film and television dramas can be generated according to the speaker information and character profile text, which can include relevant character feature descriptions in addition to dialogue content, thereby enriching the content and depth of the script and providing more comprehensive and detailed information for the secondary creation and translation of film and television dramas.

[0073] In one embodiment, the process of performing feature recognition on the video file in step S120 to obtain subtitle text carrying timestamp information and speaker facial information may include:

[0074] S121: Perform scene detection on the video file, obtain the detection results, and generate video semantic segments based on the detection results.

[0075] S122: Perform character recognition on the video semantic segment to obtain subtitle text carrying timestamp information, and perform facial lip-sync recognition on the video semantic segment to obtain speaker facial information carrying timestamp information.

[0076] In this embodiment, after the video file is split, the computer device can perform scene detection on the video file, obtain the detection result, and generate video semantic segments based on the detection result. Then, it can perform character recognition on the video semantic segments to obtain subtitle text carrying timestamp information, and perform facial lip-syncing recognition on the video semantic segments to obtain speaker facial information carrying timestamp information.

[0077] Specifically, when performing scene segmentation detection on video files, computer equipment can identify different shots and scene transition points in the video through video frame change analysis, scene feature extraction, and shot transition judgment algorithms, thereby generating detailed scene segmentation detection results. Based on these scene segmentation detection results, computer equipment can divide the video content into structured video semantic segments, where each sub-segment corresponds to a relatively continuous visual scene and character action sequence within the shot, thus forming a semantically coherent processing unit.

[0078] Furthermore, computer equipment can perform multimodal feature recognition processing on video semantic segments. On the one hand, it can use optical character recognition technology to recognize the characters in the built-in subtitles of the segment, extract their text content, and attach a precise timestamp to generate subtitle text with timestamp information. On the other hand, it can use face detection and lip-syncing algorithms to identify the facial features of the speaker in the segment, including face position, mouth movement, and expression changes, and attach timestamp information to generate speaker facial information with timestamp information. This enables accurate matching of dialogue text content with the speaker, thereby achieving simultaneous extraction of key semantic information in both textual and visual dimensions.

[0079] Understandably, when computer devices perform feature recognition on video files, the recognition results may include missing subtitle text and / or missing speaker facial information. For example, in some segments of a video file, subtitles may not be embedded, or the subtitles may be obscured, have blurry fonts, or be subject to complex background interference, causing characters to be unrecognizable or misrecognized. In this case, the recognition result for the subtitle text will be determined as missing. Furthermore, in some segments, the speaker may not be in the frame, or the speaker's face may be obscured, causing the face detection algorithm to fail to identify a valid facial region. In this case, the recognition result for the speaker's facial information will be determined as missing.

[0080] In one embodiment, the process of performing speech understanding on the audio file in step S130 to obtain speech-transcribed text and speech-understood text carrying timestamp information may include:

[0081] S131: Perform silence detection on the audio file, obtain the detection results, and generate speech activity segments based on the detection results.

[0082] S132: Perform speech-to-text conversion on the speech activity segments to obtain speech-transcribed text, and perform emotion understanding on the speech activity segments to generate speech-understood text.

[0083] In this embodiment, after the audio file is split, the computer device can perform silence detection on the audio file, obtain the detection result, and generate speech activity segments based on the detection result. Then, the speech activity segments can be converted into speech-to-text to obtain speech-transcribed text, and the speech activity segments can be subjected to emotion understanding to generate speech-understood text.

[0084] Specifically, computer devices can perform VAD (Voice Activity Detection) operations on the segmented audio files. By analyzing the energy level, spectral characteristics, and short-term amplitude changes of the audio signal, they can identify silent intervals in the audio and generate detailed silence detection results. Based on these silence detection results, the computer devices can segment continuous audio signals into structured speech activity segments, where each sub-segment corresponds to a time period in which the speaker's voice actually exists. This effectively filters out parts without speech content, reducing the amount of data and computational burden in subsequent processing.

[0085] Furthermore, computer devices can perform speech-to-text conversion on speech segments, using high-precision speech recognition models to transcribe speech signals into readable text, generating speech-transcribed text with timestamp information. This achieves a precise correspondence between speech content and the timeline, providing a reliable textual foundation for subsequent text analysis and dialogue comprehension. Simultaneously, computer devices can also employ speech understanding models to perform emotion analysis on speech segments, combining acoustic features, intonation variations, volume levels, and rhythmic patterns to generate speech-understood text, thereby revealing the speaker's emotional state and semantic tendencies in different lines or segments. The speech understanding model used here can be an open-source model such as Step-audio; no restrictions are imposed here.

[0086] In one embodiment, such as Figure 2 As shown, Figure 2 A flowchart illustrating a dialogue segmentation process provided in an embodiment of this application; Figure 2 In step S140, the process of segmenting the audio / video file into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments carrying timestamp information may include:

[0087] S141: Determine audio and video segments from audio and video files where the speaker's facial information is not missing, based on the speaker's facial information; and determine audio segments from audio files where the speaker's facial information is missing, based on the speaker's voice track and the audio and video segments.

[0088] S142: Based on the human voice track and the speaker's facial information, a multimodal segmentation model is used to segment the audio and video segments into speakers, resulting in the first dialogue segment carrying timestamp information.

[0089] S143: Based on the human voice track, a speech segmentation model is used to segment the audio segment into speakers, resulting in a second dialogue segment carrying timestamp information.

[0090] S144: Merge the first and second dialogue segments based on the timestamp information to generate a dialogue segment sequence.

[0091] In this embodiment, when segmenting dialogue fragments, the computer device can determine audio-visual fragments where the speaker's facial information is not missing from the audio-visual file based on the speaker's facial information, and determine audio fragments where the speaker's facial information is missing from the audio file based on the voice track and the audio-visual fragments. Then, based on the voice track and the speaker's facial information, a multimodal segmentation model is used to segment the audio-visual fragments by the speaker to obtain a first dialogue fragment carrying timestamp information. Based on the voice track, a speech segmentation model is used to segment the audio fragments by the speaker to obtain a second dialogue fragment carrying timestamp information. Finally, the first dialogue fragment and the second dialogue fragment can be merged according to the timestamp information to generate a dialogue fragment sequence.

[0092] Understandably, by using the speaker's facial information, computer devices can segment audio and video files to identify segments where the speaker's facial information is not missing. This means segments in which clear facial features can be detected in the video frame and which correspond to the speaker. At the same time, for segments where valid facial information cannot be obtained from the video, computer devices can analyze the audio track in the audio file to identify audio segments where the speaker's facial information is missing. This means segments in which the speaker speaks in the audio signal but cannot be detected in the corresponding video frame.

[0093] Specifically, for audio / video segments where speaker facial information is not missing, the computer device can utilize a multimodal segmentation model to comprehensively analyze facial features in the speaker's facial information and vocal features in the voice track. This allows for precise segmentation of the dialogue content corresponding to each speaker within the audio / video segment, generating a first dialogue segment with timestamp information, thus ensuring accurate correspondence between the dialogue segment and the speaker. For audio segments where speaker facial information is missing, the computer device can use a speech segmentation model to analyze vocal features in the voice track. This allows for precise segmentation of the dialogue content corresponding to each speaker within the audio segment, generating a second dialogue segment with timestamp information. This enables speaker recognition and segmentation even for segments containing only audio information. Subsequently, the computer device can use the timestamp information to precisely align and merge the first and second dialogue segments, generating a complete dialogue segment sequence, ensuring that each line corresponds to the correct speaker to the greatest extent possible.

[0094] In one embodiment, the process of aligning the standard transcribed text and the dialogue fragment sequence based on timestamp information to obtain speaker information in step S140 may include:

[0095] S145: Perform minimum semantic unit segmentation on the standard transcribed text to obtain a semantic unit sequence, and perform minimum dialogue unit segmentation on the dialogue fragment sequence to obtain a dialogue unit sequence.

[0096] S146: Align the semantic unit sequence and the dialogue unit sequence one by one according to the timestamp information to generate speaker information.

[0097] In this embodiment, the computer device can perform minimum semantic unit segmentation on the standard transcribed text to obtain a semantic unit sequence, and perform minimum dialogue unit segmentation on the dialogue fragment sequence to obtain a dialogue unit sequence. Then, the semantic unit sequence and the dialogue unit sequence are aligned one by one according to the timestamp information to generate speaker information.

[0098] Specifically, computer devices can use natural language processing (NLP) technology to perform syntactic analysis, semantic boundary recognition, and phrase structure segmentation on standard transcribed text, breaking it down into minimal semantic units with complete meaning, forming a sequence of semantic units. Each semantic unit carries start and end timestamp information to identify its position in the audio / video file. Simultaneously, computer devices can also segment dialogue fragment sequences into minimum dialogue units. By analyzing the audio features, pause information, and pronunciation rhythm of the dialogue content, it divides it into the smallest identifiable dialogue units, generating a sequence of dialogue units. Each dialogue unit also carries timestamp information, clearly indicating its start and end positions in the audio / video file. Based on this, computer devices can use the timestamp information to precisely align the semantic unit sequence and the dialogue unit sequence one-to-one, matching each semantic unit with its corresponding dialogue unit. This achieves high-precision matching between text content and speakers in audio / video, thereby generating accurately segmented and semantically coherent speaker information.

[0099] In one embodiment, step S150, which involves performing speaker profiling analysis based on video-understood text, speech-understood text, and speaker information to generate a person profile text, may include:

[0100] S151: Determine the character profile model.

[0101] S152: Input the video understanding text, speech understanding text, and speaker information into the person profiling model so that the person profiling model can perform speaker profiling analysis and output the person profiling text.

[0102] In this embodiment, when performing speaker profiling analysis, the computer device can first determine the person profiling model, and then input the video understanding text, speech understanding text and speaker information into the person profiling model so that the person profiling model can perform speaker profiling analysis and output the person profiling text.

[0103] It is understood that the character profiling model in this application refers to a model that performs character profiling analysis on input video understanding text, speech understanding text, and speaker information to obtain character profiling text. During model training, this character profiling model can use video understanding text, speech understanding text, and speaker information from different speakers as training samples, and label each training sample with a sample label, i.e., the corresponding real profiling text. Once all training samples are labeled, the labeled training samples can be input into the pre-defined character profiling model for forward propagation to train the model. During the model's backpropagation, a loss function is used to fine-tune the model's parameters. When the model meets certain training conditions or parameter convergence conditions, such as when the number of iterations reaches a set value, training is considered complete, and the trained model can be used as the final character profiling model.

[0104] Furthermore, this application can also store the trained person profiling model so that the computer device can directly call the pre-stored person profiling model to perform speaker profiling analysis during subsequent person profiling analysis. In addition, the person profiling model of this application can be trained using LLM models such as Deepseek-v3, Deepseek-r1, Qwen2.5, Qwen3, GPT, etc., without limitation.

[0105] In one embodiment, the process of generating the script information of a film or television drama based on speaker information and character profiling text in step S150 may include:

[0106] S153: Determine the speaker tag and timestamp of each smallest semantic unit in the standard transcribed text based on the speaker information, and determine the profile content corresponding to each timestamp from the profile text.

[0107] S154: Sequentially merge the speaker tags, smallest semantic units, and profile content corresponding to each timestamp to generate script information for film and television dramas.

[0108] In this embodiment, after generating speaker information, the computer device can determine the speaker tag and timestamp of each smallest semantic unit in the standard transcribed text based on the speaker information, and determine the profile content corresponding to each timestamp from the character profile text. Then, the speaker tags, smallest semantic units and profile content corresponding to each timestamp can be structurally merged in sequence to generate the script information of the film and television drama.

[0109] Specifically, computer equipment can generate speaker tags for each smallest semantic unit in the standard transcribed text based on speaker information, and annotate its corresponding timestamp, thereby establishing a mapping relationship between semantic units and specific speakers at the text level. Simultaneously, the computer equipment can extract profile content corresponding to each timestamp, i.e., each smallest semantic unit, from the character profile text. This profile content includes information such as the speaker's emotional state, psychological activities, and behavioral tendencies at the current speaking moment. Therefore, the computer equipment can structurally merge the speaker tags, smallest semantic units, and corresponding profile content corresponding to each timestamp in chronological order, automatically generating script information for film and television dramas. This script information not only includes complete dialogue text but also incorporates rich character backgrounds and personality descriptions, providing more comprehensive and detailed information for secondary creation and translation of film and television dramas.

[0110] The following describes the film and television script extraction device provided in the embodiments of this application. The film and television script extraction device described below can be referred to in correspondence with the film and television script extraction method described above.

[0111] In one embodiment, such as Figure 3 As shown, Figure 3 This application provides a schematic flowchart of a film and television script extraction device according to an embodiment of the present application. The present application also provides a film and television script extraction device, including a file splitting module 210, a video recognition module 220, an audio understanding module 230, an information alignment module 240, and a script generation module 250, specifically comprising the following:

[0112] The file splitting module 210 is used to obtain audio and video files of movies and TV series, split the audio and video files into video files and audio files, and separate the human voice tracks from the audio files.

[0113] The video recognition module 220 is used to perform feature recognition on video files, obtain subtitle text with timestamp information and speaker facial information, and perform state understanding of the speaker based on the speaker facial information to generate video understanding text.

[0114] The audio understanding module 230 is used to perform speech understanding on the audio file, obtain the speech transcribed text and speech understood text carrying timestamp information, and perform text correction on the speech transcribed text according to the subtitle text to obtain the standard transcribed text.

[0115] The information alignment module 240 is used to segment audio and video files into dialogue segments based on human voice tracks and speaker facial information to obtain a dialogue segment sequence carrying timestamp information, and to align the standard transcribed text and the dialogue segment sequence based on the timestamp information to obtain speaker information.

[0116] The script generation module 250 is used to perform speaker profiling analysis based on video understanding text, speech understanding text and speaker information, generate character profile text, and generate script information for film and television dramas based on speaker information and character profile text.

[0117] In the above embodiments, after obtaining the audio and video files of the film and television drama, the audio and video files can be split into video files and audio files, and the human voice track in the audio file can be separated, providing a clear and independent data source for subsequent feature recognition and understanding of the video and audio in different dimensions. Among them, the subtitle text with timestamp information and the speaker's facial information obtained by feature recognition of the video file can achieve a precise correspondence between text, time and speaker, ensuring the semantic coherence of the dialogue. In addition, the video understanding text generated by understanding the speaker's state based on the speaker's facial information can further enrich the semantic content. The speech transcription text with timestamp information and the speech understanding text obtained by speech understanding of the audio file can more comprehensively capture the semantic information in the speech. Subsequently, the speech transcription text can be corrected based on the subtitle text to obtain the standard transcription text, thereby improving the accuracy of text extraction. After obtaining the relevant feature recognition results of the audio and video files, this application can segment the audio and video files into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments with timestamp information, thereby reducing the amount of subsequent data processing. Then, based on the timestamp information, the standard transcribed text and dialogue segments can be aligned to accurately associate each line of dialogue with the corresponding speaker, resulting in accurately segmented and semantically coherent speaker information. Finally, based on video understanding text, speech understanding text, and speaker information, speaker profiling analysis can be performed to deeply explore the speaker's personality, emotions, behavioral characteristics, and other information, thereby generating character profile text. Based on this, script information for film and television dramas can be generated according to the speaker information and character profile text, which can include relevant character feature descriptions in addition to dialogue content, thereby enriching the content and depth of the script and providing more comprehensive and detailed information for the secondary creation and translation of film and television dramas.

[0118] In one embodiment, the video recognition module 220 may include:

[0119] The submodule for scene detection is used to perform scene detection on video files, obtain detection results, and generate video semantic segments based on the detection results.

[0120] The video recognition submodule is used to perform character recognition on video semantic segments to obtain subtitle text with timestamp information, and to perform facial lip-sync recognition on video semantic segments to obtain speaker facial information with timestamp information.

[0121] In one embodiment, the audio understanding module 230 may include:

[0122] The silence detection submodule is used to perform silence detection on audio files, obtain the detection results, and generate speech activity segments based on the detection results.

[0123] The audio understanding submodule is used to convert speech activity segments into speech-to-text, and to perform emotion understanding on speech activity segments to generate speech-understood text.

[0124] In one embodiment, the information alignment module 240 may include:

[0125] The segment determination submodule is used to determine audio and video segments from audio and video files where the speaker's facial information is not missing, based on the speaker's facial information, and to determine audio segments from audio files where the speaker's facial information is missing, based on the speaker's voice track and audio and video segments.

[0126] The first segmentation submodule is used to segment audio and video clips into speakers based on human voice tracks and speaker facial information using a multimodal segmentation model, thereby obtaining the first dialogue segment carrying timestamp information.

[0127] The second segmentation submodule is used to segment audio segments based on human voice tracks using a speech segmentation model to obtain second dialogue segments carrying timestamp information.

[0128] The segment merging submodule is used to merge the first and second dialogue segments based on timestamp information to generate a dialogue segment sequence.

[0129] In one embodiment, the information alignment module 240 may further include:

[0130] The unit splitting submodule is used to perform minimum semantic unit splitting on standard transcribed text to obtain semantic unit sequences, and to perform minimum dialogue unit segmentation on dialogue fragment sequences to obtain dialogue unit sequences.

[0131] The sequence alignment submodule is used to align the semantic unit sequence and the dialogue unit sequence one by one based on the timestamp information to generate speaker information.

[0132] In one embodiment, the script generation module 250 may include:

[0133] The model determination submodule is used to determine the character profile model.

[0134] The profiling analysis submodule is used to input video understanding text, speech understanding text, and speaker information into the person profiling model, so that the person profiling model can perform profiling analysis on the speaker and output the person profiling text.

[0135] In one embodiment, the script generation module 250 may further include:

[0136] The data determination submodule is used to determine the speaker tag and timestamp of each smallest semantic unit in the standard transcribed text based on speaker information, and to determine the profile content corresponding to each timestamp from the profile text.

[0137] The data merging submodule is used to sequentially and structurally merge the speaker tags, smallest semantic units, and profile content corresponding to each timestamp to generate script information for film and television dramas.

[0138] In one embodiment, this application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the film and television script extraction method as described in any of the above embodiments.

[0139] In one embodiment, this application also provides a computer device storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the film and television script extraction method as described in any of the above embodiments.

[0140] Indicatively, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 4 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the film and television script extraction method of any of the above embodiments.

[0141] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0142] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0143] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0144] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0145] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting film and television scripts, characterized in that, The method includes: The audio and video files of the film and television drama are obtained, and the audio and video files are split into video files and audio files. In addition, the human voice track in the audio file is separated. The video file is subjected to feature recognition to obtain subtitle text carrying timestamp information and speaker facial information. Based on the speaker facial information, the speaker's state is understood to generate video understanding text. The audio file is subjected to speech understanding to obtain speech transcribed text and speech understanding text carrying timestamp information. The speech transcribed text is then corrected according to the subtitle text to obtain standard transcribed text. The speech understanding text is the text content generated after semantic analysis, intonation recognition, emotion recognition and context understanding of the audio content, and is used to express at least one implicit information of the speaker's tone, emotion and intention at the current moment. Based on the human voice track and the speaker's facial information, the audio and video file is segmented into dialogue segments to obtain a dialogue segment sequence carrying timestamp information. Based on the timestamp information, the standard transcribed text and the dialogue segment sequence are aligned to obtain speaker information. Based on the video understanding text, the speech understanding text, and the speaker information, a speaker profiling analysis is performed to generate a character profile text, and the script information of the film and television drama is generated based on the speaker information and the character profile text. The step of segmenting the audio / video file into dialogue segments based on the human voice track and the speaker's facial information to obtain a sequence of dialogue segments carrying timestamp information includes: Based on the speaker's facial information, determine the audio and video segments from the audio and video file where the speaker's facial information is not missing, and determine the audio segments from the audio file where the speaker's facial information is missing based on the speaker's voice track and the audio and video segments; Based on the human voice track and the speaker's facial information, a multimodal segmentation model is used to segment the audio and video segments into speakers, resulting in a first dialogue segment carrying timestamp information. Based on the human voice track, a speech segmentation model is used to segment the audio segment into speakers to obtain a second dialogue segment carrying timestamp information. The first and second dialogue segments are merged based on the timestamp information to generate a dialogue segment sequence.

2. The method for extracting film and television scripts according to claim 1, characterized in that, The step of performing feature recognition on the video file to obtain subtitle text carrying timestamp information and speaker facial information includes: The video file is subjected to scene detection to obtain the detection results, and video semantic segments are generated based on the detection results; The video semantic segment is subjected to character recognition to obtain subtitle text carrying timestamp information, and the video semantic segment is subjected to facial lip-sync recognition to obtain speaker facial information carrying timestamp information.

3. The method for extracting film and television scripts according to claim 1, characterized in that, The step of performing speech understanding on the audio file to obtain speech-transcribed text and speech-understood text carrying timestamp information includes: The audio file is subjected to silence detection to obtain the detection result, and a speech activity segment is generated based on the detection result; The speech activity segment is converted into speech-to-text to obtain speech-transcribed text, and the speech activity segment is subjected to emotion understanding to generate speech-understood text.

4. The method for extracting film and television scripts according to claim 1, characterized in that, The process of aligning the standard transcribed text and the dialogue fragment sequence based on the timestamp information to obtain speaker information includes: The standard transcribed text is split into minimum semantic units to obtain a semantic unit sequence, and the dialogue fragment sequence is split into minimum dialogue units to obtain a dialogue unit sequence. The semantic unit sequence and the dialogue unit sequence are aligned one by one according to the timestamp information to generate speaker information.

5. The method for extracting film and television scripts according to claim 1, characterized in that, The step of performing speaker profiling analysis based on the video-understood text, the speech-understood text, and the speaker information to generate a person profile text includes: Determine the character profile model; The video understanding text, the speech understanding text, and the speaker information are input into the character profiling model so that the character profiling model can perform profile analysis on the speaker and output the character profile text.

6. The method for extracting film and television scripts according to claim 1, characterized in that, The step of generating the script information for the film and television drama based on the speaker information and the character profile text includes: Based on the speaker information, determine the speaker tag and timestamp of each smallest semantic unit in the standard transcribed text, and determine the profile content corresponding to each timestamp from the character profile text; The speaker tags, smallest semantic units, and profile content corresponding to each timestamp are sequentially merged in a structured manner to generate the script information of the film and television drama.

7. A device for extracting film and television scripts, characterized in that, include: The file splitting module is used to obtain audio and video files of movies and TV series, split the audio and video files into video files and audio files, and separate the human voice track from the audio file. The video recognition module is used to perform feature recognition on the video file, obtain subtitle text carrying timestamp information and speaker facial information, and perform state understanding of the speaker based on the speaker facial information to generate video understanding text; The audio understanding module is used to perform speech understanding on the audio file to obtain speech transcribed text and speech understanding text carrying timestamp information, and to perform text correction on the speech transcribed text according to the subtitle text to obtain standard transcribed text; wherein, the speech understanding text is the text content generated after semantic analysis, intonation recognition, emotion recognition and context understanding of the audio content, and is used to express at least one implicit information of the speaker's tone, emotion and intention at the current moment; The information alignment module is used to segment the audio and video file into dialogue segments based on the human voice track and the speaker's facial information to obtain a dialogue segment sequence carrying timestamp information, and to align the standard transcribed text and the dialogue segment sequence based on the timestamp information to obtain speaker information; The script generation module is used to perform profiling analysis on the speaker based on the video understanding text, the speech understanding text, and the speaker information, generate character profile text, and generate script information for the film and television drama based on the speaker information and the character profile text. The information alignment module includes: Based on the speaker's facial information, determine the audio and video segments from the audio and video file where the speaker's facial information is not missing, and determine the audio segments from the audio file where the speaker's facial information is missing based on the speaker's voice track and the audio and video segments; Based on the human voice track and the speaker's facial information, a multimodal segmentation model is used to segment the audio and video segments into speakers, resulting in a first dialogue segment carrying timestamp information. Based on the human voice track, a speech segmentation model is used to segment the audio segment into speakers to obtain a second dialogue segment carrying timestamp information. The first and second dialogue segments are merged based on the timestamp information to generate a dialogue segment sequence.

8. A storage medium, characterized in that: The storage medium stores computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the film and television script extraction method as described in any one of claims 1 to 6.

9. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the film and television script extraction method as described in any one of claims 1 to 6.

Images