A method and device for automatically extracting and retrieving script scene clip videos using film and television works and scripts

Abstract

The present invention discloses a method and device for automatically extracting and retrieving script scene fragment videos by utilizing film and television works and scripts, comprising parsing original scripts, scene contents, and film and television works, extracting video and audio parts, converting the parts into text, obtaining video fragments corresponding to matching audio, integrating the video fragments, scene formatting information, and film and television work information, and entering the integrated information into a database; utilizing basic information, scene descriptions, dialogues, and other contents of film and television works to retrieve video scenes, thereby completing the construction of a film and television work database centered on film and television work scenes; creating a retrieval system for film and television work scene fragments, reducing the difficulty and time of extracting scene fragments from complete videos, greatly saving manpower, providing relevant creators with more effective creation auxiliary tools, and providing users with richer query methods that are closer to human usage habits, thereby achieving an effective balance between the accuracy of scene extraction and the recall rate.

Description

Technical Field

[0001] The present invention relates to the technical field of text and video analysis and retrieval, and in particular to a method and device for automatically extracting and retrieving script scene fragments using film and television works and scripts, extracting video scene fragments using scripts and film and television works, and indexing the fragments in combination with text descriptions of the scenes. Background Art

[0002] In the process of creating film and television works, the crew needs to complete a lot of work from the content of the script to the final presentation effect. If the script writers can take into account the presentation effect of the final script on the screen when drawing inspiration for writing the script, it will be more conducive to the advancement of filming. At the same time, if the director and other filming-related personnel can quickly retrieve clips of previous film and television works related to the content of the script, it will provide more references and inspiration for filming and speed up the filming. The writing and shooting of scripts are based on scenes as the basic logical units, which include three parts: scene information, dialogue and description. How to retrieve and extract the scenes in the script from the video of the film and television work that has been shot, and integrate them with the structured information extracted from the scene text content and store them in the film and television work scene library, and provide them to the creators and shooters for retrieval and use is the core goal of the present invention;

[0003] Traditional film and television content retrieval methods primarily utilize video attribute-based text databases, indexing and storing text content such as the film and television title, plot summary, starring actors, and director. This retrieval method relies primarily on manual content compilation and entry, making it impossible to retrieve and locate specific segments within a video, such as a scene or shot.

[0004] In recent years, content-based video retrieval has also made some progress. It can retrieve video clips by segmenting the video, understanding the content of the frames, and merging similar frames. However, this method requires a lot of computing resources and is difficult to be practical at this stage.

[0005] Prior art patent CN201910500116.4 A method for extracting important movie clips based on script and subtitle analysis, including extracting basic element related data from the movie script including scene name, scene description, characters, character dialogues and character action related data; extracting basic element related data from the subtitles including character dialogues and the start time of character dialogues; enriching the script information and obtaining the start time of each scene; merging the relevant scenes of the movie based on character relevance, location relevance and supporting role relevance to divide the movie clips; calculating the keyword score, main character score, emotion score and action score respectively, calculating the importance value of each movie clip, and taking the higher score as the important movie clip. The present invention realizes a more in-depth analysis of semantics on the basis of dividing the movie clips; by utilizing the matching relationship between the script and the movie subtitles to segment the movie scenes, the movie can be segmented and retrieved according to the scenes. This patent can only be used for works with subtitles, and has the problem of a narrow scope of application;

[0006] Therefore, those skilled in the art are committed to developing a method and device for automatically extracting and retrieving script scene clip videos using film and television works and scripts to address the deficiencies of the above-mentioned prior art. Summary of the Invention

[0007] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the current prior art can only search and crop fragments from long videos manually or can only search through the subtitle part;

[0008] To achieve the above-mentioned object, the first aspect of the present invention provides a method for automatically extracting and retrieving script scene clip videos using film and television works and scripts, which specifically includes the following steps:

[0009] Step 1: parse the film and television script, pre-process the script, extract the scene fragments in the script, and analyze and identify the scene information content, scene description and dialogue content of the scene fragments;

[0010] Step 2: extract the video and audio content from the film and television works, convert the audio content into text, save the audio text content in the order of appearance, and add a time stamp;

[0011] Step 3: After extracting the speech content of the dialogue content in the script scene segment obtained in step 1, search for it in the audio text obtained in step 2, and use the audio text segment with the highest matching degree as the target segment, and further optimize the target segment using the global matching result;

[0012] Step 4: Using the audio text and its timestamp from step 2, obtain the start and end times of the target segment obtained in step 3 in the complete video, and extract the target scene video from the film or television work based on the start and end times;

[0013] Step 5: Use a web crawler to crawl basic information of the film and television works;

[0014] Step 6: Use the database to save the video scene clips and their corresponding script clips, formatted script scene data, basic information of the film and television work, etc. At the same time, index the scene clip text in the script to support the use of keyword technology to search for film and television work scene clips;

[0015] Furthermore, in step 1, the scene fragments in the script are extracted by parsing the film and television script using natural language processing technology based on deep learning;

[0016] Furthermore, in step 1, the natural language processing technology based on deep learning includes specific tasks such as text classification, named entity recognition and sequence labeling;

[0017] Furthermore, in step 1, the content of the session information includes session number, time, location, interior and exterior scenes, characters, weather, etc.;

[0018] Furthermore, in step 1, the conversation content includes the speaker and the speech content;

[0019] Specifically, in step 1, in actual development, the extraction results should be screened and adjusted using artificial rules;

[0020] Furthermore, in step 2, the method of extracting the video and audio content from the film and television work is to use speech recognition technology for extraction;

[0021] Furthermore, in step 2, the method of converting the audio content into text is to use speech recognition technology for conversion;

[0022] Furthermore, in step 3, searching for scene dialogue content in the audio text includes searching for similar character strings to find the audio text segment with the highest matching degree;

[0023] Furthermore, in step 5, the basic information of the film or television work includes a brief introduction, director, producer, cast list, year, shooting time, region, etc.;

[0024] Furthermore, in step 5, crawling the information of the film and television work includes crawling information from movie-related websites (such as Douban Movies, Baidu Encyclopedia, Tencent Video, etc.) using a web crawler;

[0025] Furthermore, in step 6, the established search method includes keyword search, advanced search and question-answer search;

[0026] The second aspect of the present invention provides a device for automatically extracting and retrieving script scene fragments by using film and television works and scripts, comprising a parsing unit, an extraction and conversion unit, a matching unit, an integration unit, an information capture and storage unit, and an index establishment unit; the parsing unit is connected to the extraction and conversion unit; the extraction and conversion unit is connected to the matching unit; the matching unit is connected to the integration unit; the integration unit is connected to the information capture and storage unit; the information capture and storage unit is connected to the index establishment unit;

[0027] Furthermore, the parsing unit includes performing category analysis and entity tagging on the script;

[0028] Furthermore, the extraction and conversion unit includes extracting the video and audio of the film and television work and converting the audio content into text form;

[0029] Furthermore, the matching unit includes searching the audio text for dialogue content in the script scene to find matching segments;

[0030] Furthermore, the integration unit includes searching for start and end timestamps of the matching segment in the audio file, and extracting a video segment corresponding to the scene in the complete video according to the start and end timestamps;

[0031] Furthermore, the information crawling and warehousing unit includes using a web crawler to crawl information about film and television works from film-related websites (such as Douban Movies, Baidu Encyclopedia, Tencent Video, etc.); entering basic information such as scripts, video clips, and film and television dramas into the database;

[0032] Furthermore, the indexing unit includes indexing the scene segment text in the script to facilitate users to retrieve the video scene segment;

[0033] By adopting the above scheme, the method and device disclosed in the present invention for automatically extracting and retrieving script scene clip videos using film and television works and scripts has the following advantages:

[0034] (1) The present invention uses a method and device for automatically extracting and retrieving scene fragments from film and television works and scripts. The machine automatically extracts scene fragments from the script text content and the dialogue content in the audio of the film and television work, which reduces the difficulty and time of extracting scene fragments from the complete video and greatly saves manpower.

[0035] (2) The method and apparatus of the present invention for automatically extracting and retrieving script scene fragments from film and television works and scripts can achieve an effective balance between scene extraction accuracy and recall rate by setting a threshold value related to the extraction accuracy used during extraction. Compared with the prior art, it is not limited to using subtitles for fragment extraction, and thus increases the retrieval completeness.

[0036] (3) The present invention uses film and television works and scripts to establish a method and device for automatically extracting and retrieving script scene fragments, creating a retrieval system for film and television work scene fragments, providing screenwriters, directors, and film and television drama related creative personnel with a more effective creative auxiliary tool and good applicability;

[0037] (4) The present invention utilizes film and television works and scripts to establish a method and device for automatically extracting and retrieving script scene clips. The method of combining a graph database, a relational database, and a document index to retrieve film and television work scene clips provides users with a richer and more human-like query method, which is highly practical.

[0038] In summary, the present invention discloses a method and device for automatically extracting and retrieving script scene fragments by using film and television works and scripts. The machine automatically extracts video scene fragments by using the script text content and the dialogue content in the audio of the film and television works, which reduces the difficulty and time of extracting scene fragments from the complete video and greatly saves manpower. By setting a threshold value related to the extraction accuracy used during extraction, an effective balance can be achieved between the accuracy and recall rate of scene extraction. Compared with the existing technology, it is not only limited to using subtitles for fragment extraction, but also increases the retrieval completeness, creates a retrieval system for scene fragments of film and television works, and provides screenwriters, directors and film and television drama related creators with a more effective creation auxiliary tool. It has good applicability. The method of combining graph database, relational database and document index to retrieve scene fragments of film and television works provides users with a richer and more human-like query method, which is highly practical.

[0039] The concept, specific technical solutions and technical effects of the present invention will be further described below in conjunction with specific implementation methods to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of a method for automatically extracting and retrieving script scene clips using film and television works and scripts of the present invention;

[0041] Figure 2 This is a diagram of the field head analysis model of Example 1 of the present invention;

[0042] Figure 3This is a flow chart of speech recognition according to embodiment 1 of the present invention;

[0043] Figure 4 The inverted index table (2-gram) corresponding to "Where are the other father-in-laws of Desheng Gonggong?" in Example 1 of the present invention;

[0044] Figure 5 This is a schematic diagram of searching for a candidate set of similar character strings according to embodiment 1 of the present invention;

[0045] Figure 6 This is a table header definition diagram of a relational database key table in Example 1 of the present invention;

[0046] Figure 7 This is the scene video clip list display page of Example 1 of the present invention;

[0047] Figure 8 This is the scene video clip details display page of Example 1 of the present invention; DETAILED DESCRIPTION

[0048] The following describes several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, which are for illustrative purposes only and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0049] If there are experimental methods without specific conditions, they are usually implemented according to conventional conditions, such as those in the relevant instructions or manuals.

[0050] Example 1: Automatic extraction and retrieval of script scene clips using a method and device of the present invention using film and television works and scripts

[0051] Step 1: Use deep learning-based natural language processing technology to parse film and television scripts. Specific tasks include text classification, named entity recognition, and sequence labeling to extract scene fragments from the script. For each scene fragment, analyze and identify scene information (scene number, time, location, interior and exterior scenes, characters, weather, etc.), scene descriptions, and dialogue content (speaker, speech content);

[0052] To analyze the script of a film or TV work, the script must be pre-processed first, including:

[0053] Step 1-1: Use tools to clean up the script data, such as removing illegal characters such as spaces;

[0054] Step 1-2: Divide the script into sentences;

[0055] Steps 1-3: Segment the sentences;

[0056] Steps 1-4: Use word embedding technology to convert word segmentation results into word vectors;

[0057] Steps 1-5: Perform manual tagging and use the manual tagging results of the sentences (as shown in Table 1) and the corresponding word vectors as input to the parsing model to train the model parameters;

[0058] Table 1. Script analysis content

[0059]

[0060] The deep learning parsing model adopts a multi-task model of bidirectional long short-term memory network (Bi-LSTM) + attention mechanism (Attention), and the part used to mark text sequences uses conditional random fields (CRF) to improve the accuracy of sequence labeling;

[0061] The analytical model structure is as follows Figure 2 As shown;

[0062] The script parsing model uses multi-task output: sentence classification and sequence labeling;

[0063] The goal of sequence tagging is to identify entities and label conversation content;

[0064] After the script is parsed by the model, a sentence sequence with classification and sequence tag information is obtained;

[0065] Use the sentence marked as the scene head as the starting sentence of the scene, divide the script into multiple scene segments according to the scene head, and format the content of each scene segment according to the structure in Table 1 before saving;

[0066] In addition, since the model has certain errors, it is necessary to use artificial rules to screen and adjust the model analysis results;

[0067] Step 2: Use speech recognition technology to extract video and audio content from film and television works; use speech recognition technology to convert the audio content into text form, save the audio text content in the order of appearance, and add a time stamp;

[0068] like Figure 3 As shown;

[0069] Specifically, including

[0070] Step 2-1: Use open source tools (ffmpeg) to extract the audio content of film and television works;

[0071] Step 2-2: performing noise reduction and information enhancement on the audio content, and extracting audio features from the audio content to form a feature sequence;

[0072] Step 2-3: Use the acoustic model to convert the speech feature sequence into a syllable sequence (Chinese is converted into a pinyin sequence, and English is converted into an English syllable sequence); the acoustic model uses the relatively mature DNN-HMM acoustic model;

[0073] Step 2-4: Send the obtained syllable sequence into the language model; analyze the syllable sequence through the language model, and give the text corresponding to the syllable with the highest possibility as the best recognition result, that is, the finally recognized dialogue text; the language model uses the method of Connectionist Temporal Classification (CTC), that is, an end-to-end speech recognition technology, and the entire process of continuous speech recognition is completed by a neural network, directly generating the final text sequence;

[0074] Step 2-5: The obtained film and television dialogue text requires the time stamps corresponding to each word in the text in the original video to generate a character-audio table, as shown in Table 2;

[0075] Table 2. Example of character-audio representation

[0076]

[0077]

[0078] Step 3: After extracting the speech content of the dialogue part in the script scene segment obtained in Step 1, search in the audio text obtained in Step 2, and use the audio text segment with the highest matching degree as the target segment, and further optimize the target segment using the global matching result;

[0079] Specifically, the method of searching for similar strings is adopted. That is, if the dialogue content in the scene has the highest similarity with a certain part of the audio text, the audio segment corresponding to this part of the audio text is called the audio segment matching the scene dialogue;

[0080] The measurement method of string similarity uses the edit distance. The edit distance refers to the minimum number of edit operations (insertion, deletion, replacement) required to convert one string into another string;

[0081] In this Embodiment 1, ED(r, s) is used to represent the edit distance between strings r and s. For example, r = "Yuan Chunwang looks at Wei Yingluo", s = "Yuan Chunwang looks at Yingluo", r needs to delete one character to be consistent with s, so ED(r, s) = 1;

[0082] To illustrate the algorithm, the following definitions are introduced:

[0083] Define q-gram: Given a string s and a positive integer q, the q-gram of s can be defined as the set of all substrings of s with length q. For example, the 2-gram set of the string s = "Wei Yingluo" is {"Wei Ying", "Ying Luo"}, and the 3-gram set is {"Wei Yingluo"};

[0084] Define threshold-based similar string search: Given a set of strings S, a query string q, and a threshold t, threshold-based similar string search is to find all strings s ∈ S such that ED(s, q) ≤ t.

[0085] Define top-k similar string search: Given a set of strings S and a query string q, top-k similar string search is to return a result set where |R| = k, and for any r ∈ R and s ∈ S - R, ED(r, q) ≤ ED(s, q);

[0086] For the string matching algorithm for script dialogue content and audio text, a redesigned q-gram algorithm is adopted. First, for the audio text string s, its q-gram is obtained.

[0087] The method is as follows: Use a window of length q to move backward from the starting position of the string s. Each time it moves one position, the substring in the window is intercepted, and this substring becomes a signature. The set composed of signatures becomes the q-gram of the string s.

[0088] For example, the 2-gram set corresponding to the string s = "De Sheng Gonggong, what's going on here" is {"De Sheng", "Sheng Gong", "Gonggong", "Gong Zhe", "Zhe Shi", "Shi Zen", "Zen Me", "Mei Hui", "Hui Shi"}. To more accurately represent the specific information of each signature, each signature is generally represented by a binary tuple (signature, position), where signature represents a signature and position represents the position of this signature in the string.

[0089] Therefore, the 2-gram set of the above string s with position information is {("De Sheng", 0), ("Sheng Gong", 1), ("Gonggong", 2), ("Gong Zhe", 3), ("Zhe Shi", 4), ("Shi Zen", 5), ("Zen Me", 6), ("Mei Hui", 7), ("Hui Shi", 8)}.

[0090] Figure 4 Is the inverted index corresponding to "De Sheng Gonggong, where have the other gonggongs gone";

[0091] Based on the q-gram inverted index method, the algorithm for finding the script scene dialogue string in the candidate audio text includes the following steps:

[0092] Step 3-1: For the audio text string s and the search string q representing the scenario dialogue, filter out the punctuation marks in q;

[0093] Step 3-2: Generate the q-gram corresponding to q, and only save the signature, without saving the position of the signature;

[0094] Step 3-3: Search for all signatures in the q-gram corresponding to q in the inverted index corresponding to s. If the corresponding signature is found in the inverted index table, put the signature and position information into the set t-set;

[0095] Step 3-4: Construct a binary array t-array with a length equal to the length of the s string minus 1, and mark the positions corresponding to the positions of the elements in t-set as 1 in t-array;

[0096] Step 3-5: Construct a sliding window with a width equal to the length of the string q, and move it from left to right on t-array, moving one position each time, and record the number of marked 1s contained in the sliding window at each position;

[0097] Step 3-6: Sort each position according to the number of marked 1s at each position, and obtain the top-k positions with the most 1s. top-k is the screening threshold, and the goal is to set the starting positions of the top-k candidate matching strings;

[0098] Step 3-7: For each selected position, find the corresponding starting position in the audio text, and intercept a string with a length equal to the length of the search string q from this position; Calculate the edit distance between each obtained string and the string q, and the string with the smallest edit distance is the matching string.

[0099] Figure 5 Schematic diagram for obtaining candidate strings by searching for "other eunuchs" in "Where has Eunuch Desheng gone and where are the other eunuchs". The top-k value is 3, the signature length in the q-gram is 2, and the inverted order table is as Figure 3 shown. The obtained candidate strings are "eunuch other" at position 2, "other eunuch" at position 3, and "other eunuchs" at position 4. Calculate the edit distance between the top-k candidate strings and "other eunuchs" again. The candidate string at position 5 has the smallest edit distance and is used as the matching string;

[0100] Step 4: Through the audio text and its time stamps in Step 2, obtain the start and end times of the target segment obtained in Step 3 in the complete video, and extract the target scene video in the film and television work according to the start and end times;

[0101] Specifically, it includes

[0102] Step 4-1: Sort the dialogues of all scenes from front to back in the order of appearance to form a dialogue sequence.

[0103] Step 4-2: Sort the matching strings of all scene dialogues in the audio file in ascending order according to the timestamps of their start characters to form a start-time-sequence of matching audio strings;

[0104] Step 4-3: sort the matching strings of all scene dialogues in the audio file in ascending order according to the timestamps of their end characters to form an end-time-sequence of the matching audio strings;

[0105] Step 4-4: Find all consecutive dialogues in the dialogue sequence, where the matching audio strings appear consecutively in the audio string start sequence and in the same order as in the dialogue sequence, and where the matching audio strings appear consecutively in the audio string end sequence and in the same order as in the dialogue sequence. The resulting dialogue sequence segments are called best-matching dialogue sequences, and the scenes contained in all the best-matching dialogue sequences constitute a best-matching scene set, best-scene-set. All the best-matching dialogue sequences form a best-matching dialogue sequence set, best-slice-set.

[0106] Step 4-5: Sort the elements of the best-slice-set of best-matching dialogue sequences by length from largest to smallest, and process each best-slice sequence as follows:

[0107] Step 4-5-1, find the end position end-position corresponding to the end time sequence end-time-sequence of best-slice;

[0108] Step 4-5-2, check the previous scene dialogue bs-dialog (if any) at the start position of the best-slice in dialog-sequence as a candidate dialogue;

[0109] Step 4-5-3: Recalculate the candidate audio text matching string set for bs-dialog. For a string in the matching string set, if the corresponding position of the string in the end-time-sequence is end-position-1, add bs-dialog to the best-scene-set, combine bs-dialog with best-slice to form a new sequence, and update the best-slice in the best-slice-set. If there is no candidate string corresponding to end-position-1, go to step 5.

[0110] Step 4-5-4, repeat steps 4-5-2 and 4-5-3;

[0111] Step 4-5-5, check the next scene dialogue ae-dialog (if any) after the end position of the best-slice in dialog-sequence as a candidate dialogue;

[0112] Step 4-5-6: Recalculate the candidate audio text matching string set for ae-dialog. For a string in the matching string set, if the corresponding position of the string in the end-time-sequence is end-position+1, add ed-dialog to the best-scene-set, combine ae-dialog with best-slice to form a new sequence, and update the best-slice in the best-slice-set. If there is no candidate string corresponding to the position end-position+1, the best-slice processing ends.

[0113] Step 4-5-7, repeat steps 4-5-5 and 4-5-6 until the best-slice processing is completed;

[0114] Step 4-6: All scenes corresponding to scene dialogues that do not belong to the best-scene-set are considered to be search failure scenes and are discarded;

[0115] Step 4-7: All scenes and dialogues included in the best-scene-set set are considered as the final matching scenes;

[0116] For successfully matched scenes, the start and end timestamps of the scene dialogues are readjusted;

[0117] For successfully matched scenes, obtain the start and end timestamps of the scene dialogue;

[0118] For the start timestamp, search the character audio table for all timestamps within span-threshold seconds (span-threshold is a hyperparameter that should be adjusted based on the script subject matter), calculate the time intervals between all adjacent timestamps, and select the median of the timestamp pairs with an interval of more than 1 second as the new start timestamp; if no timestamp pair with an interval of more than 1 second is found, directly update the start timestamp by starting timestamp - span-threshold / 2;

[0119] For the end timestamp, search backward in the character audio table for all timestamps within span-threshold seconds, calculate the time intervals of all adjacent timestamps, and select the median of the timestamp pairs with an interval of more than 1 second as the new end timestamp; if no timestamp pair with an interval of more than 1 second is found, directly update the end timestamp with the end timestamp + span-threshold / 2.

[0120] The video between the start timestamp and the end timestamp is captured from the film or television work as the matching video segment.

[0121] Step 5. Use web crawlers to crawl information about film and television works from movie-related websites (such as Douban Movies, Baidu Encyclopedia, Tencent Video, etc.). Information categories include series introduction, director, screenwriter, starring actor, film type, production country / region, premiere time, length, number of episodes, rating, cast, etc. The structured information obtained by parsing the script scene fragments of film and television works, the video clips corresponding to the script scenes, and the crawled film and television work information are integrated and stored in a graph database (such as Neo4j) and a relational database (such as MySQL) to build a movie clip knowledge graph. At the same time, a text index is established for the original scene text in the script, and the script scene fragments are retrieved by keywords. The main relational database table header definition is as follows Figure 6 As shown:

[0122] The main entities in the graph database include scenes, film and television works, video clips, actors, roles, positions, etc. The main entity relationships are shown in Table 4.

[0123] Table 4: Entity relationship table

[0124] entity entity Entity Relationships Scenario Film and television works belong Role Scenario Include personnel Role Played by personnel Position Serve Role Film and television works Include Scenario Film and TV clips correspond

[0125] Step 6: Use the database to save the video scene clips and their corresponding script clips, formatted script scene data, basic information of the film and television works, etc. At the same time, index the scene clip text in the script to support the use of keyword technology to search for film and television work scene clips;

[0126] The retrieval methods of the film and television scene library are divided into keyword search, high-level search and question-answer search;

[0127] The keyword search searches for the user-input keyword in the inverted index table, finds the best matching scene set and returns it to the user;

[0128] In the high-order search, the user specifies the search field, search relationship, etc. to directly search for a set of scenes that meet the conditions from the relational database and graph database;

[0129] The question-answering search uses the knowledge graph question-answering system technology to search the content of the graph database and return a set of scenarios that meet the conditions;

[0130] The video clip search results page and video clip details page are as follows Figure 7 、 Figure 8 As shown:

[0131] Results: When the method and device for automatically extracting and retrieving script scene fragment videos by using film and television works and scripts of the present invention in Example 1 are used to automatically extract script scene fragment videos, not only can the automatic extraction of scene fragments from film and television works be achieved by combining natural language processing technology with computer audio and video processing technology, thereby avoiding the need to manually search and crop fragment content from long videos, but also the script scene matching results are optimized by analyzing and adjusting the matching results of all dialogues of the script, so that the extracted fragments will not feel abrupt to the operator, and the use is convenient and comfortable.

[0132] Comparative Example 2: Using the existing technology patent CN201910500116.4, a method for extracting important movie clips based on script and subtitle analysis, to automatically extract script scene clip videos

[0133] Step 1: Extract the basic elements of the movie script, that is, extract the relevant data of scene name, scene description, characters, character dialogue and character action based on the original txt script text;

[0134] Step 2: Extract the basic elements of the subtitles. Extract the relevant data of the character dialogue and the start time of the character dialogue based on the original txt subtitle text;

[0135] Step 3: Using the data obtained in Steps 1 and 2, use the Similarity function in the gens im toolkit to calculate the similarity of the character dialogues in the script and subtitles respectively, and take the maximum value for matching, thereby enriching the script information and obtaining the start time of each scene;

[0136] Step 4: Using the data format processed in step 3, merge the relevant scenes of the movie based on character relevance, location relevance, and supporting role relevance to divide the movie segments. When the same character appears in adjacent scenes, semantically, the two scenes describe the same event and belong to one story segment, so the adjacent scenes in this case are merged; scenes are divided into interior scenes (INT.) and exterior scenes (EXT.). From interior scenes to exterior scenes, the scene switches in the script, but these scenes are all related to the same location and basically narrate events around this location. They all belong to the same story segment, so this article merges scenes that occur in the same and adjacent locations; some supporting characters in the movie only appear in a few scenes, but semantically they all belong to the same story segment. When these scenes are adjacent, they are merged and divided into one movie segment;

[0137] Step 5: Using the data format processed in step 4, calculate the keyword score from the perspective of keywords. Use the TF-IDF algorithm to extract keywords from the entire script, calculate the word frequency, and define the keyword score as follows:

[0138]

[0139] in, k 关键词 Indicates the keyword number, n 关键词 Indicates the number of keywords. Indicates the frequency of the kth keyword in the full text, Indicates the kth 关键词 The frequency of keywords in a segment i;

[0140] Step 6: Using the data format processed in Step 4, calculate the main character score from the perspective of the main characters. Determine the main characters based on the number of lines they have in the script. Take the top 8 characters with the most lines as the main characters. Count the number of main characters in each segment and the number of lines these main characters have in that segment. The main character score is defined as follows:

[0141]

[0142] Among them, |MC(i)| represents the number of main characters in the i-th segment |MCC(k 人物 )| represents the kth 人物 The number of lines a character has in a film clip;

[0143] Step 7: Using the data format processed in Step 4, calculate the sentiment score from an emotional perspective. Extract the character dialogues in each segment and determine the sentiment polarity of the dialogues to obtain positive and negative sentiment values. The closer the sentiment value is to 1, the more positive it is, and the closer the sentiment value is to 0, the more negative it is. The definition of the sentiment score is as follows:

[0144]

[0145] Among them, X represents the sentiment value of each line, n 台词 represents the number of lines in a segment, k 人物 Indicates the character dialogue number in a clip, i indicates the clip number;

[0146] Step 8: Using the data format processed in step 4, consider the action and calculate the action score: search for verbs in the extracted keywords and count the frequency of these verbs and the frequency of the keywords. The definition of the action score is as follows:

[0147]

[0148] Where V represents the frequency of verbs in the keywords belonging to a segment i, and f represents the frequency of keywords in a segment j; i 动作 Indicates the number of actions;

[0149] Step 9: Use the results of steps 5, 6, 7, and 8 to weight the importance of each movie clip, and select the n movie clips with the highest scores as the important movie clips. The importance of movie clips is defined as follows:

[0150] IMC(i)=r1·KS(i)+r2·MS(i)+r3·ES(i)+r4·AS(i)

[0151] Where IMC represents the importance value of each movie clip, KS represents the keyword score, MS represents the main character score, ES represents the emotion score, AS represents the action score, i represents the movie clip number, r1, r2, r3, r4 represent the weight function, r1+r2+r3+r4=1;

[0152] Results: After the method of Example 2 was applied, only subtitles, keywords, and other script scene fragments were extracted from the video; according to the defined scoring criteria, important fragments were extracted;

[0153] Test Example 3:

[0154] The automatic extraction and retrieval results of scene clip videos of the same script in Example 1 and Comparative Example 2 were compared; the results showed that Comparative Example 2 used movie subtitles and scripts to extract clips, and its scope of application was film and television works with subtitles, while Example 1 used speech recognition technology for dialogue recognition. Compared with Comparative Example 2, which extracted dialogue content from subtitles, it can extract clips from a large number of film and television works without subtitles, and has a wider range of applicable scenarios; compared with Comparative Example 2, Example 1 is not limited to extracting clips through subtitles, keywords, etc., and is not limited to extracting only important scene clips, but extracts target clips as a whole, with a wider extraction range and better effect; and indexing is established, which is closer to the operator's usage habits and convenient for the operator to use; in addition, Comparative Example 2 does not introduce a method for extracting script elements in step 1, while Example 1 introduces the automatic extraction method in detail, and the description is more complete; finally, Example 1 uses the method and device for automatically extracting and retrieving script scene clip videos using film and television works and scripts of the present invention to establish a film and television work scene retrieval library, while Comparative Example 2 only extracts clips, the goals are inconsistent, and Comparative Example 2 cannot achieve the construction of a retrieval library.

[0155] In summary, the present invention provides a method and device for automatically extracting and retrieving script scene fragments by using film and television works and scripts. The machine automatically extracts video scene fragments by using the script text content and the dialogue content in the audio of the film and television works, which reduces the difficulty and time of extracting scene fragments from the complete video and greatly saves manpower. By setting a threshold value related to the extraction accuracy used during extraction, an effective balance can be achieved between the accuracy and recall rate of scene extraction. Compared with the existing technology, it is not only limited to using subtitles for fragment extraction, but also increases the retrieval completeness. A retrieval system for scene fragments of film and television works is created, which provides screenwriters, directors and film and television drama related creators with a more effective creation auxiliary tool, has good applicability, and uses a method combining graph database, relational database and document index to retrieve scene fragments of film and television works, which provides users with a richer and more human-like query method, and is highly practical.

[0156] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention can be made by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art based on the concepts of the present invention through logical analysis, reasoning, or limited experimentation based on the existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for automatically extracting and retrieving script scene clips using film and television works and scripts, characterized in that: The following steps are included: Step 1: parse the film and television script, pre-process the script, extract the scene fragments in the script, and analyze and identify the scene information content, scene description and dialogue content of the scene fragments; Step 2: extract the video and audio content from the film and television works, convert the audio content into text, save the audio text content in the order of appearance, and add a time stamp; Step 3: After extracting the speech content of the dialogue content in the script scene segment obtained in step 1, search for it in the audio text obtained in step 2, and use the audio text segment with the highest matching degree as the target segment, and further optimize the target segment using the global matching result; Step 4: Using the audio text and its timestamp from step 2, obtain the start and end times of the target segment obtained in step 3 in the complete video, and extract the target scene video from the film or television work based on the start and end times; Step 5: Use a web crawler to crawl basic information of the film and television works; Step 6: Use the database to save the video scene clips and their corresponding script clips, formatted script scene data, and basic information content of the film and television work; at the same time, index the scene clip text in the script to support the use of keyword technology to search for film and television work scene clips; In the step 1, The method of extracting scene fragments from the script is to analyze the film and television script using natural language processing technology based on deep learning to extract them; The deep learning-based natural language processing technology includes text classification, named entity recognition, and sequence labeling. The event information includes the event, time, location, interior and exterior scenes, characters, and weather; The dialogue content includes the speaker and the speech content.

2. The method for automatically extracting and retrieving script scene clips using film and television works and scripts as claimed in claim 1, characterized in that: In the step 1, In actual development, artificial rules should be used to screen and adjust the extraction results.

3. The method for automatically extracting and retrieving script scene clips using film and television works and scripts as claimed in claim 1, characterized in that: In the step 2, The method of extracting the video and audio content from the film and television works is to use speech recognition technology to extract; The method of converting the audio content into text is to use speech recognition technology to convert it.

4. The method for automatically extracting and retrieving script scene clips using film and television works and scripts as claimed in claim 1, characterized in that: In the step 3, Searching for scene dialogue content in audio text includes finding the audio text segment with the highest matching degree by searching for similar character strings.

5. The method for automatically extracting and retrieving script scene clips using film and television works and scripts as claimed in claim 1, characterized in that: In the step 5, The basic information of the film or television work includes a brief introduction, director, producer, cast, year, shooting time, and region; The method of crawling the information of the film and television works includes crawling information from movie-related websites using a web crawler.

6. The method for automatically extracting and retrieving script scene clips using film and television works and scripts as claimed in claim 1, characterized in that: In step 6, The established retrieval methods include keyword search, advanced search and question-answer search.

7. A device for implementing the method for automatically extracting and retrieving script scene clips using film and television works and scripts as described in any one of claims 1 to 6, characterized in that: It includes a parsing unit, an extraction and conversion unit, a matching unit, an integration unit, an information capture and storage unit, and an index establishment unit; the parsing unit is connected to the extraction and conversion unit; the extraction and conversion unit is connected to the matching unit; the matching unit is connected to the integration unit; the integration unit is connected to the information capture and storage unit; the information capture and storage unit is connected to the index establishment unit; The parsing unit includes performing category analysis and entity tagging on the script; The extraction and conversion unit includes extracting the video and audio of the film and television work and converting the audio content into text form; The matching unit includes searching the audio text for dialogue content in the script scene to find matching segments; The integration unit includes searching for the start and end timestamps of the matching segment in the audio file, and extracting the video segment corresponding to the scene in the complete video according to the start and end timestamps; The information capture and storage unit includes using a web crawler to capture information about film and television works from movie-related websites; entering scripts, video clips, and basic information about film and television works into the database; The indexing unit includes indexing the scene segment texts in the script to facilitate users to retrieve the video scene segments.

Images