Video material clip automatic splicing method and system based on large model

Abstract

The present application relates to the field of video content generation, in particular to a video material clip automatic splicing method and system based on a large model, comprising: extracting the theme text of each segment in the video material; generating a first narrative text by fusing all theme texts; calculating the importance of each theme text therein; recording the theme text with the smallest importance as a class one text, and the rest as a class two text, and increasing the importance of the class one text based on the difference between the second narrative text generated based on the class two text and the first narrative text; extracting the common narrative features of all second narrative texts after repeating the process; dividing the theme texts into two categories according to the importance, so that the third narrative text fused by the first class theme text has the maximum similarity with the common narrative features; and splicing the corresponding video segments in the order of the first class theme text in the third narrative text. The present application automatically selects key segments from massive video materials and generates a video film that meets the user's demand.

Description

Technical Field

[0001] This invention relates to the field of video content generation, specifically to a method and system for automatic splicing of video footage based on large models. Background Technology

[0002] In the field of video editing, with the rapid increase in the amount of video footage, how to automatically filter and splice together complete videos that meet user intent from massive amounts of video clips has become a pressing technical challenge. Existing automatic editing methods largely rely on human experience, requiring the viewing, understanding, filtering, and segmentation of numerous video clips before combining and splicing together local segments – a process that demands a tremendous amount of work. While multimodal large-scale models have demonstrated powerful capabilities in understanding individual video clips, they are limited by the size of the context window and the ambiguity of the prompts. Specifically, large models cannot process hours of video content containing hundreds of clips and their complex relationships at once; furthermore, the editing intent (editing prompts) provided by users is usually highly generalized and vague. Current technologies struggle to automatically filter key segments from massive amounts of video footage and generate finished videos that meet user needs while ensuring narrative coherence and intent relevance. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method and system for automatically splicing video footage based on a large model.

[0004] The automatic video footage editing and splicing method and system based on large models of the present invention adopts the following technical solution: One embodiment of the present invention provides an automatic splicing method for video footage editing based on a large model, the method comprising the following steps: D1: Divide all video footage into several video segments, and use the image-to-text model to extract the thematic text of each video segment; use the text-to-text model to merge all thematic texts into the first narrative text based on editing prompts; calculate the importance of any thematic text in the first narrative text. D2: The least important thematic texts are categorized into Class 1 texts, and the rest are categorized into Class 2 texts. Based on the textual differences between the second narrative text formed by the fusion of Class 2 texts and the first narrative text, the importance of the thematic texts in Class 1 texts is increased. The increase is positively correlated with the textual differences. The second narrative text is then recorded. D3: After repeating D2 several times, extract the common narrative features of all recorded second narrative texts, and divide all thematic texts into two categories according to their importance. The category with the highest average importance is recorded as the first category, so that the third narrative text formed by merging all thematic texts in the first category has the greatest similarity to the common narrative features. D4: According to the order in which the thematic texts in the first category are recorded in the third narrative text, splice together the video clips corresponding to the thematic texts in the first category.

[0005] Preferably, the specific steps for calculating the importance of any thematic text in the first narrative text are as follows: The first narrative text is divided into several first text fragments; For any given topic text, convert each first text segment of the topic text and the first narrative text into vector representations, resulting in topic text vectors and first narrative segment vectors. Calculate the cosine similarity between the topic text vector and each first narrative segment vector in the first narrative text. The maximum cosine similarity between the topic text vector and all first narrative segment vectors in the first narrative text is recorded as the contribution of the topic text to the first narrative text. Normalize the contribution of all topic texts to the first narrative text, and use the normalized contribution as the importance of each topic text in the first narrative text.

[0006] Preferably, the specific steps for obtaining the textual differences between the second narrative text and the first narrative text are as follows: The second narrative text is divided into several second text fragments; Convert all second text fragments into vector representations to obtain all second narrative fragment vectors; the minimum cosine distance between each second narrative fragment vector and all first narrative fragment vectors is denoted as the local difference of each second text fragment, and the mean of the local differences of all second text fragments is denoted as the text difference.

[0007] Preferably, the specific steps for extracting the common narrative features of all recorded second narrative texts are as follows: For all second narrative texts, extract the key sentences with the highest weights from each second narrative text using the TextRank algorithm, convert all the key sentences extracted from all second narrative texts into vector representations, and denote them as key sentence vectors; cluster all key sentence vectors, and denote the mean of all key sentence vectors in each cluster as the core plot feature; the set of core plot features of all clusters is taken as the common narrative feature.

[0008] Preferably, the step of dividing all thematic texts into two categories based on importance, with the category having the highest average importance value designated as the first category, ensures that the third narrative text formed by merging all thematic texts within the first category has the greatest similarity to common narrative features. The specific steps include the following: For all thematic texts, the most important N×100% of thematic texts are taken as temporary categories, where N represents the preset division ratio. The text generation model is used to merge all thematic texts in the temporary categories into temporary narrative texts based on editing prompts. All key sentences in the temporary narrative text are converted into vector representations and denoted as temporary vectors. For all temporary vectors and any core plot feature in the common narrative features, the maximum cosine similarity between the core plot feature and all temporary vectors is calculated and denoted as the similarity index of any core plot feature. The mean of the similarity indices of all core plot features in the common narrative features is denoted as the similarity between the temporary narrative text and the common narrative features under the division ratio. The division ratio at which the similarity between the temporary narrative text and the common narrative features is maximized is obtained, and the temporary category obtained under this division ratio is taken as the first category.

[0009] Preferably, the specific steps for obtaining the order of thematic texts in the first category in the third narrative text are as follows: Divide the third narrative text into several third text fragments; Each third text fragment of the topic text and the third narrative text in the first category is converted into a vector representation, resulting in a topic text vector and a third narrative fragment vector. For any topic text in the first category, the cosine similarity between the topic text vector and the third narrative fragment vectors of all third text fragments is obtained. The third text fragment with the highest cosine similarity is obtained, and its sorting number in the third narrative text is obtained as the number of the topic text. For all topic texts in the first category, the order of the numbers from smallest to largest is used as the recording order.

[0010] Preferably, the specific steps for obtaining the division ratio at which the similarity between the temporary narrative text and the common narrative features is maximized, and the temporary category obtained under this division ratio is used as the first category, are as follows: Multiple values ​​are preset for the division ratio N. For the similarity and temporary category obtained under each division ratio value, the first total duration of the video segments corresponding to all topic texts in the temporary category is obtained. The user-preset editing duration is called the reference duration. The value index of each division ratio is obtained. This value index is positively correlated with the similarity corresponding to each division ratio value and negatively correlated with the difference between the first total duration and the reference duration. Among all possible division ratios, the division ratio with the largest value is selected, and the temporary category obtained under this division ratio is taken as the first category.

[0011] Preferably, the specific steps of dividing the first narrative text into several first text segments are as follows: The first narrative text is divided into several sub-statements using any punctuation mark as a separator; each sub-statement is the first text segment, and the several sub-statements following each sub-statement and the punctuation marks between them constitute a first text segment.

[0012] Preferably, the specific steps involved in dividing all video materials into several video segments are as follows: Each video clip is divided into several video segments based on all scene transition points; the method for obtaining scene transition points is as follows: For the i-th frame in each video clip, calculate the difference in grayscale histogram curves between the i-th frame and the (i+k)-th frame. When the difference exceeds a preset segmentation threshold, the (i+k)-th frame is determined to be a scene switching point, where k is a preset integer.

[0013] Another embodiment of the present invention provides an automatic video footage editing and splicing system based on a large model. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it implements all the steps of the above-described automatic video footage editing and splicing method based on a large model.

[0014] The beneficial effects of the technical solution of the present invention are: This invention achieves the transformation of video content into a text modality by dividing video footage into several video segments and extracting thematic text using a graph-to-text model, providing a unified data foundation for subsequent processing. Based on the graph-to-text model, all thematic texts are merged into a first narrative text, generating a complete narrative framework representing all materials that may form under user intent. By calculating the importance of thematic texts within the first narrative text, preliminary screening and sorting of massive amounts of video segments are achieved. By assigning the least important thematic texts to Category I and the rest to Category II, the importance of Category I texts is dynamically adjusted based on the textual differences between the merged second narrative text and the first narrative text. This provides a calibration mechanism for importance assessment, correcting the inaccuracies caused by interference from irrelevant segments when considering all thematic texts at once.

[0015] By repeating the importance-increasing process and extracting common narrative features from all second narrative texts, the unchanging core elements of user needs can be abstracted from the dynamic iteration process. All thematic texts are divided into two categories based on importance, ensuring that the third narrative text formed by merging all thematic texts within the first category has the greatest similarity to the common narrative features. This guarantees that the final selected core material set not only has high individual segment importance, but also that the complete narrative constructed as a whole best matches the most stable and core elements of the user's intent. Video segments are spliced ​​according to the order in which thematic texts in the first category are recorded in the third narrative text, achieving automatic execution from narrative planning in the text space to video data space, ultimately completing the automated editing into a finished film.

[0016] Overall, this invention solves the technical problem that large models cannot directly and accurately identify the most relevant segments to the core narrative from massive amounts of raw material and plan a coherent editing sequence. It achieves the technical effect of automatically selecting key segments from massive amounts of video material and generating video clips that meet user needs while ensuring narrative coherence and intent alignment. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of an automatic video footage editing and splicing method based on a large model, as provided in an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the automatic video material editing and splicing method and system based on a large model proposed in this invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details the specific solution of the automatic video material editing and splicing method and system based on a large model provided by this invention.

[0022] Please see Figure 1 The diagram illustrates a flowchart of an automatic video footage editing and splicing method based on a large model, according to an embodiment of the present invention. The method includes the following steps: Step S101: Divide all video materials into several video segments, and use the image-to-text model to extract the theme text of each video segment; use the text-to-text model to merge all theme texts into the first narrative text based on editing prompts; calculate the importance of any theme text in the first narrative text.

[0023] This embodiment performs structured segmentation of original long video footage or a large number of independent video footage to form discrete, independently analyzable video segments. This avoids the technical problem of large models processing all video content that lasts for several hours and contains hundreds of segments at once, and provides an operable data foundation for subsequent video editing and splicing.

[0024] Furthermore, this embodiment utilizes a graph-to-text model to generate a thematic text for each video segment, summarizing its core actions, intentions, conflicts, and main subjects. This process transforms the video content into a unified text modality that is easily understood and processed by the large model, avoiding the problem of excessive complexity when directly processing video data.

[0025] Furthermore, this embodiment, based on user-provided editing prompts, uses a text-to-text model to merge all thematic texts and generate a first narrative text. The process, through the integrated understanding of the semantics of all materials by the large model, generates a complete story draft that best suits the user's needs, representing the complete narrative expression framework that all current video materials can potentially form under the user's guidance.

[0026] Furthermore, this embodiment calculates the importance of any given topic text within the first narrative text. This process, by quantifying the contribution of the semantic summary of each video segment to the overall narrative draft, facilitates the initial screening and sorting of massive amounts of video segments. Importance, as a quantitative indicator, indicates that the higher the value, the more crucial the corresponding video segment is in expressing the user's intent, and the more relevant it is to the core narrative among the vast amount of original material.

[0027] Step S102: Record the few thematic texts with the least importance as Class 1 texts and the rest as Class 2 texts. Based on the textual differences between the second narrative text formed by the fusion of the two classes of texts and the first narrative text, increase the importance of the thematic texts in Class 1 texts. The amount of increase is positively correlated with the textual differences. Record the second narrative text.

[0028] Based on the preliminary importance assessment results, this embodiment divides the subject text into two sets of object objects: Category I text (low importance) and Category II text (high importance). This division operation achieves a preliminary classification of the material, temporarily separating potentially redundant or secondary narrative elements from the core narrative elements.

[0029] This embodiment uses only thematic texts classified as Category II (high importance) to re-merge and generate the second narrative text. This process not only ensures that the second narrative text best meets the user's needs, but also eliminates interference from the initially determined low-importance thematic texts (redundant thematic texts), allowing the generated second narrative text to more purely reflect the narrative content that can be constructed from the currently considered key materials.

[0030] This embodiment calculates the textual differences between the second narrative text and the first narrative text. These textual differences reflect the degree of deviation between the generated narrative and the original narrative containing all material, after excluding initially identified low-importance texts. A greater textual difference indicates a greater impact of the excluded texts on the overall narrative.

[0031] Based on this, this embodiment dynamically adjusts the importance of thematic texts within a category of texts (low-importance texts) based on textual differences, with the adjustment amount being negatively correlated with the textual differences. This process addresses the issue that the importance assessment (i.e., the importance obtained in step S101) might be inaccurate due to interference from irrelevant fragments when considering all thematic texts at once in step S101. It provides a calibration mechanism: when textual differences are large, it indicates that the excluded low-importance thematic texts actually significantly affect the narrative, therefore their importance may be underestimated to a greater extent, and thus the increase should be larger; conversely, when textual differences are small, it indicates that these low-importance thematic texts are indeed redundant, and their importance is unlikely to be overestimated, therefore the increase should be smaller.

[0032] The generated second narrative text is saved. Combined with subsequent steps, it can be seen that this operation preserves the narrative versions generated each time based on the results of different importance filtering, thus accumulating the necessary data foundation for the subsequent extraction of stable common narrative features.

[0033] Step S103: After repeating the importance increase process several times, extract the common narrative features of all recorded second narrative texts, and divide all thematic texts into two categories according to importance. The category with the highest average importance is recorded as the first category, so that the third narrative text formed by merging all thematic texts in the first category has the greatest similarity with the common narrative features.

[0034] This step executes step S102 multiple times, undergoing multiple iterations to achieve automatic screening and importance calibration of materials under the constraint of narrative consistency. This iterative process allows the importance assessment to stabilize through continuous comparison and correction, and generates multiple versions of secondary narrative texts tailored to user needs, generated from highly important texts, thus providing a data foundation for extracting stable narrative patterns.

[0035] Furthermore, this embodiment extracts common narrative features from all the second narrative texts recorded in multiple iterations. These common narrative features represent the most core and stable narrative elements (such as core plots, emotional tone, key objects, etc.) that consistently exist in the narrative constructed from highly important materials after multiple screenings and calibrations. This process abstracts the unchanging core parts of user needs from the dynamic iteration process.

[0036] Furthermore, based on the final importance values ​​after multiple rounds of revisions, all thematic texts were reclassified into two categories. This classification, based on the calibrated importance index, aims to distinguish all materials into different value sets according to a new, more accurate metric. The category with the highest average importance among the two categories is designated as the first category. This selection criterion ensures that the selected category set contains materials with the highest overall average importance, meaning that statistically, this set includes the most narratively valuable materials. Simultaneously, the third narrative text generated from the fusion of all thematic texts within the first category must have the highest similarity to the previously extracted common narrative features. This matching operation ensures that the final selected core material set (first category) not only has high individual fragment importance, but also that the complete narrative constructed as a whole aligns maximally with the most stable and core elements of the user's intent. Maximizing the similarity between the third narrative text and the common narrative features guarantees the narrative coherence and intent alignment of the final editing scheme.

[0037] Step S104: According to the order in which the thematic texts in the first category are recorded in the third narrative text, the video clips corresponding to the thematic texts in the first category are spliced ​​together.

[0038] This embodiment uses the third narrative text as the narrative blueprint for the final edit, determining the order in which the various thematic texts belonging to the first category appear. This order represents the optimal sequence of video clips planned according to the core narrative logic (the third narrative text), which directly determines the story flow and rhythm of the final generated video.

[0039] Following this order, the video clips corresponding to the thematic texts in the first category are spliced ​​together on the timeline. This operation maps and executes the narrative planning and material selection results in the text space onto the original video data space, ultimately completing the automated editing into a finished product. It achieves the goal of automatically selecting key segments from massive amounts of video material and generating a finished video that meets user needs while ensuring narrative coherence and consistency with intent.

[0040] As a preferred example, dividing all video footage into several video segments includes the following methods: Shot boundary detection technology is used to capture scene transition points in each video clip, and each video clip is divided into several video segments based on all scene transition points. In some examples, video segments shorter than two seconds are deleted.

[0041] As an example, lens boundary detection technology specifically includes: Calculate the feature differences between different frames. When the feature difference is greater than the preset segmentation threshold, it is determined as a scene switching point (scene switching).

[0042] For example, for the i-th frame in each video clip, the feature difference between the i-th frame and the (i+k)-th frame is calculated. When the feature difference exceeds a preset segmentation threshold, the (i+k)-th frame is determined to be a scene transition point, where the (i+k-1)-th frame is classified as the previous video segment, and the (i+k)-th frame is classified as the next video segment. The preferred value of k is an integer between 1 and 30. This example uses k=10 for illustration.

[0043] Specifically, if there are fewer than k frames after the i-th frame, the scene transition point will not be obtained again.

[0044] As an example, the method for obtaining feature differences is as follows: convert two frames of images to grayscale, calculate the grayscale histogram curve of each frame, and record the difference between the histogram curves as the feature differences.

[0045] As an example, the method for obtaining the difference between histogram curves is as follows: each histogram curve represents the probability of each gray level occurring (in this example, each gray level contains 5 gray values). The probabilities of all gray levels are concatenated into a vector, and the cosine distance between the vectors corresponding to the histogram curves is recorded as the difference between the histogram curves. The segmentation threshold is set to 0.65.

[0046] It should be noted that the above is a simple and efficient method for segmenting video footage. Since there are other known methods for implementing shot boundary detection technology, such as video plot segmentation based on Scene-VLM or TransNetV2 models, other methods can be used for video footage segmentation in other embodiments, and this embodiment does not limit them.

[0047] Additionally, it should be noted that in some embodiments, for video footage marked as containing conversation or speech content, the speech in the video footage is segmented using a Voice Activity Detection (VAD) algorithm, and the video frame corresponding to each segmented speech segment is considered as a video segment. The Voice Activity Detection algorithm is a well-known technology and will not be described in detail in this embodiment.

[0048] As a preferred example, the graph-to-text large model uses Gemini 2.5 Pro (multimodal large model). The system prompts for this graph-to-text large model mainly include the following: "The thematic text must strictly contain the following elements, strung together in concise language into a single paragraph: 1. Core Subject: The main character, object, or focus of the video; 2. Key Action / Event: What the subject is doing and the key changes occurring; 3. Context / Environment: The location, setting, or background of the event; 4. Potential Conflict or Intent: (If inferred) The subject's goal, the difficulties or obstacles encountered, and the emotional tone; 5. Significant Visual Features: (If critical) Mention 1-2 visual elements crucial to understanding the content, such as close-ups, dim lighting, rapid cuts, etc. Ensure the thematic text is: 50-150 words in length, avoids subjective evaluations and detailed listings, uses the present tense, and provides clear semantic anchors for subsequent narrative integration."

[0049] Other graphic-text models can be used in other embodiments, and other prompt words can be designed, or the prompt words can be optimized and adjusted based on the above prompt words. This embodiment does not impose any limitations.

[0050] As a preferred example, the Wenshengwen large model uses the DeepSeek-V3 large model. In this example, the system prompts of the Wenshengwen large model mainly include the following: Input Information: 1. Clipping Hints (User Instructions): {Insert user-provided clipping hints here}; Output Requirements: 1. Comprehensive Coverage: The generated narrative must cover all core events, main subjects, and key situations in the provided thematic text. No core information from any segment should be omitted. 2. Logical Coherence: Organically connect the information from these segments to form a logically coherent (e.g., chronological, causal, thematic progression) complete story. Implicit connections between segments are permissible, but characters or fictional events not present in the input are not allowed. 3. Intent Alignment: The style, focus, and emotional tone of the entire narrative must closely align with the user-provided clipping hints. 4. Complete Structure: The narrative should have a clear beginning (background introduction), development (conflict / process unfolding), and ending (result / elevation). 5. Concise Language: Use vivid and concise written language, avoiding length and repetition. Directly output the final narrative text without additional explanation.

[0051] Other text-to-text models can be used in other embodiments, and other prompt words can be designed, or the prompt words can be optimized and adjusted based on the above prompt words. This embodiment does not impose any limitations.

[0052] As a preferred example, the methods for calculating the importance of any thematic text within the first narrative text include: The first narrative text is divided into several first text fragments.

[0053] For any given topic text, convert each first text segment of both the topic text and the first narrative text into vector representations, resulting in a topic text vector and a first narrative segment vector. Calculate the cosine similarity between the topic text vector and each first narrative segment vector in the first narrative text. The maximum cosine similarity between the topic text vector and all first narrative segment vectors in the first narrative text is denoted as the contribution of the topic text to the first narrative text.

[0054] The contribution of all thematic texts to the first narrative text is normalized (this example uses the softmax formula for normalization). The normalized contribution is used as the importance of each thematic text in the first narrative text.

[0055] It should be noted that the vector representation of the text is implemented using a text encoder based on the Transformer architecture (such as BERT). For a Transformer-based encoder, the dimension of the output vector is 768.

[0056] In other embodiments, the vector representation of the text can employ text encoders with other Transformer architectures (such as RoBERTa), or vectorization methods based on word frequency statistics (such as TF-IDF, bag-of-words models, etc.). For statistical methods, the vector dimension is the size of the vocabulary.

[0057] As an example, dividing the first narrative text into several first text segments includes the following methods: The first narrative text is divided into several sub-statements using any punctuation mark as a separator; starting with each sub-statement, several (e.g., 5) sub-statements are obtained after each sub-statement. These sub-statements and the punctuation marks between them constitute a first text segment (containing each of the aforementioned sub-statements).

[0058] This process ensures that adjacent first text segments share common sub-statements.

[0059] Specifically, if there are fewer than 5 sub-statements following each sub-statement, all sub-statements following each sub-statement are treated as a first text segment.

[0060] As a preferred example, for all topic texts, the 40% of topic texts with the lowest importance (rounded down) are grouped into one category. Specifically, if the total number of topic texts is less than 5, then it is not necessary to run all embodiments of this invention; the text can be manually edited and spliced ​​into a single image.

[0061] As a preferred example, based on the textual differences between the second narrative text (formed by merging two types of texts) and the first narrative text, the importance of thematic texts within the first type of text is increased. This includes methods such as: For all Category II texts (high-importance topic texts), a second narrative text is generated by fusing all Category II texts based on clipping cue words using a text-to-text model. This process is the same as generating the first narrative text. The textual differences between the second and first narrative texts are calculated, reflecting the degree of change in narrative content after excluding Category I texts. The importance of each topic text in Category I is adjusted based on these textual differences. The adjusted importance equals the original importance (i.e., importance before adjustment) plus the normalized result of the increase, where the increase is positively correlated with the textual differences. Specifically, the increase is added to the original importance of each topic text in Category I to obtain the first coefficient of each topic text in Category I. Correspondingly, the importance of each topic text in Category II is set as the first coefficient of each topic text in Category II. Min-max normalization is performed on all first coefficients in both Category I and Category II texts. The normalized first coefficient of each topic text in Category I is the adjusted importance, and correspondingly, the normalized first coefficient of each topic text in Category II is also used as the adjusted importance.

[0062] In subsequent processes, regardless of whether it is a Class I or Class II text, the calculations are based on the adjusted importance.

[0063] Specifically, when all the first coefficients are the same, it indicates that this embodiment is malfunctioning. For example, the topic text or the first and second narrative texts generated by the large model are abnormal. In this case, this embodiment will stop running and an error message "data abnormal" will be returned.

[0064] As an example, calculating the textual differences between the second narrative text and the first narrative text involves the following steps: The second narrative text is divided into several second text fragments, and the specific process is the same as that for dividing the first text fragments.

[0065] Convert all second text fragments into vector representations to obtain vectors for all second narrative fragments. The minimum cosine distance between each second narrative fragment vector and all first narrative fragment vectors is denoted as the local difference of each second text fragment. A larger value indicates a significant difference between the local fragment in the second narrative text and the smallest difference in the first narrative text; a smaller value indicates a smaller difference between the local fragment in the second narrative text and a specific local fragment in the first narrative text. The mean of the local differences of all second text fragments is denoted as the text difference. A larger value indicates a significant difference between the second and first narrative texts, or a significant change in time between them.

[0066] As an example, the increase is equal to the product of the original importance and the text difference.

[0067] As another example, calculating the textual differences between the second narrative text and the first narrative text includes the following steps: The first and second narrative texts are directly converted into vector representations, resulting in a first vector and a second vector, respectively. The cosine distance between the first and second vectors is taken as the text difference.

[0068] Although this example does not take into account the local differences between the second and first narrative texts, the calculation process is relatively simple.

[0069] As an example, the number of iterations in step S102 is set to 5 to 8 times. This embodiment will describe the process with 6 iterations as an example.

[0070] As a preferred example, the common narrative features of all recorded second narrative texts are extracted, including the following methods: For all secondary narrative texts, the TextRank algorithm is used to extract the n1 key sentences with the highest weights from each secondary narrative text, resulting in a total of n1×n0 sentences (n0 being the number of all secondary narrative texts). These sentences are then converted into vector representations using the BERT model, denoted as key sentence vectors. The K-means clustering algorithm is used to cluster all key sentence vectors, and the mean of all key sentence vectors in each cluster is considered the most representative sentence, representing the core plot feature. The set of core plot features from all clusters is then considered the common narrative feature.

[0071] As an example, the preferred value range for n1 is 3 to n2, where n2 represents 75% (rounded up) of the number of key sentences in each second narrative text. This example uses n1=4 as an example. Specifically, if n2 is less than n1, then n1=n2. It should be noted that this embodiment uses punctuation marks such as semicolons, periods, and exclamation marks as sentence endings, thus splitting each second narrative text into multiple key sentences. Specifically, when n2=0, this embodiment will also stop running and return a "data error" message.

[0072] In other examples, key sentences with lower weights (e.g., less than 0.05) may be removed before extracting the n1 key sentences with the highest weights.

[0073] The TextRank algorithm is a well-known technique, and the weights mentioned are also obtained by the TextRank algorithm. This embodiment will not describe them in detail.

[0074] As an example, in the K-means clustering algorithm, the optimal value for the number of clusters K is between 5 and 15. This embodiment uses K=10 as an example. Specifically, if the total number of key sentence vectors (i.e., n1×n0) is less than K, then the set of all key sentence vectors is taken as the common narrative feature.

[0075] As a preferred example, all topic texts are divided into two categories based on importance. The category with the highest average importance is designated as the first category. This ensures that the third narrative text, formed by merging all topic texts within the first category, has the greatest similarity to common narrative features. The methods include: For all thematic texts, the most important N×100% of thematic texts are designated as temporary categories, where N represents the preset division ratio. The number of thematic texts in each temporary category is rounded down. Using a text-to-text model based on editing prompts, all thematic texts within the temporary categories are merged into a temporary narrative text.

[0076] For all key sentences in the temporary narrative text, the BERT model is used to convert these key sentences into vector representations, denoted as temporary vectors. For all temporary vectors, and for any core plot feature among the common narrative features, the maximum cosine similarity between the core plot feature and all temporary vectors is calculated, denoted as the similarity index of any core plot feature. The larger the similarity index, the more likely the core plot represented by the core plot feature is to appear in the temporary narrative text. The mean of the similarity indices of all core plot features among the common narrative features is denoted as the similarity between the temporary narrative text and the common narrative features under the partitioning ratio. The larger the similarity, the more likely the temporary narrative text is to contain all the core plots.

[0077] Thus, given a partitioning ratio, a similarity between the temporary narrative text and the shared narrative features can be obtained. The partitioning ratio that maximizes the similarity between the temporary narrative text and the shared narrative features is denoted as N0, and the temporary category obtained under this partitioning ratio N0 is taken as the first category. The result of fusing all thematic texts within this first category is denoted as the third narrative text. This third narrative text has the maximum similarity to the shared narrative features, meaning that this third narrative text contains, to the greatest extent possible, all the core plot points.

[0078] As an optional example, the method for obtaining the partition ratio when the similarity between temporary narrative text and shared narrative features is maximized includes: Multiple values ​​are preset for the division ratio N, namely 0.9, 0.8, 0.7, 0.6, and 0.5. According to the above method, each value of the division ratio corresponds to a similarity, and the division ratio with the highest similarity among all the values ​​is selected as N0.

[0079] In special cases, if there is more than one partition ratio with the highest similarity, then the partition ratio with the largest value is taken as N0.

[0080] In special cases, if all the similarity values ​​corresponding to the division ratios are the same, it means that all the video materials cannot be edited and spliced ​​together to form the video required by the user. In this case, the user should be reminded that they need to manually edit and splice the materials into a single video.

[0081] As a preferred example, other methods for obtaining the partitioning ratio that maximizes the similarity between temporary narrative texts and shared narrative features include: Multiple values ​​are preset for the division ratio N. Following the method described above, each division ratio value corresponds to a similarity and a temporary category. The first total duration of the video clips corresponding to all topic texts within this temporary category is obtained. The user-preset editing duration is called the reference duration (e.g., 3 minutes). The value index for each division ratio is obtained. This value index is positively correlated with the similarity corresponding to each division ratio value and negatively correlated with the difference between the first total duration and the reference duration.

[0082] The larger the value, the more likely the temporary category is to contain all the core plot points, and the more the final episode length meets the user's needs.

[0083] Among all possible values ​​for the division ratio, the division ratio with the largest value is selected as N0.

[0084] In special cases, if there is more than one largest division ratio for a given indicator, then the division ratio with the largest value will be taken as N0.

[0085] As an example, the formula for calculating the value index Y is: Y=y1×exp(-|t0-t| / t0), where y1 represents similarity, |t0-t| / t0 represents the difference between the first total duration t and the reference duration t0, t0 is used as the denominator to remove dimensions, and exp() represents an exponential function with the natural constant as the base.

[0086] This example takes into account all the core plot points while trying to match the user's desired runtime.

[0087] As a preferred example, the video clips corresponding to the thematic texts in the first category are spliced ​​together according to the order in which the thematic texts in the first category are recorded in the third narrative text. The methods include: The third narrative text is divided into several third text fragments (the same process as obtaining the first text fragments is used); all third text fragments are numbered with integers starting from 1 according to the order in which they appear in the third narrative text, that is, they are numbered 1, 2, 3, ... respectively.

[0088] Each third text fragment in the first category of topic text and third narrative text is converted into a vector representation, resulting in topic text vector and third narrative fragment vector.

[0089] For any topic text in the first category, obtain the cosine similarity between the topic text vector of the topic text and the third narrative fragment vectors of all third text fragments, obtain the third text fragment with the highest cosine similarity, and obtain the number of the third text fragment in the third narrative text, which is used as the number of the topic text.

[0090] For all the topic texts in the first category, arrange the video clips corresponding to all the topic texts in the first category in ascending order of the number (that is, the recording order) to obtain a video clip sequence.

[0091] To stitch together a sequence of video clips.

[0092] In special cases, if multiple topic texts have the same number, only the topic text with the highest importance will be retained.

[0093] In other examples, for multiple topic texts with the same number, they can be randomly sorted, or the user can be prompted to sort or filter the text to be retained.

[0094] As an example, methods for stitching together a sequence of video clips into a single image include: The video clips are spliced ​​together on the timeline according to the order of the video clips in the video clip sequence. During splicing, the image transition effect and audio transition effect between adjacent clips are fade-in and fade-out.

[0095] In other examples, SORA's large model's Blend feature can also be used to merge image frames between adjacent segments.

[0096] Another embodiment of the present invention provides an automatic video footage editing and splicing system based on a large model. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it implements all the methods of all the embodiments described above.

[0097] It should be noted that the system also includes a cloud server, on which all the large models, such as the image-to-text large model and the text-to-text large model, run. Specifically, when the computer program runs, it uploads relevant data (such as video clips, thematic text, etc.) to the cloud server, and the large models on the cloud server send the generated data (such as thematic text, first narrative text, second narrative text, third narrative text, etc.) back to the computer program.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatic splicing of video footage based on a large model, characterized in that, The method includes the following steps: D1: Divide all video footage into several video segments, and use the image-to-text model to extract the thematic text of each video segment; use the text-to-text model to merge all thematic texts into the first narrative text based on editing prompts; calculate the importance of any thematic text in the first narrative text. D2: The least important thematic texts are categorized into Class 1 texts, and the rest are categorized into Class 2 texts. Based on the textual differences between the second narrative text formed by the fusion of Class 2 texts and the first narrative text, the importance of the thematic texts in Class 1 texts is increased. The increase is positively correlated with the textual differences. The second narrative text is then recorded. D3: After repeating D2 several times, extract the common narrative features of all recorded second narrative texts, and divide all thematic texts into two categories according to their importance. The category with the highest average importance is recorded as the first category, so that the third narrative text formed by merging all thematic texts in the first category has the greatest similarity to the common narrative features. D4: According to the order in which the thematic texts in the first category are recorded in the third narrative text, splice together the video clips corresponding to the thematic texts in the first category.

2. The automatic splicing method for video footage editing based on a large model according to claim 1, characterized in that, The specific steps involved in calculating the importance of any thematic text within the first narrative text are as follows: The first narrative text is divided into several first text fragments; For any given topic text, convert each first text segment of the topic text and the first narrative text into vector representations, resulting in topic text vectors and first narrative segment vectors. Calculate the cosine similarity between the topic text vector and each first narrative segment vector in the first narrative text. The maximum cosine similarity between the topic text vector and all first narrative segment vectors in the first narrative text is recorded as the contribution of the topic text to the first narrative text. Normalize the contribution of all topic texts to the first narrative text, and use the normalized contribution as the importance of each topic text in the first narrative text.

3. The automatic splicing method for video footage editing based on a large model according to claim 2, characterized in that, The specific steps for obtaining the textual differences between the second narrative text and the first narrative text are as follows: The second narrative text is divided into several second text fragments; Convert all second text fragments into vector representations to obtain all second narrative fragment vectors; the minimum cosine distance between each second narrative fragment vector and all first narrative fragment vectors is denoted as the local difference of each second text fragment, and the mean of the local differences of all second text fragments is denoted as the text difference.

4. The automatic splicing method for video footage editing based on a large model according to claim 1, characterized in that, The specific steps for extracting the common narrative features of all second narrative texts in the records are as follows: For all second narrative texts, extract the key sentences with the highest weights from each second narrative text using the TextRank algorithm, convert all the key sentences extracted from all second narrative texts into vector representations, and denote them as key sentence vectors; cluster all key sentence vectors, and denote the mean of all key sentence vectors in each cluster as the core plot feature; The set of core plot features of all clusters is taken as the common narrative feature.

5. The automatic splicing method for video footage editing based on a large model according to claim 4, characterized in that, The process of dividing all thematic texts into two categories based on importance, with the category having the highest average importance value designated as the first category, and ensuring that the third narrative text formed by merging all thematic texts within the first category has the greatest similarity to common narrative features, includes the following specific steps: For all thematic texts, the most important N×100% of thematic texts are taken as temporary categories, where N represents the preset division ratio. The text generation model is used to merge all thematic texts in the temporary categories into temporary narrative texts based on editing prompts. All key sentences in the temporary narrative text are converted into vector representations and denoted as temporary vectors. For all temporary vectors and any core plot feature in the common narrative features, the maximum cosine similarity between the core plot feature and all temporary vectors is calculated and denoted as the similarity index of any core plot feature. The mean of the similarity indices of all core plot features in the common narrative features is denoted as the similarity between the temporary narrative text and the common narrative features under the division ratio. The division ratio at which the similarity between the temporary narrative text and the common narrative features is maximized is obtained, and the temporary category obtained under this division ratio is taken as the first category.

6. The automatic splicing method for video footage editing based on a large model according to claim 1, characterized in that, The specific steps for obtaining the order of the thematic texts in the first category within the third narrative text are as follows: Divide the third narrative text into several third text fragments; Each third text fragment of the topic text and the third narrative text in the first category is converted into a vector representation, resulting in a topic text vector and a third narrative fragment vector. For any topic text in the first category, the cosine similarity between the topic text vector and the third narrative fragment vectors of all third text fragments is obtained. The third text fragment with the highest cosine similarity is obtained, and its sorting number in the third narrative text is obtained as the number of the topic text. For all topic texts in the first category, the order of the numbers from smallest to largest is used as the recording order.

7. The automatic splicing method for video footage editing based on a large model according to claim 5, characterized in that, The specific steps for obtaining the division ratio that maximizes the similarity between temporary narrative text and shared narrative features, and using the temporary category obtained under this division ratio as the first category, are as follows: Multiple values ​​are preset for the division ratio N. For each value of the division ratio, the similarity and temporary category obtained are obtained. The first total duration of the video segments corresponding to all topic texts in the temporary category is obtained. The user-preset editing duration is called the reference duration. Obtain the value index for each division ratio. The value index is positively correlated with the similarity corresponding to the value of each division ratio and negatively correlated with the difference between the first total duration and the reference duration. Among all possible division ratios, the division ratio with the largest value is selected, and the temporary category obtained under this division ratio is taken as the first category.

8. The automatic splicing method for video footage editing based on a large model according to claim 2, characterized in that, The specific steps involved in dividing the first narrative text into several first text fragments are as follows: The first narrative text is divided into several sub-statements using any punctuation mark as a separator; each sub-statement is the first text segment, and the several sub-statements following each sub-statement and the punctuation marks between them constitute a first text segment.

9. The automatic splicing method for video footage editing based on a large model according to claim 1, characterized in that, The specific steps involved in dividing all video footage into several video segments are as follows: Each video clip is divided into several video segments based on all scene transition points; the method for obtaining scene transition points is as follows: For the i-th frame in each video clip, calculate the difference in grayscale histogram curves between the i-th frame and the (i+k)-th frame. When the difference exceeds a preset segmentation threshold, the (i+k)-th frame is determined to be a scene switching point, where k is a preset integer.

10. An automatic video footage editing and splicing system based on a large model, the system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor runs the computer program, it implements all the steps of the automatic splicing method for video material editing based on a large model as described in any one of claims 1 to 9.

Images