A narrative video mixing and splicing method, system and device based on ABRoll

Abstract

This application relates to the field of computer technology and provides a narrative video montage method based on ABRoll, comprising: performing temporal deep semantic analysis on the main video to generate a semantic timeline; dynamically identifying key time points based on the semantic timeline, wherein the key time points are the time points when B-Roll materials are inserted into the main video; for each key time point, matching B-Roll materials from the B-Roll material library in combination with contextual semantics; and inserting the B-Roll materials into the corresponding key time points according to a preset video style template to generate a montage video. This application eliminates the need for creators to manually perform editing tasks such as content analysis, material search, timing, and effect adjustment, thereby improving overall work efficiency.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a narrative video mixing method, system and device based on ABRoll. Background Technology

[0002] With the rapid development of the self-media industry, narrative videos such as explanations, lectures, and interviews have emerged. In order to enhance the viewing experience and narrative flow of these narrative videos, video producers usually intersperse relevant materials in the middle of the video. For this type of video creation, A / B Roll video editing technology is usually used, which uses the narrative video as the main video A-Roll and then intersperses relevant auxiliary footage B-Roll.

[0003] Existing A / B Roll editing technology typically requires video editors to manually select, edit, and insert B-Roll footage during video production. This process is not only time-consuming but also requires professional video editing skills and artistic judgment. However, with the explosive growth in demand for video content creation, the traditional manual A / B Roll video production method can hardly meet the needs of high-efficiency, large-scale video production. In particular, for content creators and small production teams, this manual operation has become a bottleneck in their workflow.

[0004] Therefore, there is an urgent need for a method that can automatically analyze video content, intelligently match materials, and generate high-quality mashup videos. Summary of the Invention

[0005] This application provides a narrative video mixing method, system, and device based on ABRoll, which can solve one of the problems of the prior art mentioned above.

[0006] In a first aspect, embodiments of this application provide a narrative video editing method based on ABRoll, including: Perform temporal deep semantic analysis on the main video to generate a semantic timeline; Based on the semantic timeline, key time points are dynamically identified, and the key time points are the time points in the main video when B-Roll material is inserted. For each of the aforementioned key time points, B-Roll materials are matched from the B-Roll material library based on the contextual semantics. Based on the preset video style template, the B-Roll footage is inserted into the corresponding key time points to generate a montage video.

[0007] Furthermore, the step of performing temporal deep semantic analysis on the main video to generate a semantic timeline includes: The main video is deconstructed to obtain a video frame sequence and an audio track, and speech recognition is performed on the audio track to generate a text sequence; The video frame sequence and the text sequence are jointly analyzed to generate multimodal semantic units, where each multimodal semantic unit corresponds to a video time segment, and a multimodal semantic unit sequence is generated. Based on the multimodal semantic unit sequence, semantic association and evolution between units are performed to identify the narrative structure nodes of the main video; By combining the multimodal semantic units and the narrative structure nodes, a semantic timeline is generated. The semantic timeline contains multiple time units, and each time unit carries a semantic description, which includes visual scene, key entities, text content summary, sentiment polarity, and narrative function tags.

[0008] Furthermore, the dynamic identification of key time points based on the semantic timeline, wherein the key time points are the times when B-Roll footage is inserted into the main video, includes: On the semantic timeline, the end timestamp of each multimodal semantic unit is used as a candidate key time point. Based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel. The insertion potential scores of each dimension are input into a decision fusion model for weighted summation to generate a comprehensive insertion recommendation score for each candidate key time point; Based on the time series of the comprehensive insertion recommendation scores, combined with the dynamic recommendation threshold, local peak points are identified; The local peak points are used as key time points for inserting B-Roll material, where each key time point is associated with the dominant insertion reason type.

[0009] Furthermore, on the semantic timeline, using the end timestamp of each multimodal semantic unit as a candidate key time point, and based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel, including: For candidate key time points that overlap with the end timestamp of a time unit, the corresponding time unit is used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively. For other candidate key time points, the corresponding multimodal semantic units are used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively.

[0010] Furthermore, the calculation of insertion potential scores from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension includes: In the semantic transformation dimension, the semantic similarity between adjacent time units or adjacent multimodal semantic units is calculated to generate a semantic transformation potential score; In the concept emphasis dimension, a concept strength potential score is generated based on the degree of repetition of entities in speech, the frequency of repetition in text, and the salience in vision in each of the time units or the multimodal semantic units. In the visual attention dimension, based on a pre-trained visual attention model, the video frame sequence of the time unit or the multimodal semantic unit is analyzed to predict the audience's attention distribution and calculate the visual attention potential score.

[0011] Furthermore, for each of the key time points, the matching of B-Roll content from the B-Roll content library, in conjunction with contextual semantics, includes: Based on the insertion reason type at the key time point, relevant key entities and visual scenes are extracted from the corresponding unit, and combined with the emotional polarity of the corresponding unit and the screen ratio of the main video, a material query vector is generated. Based on the material query vector, a search is performed in the preset B-Roll material library, and the matching score of each candidate material is calculated; The highest matching score among the candidate materials is compared with a preset matching threshold to select the B-Roll material.

[0012] Furthermore, the step of comparing the highest matching score among the candidate materials with a preset matching threshold to select the B-Roll material includes: If the highest matching score among the candidate materials is higher than the matching threshold, then the candidate material is selected as a B-Roll material; If the highest matching score among the candidate materials is lower than the matching threshold, the video generation process is triggered. When the video generation process is triggered, the material query vector is converted into video generation prompt words, and the video generation model is called to generate B-Roll video clips, which are then selected as B-Roll materials.

[0013] Furthermore, the step of inserting the B-Roll footage into corresponding key time points according to a preset video style template to generate a montage video includes: Load a preset video style template, which defines a set of compositing rules, including a transition effect rule set, a rhythm density curve, and visual integration rules; Based on the insertion reason type at each key time point in the main video and the video style template, the set of synthesis parameters is dynamically determined. The synthesis parameters include transition features, B-Roll material insertion features, and B-Roll material frame features. Based on the synthesis parameters, the main video and B-Roll footage are aligned on the timeline to generate a montage video.

[0014] Secondly, embodiments of this application provide a video mixing and editing system based on A / B Roll, including: The first processing module is used to perform temporal deep semantic analysis on the main video and generate a semantic timeline. The second processing module is used to dynamically identify key time points based on the semantic timeline, where the key time points are the time points when B-Roll footage is inserted into the main video. The third processing module is used to match B-Roll materials from the B-Roll material library for each of the key time points, taking into account the contextual semantics. The fourth processing module is used to insert the B-Roll material into the corresponding key time points according to the preset video style template, and generate a mashup video.

[0015] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described narrative video mixing method based on ABRoll.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium, including a computer program stored in the computer-readable storage medium, which, when executed by a processor, implements the above-described narrative video mixing method based on ABRoll.

[0017] The beneficial effects of the embodiments in this application compared with the prior art are: This application discloses a narrative video mashup method based on ABRoll. By performing temporal deep semantic analysis on the main video, a structured semantic timeline containing narrative structure nodes is generated. Based on this, key time points are determined by combining the contextual information of each unit in the semantic timeline for the insertion of B-Roll materials. A dominant insertion reason type is determined for each key time point, ensuring that the insertion of B-Roll materials serves and reinforces the narrative logic, rather than arbitrarily interrupting it, thus guaranteeing the narrative coherence of the final video. Simultaneously, the method combines material library retrieval with dynamic material generation. When the B-Roll material library matching is insufficient, the method can automatically call the video generation model to generate the required materials based on the contextual semantics of the corresponding key time points, thereby expanding the application boundaries of this application's video mashup method. Furthermore, during the mashup process, the most suitable transition effects and visual layout synthesis parameters are dynamically selected according to the dominant insertion reason type of each key time point, eliminating the need for manual adjustments. This ensures high-quality output of the mashup video and improves the overall efficiency of video mashup, eliminating the need for creators to manually perform editing tasks such as content analysis, material retrieval, timing, and effect adjustments, thus improving overall work efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a narrative video mixing method based on ABRoll provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a video mixing and editing system based on A / B Roll provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0022] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0024] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0026] Please see Figure 1 As shown, this invention is a narrative video editing method based on ABRoll, comprising the following steps: S100: Perform time-series deep semantic analysis on the main video to generate a semantic timeline; This application generates a structured semantic timeline containing narrative structure nodes by performing temporal deep semantic analysis on the main video. Based on this, it determines key time points by combining the contextual information of each unit of the semantic timeline for the insertion of B-Roll material, and determines the dominant insertion reason type for each key time point. This ensures that the insertion of B-Roll material serves and strengthens the narrative logic, rather than arbitrarily interrupting it, thus guaranteeing the narrative coherence of the final film.

[0027] In some embodiments, step S100 above includes: The main video is deconstructed to obtain a video frame sequence and an audio track, and speech recognition is performed on the audio track to generate a text sequence; The video frame sequence and the text sequence are jointly analyzed to generate multimodal semantic units, where each multimodal semantic unit corresponds to a video time segment, and a multimodal semantic unit sequence is generated. Based on the multimodal semantic unit sequence, semantic association and evolution between units are performed to identify the narrative structure nodes of the main video; By combining the multimodal semantic units and the narrative structure nodes, a semantic timeline is generated. The semantic timeline contains multiple time units, and each time unit carries a semantic description, which includes visual scene, key entities, text content summary, sentiment polarity, and narrative function tags.

[0028] In this embodiment, the user-uploaded main video A-Roll undergoes time-series deep analysis to generate a structured semantic timeline, which is used for subsequent mixing with B-Roll materials. Specifically, the input main video is deconstructed to obtain a video frame sequence and a separated audio track. Speech recognition is performed on the audio track to generate a text sequence with timestamps. During the speech recognition process, the audio track is processed to be more conversational, such as removing interjections like "uh" and "ah," and correcting colloquial words to specific semantic words, such as correcting "this thing" to "this device." Then, the corrected text sequence is re-timestamped with the audio.

[0029] Furthermore, the video frame sequence and text sequence are analyzed collaboratively. Specifically, a visual encoder and a text encoder are used to extract the visual features of the video frame sequence and the text features of the text sequence, respectively. Then, the visual features and text features are input into a Transformer-based cross-modal fusion layer. Through a cross-attention mechanism, the visual features and text features interact and complement each other in time, thereby generating multimodal semantic units.

[0030] Because the semantic structure and narrative rhythm of a video are dynamically changing—for example, a complete "core argument presentation" may only last 10 seconds, while a "background explanation" may last up to 1 minute—if a fixed window is used for video segmentation, complete semantic units may be fragmented, resulting in disjointed semantic descriptions. Alternatively, multiple semantic units of different natures may be mixed, causing confusion in subsequent narrative structure analysis. Therefore, in one embodiment of this application, during the generation of multimodal semantic units, the main video is divided into multiple semantic segments according to sentence boundaries in the text sequence. For each semantic segment, the visual features and text features within it are subjected to cross-modal fusion collaborative analysis. This ensures the matching between the lower-level semantic units and the higher-level narrative structure in the main video, ultimately outputting a semantic fusion vector representing the semantic segment. This semantic fusion vector corresponds to a multimodal semantic unit, and a multimodal semantic unit corresponds to a video time segment based on the time interval to which its semantic segment belongs. Furthermore, the semantic fusion vector of each multimodal semantic unit contains a scene description, entity list, and action event summary for that semantic segment, which is beneficial for the subsequent generation of narrative structure nodes in the main video.

[0031] Furthermore, the multiple multimodal semantic units defined by sentence boundaries are arranged chronologically to generate a multimodal semantic unit sequence. Semantic associations and evolutions between units within this sequence are then performed to determine the narrative structure nodes of the main video. Specifically, the semantic relevance between adjacent and spanning multimodal semantic units in the sequence is calculated, and a semantic association graph is constructed. Based on this graph, key boundary points are identified within the sequence. These key boundary points are located between adjacent units where the semantic relevance is below a preset threshold—that is, where the semantic relevance drops sharply. Finally, one or more consecutive multimodal semantic units between two adjacent key boundary points are aggregated into a narrative segment. For each narrative segment, based on the common content of one or more multimodal semantic units within it and their position in the multimodal semantic unit sequence, the corresponding narrative structure nodes are classified and labeled. These narrative structure nodes include "Introduction," "Background Explanation," "Presentation of Core Argument," "Example / Demonstration," "Argument Transition," and "Conclusion." Understandably, narrative structure nodes can also be dynamically generated by combining the common content of each narrative segment. For example, if the narrative segment is all about artificial intelligence, a narrative structure node for "Artificial Intelligence" can be generated. If the common content of the next narrative segment is about smartphone X1, then a narrative structure node for "Smartphone X1" can be generated accordingly to meet diverse video narrative scenarios.

[0032] Specifically, in one embodiment, after deconstructing and co-analyzing the main video uploaded by the user explaining artificial intelligence, the resulting multimodal semantic unit sequence is as follows: Unit 1. [00:00-00:10] Hello everyone, today we're going to talk about artificial intelligence.

[0033] Unit 2. [00:10-00:30] Artificial intelligence is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence.

[0034] Unit 3. [00:30-00:45] It has made rapid progress in the past few years.

[0035] Unit 4. [00:45-01:00] For example, in the field of image recognition, AI's accuracy has surpassed that of humans.

[0036] Understandably, in the above embodiments, the main video is divided according to sentence boundaries to generate the above four multimodal semantic units. While performing cross-modal fusion for each multimodal semantic unit, a corresponding semantic fusion vector is output. For example, the semantic fusion vector of unit 2 is: {scene description: "frontal shot of the narrator", entity list: ["artificial intelligence"], action: "defining concept"}`. Furthermore, semantic relevance analysis was performed on the sequences of multimodal semantic units. Specifically, units 1, 2, and 3 are highly semantically related, all providing a macro-level introduction to AI. However, there is a clear semantic shift between units 3 and 4, from "macro-level development" to "specific examples." Therefore, a key boundary point was identified between units 3 and 4, and units 1, 2, and 3 were aggregated into a narrative paragraph. Based on its content of "introducing the topic and explaining the definition," it was marked as a "background explanation" node. Unit 4 was treated as a separate paragraph, and based on its content of "providing specific cases," it was marked as an "example / demonstration" node. Finally, the narrative structure nodes composed of the "background explanation" and "example / demonstration" nodes were combined with the various multimodal semantic units to form a semantic timeline.

[0037] In this embodiment, each time unit in the semantic timeline corresponds to a narrative segment, and each time unit is associated with a corresponding semantic description, specifically including visual scene, key entity, text content summary, sentiment polarity, and narrative function tag. Among them, the visual scene is generated by fusing the scene descriptions of each multimodal semantic unit in the narrative segment to generate a scene description after fusion information; the key entity is the key object, character, or concept appearing in the narrative segment, which can be determined based on the entity list of each multimodal semantic unit; the text content summary is the extraction of the text content of each multimodal semantic unit in the narrative segment; the sentiment polarity is obtained by analyzing the sentiment words in the text, as well as the audio tone, speech rate, etc.; and the narrative function tag corresponds to the semantics of the narrative structure nodes of the narrative segment, such as background description, introduction, etc.

[0038] Specifically, in the aforementioned user-uploaded examples of explaining artificial intelligence, the narrative paragraphs generated after aggregation units 1, 2, and 3 form a time unit in the semantic timeline, and are associated with the following semantic descriptions: {"start": "00:00:00", "end": "00:00:45", "visual_scene": "The narrator is shown in a frontal shot." "key_entities": ["Artificial Intelligence", "Explanator"], "text_summary": "Describes the definition and development of artificial intelligence", "sentiment": "positive / expectant", "narrative_function": "Background explanation"}.

[0039] S200. Based on the semantic timeline, dynamically identify key time points, where the key time points are the times when B-Roll material is inserted into the main video. In some embodiments, step S200 above includes: On the semantic timeline, the end timestamp of each multimodal semantic unit is used as a candidate key time point. Based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel. The insertion potential scores of each dimension are input into a decision fusion model for weighted summation to generate a comprehensive insertion recommendation score for each candidate key time point; Based on the time series of the comprehensive insertion recommendation scores, combined with the dynamic recommendation threshold, local peak points are identified; The local peak points are used as key time points for inserting B-Roll material, where each key time point is associated with the dominant insertion reason type.

[0040] In this embodiment, for each multimodal semantic unit divided based on sentence boundaries and each preset time unit generated based on semantic similarity on the semantic timeline, insertion potential scores in multiple dimensions are calculated in parallel, specifically semantic transformation potential score, concept emphasis score and visual attention score. The insertion point and the inserted material are quantified from the semantic transformation dimension, concept emphasis dimension and visual attention dimension.

[0041] Furthermore, the insertion potential scores of each dimension are input into a decision fusion model for weighted summation to generate a comprehensive insertion recommendation score for each time unit. In this decision fusion model, the weights of the insertion potential scores of each dimension are dynamically adjusted according to the video type of the main video and the emotional polarity of the corresponding time unit. For example, for science tutorial videos, the concept emphasis dimension has the highest weight; for travel documentary videos, the semantic transformation dimension has the highest weight; and for emotional narrative videos, the weight of the visual attention dimension needs to be appropriately reduced to avoid disrupting the overall atmosphere of the video.

[0042] Furthermore, the comprehensive insertion recommendation scores of each candidate key time point are arranged according to the time series to generate a comprehensive recommendation score curve. It can be understood that each comprehensive insertion recommendation score corresponds to the end timestamp of the corresponding multimodal semantic unit. Then, local peak points exceeding the dynamic threshold are identified on the comprehensive recommendation score curve and used as key time points for inserting B-Roll material. Each key time point is associated with its dominant insertion reason type.

[0043] Specifically, for local peak points, a sliding window method is used to identify peak points in the comprehensive recommendation score curve, and the height difference between each peak point and its left and right neighboring valleys is calculated as the significance of that peak point. For the dynamic recommendation threshold, it is determined based on the global statistical characteristics of the comprehensive recommendation score curve. For example, in one embodiment, T = μ + k × σ, where μ is the average value of the comprehensive recommendation score curve, σ is its standard deviation, and k is an adjustable sensitivity coefficient. It is understandable that for a video with varied rhythms and frequent climaxes, its comprehensive recommendation score curve fluctuates greatly, and the threshold T will be increased accordingly, thus selecting the most prominent peaks as local peak points. Conversely, for... A video with a smooth rhythm has small curve fluctuations, and the threshold T will be lower. More peak points can be selected for inserting materials to enrich the video. In addition, the higher the overall insertion recommendation score, the more beneficial the insertion of B-Roll materials is to the semantic, conceptual, or visual transformation and supplementation of the main video. Therefore, peak points with a significance higher than the dynamic recommendation threshold are regarded as local peak points, and the corresponding local peak points can be used as key time points for B-Roll material insertion. Specifically, when associating the insertion reason type, the insertion potential score of the corresponding key time point in each dimension is traced back, and the dimension with the greatest contribution is marked as the dominant insertion reason type of that key time point.

[0044] In some embodiments, on the semantic timeline, using the end timestamp of each multimodal semantic unit as a candidate key time point, and based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel, including: For candidate key time points that overlap with the end timestamp of a time unit, the corresponding time unit is used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively. For other candidate key time points, the corresponding multimodal semantic units are used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively.

[0045] In one embodiment of this application, in terms of semantic time, the end timestamp of each multimodal semantic unit is used as a candidate key time point, and the insertion potential score of each candidate key time point is calculated in parallel in multiple dimensions. For the candidate key time points that overlap with the end timestamp of the time unit, they are located at the transition points of different narrative structure nodes in the main video narrative flow. Therefore, when calculating the insertion potential score of each dimension here, the semantic similarity between time units, the emphasis intensity of entities, and the video frame sequence are used as the calculation basis to measure the potential of inserting material between each narrative structure node from the perspective of the narrative flow.

[0046] Furthermore, the multiple dimensions are specifically the semantic transformation dimension, the concept emphasis dimension, and the visual attention dimension. The semantic transformation dimension is used to strengthen the narrative flow in the main video and measure the necessity of inserting materials at each semantic node. The semantic transformation dimension is quantified based on the semantic similarity between each multimodal semantic unit or the semantic similarity between each time unit. The concept intensity dimension is used to capture the key concepts in each time unit and each multimodal semantic unit, which can be used to select inserted materials. The visual attention dimension is used to compensate for the visual monotony in the time unit, and at the same time to prevent visual interference caused by inserting shots when the video screen is already visually attractive, so as to ensure the overall visual harmony.

[0047] In some embodiments, calculating the insertion potential score from the semantic transformation dimension, the concept emphasis dimension, and the visual attention dimension respectively includes: In the semantic transformation dimension, the semantic similarity between adjacent time units or adjacent multimodal semantic units is calculated to generate a semantic transformation potential score; In the concept emphasis dimension, a concept strength potential score is generated based on the degree of repetition of entities in speech, the frequency of repetition in text, and the salience in vision in each of the time units or the multimodal semantic units. In the visual attention dimension, based on a pre-trained visual attention model, the video frame sequence of the time unit or the multimodal semantic unit is analyzed to predict the audience's attention distribution and calculate the visual attention potential score.

[0048] Specifically, in the semantic transformation dimension, the cosine similarity of the text content summaries of two narrative paragraphs before and after a narrative structure node is calculated, which is equivalent to calculating the semantic similarity between two adjacent time units. Similarly, the cosine similarity calculation method is used to calculate the semantic similarity between two adjacent multimodal semantic units. The semantic transformation potential score is negatively correlated with this cosine similarity, indicating that the higher the semantic association between two units, the lower their semantic transformation potential, which means they are less suitable for inserting materials and will disrupt the corresponding narrative rhythm in the main video.

[0049] Furthermore, in the concept strength dimension, the emphasis of each entity in each unit is calculated in terms of speech repetition, text repetition frequency, and visual salience, quantifying the emphasis intensity of the core concept in that time unit. Specifically, for each entity, audio emphasis signals, text emphasis signals, and visual emphasis signals are calculated in parallel. The audio emphasis signal is calculated by analyzing the entity's audio data within the corresponding unit. Specifically, when emphasizing a word, the pitch and volume of that word are usually unconsciously increased, corresponding to changes in the fundamental frequency of intonation. Therefore, the fundamental frequency variance of the word during the pronunciation period is calculated and compared with the baseline level of the sentence containing the word to quantify its repetition score. Thus, the repetition score corresponding to the audio emphasis signal is measured based on the deviation of the entity's fundamental frequency variance from the sentence baseline over the duration. For the text emphasis signal, an important concept is usually repeatedly mentioned in the local context; therefore, by analyzing the relevant... The repetition frequency score is calculated by measuring the occurrence of key entities within the corresponding unit. For visual emphasis signals, when an entity is presented in the video through close-ups, centered composition, or by guiding the viewer's eye, it indicates that the entity has a significant position in the video. Therefore, its visual saliency score can be calculated by analyzing how the entity is presented in the corresponding video frame. Specifically, the size and centrality of the target detection box of the entity in the video frame are determined. Through a pre-trained visual saliency model, the overlap of the attention hotspot of the entity is output to obtain the visual saliency score. Finally, for each entity, the quantized scores of the above emphasis signals are weighted and summed to obtain the comprehensive emphasis score of the entity. The highest comprehensive emphasis score of all entities in the unit is taken as the concept emphasis potential score of the unit. For units with higher concept emphasis potential scores, it indicates that the explanation of the entity is the focus in that time unit, and the entity can be inserted when inserting material.

[0050] Furthermore, in the visual attention dimension, based on the pre-trained visual attention model, the video frame sequence within the current unit is sampled to predict the visual attention heatmap for each frame. This visual attention heatmap represents the probability distribution of the viewer's attention on that frame. Based on the aforementioned visual attention heatmap sequence, the visual attractiveness score for the current unit is calculated. The visual attractiveness score is measured by average attention entropy and temporal dynamics score. The average attention entropy is obtained by calculating the Shannon entropy of the attention heatmap for each frame and taking the average value within the unit. Thus, the average attention entropy quantifies the degree of attention dispersion; a higher entropy value indicates more dispersed attention and lower visual attractiveness. The temporal dynamics score is calculated by determining the difference between the attention heatmaps of consecutive frames and taking the average value within the time unit. A low difference indicates monotonous visual content, which also results in lower visual attractiveness. Finally, the visual attractiveness score obtained through average attention entropy and temporal dynamics score is mapped to a visual attention potential score. The mapping relationship is negatively correlated, meaning that a lower visual attractiveness score indicates a higher visual attention potential score. In this case, relevant materials need to be inserted to enhance the visual experience.

[0051] S300. For each of the key time points, based on the contextual semantics, match B-Roll materials from the B-Roll material library; In some embodiments, step S300 above includes: Based on the insertion reason type at the key time point, relevant key entities and visual scenes are extracted from the corresponding unit, and combined with the emotional polarity of the corresponding unit and the screen ratio of the main video, a material query vector is generated. Based on the material query vector, a search is performed in the preset B-Roll material library, and the matching score of each candidate material is calculated; The highest matching score among the candidate materials is compared with a preset matching threshold to select the B-Roll material.

[0052] In this embodiment, for each key time point, relevant contextual semantic information is extracted from its associated multimodal semantic unit or time unit to construct a material query vector. The material query vector includes core visual concepts, emotional style constraints, and editing intent descriptions. Specifically, the core visual concepts are extracted from the most relevant key entities and visual scene descriptions from the corresponding multimodal semantic unit or time unit based on the dominant insertion reason type of the key time point. Specifically, if the dominant insertion reason type is a semantic transformation dimension, the scene description of the next multimodal semantic unit or the visual scene of the next time unit at that key time point is extracted as the core visual concept. If the dominant insertion reason type is a concept emphasis dimension, the comprehensive information from each entity in the corresponding unit at that key time point is extracted. The entity with the highest score is marked as the key entity and used as the core visual concept. If the dominant insertion reason type is the visual attention dimension, the scene description of the multimodal semantic unit or the visual scene of the time unit corresponding to the key time point is extracted as the core visual concept. For emotional style constraints, for key time points that overlap with the end timestamp of the time unit, the emotional polarity of the corresponding time unit is used as the emotional style constraint. For key time points that do not overlap with the end timestamp of the time unit, the time unit to which the multimodal semantic unit corresponding to the key time point belongs is searched back, and the emotional polarity of the time unit is used as the emotional style constraint. For the description of editing intent, specifically the aspect ratio of the main video, in a preferred embodiment, it also includes a preset target duration of the inserted material.

[0053] Furthermore, using the material query vector as input, a search is performed in the B-Roll material library. Specifically, the comprehensive matching degree of each candidate material in terms of visual similarity and emotional consistency is calculated. For visual similarity, it is obtained by comparing the core visual concepts in the material query vector with the text tags or visual feature embeddings of the candidate B-Roll materials. For emotional consistency, it is obtained by calculating the matching degree between the emotional constraint style in the material query vector and the emotional tags of the candidate B-Roll materials. The weighted sum of visual similarity and emotional consistency is calculated to obtain the final matching degree score. The editing intent description in the material query vector is used to constrain the screen ratio and video length of the B-Roll materials to ensure seamless compatibility with the main video in the subsequent mixing and compositing process.

[0054] Furthermore, this application combines material library retrieval with dynamic material generation. When the B-Roll material library matching degree is insufficient, it can automatically call the video generation model to generate the required materials based on the contextual semantics of the corresponding key time points, thereby expanding the application boundaries of video mashup in this application.

[0055] In some embodiments, the step of comparing the highest matching score among the candidate materials with a preset matching threshold to select the B-Roll material includes: If the highest matching score among the candidate materials is higher than the matching threshold, then the candidate material is selected as a B-Roll material; If the highest matching score among the candidate materials is lower than the matching threshold, the video generation process is triggered. When the video generation process is triggered, the material query vector is converted into video generation prompt words, and the video generation model is called to generate B-Roll video clips, which are then selected as B-Roll materials.

[0056] Furthermore, the highest matching score among the candidate materials is compared with a preset matching threshold. If it is higher than the matching threshold, the candidate material is selected as the B-Roll material for supply. If it is lower than the matching threshold, the video generation process is triggered to generate a new material for insertion at that key time point. When the video generation process is triggered, the material query vector is converted into a structured video generation prompt, and the video generation model is called to generate a B-Roll video clip that meets the requirements. The format of the video generation prompt is: [Main Action Description], [Scene and Environment], [Visual Style and Tone], [Emotional Atmosphere], where each part is filled by the corresponding field in the material query vector.

[0057] In addition, for each selected B-Roll material, it is associated with the corresponding key time point and a material utility tag is generated. The material utility tag records whether the material comes from the B-Roll material library or is generated in real time, as well as its matching score or generation confidence.

[0058] S400. According to the preset video style template, insert the B-Roll material into the corresponding key time points to generate a montage video.

[0059] During the video mixing process, this application dynamically selects the most suitable transition effects and visual layout composition parameters based on the dominant insertion reason type at each key time point, without the need for manual adjustments. This ensures high-quality output of the video mixing and improves the overall efficiency of video mixing, eliminating the need for creators to manually perform editing tasks such as content analysis, finding materials, timing, and adjusting effects, thus improving overall work efficiency.

[0060] In some embodiments, step S400 above includes: Load a preset video style template, which defines a set of compositing rules, including a transition effect rule set, a rhythm density curve, and visual integration rules; Based on the insertion reason type at each key time point in the main video and the video style template, the set of synthesis parameters is dynamically determined. The synthesis parameters include transition features, B-Roll material insertion features, and B-Roll material frame features. Based on the synthesis parameters, the main video and B-Roll footage are aligned on the timeline to generate a montage video.

[0061] In this embodiment, the video style template is specifically selected by the user or recommended based on the main video type. The video style template is a structured configuration file that defines a set of composition rules, including at least a transition effect rule set, a rhythm density curve, and visual integration rules. The transition effect rule set defines which transitions to use under what insertion cause type, with specific transition methods including "dissolve," "slide in," and "zoom in." The rhythm density curve is a curve that changes over time, defining the appropriate editing rhythm for different narrative structure nodes in the video, such as a slow rhythm in the introduction and a fast rhythm in the climax. The visual integration rules define the aesthetic principles of how B-Roll footage coexists with the main video frame. For example, in one embodiment, based on... The visual complexity of frames at key time points is dynamically adjusted to optimize the proportion and position of B-Roll footage, ensuring readability and aesthetics for both the main video and B-Roll materials. Specifically, if the main video is simple, such as a presenter standing against a solid-color background, the B-Roll footage is inserted in full-screen or large picture-in-picture mode. If the main video is already complex, such as a PPT slide containing a lot of information, the B-Roll footage is adjusted to a smaller scale and placed in a corner that does not obscure key information, or it is inserted using advanced effects such as semi-transparent overlay to avoid visual clutter and information obstruction. Thus, by using the aforementioned video style templates, combined with the insertion reason type and time unit of the corresponding key time points, the corresponding set of composition parameters can be determined.

[0062] Specifically, the compositing parameters include transition features, B-Roll insert features, and B-Roll image features. Transition features specifically refer to the transition type and duration, B-Roll insert features specifically refer to the precise in-point and out-point of the B-Roll material, and B-Roll image features specifically refer to the position, proportion, and transparency of the B-Roll material in the main video.

[0063] Specifically, the transition features are determined based on the insertion reason type of key time points and mapped from the transition effect rule set of the video style template. In one embodiment, for the semantic transformation dimension, transition effects with a sense of scene switching are prioritized, such as triggering a "radial wipe" transition or visually "turning pages". For the concept emphasis dimension, smooth transition effects that focus on the center of the screen are prioritized, such as using a "zoom" transition to push the video focus to the corresponding key entity. For the visual attention dimension, transition effects with dynamic appeal are prioritized.

[0064] Furthermore, the main video track and the B-Roll material track are precisely aligned on the timeline, and the above-mentioned compositing parameters are automatically applied to generate a mashup video. During the compositing process, the set of compositing parameters and their corresponding decision-making basis and key time points are recorded and output in association with the mashup video for easy querying by users later.

[0065] Please see Figure 2 As shown, the present invention also provides a video mixing and editing system based on A / B Roll, the system comprising: First processing module 201: used to perform temporal deep semantic analysis on the main video and generate a semantic timeline; The second processing module 202 is used to dynamically identify key time points based on the semantic timeline, wherein the key time points are the time points when B-Roll material is inserted into the main video; The third processing module 203 is used to match B-Roll materials from the B-Roll material library for each of the key time points, taking into account the contextual semantics. The fourth processing module 204 is used to insert the B-Roll material into the corresponding key time points according to the preset video style template to generate a montage video.

[0066] It is understandable that, such as Figure 1 The content of the narrative video mixing method embodiment based on A / B Roll shown is applicable to this A / B Roll-based video mixing system embodiment. The specific functions implemented by this A / B Roll-based video mixing system embodiment are the same as those shown below. Figure 1 The illustrated example of the narrative video mixing method based on ABRoll is the same, and the beneficial effects achieved are the same as those shown. Figure 1 The beneficial effects achieved by the ABRoll-based narrative video mixing method embodiment shown are also the same.

[0067] It should be noted that the information interaction and execution process between the above systems are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] Please see Figure 3 As shown, this embodiment of the invention also provides a computer device 3, including: a memory 302 and a processor 301, and a computer program 303 stored on the memory 302. When the computer program 303 is executed on the processor 301, it implements the narrative video mixing method based on ABRoll as described in any of the above methods.

[0070] The computer device 3 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 3 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 3 The computer device 3 is merely an example and does not constitute a limitation on the computer device 3. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0071] The processor 301 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0072] In some embodiments, the memory 302 may be an internal storage unit of the computer device 3, such as a hard disk or memory of the computer device 3. In other embodiments, the memory 302 may be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 3. Furthermore, the memory 302 may include both internal and external storage units of the computer device 3. The memory 302 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0073] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the narrative video mixing method based on ABRoll as described in any of the above methods.

[0074] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / computer device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0075] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for ABRoll-based narrative video mixing and splicing, characterized in that, include: Perform temporal deep semantic analysis on the main video to generate a semantic timeline; Based on the semantic timeline, key time points are dynamically identified, and the key time points are the time points in the main video when B-Roll material is inserted. For each of the aforementioned key time points, B-Roll materials are matched from the B-Roll material library based on the contextual semantics. Based on the preset video style template, the B-Roll footage is inserted into the corresponding key time points to generate a montage video.

2. The method as described in claim 1, characterized in that, The step of performing temporal deep semantic analysis on the main video to generate a semantic timeline includes: The main video is deconstructed to obtain a video frame sequence and an audio track, and speech recognition is performed on the audio track to generate a text sequence; The video frame sequence and the text sequence are jointly analyzed to generate multimodal semantic units, where each multimodal semantic unit corresponds to a video time segment, and a multimodal semantic unit sequence is generated. Based on the multimodal semantic unit sequence, semantic association and evolution between units are performed to identify the narrative structure nodes of the main video; By combining the multimodal semantic units and the narrative structure nodes, a semantic timeline is generated. The semantic timeline contains multiple time units, and each time unit carries a semantic description, which includes visual scene, key entities, text content summary, sentiment polarity, and narrative function tags.

3. The method as described in claim 2, characterized in that, The dynamic identification of key time points based on the semantic timeline, wherein the key time points are the times when B-Roll footage is inserted into the main video, includes: On the semantic timeline, the end timestamp of each multimodal semantic unit is used as a candidate key time point. Based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel. The insertion potential scores of each dimension are input into a decision fusion model for weighted summation to generate a comprehensive insertion recommendation score for each candidate key time point; Based on the time series of the comprehensive insertion recommendation scores, combined with the dynamic recommendation threshold, local peak points are identified; The local peak points are used as key time points for inserting B-Roll material, where each key time point is associated with the dominant insertion reason type.

4. The method as described in claim 3, characterized in that, On the semantic timeline, using the end timestamp of each multimodal semantic unit as a candidate key time point, and based on the time unit and the multimodal semantic unit, the insertion potential score of each candidate key time point in multiple dimensions is calculated in parallel, including: For candidate key time points that overlap with the end timestamp of a time unit, the corresponding time unit is used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively. For other candidate key time points, the corresponding multimodal semantic units are used as the basis for analysis, and insertion potential scores are calculated from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension, respectively.

5. The method as described in claim 4, characterized in that, The calculation of insertion potential scores from the semantic transformation dimension, concept emphasis dimension, and visual attention dimension includes: In the semantic transformation dimension, the semantic similarity between adjacent time units or adjacent multimodal semantic units is calculated to generate a semantic transformation potential score; In the concept emphasis dimension, a concept strength potential score is generated based on the degree of repetition of entities in speech, the frequency of repetition in text, and the salience in vision in each of the time units or the multimodal semantic units. In the visual attention dimension, based on a pre-trained visual attention model, the video frame sequence of the time unit or the multimodal semantic unit is analyzed to predict the audience's attention distribution and calculate the visual attention potential score.

6. The method as described in claim 1, characterized in that, For each of the key time points, the process of matching B-Roll content from the B-Roll content library, taking into account the contextual semantics, includes: Based on the insertion reason type at the key time point, relevant key entities and visual scenes are extracted from the corresponding unit, and combined with the emotional polarity of the corresponding unit and the screen ratio of the main video, a material query vector is generated. Based on the material query vector, a search is performed in the preset B-Roll material library, and the matching score of each candidate material is calculated; The highest matching score among the candidate materials is compared with a preset matching threshold to select the B-Roll material.

7. The method as described in claim 6, characterized in that, The step of comparing the highest matching score among the candidate materials with a preset matching threshold to select the B-Roll material includes: If the highest matching score among the candidate materials is higher than the matching threshold, then the candidate material is selected as a B-Roll material; If the highest matching score among the candidate materials is lower than the matching threshold, the video generation process is triggered. When the video generation process is triggered, the material query vector is converted into video generation prompt words, and the video generation model is called to generate B-Roll video clips, which are then selected as B-Roll materials.

8. The method as described in claim 1, characterized in that, The step of inserting the B-Roll footage into corresponding key time points according to a preset video style template to generate a montage video includes: Load a preset video style template, which defines a set of compositing rules, including a transition effect rule set, a rhythm density curve, and visual integration rules; Based on the insertion reason type at each key time point in the main video and the video style template, the set of synthesis parameters is dynamically determined. The synthesis parameters include transition features, B-Roll material insertion features, and B-Roll material frame features. Based on the synthesis parameters, the main video and B-Roll footage are aligned on the timeline to generate a montage video.

9. A video mixing and editing system based on A / B Roll, characterized in that, include: The first processing module is used to perform temporal deep semantic analysis on the main video and generate a semantic timeline. The second processing module is used to dynamically identify key time points based on the semantic timeline, where the key time points are the time points when B-Roll footage is inserted into the main video. The third processing module is used to match B-Roll materials from the B-Roll material library for each of the key time points, taking into account the contextual semantics. The fourth processing module is used to insert the B-Roll material into the corresponding key time points according to the preset video style template, and generate a mashup video.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images