Automatic movie short video explanation editing method and device, server and storage medium

Abstract

The invention discloses an automatic movie short video explanation editing method and device, a server and a storage medium, and relates to the field of multimedia video editing, and the method comprises the steps: analyzing each plot content of an original movie video; performing shot fragmentation on the original movie video, extracting shot information, and marking starting and ending timestamps of each shot; matching the preset commentary with the shot information in the film; based on a text-to-speech model, converting the matched commentary into a corresponding commentary audio, synchronizing the corresponding commentary audio with a corresponding movie shot, and providing background commentary for the short video; and dynamically planning the rhythm of lens switching based on the identified plot content, the lens matching information and the synchronized explanation audio, and intelligently inserting a transition special effect to assemble the mixed video. According to the invention, by fusing multi-modal feature analysis and intelligent editing logic generation, efficient and high-quality video automatic production is realized, and the manufacturing cost is reduced while the content attraction is improved.

Description

Technical Field

[0001] This invention relates to the field of multimedia video editing technology, and in particular to an automatic editing method, apparatus, server, and storage medium for short video narration. Background Technology

[0002] With the explosive growth of short video content, users' demands for video editing efficiency and quality are increasing. Currently, mainstream video editing technologies mainly fall into two categories: traditional manual editing and automated editing based on single features. Traditional editing relies on manual operation, requiring a significant amount of time to select materials, adjust shot order, and add effects, resulting in low efficiency and high costs.

[0003] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an automated video editing method, apparatus, server, and storage medium for short video narration. This invention offers an automated video editing method based on plot analysis, narration shot matching, and TTS (Text-to-Speech) conversion. Through collaborative analysis, it solves the problems of content logic breaks and repetitive shots in automated editing. Simultaneously, it introduces a multi-objective optimization mechanism to improve editing efficiency while meeting the needs of personalized content production. By integrating multimodal feature analysis and intelligent editing logic generation, it achieves efficient and high-quality automated video production, increasing content appeal while reducing production costs.

[0005] The technical solution of this application is as follows: An automatic editing method for short video narration, comprising: A acquires the original video that needs to be edited and explained, and uses a multimodal feature extraction method to extract multiple modal information, including visual and textual information, from the original video. B analyzes the original video to identify the various plot points of the video; and segments the original video into shots, extracts all shot information, and marks the start and end timestamps of each shot. C matches the preset narration with the shot information in the film; D is based on a text-to-speech model, which converts the matched narration into corresponding audio narration and synchronizes it with the corresponding video footage to provide background narration for short videos; Based on the identified plot content, shot matching information, and synchronized narration audio, E dynamically plans the rhythm of shot transitions and intelligently inserts transition effects to assemble a mixed video.

[0006] The aforementioned automatic editing method for short video narration includes, prior to the step of obtaining the original video to be edited and narrated, and extracting multimodal information including visual and textual information from the original video using a multimodal feature extraction method: Obtain external explanatory text, clean the explanatory text to conform to the third-person narration format, and use a preset large model to divide the cleaned explanatory text into multiple independent semantic units to form explanatory sentences.

[0007] The aforementioned automatic editing method for short video narration, wherein the step of analyzing the original video and identifying the various plot points of the video includes: One frame is extracted from the original video every second as an image, and the extracted images and relevant information from the original video are input into a preset visual model for analysis to identify various plot points of the video.

[0008] The aforementioned automatic editing method for short video narration includes the step of matching preset narration text with shot information in the video, comprising: Using a deep learning-based visual model, the system automatically searches for memorable scenes from the original video. Based on the aforementioned famous scene information, narration segmentation information, and shot information, each famous scene information is matched with the corresponding narration, and the start and end times in the film are determined.

[0009] The aforementioned automatic editing method for short video narration includes the following steps: converting the matched narration text into corresponding audio narration based on a text-to-speech model and synchronizing it with the corresponding video footage to provide background narration for the short video. The matched narration is input into a preset text-to-speech model to generate corresponding narration audio, which is then used as background narration audio for the short video. Based on the generated audio narration, subtitle information for the narration video is generated and used as subtitles for the short video footage.

[0010] The aforementioned automatic editing method for short video narration includes the following steps: dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the mixed-edit video based on the identified plot content, shot matching information, and synchronized narration audio. The narration segments that match individual sentences and the narration segments that match the original video information are then edited and combined into a single video. Initially, the individual sentence narration segments did not contain audio information, while the original video information segments contained the original audio. Add corresponding audio explanations to single-sentence segments, providing complete audio content; Add corresponding subtitles to the entire commentary video; The video is assembled, audio and video fade-in / fade-out effects are added, and volume equalization is performed to ultimately generate a short video with coherent and smooth narration.

[0011] The aforementioned automatic editing method for short video narration, wherein the steps of dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the mixed-edit video based on the identified plot content, shot matching information, and synchronized narration audio further include: Sentiment analysis was performed on each sentence in the explanatory text to identify the sentiment and intensity. Using a large visual model, emotion recognition is performed on each shot segment to analyze the emotional tendency and intensity contained in the visual elements of the shot; Compare the sentiment analysis results of the matched narration with the visual sentiment analysis results of the corresponding shot segments; When there is a significant difference in type or intensity between the emotion of the narration and the emotion of the shot, it is marked as an emotion mismatch; for shot segments with emotion mismatch, try to find other shot segments in the same plot or adjacent plots that match the emotion of the current narration and replace them; If a matching shot cannot be found, the tone, speed and timbre of the narration audio are dynamically adjusted based on the sentiment analysis results of the narration, so that the narration audio matches the emotional expression of the narration. For segments where the emotions don't match, the system intelligently selects or adjusts transition effects and background music to compensate for the lack of emotion and enhance overall harmony.

[0012] An automatic editing device for short video narration, wherein the device includes: The multimodal feature extraction module is used to acquire the original video of the video to be edited and explained, and to extract multiple modal information, including visual and textual information, from the original video using a multimodal feature extraction method. The plot analysis unit is used to analyze the original video and identify the various plot contents of the video. The shot segmentation unit is used to segment the original video into shots, extract all shot information, and mark the start and end timestamps of each shot. The narration shot matching unit is used to match preset narration with shot information in the film; The TTS (Text-to-Speech) unit is used to convert the matched narration into corresponding audio narration based on the text-to-speech model, and synchronize it with the corresponding video footage to provide background narration for short videos. The editing logic generation module is used to dynamically plan the rhythm of shot transitions based on the identified plot content, shot matching information, and synchronized narration audio, and intelligently insert transition effects to assemble the mixed video.

[0013] A server includes a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including steps for performing any of the methods described herein.

[0014] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it enables an electronic device to perform the steps of any of the methods described.

[0015] As can be seen from the above, the automatic editing method, device, server, and storage medium for short video narration provided in this application achieve efficient and high-quality automated video production by integrating multimodal feature analysis and intelligent editing logic generation. This invention can be applied to scenarios such as short video creation and the generation and production of film and television trailers, improving content appeal while reducing production costs. Compared with existing technologies, editing time is reduced by more than 60%, and the click-through rate and completion rate of the generated videos are significantly improved. Furthermore, this technology can be extended to scenarios such as advertising placement and paid content previews, balancing content appeal and commercial value conversion through intelligent editing strategies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the automatic editing method for short video narration according to Embodiment 1 of the present invention.

[0018] Figure 2 This is a detailed flowchart of step S500 of the automatic editing method for short video narration in Embodiment 1 of the present invention.

[0019] Figure 3 This is a flowchart illustrating the automatic editing method for short video narration according to Embodiment 2 of the present invention.

[0020] Figure 4 The principle block diagram of the automatic editing device for short video narration provided by the present invention.

[0021] Figure 5 This is a block diagram illustrating the internal structure of the server provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0024] With the explosive growth of short video content, users' demands for video editing efficiency and quality are increasing. Currently, mainstream video editing technologies mainly fall into two categories: traditional manual editing and automated editing based on single features. Traditional manual editing relies on manual operation, requiring a significant amount of time to select materials, adjust shot order, and add effects, resulting in low efficiency and high costs. Existing automated editing technologies generally suffer from the following drawbacks: 1) Multimodal information fragmentation: The text and visual features of existing video editing technologies are not deeply integrated, making it difficult to balance the content coherence and visual expressiveness of the editing results.

[0025] 2) Insufficient personalization: Existing video editing technology does not dynamically optimize based on user preferences, such as video rhythm, duration, and style, resulting in monotonous output content.

[0026] 3) Efficiency bottleneck: The automated editing algorithms of existing video editing technologies are highly complex and require high-performance hardware when processing long videos, making them difficult to adapt to mobile devices or lightweight scenarios.

[0027] It is evident that these problems with existing video editing technologies collectively lead to issues such as logical breaks in content, repetitive shots, low editing efficiency, insufficient personalization, and high hardware resource consumption in automated editing, failing to meet users' demands for efficient, high-quality, and personalized short video content production.

[0028] To address the aforementioned technical problems, this invention provides an automatic editing method for short video narration, as detailed in the following embodiments.

[0029] Example 1 like Figure 1 As shown, an embodiment of the present invention provides an automatic editing method for short video narration, comprising the following steps: Step S100: Obtain the original video of the video to be edited and explained, and use a multimodal feature extraction method to extract multiple modal information including visual and textual information from the original video. The multimodal feature extraction in this embodiment refers to the use of technical means to simultaneously extract multiple different forms of information features from the original video, rather than extracting from a single dimension, thereby achieving comprehensive information capture. These multiple forms of information features include visual (images), text (subtitles, embedded text), and audio (background sounds, voices), etc.

[0030] The visual modal information refers to the core information at the video frame level, including the content of the shot such as people, scenes, and actions, as well as features that can be perceived visually, such as the color, brightness, camera movement trajectory such as push, pull, pan, and tilt, and image clarity.

[0031] The text modal information refers to the text-based information contained in the video, including embedded subtitles, text elements in the screen such as poster text, dialogue subtitles, video title / description related text, and other information that can be converted into text.

[0032] Step S100 involves acquiring all necessary visual and textual information from the original video. Specifically, this involves first acquiring the original video that needs to be edited and have narration added, which can be various video sources such as long video clips, full-length films and TV dramas, and live-action footage. Then, multimodal feature extraction technology is used to overcome the limitations of single feature extraction in existing technologies, simultaneously capturing visual modal information (such as core elements of the image, motion features, etc.) and textual modal information (such as subtitles, on-screen text, etc.) from the video, forming a comprehensive video information dataset, providing basic data support for subsequent editing and analysis.

[0033] For example, when editing a short video narration of a TV series "XX", the first step is to obtain the complete 10th episode of "XX" as the original video. Using multimodal feature extraction technology, visual modal information is extracted, identifying the main character A, supporting character B, scenes (such as bedrooms and streets), actions (such as dialogue and running), camera movement trajectories (such as close-up shots during dialogue and follow shots during running), and color schemes (such as warm colors in the bedroom and cool colors in the street). Textual modal information is also extracted, capturing embedded subtitles (such as the main character A's line "I must find him") and textual elements in the video (such as the "XX Convenience Store" sign on the street). Finally, this visual and textual information is integrated to form a multimodal data set of the episode, which can then be used to analyze the plot and match narration.

[0034] In this step, visual and textual information are extracted simultaneously to achieve preliminary fusion of the two core modalities, avoiding insufficient content coherence caused by single feature extraction. This lays a data foundation for subsequent shot matching and logical assembly, and can solve the problem of fragmented multimodal information in existing technologies. Furthermore, compared to the limitations of traditional manual material selection and single feature extraction, multimodal extraction can comprehensively capture core video information, reduce the omission of key materials, ensure the quality of subsequent editing, and improve the completeness of information coverage.

[0035] Moreover, this invention eliminates the need for manual screening of materials and extraction of key information, automating information collection, significantly improving the efficiency of pre-editing preparation, and reducing the cost of manual intervention.

[0036] Step S200: Use a large visual model to perform in-depth analysis on the original video, identify and summarize the various plot contents of the video; and divide the original video into shot segments, extract all shot information, and mark the start and end timestamps of each shot. The large visual model in this embodiment refers to a large-scale visual processing model trained based on deep learning technology. It has powerful image analysis, content recognition and semantic understanding capabilities, and can accurately identify complex information such as scenes, people, actions and plot relationships in video images, far exceeding the analysis depth of traditional single visual recognition algorithms.

[0037] The plot content recognition in this embodiment refers to the continuous analysis of video frame sequences through a large visual model, mining the logical connections between shots, and sorting out the core storyline and key plot nodes such as the opening, development, climax, and ending contained in the video.

[0038] The shot segmentation of this invention refers to the process of dividing a complete video into independent shot segments according to the shot switching rules. A shot is the basic unit of video. For example, a continuously shot scene is a shot. Switching methods include cut, fade in and fade out, dissolve, etc.

[0039] In this embodiment, the timestamp marking refers to marking the precise start time (the time point when the shot starts) and end time (the time point when the shot switches) for each split shot, forming standardized shot time information, and realizing precise positioning of each shot.

[0040] Step S200 utilizes a large visual model to conduct in-depth analysis of the film, identifying and summarizing its core plot elements to provide a macro-level narrative structure for subsequent editing; and then performs shot segmentation. Specifically, firstly, the large visual model is used to perform in-depth analysis of the original video, not only identifying the content of individual frames but also summarizing the core plot elements and plot nodes through the correlation analysis of consecutive frames, achieving a structured understanding of the video content; secondly, based on the large visual model's shot transition recognition capability, the original video is split into multiple independent shots, extracting the core information of each shot, such as shot content, movement method, and duration, and marking each shot with precise start and end timestamps, forming a precisely accessible shot material library.

[0041] In this embodiment of the invention, the structured logic of the video is organized through plot content recognition, avoiding the logical chaos caused by the stacking of shots in existing automated editing, ensuring the continuity of subsequent editing content, and solving the problem of content logic breakage. Furthermore, this invention enables precise shot control because the timestamp markings provide accurate positioning for each shot, allowing for rapid matching of narration and adjustment of shot order, significantly improving editing efficiency and solving the problem of time-consuming manual shot splitting. Moreover, this embodiment of the invention splits long videos into independent shot footage, which can be flexibly used according to plot requirements, avoiding the reuse of footage, solving the defect of repeated footage in existing technologies, and improving the utilization rate of materials.

[0042] Step S300: Use a deep learning visual model to match the preset narration with the shot information in the video; The deep learning visual big model in this embodiment refers to the integration of deep learning's semantic association capabilities on the basis of the visual big model, which can realize semantic matching between text (narration) and visuals (camera shots). It can not only identify surface information, but also understand the deep association between the meaning of the text and the content of the image.

[0043] The preset narration text in this embodiment refers to the video narration text set by the user in advance. It can be general narration, scene-based narration, such as narration of movie trailers, narration of short video science videos, or narration text automatically generated according to the video theme.

[0044] The aforementioned shot information matching refers to the correlation analysis based on text semantics and image semantics, which matches each segment of the narration with the most suitable shot to ensure a high degree of matching between the narration and the image content. For example, when the narration describes the protagonist's appearance, it matches the shot in which the protagonist appears.

[0045] In the specific implementation of step S300, a deep learning visual model with semantic understanding capabilities is first invoked. On the one hand, it analyzes the text semantics and paragraph structure of the preset narration, such as the core meaning of each sentence and the plot nodes corresponding to the paragraphs. On the other hand, it combines the shot information obtained in step S200, such as the shot content, timestamp, and plot, and establishes a precise correspondence between the narration and the shot through a semantic association algorithm. That is, each narration is matched with a shot that is appropriate in content and duration, realizing the synchronous linkage between the narration content and the screen content.

[0046] As can be seen, this invention can achieve deep semantic fusion of text (narration) and visuals (camera shots), balancing content coherence and visual expressiveness, avoiding the problem of disconnect between visuals and text in existing technologies, and completely solving the problem of multimodal information fragmentation. Furthermore, this invention achieves precise matching of narration and camera shots, allowing viewers to obtain information through visuals and deepen their understanding through narration, enhancing the efficiency of video information delivery and the viewing experience. Moreover, this invention eliminates the need for manual, sentence-by-sentence matching of narration and camera shots, automating precise matching, significantly improving editing efficiency compared to traditional manual editing, and reducing manual matching costs.

[0047] Step S400: Based on the text-to-speech model, the matched narration is converted into corresponding narration audio and synchronized with the corresponding video footage to provide background narration for the short video; The text-to-speech (TTS) model in this embodiment refers to an AI model based on speech synthesis technology that can automatically convert text information (narration) into natural and fluent human voice audio, and supports adjusting parameters such as speech rate, tone, and timbre to simulate the effect of real human narration.

[0048] In this embodiment, the audio narration synchronization refers to precisely aligning the converted audio narration with the matched shot according to the shot timestamp marked in step S200, ensuring that the playback progress of the audio narration matches the playback progress of the shot image. For example, the audio narration starts synchronously when the shot begins and ends synchronously when the shot ends.

[0049] The background narration mentioned in this embodiment refers to audio narration adapted to short video scenarios, existing as background sound in the video to supplement on-screen information, guide audience understanding, and enhance the video's atmosphere.

[0050] In the specific implementation of step S400, the text-to-speech model first converts the matched narration from step S300 into narration audio that matches the video style. Specifically, the timbre and speaking speed can be adjusted according to needs, such as using a calm timbre for science videos and a lively timbre for entertainment videos. Then, based on the shot timestamp marked in step S200, the narration audio is precisely synchronized with the corresponding shot to ensure that the audio and video rhythms are consistent. When the shot changes, the narration audio can also be synchronously connected to avoid audio and video misalignment. Finally, a suitable background narration is generated for the short video, completing the three-modal fusion of image, text and audio.

[0051] This invention improves editing efficiency and quality because it eliminates the need for manual recording of narration audio and manual adjustment of audio-visual synchronization. It automates audio generation and alignment, avoiding audio-visual misalignment and inefficiency caused by manual operation, significantly reducing editing time. Furthermore, this invention reduces production costs because it eliminates the need for professional voice-over artists; AI automatically generates audio, drastically reducing the labor costs of short video production.

[0052] Step S500: Based on the identified plot content, shot matching information and synchronized narration audio, dynamically plan the rhythm of shot switching and intelligently insert transition effects to assemble the mixed video.

[0053] The dynamic planning of shot switching rhythm in this embodiment refers to the intelligent adjustment of the frequency and timing of shot switching based on the development of the video plot and the rhythm of the narration audio, such as the speed of speech and pauses. For example, fast-paced switching is matched with climax scenes, and slow-paced switching is matched with calm scenes, so as to achieve dynamic adaptation of shot rhythm with content and audio.

[0054] The transition effects in this embodiment refer to the transition effects between shots, including basic transitions such as cut and fade in / out, and special effects transitions such as dissolve, flash, and rotation, which are used to improve the smoothness and visual appeal of shot switching.

[0055] The assembled video editing in this embodiment refers to combining shots with matched narration audio, inserting appropriate transition effects according to a planned rhythm sequence, and integrating them into a complete short video product, thus achieving intelligent multi-shot video editing.

[0056] In the specific implementation of step S500, the plot content identified in step S200 is used as the logical framework, the shot matching information in step S300 is used as the material basis, and the synchronous narration audio in step S400 is used as the rhythm. The intelligent algorithm dynamically plans the shot switching rhythm to ensure that the shot switching not only conforms to the plot development logic, but also matches the speech speed and pauses of the narration audio. Then, according to the style and rhythm requirements of the shot content, the appropriate transition effects are intelligently selected, such as using dissolve transitions for emotional videos and flash white transitions for fast-paced videos to avoid abrupt transition effects. Finally, all shots, audio, and transition effects are integrated and assembled to generate a complete mashup short video product.

[0057] This invention enables high-quality personalized editing because it dynamically plans shot rhythm and intelligently selects transition effects, adapting different editing effects to the video plot and narration style, thus overcoming the shortcomings of existing technologies in terms of insufficient personalization and content homogenization. Furthermore, this invention overcomes efficiency bottlenecks by automating rhythm planning, transition insertion, and video assembly, eliminating the need for manual adjustments to shot order and transition additions. Editing time is reduced by over 60%, and it does not rely on high-performance hardware, making it suitable for mobile devices and lightweight scenarios.

[0058] In a further embodiment of the present invention, the method further includes the following steps before step S100: S101. Obtain external explanatory text, clean the explanatory text to conform to the third-person explanatory format, and use a preset large model to divide the cleaned explanatory text into multiple independent semantic units to form explanatory sentences.

[0059] In this embodiment of the invention, high-quality external narration text needs to be acquired and cleaned to conform to the third-person narration format. Subsequently, a large model is used to segment the text into multiple independent semantic units, i.e., narration sentences, for subsequent shot matching.

[0060] For example, when creating a 3-minute short video narration for the movie "X-Wave Earth," the first step is to acquire high-quality external narration text for "X-Wave Earth." This text is cleaned to ensure it conforms to the third-person narration format and is divided into multiple independent semantic units by a large model, such as: "Humanity launches the wandering project," "Planetary engines ignite, Earth heads into the depths of space," and "Jupiter's gravitational crisis, the protagonist team turns the tide."

[0061] As can be seen, this invention, through preprocessing and structuring the narration text, solves problems such as non-standard formatting and unclear semantics that may exist in the original text, ensuring the accuracy and efficiency of subsequent narration matching with shots. The cleaned and segmented text can more accurately correspond to the film's plot and shots, avoiding logical breaks or improper image matching caused by text quality issues, thereby improving the overall quality of automated editing and user experience.

[0062] In a further embodiment of the present invention, the automatic editing method for short video narration, wherein the step of using a large visual model to perform in-depth analysis of the original video and identifying and summarizing the various plot contents of the video includes: S201. Extract one frame per second from the original video as an image, and input the extracted images and relevant information from the original video into a preset visual model for in-depth analysis, to identify and summarize the various plot contents of the video.

[0063] This step primarily involves scene and plot analysis. For example, one frame per second is extracted from the complete film "X-Wave Earth" and sent along with relevant film information, such as plot synopsis and main characters, to the visual big data model. The big data model analyzes this information to summarize the film's core plot points, such as: "The Wandering Project Launched," "Planetary Engine Ignition," "Jupiter's Gravitational Crisis," and "The Protagonist Team's Rescue Operation." This invention utilizes the visual big data model to conduct in-depth analysis of the film, identifying and summarizing its core plot elements, providing a macro-level narrative structure for subsequent editing.

[0064] S202. Segment the film into shots, extract all shots and mark their start and end timestamps; during shot analysis, avoid switching during shot flickering, and extend the clips if necessary to avoid flickering.

[0065] For example, following the previous steps, the footage of "X-Wave Earth" is segmented into segments, all shots are extracted, and the start and end timestamps of each shot are marked. For instance, the shot segment corresponding to "planetary engine ignition" (from 00:15:20 to 00:16:00) is identified. Simultaneously, switching during shot flickering is avoided, and segments are extended if necessary to avoid flickering.

[0066] This embodiment of the steps solves the problem of abrupt and unnatural scene transitions that may occur in automated editing by accurately segmenting shots and avoiding flickering. Avoiding transitions during flickering significantly improves the visual smoothness and viewing experience of the edited video, reduces viewer discomfort, and thus enhances the professionalism and high-quality performance of the edited video.

[0067] In a further embodiment of the present invention, the automatic editing method for short video narration, wherein step S300 specifically includes: S301. Using a deep learning visual model, automatically search for famous scenes from the original video. For example, by using a deep learning-based visual model, the system intelligently searches the internet for memorable scenes from the film *X-Wave Earth*, such as "Igniting Jupiter" and "Frozen Shanghai," and uses the corresponding film clips as important editing material. This invention solves the problem of automated editing missing key moments in a film due to relying solely on internal analysis by introducing external information about memorable scenes. Memorable scenes are usually the most attractive and shareable parts of a film; using them as important editing material can significantly improve the appeal, click-through rate, and completion rate of short videos, thereby better meeting users' demands for high-quality, personalized short video content.

[0068] S302. Based on the famous scene information, narration text segmentation information, and shot information, match each famous scene information with the corresponding narration text, and determine the start and end times in the film.

[0069] In this step, based on information about iconic scenes, segmented narration, and detailed shot information, each iconic scene is matched with its corresponding narration, and its start and end times in the film are determined. For example, the narration "Faced with a crisis, humanity launches a relocation plan" is matched with the shot segment in the film depicting the crisis and the launch of the relocation plan. Similarly, for each narration sentence, such as "Planetary engines ignite, Earth heads into the depths of space," a start and end time in the original film is matched, creating a short video clip for automatic editing.

[0070] In one embodiment, step S400 specifically includes: S401. Input the matched narration into the preset text-to-speech model to generate the corresponding narration audio, which will be used as the background narration audio for the short video; S402. Based on the generated narration audio, generate subtitle information for the narration video as subtitles for the short video footage.

[0071] In this embodiment of the invention, the matched narration text is sent to a text-to-speech (TTS) model to generate audio information for the narration, which serves as the background audio for the short video. For example, "Humanity launches a homeless program" is converted into corresponding speech. Then, based on the generated TTS audio information, subtitle information for the narration video is generated, serving as subtitles for the short video footage.

[0072] In this embodiment of the invention, the problem of information acquisition when watching short videos in a silent environment or when users have hearing impairments is solved by automatically generating SRT subtitles (subtitle file format). Adding subtitles improves video accessibility and user experience, especially when watching on mobile devices or in public places. Users can understand the video content even without sound, thereby expanding the video's audience and dissemination effect.

[0073] In a further embodiment of the present invention, the automatic editing method for short video narration is as follows: Figure 2 As shown, step S500 includes: S501. Obtain the narration segment matching the single narration sentence and the narration segment matching the original video information, and perform segment editing and synthesis to synthesize a video; wherein, the narration single sentence segment initially has no audio information, while the original video information segment contains the original audio; The narration segment matched with each narration sentence in this step refers to the original video shot segment that is precisely matched with each narration sentence in step S300. This segment only retains the visuals and does not add any audio, including the narration audio and the original video audio. Its purpose is to carry the visual images corresponding to the narration content.

[0074] The narration clips that match the original film information refer to supplementary shot clips extracted from the original film that match the video plot or visual atmosphere, retaining the original audio, such as the background music, dialogue, and environmental sound effects of the original film, to enrich the layers of the video content.

[0075] The editing and compositing in this embodiment refers to splicing together two different types of shot clips, such as a single-sentence narration clip without audio and a supplementary clip with original audio, according to a preset plot logic and time sequence, into a preliminary semi-finished video product.

[0076] In the specific implementation of step S501, two types of key segments are extracted from the shot matching results of S300: one type is a single-sentence narration segment without audio that corresponds one-to-one with each narration sentence, such as the shot of the protagonist entering the secret room matching the narration "The protagonist enters the secret room"; the other type is supplementary segments with original audio that match the overall plot of the video, such as the dripping sound in the secret room environment and the original shot corresponding to the background music. Then, according to the shot switching rhythm planned in S500, these two types of segments are sequentially spliced ​​together. For example, the "door opening shot" without audio is played first, followed by the "panoramic shot of the secret room" with original ambient sound, forming a semi-finished video containing the core image and supplementary images.

[0077] This invention processes the core visuals carrying the narration separately from the original footage containing rich atmosphere, avoiding interference from the original audio with the subsequent addition of narration audio. Simultaneously, it enhances the realism and immersion of the video by supplementing segments. Furthermore, the segment-by-segment splicing method of this invention allows for individual adjustment of the order or duration of specific segments without requiring a complete rework, improving the efficiency of refined editing and reducing the margin for error in editing.

[0078] S502. Add corresponding audio explanations to single-sentence segments to provide complete audio content; The corresponding narration audio in this embodiment refers to the human voice audio generated in step S400 by a text-to-speech model from the narration sentence text, which perfectly matches the duration of the narration sentence. The complete audio content in this embodiment refers to the final audio track of the video, which integrates the narration audio and the original audio of the original video information segments to form a two-layer audio structure of narration voice + original video background sound / sound effects.

[0079] In this step, the narration audio generated in S400 corresponding to each narration segment is retrieved. Based on the shot timestamp marked in S200, the narration audio is precisely overlaid onto the corresponding narration segment without audio, ensuring that the start time of the narration audio is completely consistent with the start time of the shot, and the end time of the narration is synchronized with the shot transition time. The original audio of the matching segments is preserved, allowing the audio of the two types of segments to connect naturally, forming a complete audio track of the narrator explaining the core scene and the original audio complementing the supplementary scene.

[0080] This invention completely avoids the problem of narration and visual asynchrony by aligning timestamps, ensuring a superior viewing experience and achieving precise audio-visual synchronization. Furthermore, this invention eliminates the need for manual dubbing and audio duration adjustment, automating the entire audio overlay process, significantly reducing editing time and increasing automation.

[0081] S503. Add corresponding subtitle information to the entire narration video; The corresponding subtitle information in this embodiment refers to text subtitles that completely match the video and audio content. It includes two parts: narration subtitles, which are the text content of the narration audio, and / or the narration subtitles of this invention may also include original film subtitles, which are the text content of the characters' dialogue and lines in the original film information segments.

[0082] In this embodiment, the subtitle synchronization means that the display time of the subtitles corresponds precisely to the playback time of the audio. When the audio plays a certain sentence, the corresponding subtitle appears synchronously; when the audio ends, the subtitle disappears synchronously.

[0083] In specific implementation of step S503, based on the narration text and generated narration audio from S400, narration subtitles are automatically generated. The subtitle display segments are then split according to the audio's pace and pauses; for example, a long narration sentence is split into 2-3 lines of subtitles to avoid excessive length. Then, based on the original audio of the original video information segments, such as dialogue, original video subtitles are generated using speech recognition technology, preserving the style of the original subtitles, such as the position and font of the dialogue subtitles. Finally, both types of subtitles are precisely embedded into designated positions on the video screen according to timestamps; for example, the narration subtitles are at the bottom of the screen, and the original video dialogue subtitles are in the lower center of the screen, achieving complete synchronization between audio playback and subtitle display. Of course, in specific implementations of this invention, the original video dialogue subtitles can also be deleted as needed, retaining only the text content subtitles of the narration audio.

[0084] As can be seen, the subtitles in this embodiment of the invention can help viewers understand the narration and original dialogue, and are especially suitable for quiet viewing scenarios (such as subways and offices), expanding the applicable scenarios of videos and improving the efficiency of video information transmission. Furthermore, this invention can automatically generate and synchronize subtitles, eliminating the need for manual line-by-line creation and adjustment of the subtitle timeline, thus solving the pain point of time-consuming subtitle production in traditional manual editing and reducing labor costs.

[0085] S504. Perform video assembly, add fade-in and fade-out effects for audio and video, and perform volume equalization to finally generate a short video with coherent and smooth narration content.

[0086] In this embodiment, the fade-in / fade-out effect refers to the transition effect between video and audio. Video fade-in means that the picture gradually becomes clear from a black screen at the beginning of the shot, and video fade-out means that the picture gradually becomes black again at the end of the shot. Audio fade-in / fade-out means that the audio volume gradually transitions from zero to one and from one to zero, avoiding sudden changes in volume.

[0087] The volume equalization in this embodiment refers to automatically adjusting the volume of different audio segments in the video through an algorithm to coordinate the volume ratio of the narration audio, the original background sound, and the sound effects. For example, the volume of the narration audio is slightly higher than the background sound to ensure that it is clearly audible.

[0088] The final video assembly described in this embodiment integrates all video clips and special effects that have been edited, audio overlaid, and subtitled, and renders them into a complete short video file that can be directly published.

[0089] In the specific implementation of step S504, video transition optimization is first performed. Specifically, fade-in and fade-out effects are added between shots to replace abrupt, direct transitions and improve the visual smoothness of the image. For key scenes, such as climaxes, the duration of fade-in and fade-out can be increased to highlight the rhythm of the plot. Next, audio volume equalization is performed. Specifically, algorithms detect the volume peaks of the narration audio and the original video audio, automatically adjusting the volume of each audio track, reducing the volume of the original video background noise to ensure clear narration, and unifying the volume of different segments to avoid inconsistent volume.

[0090] Then, all the optimized visuals, audio, subtitles, and special effects are integrated to generate a file format (such as MP4) that meets the publishing standards of short video platforms, thus completing the final production of the narration short video.

[0091] This invention employs fade-in / fade-out effects to avoid abrupt scene transitions, and balanced volume ensures auditory comfort, significantly improving the overall quality of the video. Furthermore, this invention can output standardized video files, eliminating the need for manual adjustments to format and volume, allowing direct publishing to short video platforms. Moreover, this invention forms an automated editing closed loop from material acquisition to final output, improving editing efficiency several times over compared to traditional manual methods, while ensuring stable video quality and achieving full-process automation.

[0092] As can be seen from the above, steps S501-S504 in this invention solve the problems of audio-visual asynchrony, audio clutter, lack of subtitles, and abrupt transitions in automated editing through four steps: segment splicing, audio overlay, subtitle addition, and special effects optimization. The final output is a short narration video with coherent content, harmonious audio-visual presentation, and a good user experience.

[0093] The present invention will be further described in detail below through specific application examples: like Figure 3 As shown in this specific application embodiment, the automatic editing method for short video narration includes the following steps: S10. Obtain the submitted videos that require narration, and proceed to S11, S21, and S31 respectively; S11. Obtain the explanatory text and proceed to S12; S12, Clean up the copy and proceed to S13; S13. Perform text segmentation, then proceed to S32; In this embodiment of the invention, high-quality external narration text is obtained, and the narration text is cleaned to conform to the format of third-person narration. It is then sent to a large model for narration text segmentation, making it into narration sentences with independent semantic units, which are used for subsequent shot matching.

[0094] S21. Divide the video into segments and proceed to S22; S22, Internal lens analysis, and proceed to S23; In this embodiment of the invention, a frame can be extracted from a complete film every second as an image, and the image and relevant information about the film can be sent to a visual big model, allowing it to search for information on the Internet and summarize the various plots of the entire film based on the image information.

[0095] When segmenting shots, the specified video is segmented into shots, and all shots in the video are extracted. The segment between each two scene changes is considered as one shot. Shot analysis is to avoid the position of the shot change in the premarital examination video being in the shot flickering stage. If it is in the shot flickering stage, the position of the segment will be appropriately extended by using shot segmentation to avoid the flickering position.

[0096] S23. Perform video frame extraction; and proceed to S24; S24. Perform scenario analysis, then proceed to S25; S25, refine the lens details, then proceed to S32; In this embodiment of the invention, when performing detailed shot analysis, the shot information contained in the plot analysis is analyzed based on the plot analysis and shot segmentation results, and the start and end timestamps of the shots are marked for subsequent shot matching.

[0097] S31. Generate a famous meme, then proceed to S32; In this embodiment of the invention, information on famous scenes from the film will also be sought; specifically, the large model will automatically search for relevant information on the Internet to find famous scenes from the film, which will be used as clips from the original film and added to the final edited narration video.

[0098] S32, wait for the previous steps to complete (blocked), then proceed to S41 and S42; S41. Generate the original image information, and then proceed to S43; In this embodiment, the original film information is generated by matching the famous scene information, narration script segmentation information, and shot detailing information obtained earlier. The famous scene information is then matched with the corresponding narration text and its start and end times in the film, and used as a segment of the automatically edited narration short video.

[0099] S42, Camera matching, then proceed to S43; Based on a deep learning-based visual model, the narration is matched with the video footage. In practice, based on the segmentation information of the narration text and the detailed information of the footage obtained earlier, each narration sentence is matched with a start and end time in the original video. The time matched for each narration sentence is used as a segment of the automatically edited narration video.

[0100] S43. Wait for the previous steps to complete (blocked), then proceed to S44; S44, TTS, based on a text-to-speech model, converts narration into audio, matches the shots, and then proceeds to S45; In this embodiment, TTS sends the narration text information to a text-to-speech model to generate audio information for the narration, which serves as background audio for the narration of the short video.

[0101] S45. Generate subtitle information and proceed to S46; In this embodiment, subtitle information for the narration video is generated based on the generated TTS audio information, which serves as the subtitles for the narration short video.

[0102] S46. Narration video generated; then the process ends.

[0103] This step involves assembling the montage video, specifically including: 1) Obtaining narration segments matching individual narration sentences and narration segments matching the original video information, editing them to create a single video. The individual narration sentence segments have no audio information, while the original video information segments do; 2) Adding generated TTS information to the individual narration sentences to provide audio; 3) Adding SRT subtitle information to the narration video; 4) Assembling the video and adding audio and video effects: volume equalization, and adding fade-in / fade-out effects for audio and video. Then the process ends.

[0104] In a further embodiment of the present invention, in order to solve the problem of abrupt viewing caused by the mismatch between the narration and the emotional content of the visuals in the automated generation of short videos for film and television commentary, the present invention further proposes an optimization scheme, as described in Embodiment 3.

[0105] Example 3 The automatic editing method for short video narration provided in this embodiment three also includes the following steps: S901. Perform sentiment analysis on each single sentence in the explanatory text to identify the sentiment and intensity; In this third embodiment, after the step of obtaining and processing the explanatory text, a new step of sentiment analysis of the explanatory text is added. Natural language processing (NLP) technology is used to analyze the sentiment tendency of each explanatory sentence and identify its main emotions such as excitement, sadness, calmness, tension and their intensity.

[0106] S902. Use a large visual model to perform emotion recognition on each shot segment and analyze the emotional tendency and intensity contained in the visual elements of the shot. In this embodiment, after the plot analysis and shot segmentation and refinement analysis steps, a visual emotion analysis step is added. The visual big model is used to perform emotion recognition on each shot segment and analyze the emotional tendency and intensity contained in visual elements such as screen color, character expression, action, scene atmosphere, etc.

[0107] S903. Compare the sentiment analysis results of the matched narration with the visual sentiment analysis results of the corresponding shot segments; In this embodiment, after the initial matching of narration and shots is completed, an emotional consistency verification step is added, which compares the emotional analysis results of the matched narration with the visual emotional analysis results of the corresponding shot segments.

[0108] S904. When there is a significant difference in type or intensity between the emotion of the narration and the emotion of the shot, it is marked as an emotion mismatch. For shot segments with emotion mismatch, try to find other shot segments in the same plot or adjacent plots that match the emotion of the current narration and replace them. In this embodiment, if a significant difference in type or intensity is detected between the emotion of the narration and the emotion of the scene, for example, the narration is "excited" while the scene is "calm"; or the narration has a high emotional intensity while the scene has a low emotional intensity, then it is marked as an emotion mismatch.

[0109] In this embodiment, for segments with mismatched emotions, the system prioritizes searching for other shot segments within the same or adjacent scenes that better match the current narration's emotional tone for replacement. For example, if the narration for "facing a crisis" conveys the emotion of "tension," but the currently matched shot is a calm cityscape in the distance, the system will attempt to find a more alarming scene, such as a scene of panic, for replacement.

[0110] S905. When a matching shot cannot be found, the tone, speed and timbre of the narration audio are dynamically adjusted according to the sentiment analysis results of the narration, so that the narration audio conforms to the emotional expression of the narration. In this embodiment, if a more matching shot cannot be found, or the emotional difference is small, the tone, speed, and timbre of the TTS speech are dynamically adjusted in the TTS text-to-speech unit based on the sentiment analysis results of the narration, making it more consistent with the emotional expression of the narration. For example, for an exciting narration describing the ignition of a planetary engine, the TTS speech in this embodiment of the invention will use a more impassioned tone.

[0111] S906. For segments with mismatched emotions, control the intelligent selection or adjustment of transition effects and background music to compensate for the lack of emotion and enhance the overall harmony.

[0112] In this embodiment, for segments with mismatched emotions, transition effects such as fast cuts and slow fade-in / fade-out can be intelligently selected or adjusted, as well as background music such as rousing, soothing, or tense to compensate for the lack of emotion and enhance the overall harmony. For example, fast cuts and tense background music can be added before tense scenes.

[0113] As can be seen from the above, this third embodiment, by introducing an emotional dimension, expands the original editing logic based on content and time matching into a three-dimensional matching and optimization integrating content, time, and emotion. By performing independent emotional analysis on the narration and visual images, and verifying emotional consistency during the matching stage, emotional inconsistencies can be effectively identified and corrected. When an emotional mismatch is detected, this invention, according to a preset adjustment strategy, prioritizes re-matching shots to address the root cause, and then assists in improving emotional consistency by adjusting the emotional expression of the TTS voice and optimizing transition effects and background music. In this way, overly complex artificial intelligence algorithms are avoided; instead, functional extensions are based on existing NLP and large visual model technologies, resulting in high rationality and practicality. This significantly improves the emotional expressiveness and user experience of automatically generated short videos, resolving the abrupt viewing experience caused by emotional mismatch between narration and visuals.

[0114] Exemplary device like Figure 4 As shown, this embodiment of the invention provides an automatic editing device for short video narration, the device comprising: The multimodal feature extraction module 310 is used to acquire the original video of the video to be edited and explained, and to extract multiple modal information including visual and textual information from the original video using a multimodal feature extraction method. The plot analysis unit 320 is used to perform in-depth analysis of the original video using a large visual model, and to identify and summarize the various plot contents of the video. The shot segmentation unit 330 is used to segment the original video into shots, extract all shot information, and mark the start and end timestamps of each shot. The narration shot matching unit 340 is used to match the preset narration with the shot information in the film using a deep learning visual model. The TTS text-to-speech unit 350 is used to convert the matched narration into corresponding narration audio based on the text-to-speech model and synchronize it with the corresponding video footage to provide background narration for short videos. The editing logic generation module 360 ​​is used to dynamically plan the rhythm of shot switching based on the identified plot content, shot matching information and synchronized narration audio, and intelligently insert transition effects to assemble the mixed video, as described above.

[0115] Based on the above embodiments, the present invention also provides a server, which can be a smart TV, and its principle block diagram can be as follows. Figure 5 As shown. The server includes a processor, memory, network interface, display screen, and database connected via a system bus.

[0116] The memory stores one or more programs configured to be executed by a processor to implement the automatic editing method for short video narration described in the above embodiments.

[0117] In this context, "server" refers to an intelligent computer or similar device with data processing capabilities. "Memory" can be internal memory, flash memory, hard disk, or cloud storage, used to store program code, pre-acquired external narration text, identified plot details, shot matching information, synchronized narration audio, and other data. "Processor" can be a central processing unit (CPU), used to execute the algorithmic logic within the program. The program includes an automatic editing method for short video narration.

[0118] In a further embodiment, a server of this embodiment includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for performing the following operations: A acquires the original video that needs to be edited and explained, and uses a multimodal feature extraction method to extract multiple modal information, including visual and textual information, from the original video. B uses a large visual model to conduct in-depth analysis of the original video, identify and summarize the various plot contents of the video; and divides the original video into shot segments, extracts all shot information, and marks the start and end timestamps of each shot; C uses a deep learning visual model to match the pre-set narration with the shot information in the film; D is based on a text-to-speech model, which converts the matched narration into corresponding audio narration and synchronizes it with the corresponding video footage to provide background narration for short videos; Based on the identified plot content, shot matching information, and synchronized narration audio, E dynamically plans the rhythm of shot transitions and intelligently inserts transition effects to assemble the mixed video, as described above.

[0119] The step of obtaining the original video for editing and narration, and extracting multimodal information including visual and textual information from the original video using a multimodal feature extraction method, includes the following steps before the following: Obtain external explanatory text, clean the explanatory text to conform to the third-person narration format, and use a preset large model to divide the cleaned explanatory text into multiple independent semantic units to form explanatory sentences.

[0120] The step of using a large visual model to perform in-depth analysis of the original video and identify and summarize the various plot points of the video includes: One frame is extracted from the original video every second as an image, and the extracted images and relevant information from the original video are input into a preset visual model for in-depth analysis to identify and summarize the various plot contents of the video.

[0121] The step of matching the preset narration with the shot information in the film using a deep learning visual model includes: Using a deep learning-based visual model, the system automatically searches for memorable scenes from the original video. Based on the aforementioned famous scene information, narration segmentation information, and shot information, each famous scene information is matched with the corresponding narration, and the start and end times in the film are determined.

[0122] The step of converting the matched narration into corresponding audio based on a text-to-speech model and synchronizing it with the corresponding video footage to provide background narration for the short video includes: The matched narration is input into a preset text-to-speech model to generate corresponding narration audio, which is then used as background narration audio for the short video. Based on the generated audio narration, subtitle information for the narration video is generated and used as subtitles for the short video footage.

[0123] The steps of dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the montage video based on the identified plot content, shot matching information, and synchronized narration audio include: The narration segments that match individual sentences and the narration segments that match the original video information are then edited and combined into a single video. Initially, the individual sentence narration segments did not contain audio information, while the original video information segments contained the original audio. Add corresponding audio explanations to single-sentence segments, providing complete audio content; Add corresponding subtitles to the entire commentary video; The video is assembled, audio and video fade-in / fade-out effects are added, and volume equalization is performed to ultimately generate a short video with coherent and smooth narration.

[0124] The step of dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the montage video based on the identified plot content, shot matching information, and synchronized narration audio further includes: Sentiment analysis was performed on each sentence in the explanatory text to identify the sentiment and intensity. Using a large visual model, emotion recognition is performed on each shot segment to analyze the emotional tendency and intensity contained in the visual elements of the shot; Compare the sentiment analysis results of the matched narration with the visual sentiment analysis results of the corresponding shot segments; When there is a significant difference in type or intensity between the emotion of the narration and the emotion of the shot, it is marked as an emotion mismatch; for shot segments with emotion mismatch, try to find other shot segments in the same plot or adjacent plots that match the emotion of the current narration and replace them; If a matching shot cannot be found, the tone, speed and timbre of the narration audio are dynamically adjusted based on the sentiment analysis results of the narration, so that the narration audio matches the emotional expression of the narration. For segments where the emotions are mismatched, the system intelligently selects or adjusts transition effects and background music to compensate for the lack of emotion and enhance the overall harmony, as described above.

[0125] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, enables an electronic device to perform the steps of the method described in any of the above embodiments.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An automatic editing method for short video narration, characterized in that, include: Obtain the original video of the video to be edited and explained, and use a multimodal feature extraction method to extract multiple modal information including visual and textual information from the original video. The original video is analyzed to identify the various plot points of the video; the original video is then segmented into shots, all shot information is extracted, and the start and end timestamps of each shot are marked. Match the pre-set narration with the shot information in the film; Based on a text-to-speech model, the matched narration is converted into corresponding audio narration and synchronized with the corresponding video footage to provide background narration for short videos; Based on the identified plot content, shot matching information, and synchronized narration audio, the rhythm of shot switching is dynamically planned, and transition effects are intelligently inserted to assemble the mixed video.

2. The automatic editing method for short video narration according to claim 1, characterized in that, Before the step of obtaining the original video clip to be edited and explained, and extracting multimodal information including visual and textual information from the original video clip using a multimodal feature extraction method, the following steps are included: Obtain external explanatory text, clean the explanatory text to conform to the third-person narration format, and use a preset large model to divide the cleaned explanatory text into multiple independent semantic units to form explanatory sentences.

3. The automatic editing method for short video narration according to claim 1, characterized in that, The steps of analyzing the original video and identifying the various plot points of the video include: One frame is extracted from the original video every second as an image, and the extracted images and relevant information from the original video are input into a preset visual model for analysis to identify various plot points of the video.

4. The automatic editing method for short video narration according to claim 2, characterized in that, The step of matching the preset narration with the shot information in the film includes: Using a deep learning-based visual model, the system automatically searches for memorable scenes from the original video. Based on the aforementioned famous scene information, narration segmentation information, and shot information, each famous scene information is matched with the corresponding narration, and the start and end times in the film are determined.

5. The automatic editing method for short video narration according to claim 4, characterized in that, The steps of converting the matched narration into corresponding audio based on the text-to-speech model and synchronizing it with the corresponding video footage to provide background narration for the short video include: The matched narration is input into a preset text-to-speech model to generate corresponding narration audio, which is then used as background narration audio for the short video. Based on the generated audio narration, subtitle information for the narration video is generated and used as subtitles for the short video footage.

6. The automatic editing method for short video narration according to claim 1, characterized in that, The steps of dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the montage video based on the identified plot content, shot matching information, and synchronized narration audio include: The narration segments that match individual sentences and the narration segments that match the original video information are then edited and combined into a single video. Initially, the individual sentence narration segments did not contain audio information, while the original video information segments contained the original audio. Add corresponding audio explanations to single-sentence segments, providing complete audio content; Add corresponding subtitles to the entire commentary video; The video is assembled, audio and video fade-in / fade-out effects are added, and volume equalization is performed to ultimately generate a short video with coherent and smooth narration.

7. The automatic editing method for short video narration according to claim 1, characterized in that, The steps of dynamically planning the rhythm of shot transitions and intelligently inserting transition effects to assemble the montage video based on the identified plot content, shot matching information, and synchronized narration audio also include: Sentiment analysis was performed on each sentence in the explanatory text to identify the sentiment and intensity. Using a large visual model, emotion recognition is performed on each shot segment to analyze the emotional tendency and intensity contained in the visual elements of the shot; Compare the sentiment analysis results of the matched narration with the visual sentiment analysis results of the corresponding shot segments; When there is a significant difference in type or intensity between the emotion of the narration and the emotion of the shot, it is marked as an emotion mismatch; for shot segments with emotion mismatch, try to find other shot segments in the same plot or adjacent plots that match the emotion of the current narration and replace them; If a matching shot cannot be found, the tone, speed and timbre of the narration audio are dynamically adjusted based on the sentiment analysis results of the narration, so that the narration audio matches the emotional expression of the narration. For segments where the emotions don't match, the system intelligently selects or adjusts transition effects and background music to compensate for the lack of emotion and enhance overall harmony.

8. An automatic editing device for short video narration, characterized in that, The device includes: The multimodal feature extraction module is used to acquire the original video of the video to be edited and explained, and to extract multiple modal information, including visual and textual information, from the original video using a multimodal feature extraction method. The plot analysis unit is used to analyze the original video and identify the various plot contents of the video. The shot segmentation unit is used to segment the original video into shots, extract all shot information, and mark the start and end timestamps of each shot. The narration shot matching unit is used to match preset narration with shot information in the film; The TTS (Text-to-Speech) unit is used to convert the matched narration into corresponding audio narration based on the text-to-speech model, and synchronize it with the corresponding video footage to provide background narration for short videos. The editing logic generation module is used to dynamically plan the rhythm of shot transitions based on the identified plot content, shot matching information, and synchronized narration audio, and intelligently insert transition effects to assemble the mixed video.

9. A server, characterized in that, It includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, wherein the one or more programs include steps for performing the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it enables the electronic device to perform the steps of the method as described in any one of claims 1-7.

Images