Video editing method and related equipment

Abstract

The invention discloses a video editing method and related equipment, and the method comprises the steps: receiving an editing intention of a user, carrying out the semantic analysis of an intention analysis model, and generating an editing copywriting, a style and a structured instruction; the edited copywriting is split according to the edited copywriting; performing cross-modal analysis on video materials by using a video understanding model trained based on an open-source multi-modal model, and accurately positioning sub-lens segments; secondly, editing the sub-lens segments through an editing model to generate a preliminary sheet, performing quantitative scoring through a video scoring model, and obtaining a scoring result according to a multi-dimensional index; and finally, the result is fed back to the editing model, and iterative adjustment is carried out until a final slice is produced. According to the method, the editing efficiency is improved, the manual operation time is shortened, the material adaptability is enhanced, the personalized requirement is met, the editing effect consistency is improved, the video content understanding is deepened, the editing effect is optimized, the technical threshold is reduced, the existing technical problems are solved comprehensively, and an efficient, intelligent and personalized editing scheme is provided.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and more specifically, to a video editing method and related equipment. Background Art

[0002] In today's Internet era, video editing has penetrated widely into various fields such as e-commerce live streaming, education and teaching, and the entertainment industry, and the demand for video editing is showing a blowout growth.

[0003] Existing editing technologies fall into two main categories. The first is traditional manual editing. In this approach, editors rely on their personal experience to analyze video content frame by frame, manually screen footage, determine editing points, and apply special effects. This process not only consumes significant time and effort, resulting in lengthy editing cycles, but also relies heavily on the editor's individual skills. Different editors can produce vastly different results when processing the same footage, making it difficult to ensure consistent video quality. The second category relies on video editing software or automated editing templates. While professional video editing software like Adobe Premiere Pro, Final Cut Pro, and DaVinci Resolve offer a wealth of editing tools and features, they still require significant manual effort for core steps like organizing footage and making editing decisions. This inherently hinders the inefficiencies and inconsistent quality associated with manual intervention. While automated editing tools based on rules or templates, such as some simple editing software based on visual and audio features, attempt to improve editing efficiency, they lack a deep understanding of video content and a precise understanding of user needs. Consequently, they can only process specific types of video footage, and the editing results often fail to meet diverse user needs.

[0004] Overall, existing technologies have problems such as low editing efficiency, insufficient automation, unstable editing effects, limited intelligence level and high technical threshold, making it difficult to adapt to the needs of the rapidly developing video content creation market.

[0005] Based on this, the present application provides an efficient, intelligent and personalized video editing solution, which achieves accurate editing while significantly reducing editing time and effectively avoids various problems existing in the existing technology. Summary of the Invention

[0006] The present application provides a video editing method and related equipment, which achieves improved editing efficiency, reduced manual operation time, enhanced material adaptability, satisfaction of personalized needs, improved consistency of editing effects, deepened understanding of video content, optimized editing effects and lowered technical barriers.

[0007] A video editing method, comprising:

[0008] Receive user editing intentions, and perform semantic analysis on the user editing intentions through the intention analysis model to generate editing copy, editing style, and structured instructions including time allocation weights and scene logic association rules;

[0009] Performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies;

[0010] Performing cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each of the storyboards. The video understanding model is based on an open-source multimodal model and is trained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards.

[0011] The storyboard clips are edited with special effects using an editing model to generate a preliminary finished film, and the preliminary finished film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user intention matching, special effect naturalness, and style consistency;

[0012] The scoring results are fed back to the editing model to iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0013] Optionally, the editing copy is subjected to storyboard splitting processing based on the structured instruction and the editing style to obtain multiple storyboards, including:

[0014] Dividing the editing copy into a plurality of time intervals according to a preset time threshold according to the time allocation weight in the structured instruction;

[0015] In each time interval, a scene switching trigger point is identified according to the scene logic association rule in the structured instruction, and a plurality of scene units are obtained based on the scene switching trigger point, wherein the trigger point includes a scene keyword or an emotional transition word;

[0016] Assigning an emotional label to each of the scene units based on the editing style, and determining transition connection rules between the scene units;

[0017] Generate storyboards with emotional labels and transition rules.

[0018] Optionally, the cross-modal analysis process of the video understanding model includes:

[0019] Separately extracting semantic text features of the storyboard copy, spatiotemporal visual features, and audio emotional features of the video material;

[0020] Mapping the semantic text features, the spatiotemporal visual features, and the audio emotion features to the same latent space for joint encoding;

[0021] The video material is traversed, the cross-modal similarity between each video clip in the video material and the storyboard is calculated and dynamically weighted according to the editing style to obtain a matching score, and the video clip with the highest matching score is determined as the storyboard clip that matches the storyboard.

[0022] Optionally, the process of the video scoring model performing quantitative scoring on the input preliminary film includes:

[0023] Determining the user intent matching degree by comparing the semantic overlap and emotional label consistency of the preliminary finished film subtitles with the storyboard copy;

[0024] Determining the naturalness of special effects by analyzing the motion continuity and audio rhythm synchronization of the transition area in the preliminary film;

[0025] Determining the style consistency by detecting the difference between the color tone, shot duration, and music type of the preliminary film and the preset style library;

[0026] A scoring result is generated based on the user intention matching degree, the special effect naturalness and the style consistency.

[0027] Optionally, the iterative optimization process of the editing model includes:

[0028] The scoring result is used as a feedback adjustment signal to dynamically adjust the probability of transition special effect selection, shot playback rate and scene switching direction, and the parameters of the editing model are updated through a policy gradient algorithm until the scoring result meets the threshold.

[0029] Optionally, also include:

[0030] A user real-time feedback interface is introduced in the iterative process, and the user's manual correction instructions for the preliminary film are received through the user real-time feedback interface, and the manually corrected correction data is added to the training set. The model parameters of the intent analysis model, the video understanding model and the editing model are updated through online learning.

[0031] A video editing device, comprising:

[0032] The editing intention module is used to receive the user's editing intention and perform semantic analysis on the user's editing intention through the intention analysis model to generate editing copy, editing style and structured instructions including time allocation, scene logic and emotional expression;

[0033] A storyboard splitting module is used to perform storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies;

[0034] A segment location module is used to perform cross-modal analysis on the video material using a video understanding model to locate the storyboard segments that match the storyboards in the video material. The video understanding model is based on an open source multimodal model and is obtained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards.

[0035] A quantitative scoring module is used to edit the storyboard clips with special effects using an editing model to generate a preliminary finished film, and input the preliminary finished film into a video scoring model for quantitative scoring, generating a scoring result based on the user intention matching, special effect naturalness, and style consistency;

[0036] The iterative editing module is used to feed back the scoring results to the editing model, iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0037] A video editing device comprising a memory and a processor;

[0038] The memory is used to store programs;

[0039] The processor is used to execute the program to implement each step of the video editing method as described in any one of the above items.

[0040] A readable storage medium stores a computer program thereon, wherein when the computer program is executed by a processor, each step of the video editing method as described above is implemented.

[0041] A computer program product includes a computer program, wherein the computer program executes each step of any of the above-mentioned video editing methods when the computer program is executed by a processor.

[0042] It can be seen from the above technical solutions that a video editing method and related equipment provided by an embodiment of the present application first receives the user's editing intention, and performs semantic analysis on it with the help of an intention analysis model to generate editing text, editing style, and structured instructions containing time allocation weights and scene logic association rules. Then, based on the structured instructions and editing style, the editing text is split into storyboards to obtain multiple storyboards. Afterwards, a video understanding model trained based on an open source multimodal model is used to perform cross-modal analysis on the video material, and the storyboard segments that match each storyboard are accurately located from the video material. The storyboard segments are then edited with special effects through the editing model to generate a preliminary film, and the preliminary film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user's intention matching, special effect naturalness, and style consistency. Finally, the scoring result is fed back to the editing model, and the editing order, transition effect, and special effect parameters are iteratively adjusted until the final editing film is output.

[0043] This application significantly improves the efficiency of video editing and greatly reduces manual operation time by automating the processing of user intent analysis, storyboard splitting, video material matching and editing optimization. Based on the intention analysis model and the video understanding model, intelligent video editing decisions are implemented, which can adapt to different types of video materials and meet the personalized needs of users. Through the guidance of structured instructions and editing styles, the consistency of the storyboard splitting and editing process is ensured, the subjective differences caused by manual intervention are reduced, and the stability of video quality is improved. Combined with multimodal feature analysis and reinforcement learning algorithms, the present invention achieves a deep semantic understanding of video content, accurately identifies key clips and emotional expressions, and optimizes the editing effect. In addition, through the automated and intelligent editing process, the user's dependence on professional skills is reduced, so that non-professionals can easily complete high-quality video editing.

[0044] To sum up, the present application provides an efficient, intelligent and personalized video editing solution, which improves editing efficiency, reduces manual operation time, enhances material adaptability, meets personalized needs, improves consistency of editing effects, deepens understanding of video content, optimizes editing effects and lowers technical barriers. It effectively solves the problems existing in the existing technology and promotes the development of video editing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0046] Figure 1 This is a flowchart of a video editing method disclosed in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a video editing method disclosed in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a video understanding model training and processing process disclosed in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the training and processing process of a video scoring model and editing model disclosed in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a video editing device disclosed in an embodiment of the present application;

[0051] Figure 6This is a hardware structure block diagram of a video editing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The present application can be used in a variety of general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, and distributed computing environments including any of the above devices or devices.

[0054] Next, we will introduce the application scheme. This application proposes the following technical scheme, please see below for details.

[0055] Figure 1 This is a flowchart of a video editing method disclosed in an embodiment of the present application.

[0056] Figure 2 A schematic diagram of a video editing method disclosed in an embodiment of the present application.

[0057] like Figure 1 and Figure 2 As shown, the method may include:

[0058] Step S1: Receive the user's editing intention, and perform semantic analysis on the user's editing intention through the intention analysis model to generate editing copy, editing style and structured instructions including time allocation weights and scene logic association rules.

[0059] Specifically, users enter their editing intent via text or voice through the system's interactive interface. This includes key information such as the desired editing duration, desired editing style, and core editing content. Upon receiving this editing intent, the system transmits it to an intent analysis model, which can be a large language model (LLM). Based on its massive pre-trained text data and advanced natural language processing technology, the intent analysis model performs a deep semantic analysis of the user's editing intent. First, the model optimizes and polishes the original editing content with grammar and semantics, making it more precise and professional, in line with the conventional expression practices of video production. Second, based on the user's inputted fuzzy style description and combined with its internal knowledge base of style features, the model matches a specific editing style that is highly compatible with it, such as retro, modern minimalist, or lively and dynamic. Furthermore, the model analyzes the user's potential requirements for the overall video structure and time allocation, generating structured instructions that include time allocation weights and scene-by-scene logical association rules. For example, if a user wants to create a 5-minute travel video that focuses on scenic spots and local food, the model may generate the following structured instructions: 3 minutes, or 60%, are allocated to the scenic spots; 2 minutes, or 40% are allocated to the food; the transition between the scenic spots and the food scenes is made with a fade-in and fade-out transition, etc.

[0060] Step S2: performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies.

[0061] Specifically, based on the structured instructions and editing style generated in step S1, the polished editing copy is split into storyboards. First, according to the scene logic association rules in the structured instructions, the editing copy is divided into different theme scene modules, such as the scenic scene module and the local food scene module in the above-mentioned travel video case. For each scene module, the split is further refined in combination with the established editing style. For example, under the scenic scene module, if the editing style is lively and dynamic, it may be split into the following storyboards: "The camera quickly zooms in to show the magnificent mountain panorama, and exciting background music plays synchronously", "Follow the footsteps of tourists and show the scenes of walking through the ancient woods from a follow-up perspective. The pictures are bright and colorful", etc. In the splitting process, the time allocation weight in the structured instructions is strictly followed, and the approximate duration corresponding to each storyboard copy is reasonably determined to ensure that the rhythm of the entire video is reasonable and the narrative is smooth.

[0062] Step S3: Use a video understanding model to perform cross-modal analysis on the video material, and locate the storyboard segments that match the storyboard texts from the video material. The video understanding model is based on an open source multimodal model and is obtained through large-scale supervised training using sample video materials, sample storyboard texts, and sample storyboard texts.

[0063] Specifically, such as Figure 3 As shown, the video understanding model is trained based on an open-source multimodal model. This training process utilizes a large amount of sample video footage (video clips covering various scenes, styles, and filming techniques), sample storyboards (text descriptions of storyboards corresponding to different themes and styles), and labeled data (i.e., accurately labeled storyboard segments that match the storyboards, belonging to the text modality) for large-scale supervised training. In practical applications, the video footage input by the user and the storyboards obtained in step S2 are fed into the video understanding model. Leveraging the cross-modal association capabilities learned during training, the model analyzes the video footage frame by frame, extracting visual features (such as color, composition, and object motion) and auditory features (such as background music, ambient sound effects, and character dialogue) from the video. It also extracts textual features from the storyboards. Using a sophisticated feature matching algorithm, it accurately locates the video clip within the video footage that best matches each storyboard. For example, for a storyboard copy "The camera quickly zooms in to show a magnificent mountain panorama, and exciting background music plays synchronously", the model will search the video material for clips containing a mountain panorama, fast camera movement, and exciting background music, and cut them out as the material clips for the corresponding storyboard.

[0064] Step S4: Edit the storyboard clips with special effects through the editing model to generate a preliminary film, and input the preliminary film into the video scoring model for quantitative scoring, and generate a scoring result based on the user intention matching, special effect naturalness and style consistency.

[0065] Specifically, such as Figure 4 As shown, the editing model is an intelligent model built using reinforcement learning technology. Its construction process involves first writing a series of agent functions, such as those for handling video transitions, scene switching, and camera acceleration and deceleration. Each of these functions is responsible for a specific editing operation. Storyboards are then fed into the editing model, and a scheduling agent coordinates the operations of the various agent functions. It flexibly determines which functions to call and how to combine these operations based on information such as the storyboard content, editing style, and user editing intent.

[0066] During the training phase, a large number of video clips of various types are collected from the internet and then crowdsourced for rating by human editors. A video scoring model is trained based on this rated data. The editing model then uses this video scoring model to score the multiple finished versions it generates. Based on reinforcement learning, the model rewards high-scoring clips and penalizes low-scoring ones. Through multiple rounds of training and parameter tuning, the editing model masters editing techniques that are difficult to define using rules. It can consistently produce high-quality, smooth clips that align with the editing intent and style, achieving results that match or even surpass those of human editing.

[0067] The storyboard segments located in step S3 are input into the editing model. Based on pre-set editing rules and information such as the editing style analyzed from the user's editing intent, the editing model performs editing operations on the storyboard segments, such as adding filters, adjusting color contrast, adding subtitles, and applying transition effects. For example, if the editing style is retro, the editing model may add filter effects such as yellowing and graininess to the storyboard segments, as well as use transition effects similar to switching between old film reels. After a series of editing processes, the individual storyboard segments are spliced together in sequence to generate a preliminary film.

[0068] The video scoring model is a multimodal model that aims to quantitatively evaluate the quality of video editing. Figure 4 As shown, during training, a large amount of collected video editing data with ratings from editors is used to allow the model to learn the relationship between the visual, auditory, and other multimodal features of different videos and the final score. During the evaluation phase, the model analyzes and scores the input preliminary film from multiple dimensions: by comparing the semantic overlap and emotional label consistency between the film's subtitles and the storyboard copy, the user intent match is determined; the motion continuity and audio rhythm synchronization in the transition area are analyzed to determine the naturalness of the special effects; and the color tone, shot length, and music type of the film are detected to measure the difference with the preset style library to measure the consistency of style. Finally, these dimensions are combined and the final score is calculated according to the preset weights, providing feedback for the optimization of the editing model.

[0069] The process of the video scoring model quantitatively scoring the input preliminary film includes:

[0070] ① Determine the user intent matching degree by comparing the semantic overlap and emotional label consistency between the subtitles of the preliminary film and the storyboard copy;

[0071] ② Determine the naturalness of the special effects by analyzing the motion continuity and audio rhythm synchronization of the transition area in the preliminary film;

[0072] ③ Determine the style consistency by detecting the difference between the color tone, shot duration and music type of the preliminary film and the preset style library;

[0073] ④ Generate a scoring result based on the user intention matching, the naturalness of the special effects and the style consistency.

[0074] Specifically, the video scoring model obtains the content of the subtitles of the preliminary film and compares it with the storyboard copy obtained in step S2. Natural language processing technology is used to calculate the semantic overlap between the two, such as through lexical analysis, syntactic analysis, and semantic similarity algorithms (such as cosine similarity algorithm) to measure the degree of semantic similarity. At the same time, the emotional labels (such as positive, negative, neutral and other emotional categories) contained in the storyboard copy and the film subtitles are analyzed to determine the consistency of the emotional labels. The two dimensions of semantic overlap and emotional label consistency are combined to determine the user intention matching degree. For example, if the film subtitles accurately convey the scene information described in the storyboard copy and the emotional tone is consistent, then the user intention matching degree is high.

[0075] For the transition areas in the preliminary film, the video scoring model analyzes the motion continuity of objects in the video. For example, when switching from one scene to another, the model observes whether the object's motion trajectory is smooth and natural, and whether there are any abrupt jumps or pauses. At the same time, the model analyzes the synchronization between the audio rhythm and the transition, for example, whether the transition of the background music during the transition is natural, and whether the sound effects match the picture switching. By evaluating the motion continuity and audio rhythm synchronization, the naturalness of the special effects is determined. If the transition between the picture and audio is smooth and there is no sense of disobedience, the special effect is highly natural.

[0076] The video scoring model examines the color palette of the initial film and compares it to the standard tonal range of the corresponding style in the preset style library, such as warm yellow for retro style or light blue for fresh style. It also checks whether the shot duration matches the rhythmic characteristics of the preset style. For example, a fast-paced style may have short shots with frequent switching, while a slow-paced style may have longer shots. Furthermore, the model analyzes whether the music genre matches the preset style, such as classical music for a classical style or electronic pop for a modern style. By measuring the differences in these aspects from the preset style library, the model determines the degree of stylistic consistency.

[0077] Based on the user intention matching, special effects naturalness and style consistency determined above, a weighted calculation is performed according to the pre-set weights to generate the final scoring result.

[0078] Step S5: Feedback the scoring result back to the editing model, iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0079] Specifically, the iterative optimization process of the editing model includes:

[0080] The scoring result is used as a feedback adjustment signal to dynamically adjust the probability of transition special effect selection, shot playback rate and scene switching direction, and the parameters of the editing model are updated through a policy gradient algorithm until the scoring result meets the threshold.

[0081] Dynamic adjustment based on scoring results: The scoring results obtained in step S4 are input into the editing model as feedback adjustment signals. The editing model dynamically adjusts relevant editing parameters based on the scoring results. For example, if the scoring results show that the naturalness of the special effects is low, it may be that the transition effects are too abrupt. In this case, the editing model will adjust the probability of selecting the transition effects to increase the probability of selecting more natural transition effects. If it is found that the shot playback rate does not match the overall rhythm, resulting in a loss of style consistency, the shot playback rate will be adjusted. If the unreasonable shot switching direction affects the user's intention matching, the shot switching direction will be adjusted.

[0082] Parameter Update via Policy Gradient Algorithm: The editing model's parameters are updated using a policy gradient algorithm. Based on the current scoring feedback, the policy gradient algorithm calculates the direction and magnitude of model parameter adjustments, enabling optimization toward improved scoring results during subsequent editing. For example, over multiple iterations, by continuously adjusting parameters, the editing model gradually learns how to select more appropriate transition effects, set more appropriate shot playback rates, and change camera angles to meet scoring requirements. This process continues until the scoring meets a pre-set threshold.

[0083] In addition, this application can also introduce a user real-time feedback interface during the iteration process.

[0084] The user's manual correction instructions for the preliminary film are received through the user real-time feedback interface, and the manually corrected correction data is added to the training set, and the model parameters of the intent analysis model, the video understanding model and the editing model are updated through online learning.

[0085] Specifically, during the iteration process of the editing model, the system provides users with a real-time feedback interface, through which users can watch and evaluate the preliminary film, and then input manual correction instructions through the interface, such as "the transition effect here is softer", "the duration of this shot needs to be shortened", etc. The system collects the correction data generated by the user's manual correction and adds it to the training set. Then, through online learning, this new data is used to update the model parameters of the intent analysis model, video understanding model and editing model. For example, the intent analysis model can learn from user feedback a more accurate understanding of the description of certain specific editing intentions; the video understanding model can further optimize the accuracy of matching video materials and storyboards; and the editing model can better adjust the editing operation based on user feedback, thereby continuously improving the performance and effectiveness of the entire video editing system.

[0086] It can be seen from the above technical solutions that a video editing method and related equipment provided by an embodiment of the present application first receives the user's editing intention, and performs semantic analysis on it with the help of an intention analysis model to generate editing text, editing style, and structured instructions containing time allocation weights and scene logic association rules. Then, based on the structured instructions and editing style, the editing text is split into storyboards to obtain multiple storyboards. Afterwards, a video understanding model trained based on an open source multimodal model is used to perform cross-modal analysis on the video material, and the storyboard segments that match each storyboard are accurately located from the video material. The storyboard segments are then edited with special effects through the editing model to generate a preliminary film, and the preliminary film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user's intention matching, special effect naturalness, and style consistency. Finally, the scoring result is fed back to the editing model, and the editing order, transition effect, and special effect parameters are iteratively adjusted until the final editing film is output.

[0087] This application significantly improves the efficiency of video editing and greatly reduces manual operation time by automating the processing of user intent analysis, storyboard splitting, video material matching and editing optimization. Based on the intention analysis model and the video understanding model, intelligent video editing decisions are implemented, which can adapt to different types of video materials and meet the personalized needs of users. Through the guidance of structured instructions and editing styles, the consistency of the storyboard splitting and editing process is ensured, the subjective differences caused by manual intervention are reduced, and the stability of video quality is improved. Combined with multimodal feature analysis and reinforcement learning algorithms, the present invention achieves a deep semantic understanding of video content, accurately identifies key clips and emotional expressions, and optimizes the editing effect. In addition, through the automated and intelligent editing process, the user's dependence on professional skills is reduced, so that non-professionals can easily complete high-quality video editing.

[0088] To sum up, the present application provides an efficient, intelligent and personalized video editing solution, which improves editing efficiency, reduces manual operation time, enhances material adaptability, meets personalized needs, improves consistency of editing effects, deepens understanding of video content, optimizes editing effects and lowers technical barriers. It effectively solves the problems existing in the existing technology and promotes the development of video editing technology.

[0089] In some embodiments of the present application, the process of step S2, performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboards, is introduced, and may specifically include:

[0090] Step S21: Divide the editing copy into multiple time intervals according to the preset time length threshold according to the time allocation weight in the structured instruction;

[0091] Step S22: In each time interval, identifying scene switching trigger points according to the scene logic association rules in the structured instruction, and splitting the scene switching trigger points into a plurality of scene units, wherein the trigger points include scene keywords or emotional transition words;

[0092] Step S23: assigning an emotion label to each of the scene units based on the editing style, and determining a transition rule between the scene units;

[0093] Step S24: Generate a storyboard with emotional labels and transition rules.

[0094] Specifically, the structured instructions include time allocation weight information. For example, when making a 5-minute video, it may be stipulated that the opening part accounts for 10% of the time, the main content display accounts for 70% of the time, and the ending summary accounts for 20% of the time. Based on these weights and the preset time threshold, the video length corresponding to the entire editing copy is divided into multiple time intervals. Each time interval corresponds to a certain proportion of video content, and based on the time allocation weight, the approximate content direction of the interval is clarified. For example, a certain interval is mainly used to display core events, and another interval is used for transition preparation, etc.

[0095] Within each defined time interval, scene-change triggers are identified based on the scene logic association rules in the structured instructions. Scene keywords are words that clearly identify scene transitions. For example, in a video clip about campus life, "classroom," "playground," and "library" are scene keywords. When the editing text shifts from a description related to "classroom" to one related to "playground," it indicates a scene change. Emotional transition words, such as "however," "but," and "suddenly," convey emotional change. For example, when the editing text shifts from describing a pleasant classroom atmosphere to an emotional transition word like "however, something unexpected happened," it indicates a shift in scene or plot. By identifying these trigger points, the editing text within each time interval is further broken down into several scene units. For example, within a time interval, the original continuous text describes students in a classroom. When a sentence containing the scene keyword "playground" appears, such as "the bell rings, and students head for the playground," this sentence is used as the trigger to split the time interval into two scene units: "classroom class" and "heading for the playground."

[0096] Assign emotional tags to each scene unit based on the determined editing style. If the editing style is lively and cheerful, then for a scene unit like "students playing on the playground" in a campus life video, it may be assigned an emotional tag of "happy and joyous"; if the editing style is nostalgic and lyrical, a scene unit like "the teacher recalling past years on the podium" may be given an emotional tag of "nostalgic and sentimental". At the same time, determine the transition rules between each scene unit. For a video with a bright style, the transition between scene units may adopt a fast-switching and brightly colored transition method, such as a flash cut effect; while for a video with a slow style, a fade-in / fade-out and music transition method may be adopted to make the scene transition more gentle and natural.

[0097] Based on the above operations, generate a storyboard script with emotional tags and transition rules. For example, "(Scene unit 1: Class in the classroom, emotional tag: focused and serious, transition rule: fade-in, connecting music gradually gets louder) The teacher is vividly lecturing on the podium, and the students are listening attentively... (Scene unit 2: Walking towards the playground, emotional tag: relaxed and happy, transition rule: flash cut, lively music starts) The class bell rings, and the students run towards the playground cheerfully like little birds...". The generated storyboard script not only clarifies the content of each storyboard but also includes the emotional atmosphere and transition requirements, providing detailed and accurate guidance for subsequent video material matching and editing work.

[0098] In some embodiments of the present application, the cross-modal analysis process of the video understanding model described in step S3 is introduced, which may specifically include:

[0099] Step S31: Extract the semantic text features of the storyboard script, the spatio-temporal visual features of the video material, and the audio emotional features respectively.

[0100] Specifically, use natural language processing (NLP) technology to process the storyboard script. First, perform word segmentation operations to split the script into individual words, and then remove stop words (such as words without actual semantics like "of", "is", "in", etc.). Then, use a word embedding model (such as Word2Vec, GloVe, etc.) to convert each word into a corresponding vector representation. Next, through aggregation operations (such as average pooling, max pooling, etc.) on these word vectors, obtain the semantic text feature vector of the entire storyboard script. This vector can reflect the core semantic information of the storyboard script, such as the scenes, actions, emotions, etc. described in the script.

[0101] To extract spatiotemporal visual features from video footage, 3D convolutional neural networks (such as C3D and I3D) are used. These networks can simultaneously process both the spatial dimension (image information within each frame) and the temporal dimension (dynamic changes between frames). The network processes the video frame by frame, extracting image features from each frame while also considering the temporal relationship between adjacent frames to capture motion information within the video. The resulting spatiotemporal visual feature vectors contain visual information at different moments and spatial locations in the video, such as object motion trajectories and scene changes.

[0102] Audio processing techniques are used to analyze the audio content of video footage. First, the audio is preprocessed, including noise reduction and frame segmentation. Then, deep neural networks (such as convolutional neural networks and recurrent neural networks) are used to extract audio features. These networks learn acoustic features (such as pitch, timbre, and rhythm) in the audio and map them to an emotional space, generating an emotional feature vector for the audio. This vector reflects the emotional information conveyed by the audio, such as joy, sadness, or tension.

[0103] Step S32: Map the semantic text features, the spatiotemporal visual features, and the audio emotion features to the same latent space for joint encoding.

[0104] Specifically, the extracted semantic text features, spatiotemporal visual features, and audio emotional features are mapped to the same latent space for joint encoding. This process is typically implemented using a multimodal encoder, which can be a model based on the Transformer architecture. In the latent space, features from different modalities can be represented in the same feature space, facilitating subsequent similarity calculations. The joint encoding process learns the associations and complementary information between features from different modalities, enabling the model to better understand the correspondence between video footage and storyboard copy.

[0105] Step S33: traverse the video material, calculate the cross-modal similarity between each video clip in the video material and the storyboard copy, and dynamically weight it according to the editing style to obtain a matching score, and determine the video clip with the highest matching score as the storyboard clip that matches the storyboard copy.

[0106] Specifically, the video material is traversed and divided into small video segments. For each video segment, its cross-modal similarity with the storyboard copy is calculated. Cross-modal similarity can be calculated using methods such as cosine similarity and Euclidean distance to measure the similarity in the latent space between the spatiotemporal visual features and audio emotional features of the video segment and the semantic textual features of the storyboard copy.

[0107] Dynamically weight cross-modal similarity based on editing style. Different editing styles may prioritize visual, audio, and textual information differently. For example, a style focused on visuals might prioritize spatiotemporal visual features, while a style focused on emotional expression might prioritize audio emotional features. Dynamic weighting more accurately reflects the impact of editing style on matching scores.

[0108] The video clip with the highest dynamically weighted matching score is then determined as the storyboard clip that matches the storyboard copy. This allows the video understanding model to accurately locate the video clip that best matches each storyboard copy within the video footage, providing a foundation for subsequent video editing.

[0109] A video editing device provided in an embodiment of the present application is described below. The video editing device described below and the video editing method described above can refer to each other.

[0110] See also Figure 5 , Figure 5 A schematic diagram of a video editing device disclosed in an embodiment of the present application.

[0111] like Figure 5 As shown, the video editing device may include:

[0112] The editing intention module 110 is used to receive the user's editing intention and perform semantic analysis on the user's editing intention through the intention analysis model to generate editing copy, editing style and structured instructions including time allocation, scene logic and emotional expression;

[0113] A storyboard splitting module 120 is configured to perform storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain a plurality of storyboard copies;

[0114] A segment location module 130 is configured to perform cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each storyboard copy. The video understanding model is based on an open-source multimodal model and is obtained through large-scale supervised training using sample video material, sample storyboard copy, and sample storyboard copy.

[0115] A quantitative scoring module 140 is configured to perform editing and special effects editing on the storyboard segments using an editing model to generate a preliminary finished film, and input the preliminary finished film into a video scoring model for quantitative scoring, generating a scoring result based on user intent matching, special effects naturalness, and style consistency;

[0116] The iterative editing module 150 is used to feed back the scoring results to the editing model, iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0117] It can be seen from the above technical solutions that a video editing method and related equipment provided by an embodiment of the present application first receives the user's editing intention, and performs semantic analysis on it with the help of an intention analysis model to generate editing text, editing style, and structured instructions containing time allocation weights and scene logic association rules. Then, based on the structured instructions and editing style, the editing text is split into storyboards to obtain multiple storyboards. Afterwards, a video understanding model trained based on an open source multimodal model is used to perform cross-modal analysis on the video material, and the storyboard segments that match each storyboard are accurately located from the video material. The storyboard segments are then edited with special effects through the editing model to generate a preliminary film, and the preliminary film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user's intention matching, special effect naturalness, and style consistency. Finally, the scoring result is fed back to the editing model, and the editing order, transition effect, and special effect parameters are iteratively adjusted until the final editing film is output.

[0118] This application significantly improves the efficiency of video editing and greatly reduces manual operation time by automating the processing of user intent analysis, storyboard splitting, video material matching and editing optimization. Based on the intention analysis model and the video understanding model, intelligent video editing decisions are implemented, which can adapt to different types of video materials and meet the personalized needs of users. Through the guidance of structured instructions and editing styles, the consistency of the storyboard splitting and editing process is ensured, the subjective differences caused by manual intervention are reduced, and the stability of video quality is improved. Combined with multimodal feature analysis and reinforcement learning algorithms, the present invention achieves a deep semantic understanding of video content, accurately identifies key clips and emotional expressions, and optimizes the editing effect. In addition, through the automated and intelligent editing process, the user's dependence on professional skills is reduced, so that non-professionals can easily complete high-quality video editing.

[0119] To sum up, the present application provides an efficient, intelligent and personalized video editing solution, which improves editing efficiency, reduces manual operation time, enhances material adaptability, meets personalized needs, improves consistency of editing effects, deepens understanding of video content, optimizes editing effects and lowers technical barriers. It effectively solves the problems existing in the existing technology and promotes the development of video editing technology.

[0120] Optionally, the editing copy is subjected to storyboard splitting processing based on the structured instruction and the editing style to obtain multiple storyboards, including:

[0121] Dividing the editing copy into a plurality of time intervals according to a preset time threshold according to the time allocation weight in the structured instruction;

[0122] In each time interval, a scene switching trigger point is identified according to the scene logic association rule in the structured instruction, and a plurality of scene units are obtained based on the scene switching trigger point, wherein the trigger point includes a scene keyword or an emotional transition word;

[0123] Assigning an emotional label to each of the scene units based on the editing style, and determining transition connection rules between the scene units;

[0124] Generate storyboards with emotional labels and transition rules.

[0125] Optionally, the cross-modal analysis process of the video understanding model includes:

[0126] Separately extracting semantic text features of the storyboard copy, spatiotemporal visual features, and audio emotional features of the video material;

[0127] Mapping the semantic text features, the spatiotemporal visual features, and the audio emotion features to the same latent space for joint encoding;

[0128] The video material is traversed, the cross-modal similarity between each video clip in the video material and the storyboard is calculated and dynamically weighted according to the editing style to obtain a matching score, and the video clip with the highest matching score is determined as the storyboard clip that matches the storyboard.

[0129] Optionally, the process of the video scoring model performing quantitative scoring on the input preliminary film includes:

[0130] Determining the user intent matching degree by comparing the semantic overlap and emotional label consistency of the preliminary finished film subtitles with the storyboard copy;

[0131] Determining the naturalness of special effects by analyzing the motion continuity and audio rhythm synchronization of the transition area in the preliminary film;

[0132] Determining the style consistency by detecting the difference between the color tone, shot duration, and music type of the preliminary film and the preset style library;

[0133] A scoring result is generated based on the user intention matching degree, the special effect naturalness and the style consistency.

[0134] Optionally, the iterative optimization process of the editing model includes:

[0135] The scoring result is used as a feedback adjustment signal to dynamically adjust the probability of transition special effect selection, shot playback rate and scene switching direction, and the parameters of the editing model are updated through a policy gradient algorithm until the scoring result meets the threshold.

[0136] Optionally, also include:

[0137] A user real-time feedback interface is introduced in the iterative process, and the user's manual correction instructions for the preliminary film are received through the user real-time feedback interface, and the manually corrected correction data is added to the training set. The model parameters of the intent analysis model, the video understanding model and the editing model are updated through online learning.

[0138] The video editing apparatus provided in the embodiments of the present application can be applied to video editing equipment. Figure 6 The hardware structure diagram of the video editing device is shown in FIG. Figure 6 ,The hardware structure of the video editing device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0139] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0140] The processor 1 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0141] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory;

[0142] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0143] Receive user editing intentions, and perform semantic analysis on the user editing intentions through the intention analysis model to generate editing copy, editing style, and structured instructions including time allocation weights and scene logic association rules;

[0144] Performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies;

[0145] Performing cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each of the storyboards. The video understanding model is based on an open-source multimodal model and is trained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards.

[0146] The storyboard clips are edited with special effects using an editing model to generate a preliminary finished film, and the preliminary finished film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user intention matching, special effect naturalness, and style consistency;

[0147] The scoring results are fed back to the editing model to iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0148] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0149] The present application also provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0150] Receive user editing intentions, and perform semantic analysis on the user editing intentions through the intention analysis model to generate editing copy, editing style, and structured instructions including time allocation weights and scene logic association rules;

[0151] Performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies;

[0152] Performing cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each of the storyboards. The video understanding model is based on an open-source multimodal model and is trained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards.

[0153] The storyboard clips are edited with special effects using an editing model to generate a preliminary finished film, and the preliminary finished film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user intention matching, special effect naturalness, and style consistency;

[0154] The scoring results are fed back to the editing model to iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0155] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0156] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the execution method is as follows:

[0157] Receive user editing intentions, and perform semantic analysis on the user editing intentions through the intention analysis model to generate editing copy, editing style, and structured instructions including time allocation weights and scene logic association rules;

[0158] Performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies;

[0159] Performing cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each of the storyboards. The video understanding model is based on an open-source multimodal model and is trained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards.

[0160] The storyboard clips are edited with special effects using an editing model to generate a preliminary finished film, and the preliminary finished film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user intention matching, special effect naturalness, and style consistency;

[0161] The scoring results are fed back to the editing model to iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

[0162] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0163] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0164] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0165] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video editing method, characterized in that: include: Receive user editing intentions, and perform semantic analysis on the user editing intentions through the intention analysis model to generate editing copy, editing style, and structured instructions including time allocation weights and scene logic association rules; Performing storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies; Performing cross-modal analysis on the video material using a video understanding model to locate, from the video material, each storyboard segment that matches each of the storyboards. The video understanding model is based on an open-source multimodal model and is trained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards. The storyboard clips are edited with special effects using an editing model to generate a preliminary finished film, and the preliminary finished film is input into a video scoring model for quantitative scoring, and a scoring result is generated based on the user intention matching, special effect naturalness, and style consistency; The scoring results are fed back to the editing model to iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

2. The method according to claim 1, characterized in that The editing copy is subjected to storyboard splitting processing based on the structured instruction and the editing style to obtain multiple storyboards, including: Dividing the editing copy into a plurality of time intervals according to a preset time threshold according to the time allocation weight in the structured instruction; In each time interval, a scene switching trigger point is identified according to the scene logic association rule in the structured instruction, and a plurality of scene units are obtained based on the scene switching trigger point, wherein the trigger point includes a scene keyword or an emotional transition word; Assigning an emotional label to each of the scene units based on the editing style, and determining transition connection rules between the scene units; Generate storyboards with emotional labels and transition rules.

3. The method according to claim 1, characterized in that The cross-modal analysis process of the video understanding model includes: Separately extracting semantic text features of the storyboard copy, spatiotemporal visual features, and audio emotional features of the video material; Mapping the semantic text features, the spatiotemporal visual features, and the audio emotion features to the same latent space for joint encoding; The video material is traversed, the cross-modal similarity between each video clip in the video material and the storyboard is calculated and dynamically weighted according to the editing style to obtain a matching score, and the video clip with the highest matching score is determined as the storyboard clip that matches the storyboard.

4. The method according to claim 1, wherein The process of the video scoring model quantitatively scoring the input preliminary film includes: Determining the user intent matching degree by comparing the semantic overlap and emotional label consistency of the preliminary finished film subtitles with the storyboard copy; Determining the naturalness of special effects by analyzing the motion continuity and audio rhythm synchronization of the transition area in the preliminary film; Determining the style consistency by detecting the difference between the color tone, shot duration, and music type of the preliminary film and the preset style library; A scoring result is generated based on the user intention matching degree, the special effect naturalness and the style consistency.

5. The method according to claim 3, characterized in that The iterative optimization process of the clipping model includes: The scoring result is used as a feedback adjustment signal to dynamically adjust the probability of transition special effect selection, shot playback rate and scene switching direction, and the parameters of the editing model are updated through a policy gradient algorithm until the scoring result meets the threshold.

6. The method according to claim 1, characterized in that Also includes: A user real-time feedback interface is introduced in the iterative process, and the user's manual correction instructions for the preliminary film are received through the user real-time feedback interface, and the manually corrected correction data is added to the training set. The model parameters of the intent analysis model, the video understanding model and the editing model are updated through online learning.

7. A video editing device, characterized in that: include: The editing intention module is used to receive the user's editing intention and perform semantic analysis on the user's editing intention through the intention analysis model to generate editing copy, editing style and structured instructions including time allocation, scene logic and emotional expression; A storyboard splitting module is used to perform storyboard splitting processing on the editing copy based on the structured instruction and the editing style to obtain multiple storyboard copies; A segment location module is used to perform cross-modal analysis on the video material using a video understanding model to locate the storyboard segments that match the storyboards in the video material. The video understanding model is based on an open source multimodal model and is obtained through large-scale supervised training using sample video material, sample storyboards, and sample storyboards. A quantitative scoring module is used to edit the storyboard clips with special effects using an editing model to generate a preliminary finished film, and input the preliminary finished film into a video scoring model for quantitative scoring, generating a scoring result based on the user intention matching, special effect naturalness, and style consistency; The iterative editing module is used to feed back the scoring results to the editing model, iteratively adjust the editing sequence, transition effects and special effect parameters until the final editing is output.

8. A video editing device, characterized in that: including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the video editing method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the video editing method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the video editing method according to any one of claims 1 to 6 is executed.

Images