Video editing method and device

Abstract

The embodiment of the invention provides a video editing method and device, and belongs to the technical field of video editing. The video editing method comprises the following steps: acquiring a to-be-edited video; inputting the video to be edited into the video editing model, and outputting a target edited video through the video editing model; obtaining target feedback data for the target edited video; and adjusting the video editing model based on the target feedback data, wherein the adjusted video editing model is used for processing the next to-be-edited video. According to the embodiment of the invention, the to-be-edited video is acquired and input into the video editing model, and the video editing model is dynamically adjusted in combination with the feedback data of the target edited video, so that the self-optimization capability of the video editing model is realized. The video editing model is continuously optimized by constructing the data closed loop, so that the video editing model can adapt to diversified editing requirements, the generalization ability of the video editing model is improved, and the intelligent video editing effect is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video editing technology, and in particular, to a video editing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of artificial intelligence technology, large language models (LLMs) have made significant progress in understanding and generating text. Large language models can understand instructions and generate high-quality text. Large language models can be applied to intelligent video editing, thereby replacing traditional manual editing. Large language models can generate intelligent video editing solutions by understanding the text of video content. However, large language models are easily affected by model training data and lack knowledge of editing specific video types, resulting in poor results in intelligent video editing.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0004] The embodiments of the present application provide a video editing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides a video editing method, the method comprising: Get the video to be edited; Inputting the video to be edited into a video editing model to output a target edited video through the video editing model; Acquire target feedback data for the target clip video; The video editing model is adjusted based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

[0006] Optionally, the video editing model includes: An input module, used for receiving the video to be edited, and extracting multimodal data from the video to be edited; wherein the multimodal data includes video data, audio data and / or text data of the video to be edited; An intelligent editing module, used for outputting corresponding semantic analysis results according to the input multimodal data; A script acquisition module, used for generating the editing script according to the semantic analysis result, and used for instructing to edit the video to be edited into the target editing video; An optimization module, used for adjusting the intelligent editing module in the video editing model according to the target feedback data; The output module is used to provide file output based on the target clip video.

[0007] Optionally, the multimodal data is input into the intelligent editing module for multiple rounds, and the intelligent editing module outputs multiple semantic analysis results, one semantic analysis result corresponds to one editing script, and one editing script corresponds to generating one target editing video; the method further includes: Acquiring target feedback data of each target clip video through the optimization module; Based on the target feedback data of each target clip video, evaluating each clip script; Determining a preset number of target editing scripts from a plurality of the editing scripts according to evaluation parameters of the editing scripts; The target editing script is used for subsequent video editing.

[0008] Optionally, adjusting the video editing model based on the target feedback data includes: Based on the target feedback data, generating adjustment instructions through the optimization module to generate a preset form value; The preset form value is provided to the intelligent editing module through the optimization module to adjust the parameters of the video editing model.

[0009] Optionally, the intelligent editing module includes a first convolutional layer, and the first convolutional layer includes a plurality of first matrices; adjusting the video editing model based on the target feedback data includes: Adding a second matrix in the first convolutional layer of the smart editing module; the dimension of the second matrix is ​​lower than that of the first matrix; Based on the target feedback data, the parameters of the second matrix are adjusted to optimize the video editing model.

[0010] Optionally, the method further comprises one or more features: (1) Target feedback data includes: viewing time, click-through rate, sharing rate, like rate and / or interactive information of the target video clip; (2) The target feedback data includes the emotion type of the target video clip; the emotion type is determined according to the interaction information.

[0011] Another aspect of an embodiment of the present application provides a video editing device, the device comprising: A first acquisition module is used to acquire the video to be edited; A video input module, used for inputting the video to be edited into a video editing model, so as to output a target edited video through the video editing model; A second acquisition module is used to acquire target feedback data for the target clip video; The adjustment module is used to adjust the video editing model based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

[0012] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0013] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0014] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0015] The above technical solution adopted in the embodiment of the present application may have the following advantages: By obtaining the video to be edited and inputting it into the video editing model, the video editing model is dynamically adjusted in combination with the feedback data of the target edited video, thereby realizing the self-optimization capability of the video editing model during use. By building a data closed loop to continuously optimize the video editing model, the video editing model can adapt to diverse editing needs, improve the generalization ability of the video editing model, and thus improve the effect of intelligent video editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 The flowchart of the video editing method according to the first embodiment of the present application is schematically shown; Figure 2 The newly added flow chart of the video editing method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 1 Flow chart of sub-steps of step S106; Figure 4 Schematically shows Figure 1 Another sub-step flow chart of step S106; Figure 5 A diagram schematically shows an application example of the video editing method according to the first embodiment of the present application; Figure 6 Another application example diagram of the video editing method according to the first embodiment of the present application is schematically shown; Figure 7 A diagram schematically shows another application example of the video editing method according to the first embodiment of the present application; Figure 8 A block diagram schematically shows a video editing device according to the second embodiment of the present application; and Fig. 9 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0019] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0020] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0021] First, the following terms are explained: Multimodal large language model: is an artificial intelligence model that can process multiple forms of data (such as text, images, audio and video). Unlike traditional large language models that mainly process text data, multimodal large language models can understand and generate multiple forms of information at the same time.

[0022] Data closed loop: refers to the process of driving continuous self-optimization of the system through a continuous data feedback loop.

[0023] Editing script: refers to a text description file used to indicate the video editing process, which may include editing requirements, plot arrangement, rhythm control, etc. Editing scripts can be customized by professionals and are the basis for realizing video content editing.

[0024] Low-rank adaptation: It is a method for optimizing deep learning models. The idea of ​​low-rank adaptation is to insert low-rank matrices between certain layers of the pre-trained model. The model can be adjusted by introducing a small number of additional parameters, thereby greatly reducing the amount of computation and storage requirements.

[0025] Instruction fine-tuning: is a method of optimizing a pre-trained model by providing a set of examples consisting of instructions and corresponding outputs, allowing the model to learn how to perform a specific task more accurately.

[0026] Secondly, to facilitate those skilled in the art to understand the technical solutions provided by the embodiments of the present application, the relevant technologies are described below: In an era of rapid growth in video content, automated video editing technology has gradually become an important tool for meeting user needs and improving content production efficiency. With the development of video technology, more and more intelligent editing systems have been developed to replace traditional manual editing methods. For example, traditional automatic editing systems can achieve automated editing based on predefined rules and templates through key frame extraction, scene segmentation, and editing point identification. The main limitation of such methods is the lack of understanding of the deep semantics of video content, making it difficult to handle complex editing tasks. For example, when it comes to optimizing plot coherence, emotional expression, or audience experience, rule-based systems often perform poorly and cannot meet high standards for editing.

[0027] With the significant progress made in understanding and generating text by large language models (LLMs), large language models can be applied to intelligent video editing, thus replacing traditional manual editing. Large language models can generate intelligent video editing solutions by understanding the text of video content. However, large language models are easily affected by model training data and lack knowledge of specific video type editing, resulting in poor results of intelligent video editing.

[0028] To this end, an embodiment of the present application provides a video editing technical solution. In this technical solution, by obtaining the video to be edited and inputting it into the video editing model, the video editing model is dynamically adjusted in combination with the feedback data of the target edited video, thereby realizing the self-optimization capability of the video editing model. By building a data closed loop to continuously optimize the video editing model, the video editing model can adapt to diverse editing needs, improve the generalization ability of the video editing model, and thus improve the effect of intelligent video editing. See below for details.

[0029] Embodiment 1 Figure 1 The flowchart of the video editing method according to the first embodiment of the present application is schematically shown.

[0030] like Figure 1 As shown, the video editing method may include steps S100 to S106, wherein: Step S100, obtaining a video to be edited; Step S102, inputting the video to be edited into a video editing model, so as to output a target edited video through the video editing model; Step S104, obtaining target feedback data for the target clip video; Step S106, adjusting the video editing model based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

[0031] The video editing method provided in this embodiment obtains the video to be edited and inputs it into the video editing model, and dynamically adjusts the video editing model in combination with the feedback data of the target edited video, thereby realizing the self-optimization capability of the video editing model during use. By building a data closed loop to continuously optimize the video editing model, the video editing model can adapt to diverse editing needs, improve the generalization capability of the video editing model, and thus improve the effect of intelligent video editing.

[0032] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0033] Step S100 , get the video to be edited.

[0034] In some embodiments, the video to be edited can be a local file, a cloud video file, or a real-time recorded video file. For example, a local video file can be loaded, a device camera or a screen recording tool can be connected to capture a video stream in real time and store it as a video file to be edited, or the video to be edited can be obtained through a network protocol.

[0035] Step S102, input the video to be edited into the video editing model to output the target edited video through the video editing model.

[0036] By inputting the video data of the video to be edited, the video editing model can parse the content of the video and output the target edited video according to the editing requirements.

[0037] In an optional embodiment, the video editing model includes: an input module, an intelligent editing module, a script acquisition module, an optimization module and an output module.

[0038] An input module is used to receive the video to be edited and extract multimodal data from the video to be edited; wherein the multimodal data includes video data, audio data and / or text data of the video to be edited.

[0039] The input module can receive videos to be edited in multiple formats, such as MP4, AVI, MKV, etc., without the need for users to convert file formats. In addition, the input module can also support videos to be edited in different resolutions, including standard definition, high definition and / or ultra-high definition videos.

[0040] The input module can also pre-process the video to be edited to ensure the smoothness and accuracy of subsequent processing. For example, the input module can decode the video file to be edited and convert it into a raw data format suitable for processing. The frame rate of the video to be edited can also be standardized, for example, videos to be edited with different frame rates can be uniformly adjusted to the target frame rate to avoid freezing or information loss in the video to be edited due to frame rate differences, thereby ensuring the consistency of the editing effect. In addition, the video to be edited can also be synchronized with audio and video, and the audio and video tracks of the video to be edited can be accurately aligned to ensure the consistency of sound and picture.

[0041] For example, if the video to be edited is a recording of a basketball game, the input module first receives the video file and starts to execute the multimodal data extraction process. The multimodal video data extraction process includes video data extraction, for example, extracting each frame of the video through a decoder to generate a video frame sequence; audio data extraction, extracting the audio stream from the video and analyzing the sound effect data, such as the sound of the player shooting and the cheers of the audience; text data extraction, if there are subtitles or commentary in the video, the input module can extract the text information of "Player A completed a three-point shot" through OCR (Optical Character Recognition) technology. After extracting multiple multimodal data, data fusion can be performed to integrate video, audio and text data together to form a multimodal data input set for subsequent video editing model analysis and processing.

[0042] The intelligent editing module is used to output corresponding semantic analysis results based on the input multimodal data.

[0043] The intelligent editing module can be built based on a multimodal large language model. Through the multimodal large language model's deep understanding of multimodal data and the fusion of cross-modal information, accurate semantic analysis results (regularized text) are output. The multimodal large language model can understand and extract complex information by analyzing multimodal data such as video, audio, and text. The analysis process may include visual understanding process (scene recognition, object detection), audio analysis process (emotion recognition, speech-to-text), and text analysis process (keyword extraction, plot understanding). Through the collaborative analysis of multimodal data, the multimodal large language model can accurately identify the wonderful moments, emotional turning points, and key plots in the video to be edited. Based on this, the multimodal large language model can output relatively accurate semantic analysis results. The content of the semantic analysis results may include editing strategy information, which may include editing time nodes, editing segment lengths, and transition special effects descriptions.

[0044] A script acquisition module is used to generate the editing script according to the semantic analysis result, and is used to instruct to edit the video to be edited into the target editing video.

[0045] The script acquisition module generates an editing script based on the editing time node, clip length, transition effect description, subtitle style and other information. The generated editing script can support user customization. Through natural language or preset instructions, users can add, modify or delete specific editing instructions in the script to meet personalized needs. The editing script and the video to be edited are input into the editing tool for automatic editing. The editing tool can automatically edit the video to be edited into a target editing video that meets the requirements by guiding the editing script.

[0046] An optimization module is used to adjust the intelligent editing module in the video editing model according to the target feedback data.

[0047] Exemplarily, the target feedback data shows that the audience is less enthusiastic about the game editing, and the reasons include not accurately capturing the climax of the game during editing, and editing some bland scenes as exciting moments. The optimization module identifies that the selection of editing points is not accurate enough by analyzing the audience's viewing behavior (such as the frequency of skipping specific clips) and user feedback (such as dissatisfied comments on certain clips). The optimization module can adjust the editing points to ensure that the clips are sliced ​​at the moments that the audience is most interested in (such as scoring or key strikes) instead of editing some relatively bland moments. In this way, the smoothness and viewing experience of the edited video are improved, and too fast or too slow scene switching is avoided. The target feedback data is actively provided or authorized to be collected by the user.

[0048] The output module is used to provide file output based on the target clip video.

[0049] The output module can support the output of multiple video formats to meet the needs of different devices and platforms. For example, MP4, AVI, MOV, etc. can output different video formats according to different application scenarios. For example, some platforms may require a higher compression ratio to reduce the file size, while some professional video editing platforms may require higher resolution and frame rate.

[0050] The output module also provides a preview function, so users can check the editing effect before finally exporting the edited video. For example, users can check the connection between multiple clips of the target edited video, the visual presentation effect, and the audio synchronization to ensure that the video content meets expectations. Based on the preview results, users can also make further adjustments or modifications, such as re-adjusting the editing timeline, replacing or adding special effects, modifying the background music, or adjusting the subtitle style.

[0051] In this embodiment, the video editing model uses multiple modules to collaboratively process the video to be edited, uses the editing script corresponding to the multimodal data to indicate the editing process of the video to be edited, achieves more accurate and intelligent video editing, and continuously optimizes the editing model according to feedback data, thereby improving the intelligence level and editing effect.

[0052] Step S104 , obtaining target feedback data for the target clip video.

[0053] By collecting users' evaluations or interactive behaviors on the target video clips and extracting effective feedback on the editing effects, we can provide a basis for adjusting the subsequent video editing model. Feedback data can reflect the performance of the video in terms of user experience, effect presentation, content quality, etc., thereby guiding the model to make more expected improvements in the next editing.

[0054] Exemplarily, the target feedback data can come from the user's behavioral feedback, such as collecting the user's behavioral data on the platform, such as playback time, completion rate, number of likes, forwarding volume, number of comments, etc., to quantify the popularity of the video. The target feedback data can also be direct ratings from users, such as designing a rating function in the playback interface to allow users to actively rate the edited video, and use the average score as an evaluation indicator. The target feedback data can also be the emotional characteristics represented by the interactive information posted by the user. For example, the text content of the user in the comment area can be collected and analyzed, and the comment sentiment value (such as positive, negative or neutral) can be extracted using natural language processing technology to evaluate the editing effect. If a large number of words such as "wonderful" and "shocking" appear in the user's comment content, and the sentiment analysis results show that the proportion of positive emotions reaches 85%, the editing logic of similar styles can be retained first. It should be noted that the above data needs to be collected after the user's authorization and desensitized.

[0055] Step S106 , adjusting the video editing model based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

[0056] When a new video clip is output, the video clip model is optimized and adjusted again by obtaining the corresponding feedback data to process the next video to be edited, forming a data closed-loop editing mode that can continuously optimize the editing strategy. The adjusted video clip model can quickly adapt to user needs based on actual feedback, ensuring that the output of each video to be edited is more in line with user preferences, content quality and platform requirements.

[0057] In some embodiments, the decision-making process of the model in generating video clips can be optimized by fine-tuning the training parameters of the video clip model. During the fine-tuning process, the target feedback data is used as annotation information to adjust the key parameters of the video clip model so that the video clip model can better adapt to the video clip task. For example, based on the number of user likes and viewing time data, the editing strategy of the video clip model is fine-tuned to reduce redundant parts and increase high-energy clips in the next editing.

[0058] In some embodiments, incremental learning or continuous learning can be performed using target feedback data based on the original video editing model, so that the entire video editing model does not need to be retrained. Instead, the video editing model can be continuously adapted to new data by specifically adjusting some weights of the video editing model. This reduces training time and improves the model's adaptability to new feedback data.

[0059] In some embodiments, reinforcement learning can be used to optimize the video editing model. Specifically, based on the target feedback data, the editing strategy of the video editing model is gradually optimized through continuous experiments and feedback. The video editing model can adjust the editing strategy according to the reward or punishment signal of the environment (such as the user's likes, viewing time, etc.), and finally achieve a better video editing effect, so that the video is more in line with the user's viewing habits and needs.

[0060] In an optional embodiment, the multimodal data is input into the intelligent editing module for multiple rounds, and the intelligent editing module outputs multiple semantic analysis results, one semantic analysis result corresponds to one editing script, and one editing script corresponds to generating one target editing video; Figure 2 As shown, the method also includes: Step 200: obtaining target feedback data of each target clip video through the optimization module.

[0061] Step 202: Evaluate each of the editing scripts based on the target feedback data of each of the target editing videos.

[0062] Step 204, determining a preset number of target editing scripts from the plurality of editing scripts according to the evaluation parameters of the editing scripts; wherein the target editing scripts are used for subsequent video editing.

[0063] By repeatedly inputting the multimodal data into the intelligent editing module, the intelligent editing module outputs multiple semantic analysis results (which may have different contents), and then generates multiple different editing scripts through the script module. That is, for the same video to be edited, multiple different editing scripts are generated. Then the performance of these different editing scripts is comprehensively evaluated, and the effect of the editing script is continuously optimized through the horse racing mechanism. The horse racing mechanism can enable multiple editing scripts to compete in practical applications, and evaluate the pros and cons of the performance of the editing scripts based on the results of the competition. The editing scripts can be screened and optimized based on the user's interactive data (such as viewing time, likes, comments, click-through rate, etc.). Excellent editing scripts can get more recommendations, while poorly performing editing scripts are eliminated or improved, ultimately forming a continuously optimized editing solution that adapts to user needs.

[0064] For example, the video to be edited of a sports game can be edited by multiple editing scripts, and each editing script focuses on different game highlights (such as goals, fouls, wonderful counterattacks, etc.). Through the horse racing mechanism, the corresponding target editing videos generated by multiple editing scripts are submitted to the video platform, and the user's feedback data on the above target editing videos is collected. For example, the target editing video corresponding to editing script A can attract more viewing time and likes, and stimulate more discussions; the target editing video corresponding to editing script B has a low viewing rate and less likes. The interaction. According to the feedback data, the editing script is scored. For example, editing script A can get 90 points, while editing script B with poor feedback can get 50 points. Further, it can be determined that the script with a score of more than 90 points is an excellent script and is retained for subsequent video editing. On the contrary, editing script B with a score below 90 is eliminated and will not be used in subsequent video editing.

[0065] In this embodiment, by evaluating the performance of multiple editing scripts of the same video to be edited, a target editing script with excellent performance can be screened out from the multiple editing scripts and used for subsequent video editing. This not only ensures that the content quality of the subsequent edited videos is further improved, but also makes the target editing videos more in line with the user's viewing preferences.

[0066] In an optional embodiment, if Figure 3 As shown, adjusting the video editing model based on the target feedback data includes: Step S300, based on the target feedback data, generating adjustment instructions through the optimization module to generate a preset form value.

[0067] Step S302: providing the preset form value to the intelligent editing module through the optimization module to adjust the parameters of the video editing model.

[0068] In some embodiments, the target feedback data of the target clip video is received by the optimization module. The target feedback data may include various feedback information of the user on the clip result, such as viewing time, number of likes, comment content, interaction frequency, etc. According to the target feedback data, the optimization module can analyze the user's preferences and behavior patterns to generate corresponding adjustment instructions. Further, the optimization module can be used to optimize the generation of preset form numerical adjustment instructions using mathematical modeling, data analysis and algorithm. The adjustment instructions may include changing the weight of certain clip points, adjusting the rhythm of the clip and the timing of the lens switching. The generated preset form numerical values ​​are provided to the intelligent clip module to adjust the multimodal large language model in the intelligent clip module. The preset form numerical values ​​in the adjustment process can appear in different forms, such as numerical vectors, and the clip parameters can be adjusted by vectors to generate clips that are more in line with user preferences. For example, matrix form information, through matrix data, the multidimensional parameters of the clip model can be adjusted more comprehensively. For example, tensor form information, for more complex clip scenes, especially those involving multiple modal data, tensor form information can enable the intelligent clip module to better understand and optimize multi-dimensional data features.

[0069] In this embodiment, the parameters of the video editing model can be fine-tuned by generating adjustment instructions through techniques such as command fine-tuning (SFT). The parameters of the video editing model can be flexibly, efficiently and accurately adjusted according to user behavior feedback, thereby achieving efficient improvement of the video editing model and further improving the overall editing quality and effect.

[0070] In an optional embodiment, the intelligent editing module includes a first convolution layer, and the first convolution layer includes a plurality of first matrices; Figure 4 As shown, adjusting the video editing model based on the target feedback data includes: Step S400, adding a second matrix in the first convolutional layer of the intelligent editing module; the dimension of the second matrix is ​​lower than that of the first matrix.

[0071] Step S402: Based on the target feedback data, adjust the parameters of the second matrix to optimize the video editing model.

[0072] The first convolution layer of the intelligent editing module can extract feature information from the input video data to be edited. The first convolution layer consists of multiple convolution kernels, and the size and shape of each convolution kernel are used to determine the way to extract features. The first matrix represents the weight matrix of the convolution kernel, which is used to extract different types of spatial information and features in the video frame through convolution operations.

[0073] In some embodiments, the intelligent editing module can extract complex features of the video through a high-dimensional matrix (first matrix), and continuously adjust the low-dimensional matrix (second matrix) during the model optimization process, and simulate more complex features in low dimensions by combining with the first matrix. The second matrix can capture some potential patterns or features through low-rank approximation, thereby reducing the number of parameters that the network needs to learn and avoiding excessive consumption of computing resources. Since the second matrix is ​​sparser and less efficient than the first matrix, by adjusting the second matrix with a lower dimension, the number of parameters that the convolutional layer needs to process is reduced, thereby reducing the computational cost.

[0074] For example, in the first convolutional layer of the video editing model, the first matrix is ​​used to extract a large amount of detail information and spatial features (such as the edges of objects, color changes, etc.) from the video to be edited. The introduced second matrix can capture more abstract features in the video through low-rank adaptation, such as changes in certain special scenes or emotional atmosphere in the video. In this way, the low-dimensional features of the second matrix can accelerate the training and reasoning process of the model and improve the diversity and accuracy of the editing effect.

[0075] In this embodiment, by adding a first matrix with a lower dimension to the first convolutional layer for adaptive adjustment, a low-rank adaptation method is adopted to improve the efficiency and accuracy of the video editing model in processing videos. Thus, while reducing the computational complexity, it is possible to ensure that the editing model can give full play to its video content analysis and editing capabilities, thereby achieving a more efficient and accurate editing effect.

[0076] In an optional embodiment, the method further comprises one or more features: (1) Target feedback data includes: viewing time, click-through rate, sharing rate, like rate and / or interactive information of the target video clip; The viewing time reflects the user's overall interest in the target video clip and the willingness to continue watching it, which can indirectly evaluate the attractiveness of the video content and user stickiness; the click-through rate can reflect the attractiveness of the target video clip when the user first contacts it; the sharing rate reflects the user's recognition of the target video clip and its dissemination potential in social scenarios, which can be used as an important indicator to measure the attractiveness of the video; the like rate and interaction information obtain data from the user's subjective feedback and direct operations, which can be used to describe the positive evaluation of the video, the content of user comments, and the degree of fit with the video editing strategy.

[0077] In some embodiments, the target feedback data may also include expert evaluation. For example, experienced editors may rate the target video clip on dimensions such as fluency, visual appeal, and narrative coherence. Expert ratings can more accurately assess the quality of the editing effect and provide more detailed guidance for optimizing the editing model.

[0078] In some embodiments, the feedback data can be organized into a structured data form so that the video editing model can efficiently process and analyze the feedback data. For example, the structured feedback data can be in the following form: {"watching behavior data":{"watching time":"75%","click rate":"18%","sharing rate":"10%"}, "user evaluation":{"likes":1250,"comments":300,}}.

[0079] (2) The target feedback data includes the emotion type of the target video clip; the emotion type is determined according to the interaction information.

[0080] In some embodiments, the emotion type may refer to the emotional tendency of the audience when watching the target video clip, such as pleasure, emotion, tension, excitement, anger or sadness. By analyzing the emotion type of the target video clip, clear direction suggestions can be provided for the editing strategy, such as adding high-energy scenes with a tight rhythm or extending the transition time of emotional rendering. In addition, by combining the emotion type with user behavior analysis, the content style and emotional expression that users prefer can be explored, thereby optimizing subsequent video editing.

[0081] In some embodiments, the interactive information such as the bullet screen content or comment content sent by the audience can be analyzed by natural language processing technology to extract emotional keywords, such as "too funny", "so touching", "too exciting", etc. After the emotional keywords are extracted, the emotional keywords can be mapped to a predefined set of emotional classification labels, such as joy, sadness, and neutral. Then, the emotional keywords and behavior data are clustered and classified through a machine learning model or a rule engine to generate specific emotional type labels.

[0082] In this embodiment, by collecting target feedback data including behavioral information such as viewing time, click-through rate, sharing rate, and like rate, and determining the emotional type of the target video clip based on the interactive information, the audience response to the target video clip can be comprehensively evaluated. Not only can the attractiveness and communication effect of the target video clip be quantified, but the key factors of emotional resonance can also be revealed, thereby optimizing the video clip model more comprehensively.

[0083] In order to make this application easier to understand, the following Figures 5 to 7 Some exemplary applications are provided.

[0084] S11, inputting the original video into the intelligent editing model.

[0085] Expert knowledge-based intelligent editing models include multimodal large language models (MLLM) or large language models (LLM).

[0086] S12, the intelligent editing model outputs the editing script corresponding to the original video.

[0087] S13, outputting the intelligent editing into a film according to the instructions of the editing script.

[0088] S14, the up-host (video producer) will submit the intelligent editing video to Bilibili (video playback platform).

[0089] S15, the up-host (video producer) determines the satisfaction index for the intelligent editing video and outputs the professional evaluation.

[0090] S16, outputs an attractiveness judgment based on the audience's consumption behavior indicators and viewing behavior indicators for the intelligently edited film.

[0091] S17, optimizing the intelligent editing model according to the attractiveness evaluation and professional evaluation corresponding to the intelligent editing film, so that the intelligent editing model can output a better editing script, forming an optimization model mechanism of a closed data loop.

[0092] Through the methods of command fine-tuning (SFT) and low-rank adaptation (LoRA), the intelligent editing model is optimized using indicator data such as attractiveness judgment and professionalism judgment.

[0093] In this exemplary application, by obtaining the video to be edited and inputting it into the video editing model, the video editing model is dynamically adjusted in combination with the feedback data of the target edited video, thereby realizing the self-optimization capability of the video editing model. By building a data closed loop to continuously optimize the video editing model, the video editing model can adapt to diverse editing needs, improve the generalization ability of the video editing model, and thus improve the effect of intelligent video editing.

[0094] Embodiment 2 Figure 8 The block diagram of the video editing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 8 As shown, the device 800 may include: a first acquisition module 810, an input module 820, a second acquisition module 830, and an adjustment module 840, wherein: A first acquisition module 810 is used to acquire a video to be edited; The video input module 820 is used to input the video to be edited into the video editing model, so as to output a target edited video through the video editing model; A second acquisition module 830 is used to acquire target feedback data for the target clip video; The adjustment module 840 is used to adjust the video editing model based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

[0095] In an optional embodiment, the video editing model includes: An input module, used for receiving the video to be edited, and extracting multimodal data from the video to be edited; wherein the multimodal data includes video data, audio data and / or text data of the video to be edited; An intelligent editing module, used for outputting corresponding semantic analysis results according to the input multimodal data; A script acquisition module, used for generating the editing script according to the semantic analysis result, and used for instructing to edit the video to be edited into the target editing video; An optimization module, used for adjusting the intelligent editing module in the video editing model according to the target feedback data; The output module is used to provide file output based on the target clip video.

[0096] In an optional embodiment, the multimodal data is input into the intelligent editing module for multiple rounds, and the intelligent editing module outputs multiple semantic analysis results, one semantic analysis result corresponds to one editing script, and one editing script corresponds to generating one target editing video; the device 800 further includes a script selection module, which is used to: Acquiring target feedback data of each target clip video through the optimization module; Based on the target feedback data of each target clip video, evaluating each clip script; Determining a preset number of target editing scripts from a plurality of the editing scripts according to evaluation parameters of the editing scripts; The target editing script is used for subsequent video editing.

[0097] In an optional embodiment, adjusting the video editing model based on the target feedback data includes: Based on the target feedback data, generating adjustment instructions through the optimization module to generate a preset form value; The preset form value is provided to the intelligent editing module through the optimization module to adjust the parameters of the video editing model.

[0098] In an optional embodiment, the intelligent editing module includes a first convolutional layer, the first convolutional layer includes a plurality of first matrices; adjusting the video editing model based on the target feedback data includes: Adding a second matrix in the first convolutional layer of the smart editing module; the dimension of the second matrix is ​​lower than that of the first matrix; Based on the target feedback data, the parameters of the second matrix are adjusted to optimize the video editing model.

[0099] In an optional embodiment, (1) the target feedback data includes: viewing time, click-through rate, sharing rate, like rate and / or interactive information of the target clip video; (2) The target feedback data includes the emotion type of the target video clip; the emotion type is determined according to the interaction information.

[0100] Embodiment 3 Fig. 9 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing a video editing method according to the third embodiment of the present application is shown schematically. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig. 9 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed in the computer device 10000, such as program codes of a video editing method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0101] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0102] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0103] It should be pointed out that Fig. 9 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0104] In this embodiment, the video editing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0105] Embodiment 4 An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video editing method in the embodiment are implemented.

[0106] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the video editing method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0107] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0108] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0109] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video editing method, characterized in that: The method comprises: Get the video to be edited; Inputting the video to be edited into a video editing model to output a target edited video through the video editing model; Acquire target feedback data for the target clip video; The video editing model is adjusted based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

2. The method according to claim 1, characterized in that The video clip model includes: An input module, used for receiving the video to be edited, and extracting multimodal data from the video to be edited; wherein the multimodal data includes video data, audio data and / or text data of the video to be edited; An intelligent editing module, used for outputting corresponding semantic analysis results according to the input multimodal data; A script acquisition module, used for generating the editing script according to the semantic analysis result, and used for instructing to edit the video to be edited into the target editing video; An optimization module, used for adjusting the intelligent editing module in the video editing model according to the target feedback data; The output module is used to provide file output based on the target clip video.

3. The method according to claim 2, characterized in that Inputting the multimodal data into the intelligent editing module for multiple rounds, the intelligent editing module outputting a plurality of the semantic analysis results, one of the semantic analysis results corresponds to one of the editing scripts, and one of the editing scripts corresponds to generating one of the target editing videos; The method further comprises: Acquiring target feedback data of each target clip video through the optimization module; Based on the target feedback data of each target clip video, evaluating each clip script; Determining a preset number of target editing scripts from a plurality of the editing scripts according to evaluation parameters of the editing scripts; The target editing script is used for subsequent video editing.

4. The method according to claim 2, characterized in that: Adjusting the video editing model based on the target feedback data includes: Based on the target feedback data, generating adjustment instructions through the optimization module to generate a preset form value; The preset form value is provided to the intelligent editing module through the optimization module to adjust the parameters of the video editing model.

5. The method according to claim 2, characterized in that: The intelligent editing module includes a first convolutional layer, and the first convolutional layer includes a plurality of first matrices; adjusting the video editing model based on the target feedback data includes: Adding a second matrix in the first convolutional layer of the smart editing module; the dimension of the second matrix is ​​lower than that of the first matrix; Based on the target feedback data, the parameters of the second matrix are adjusted to optimize the video editing model.

6. The method according to claim 1, characterized in that The method may further comprise one or more of the following features: (1) Target feedback data includes: viewing time, click-through rate, sharing rate, like rate and / or interactive information of the target video clip; (2) The target feedback data includes the emotion type of the target video clip; the emotion type is determined according to the interaction information.

7. A video editing device, characterized in that: The device comprises: A first acquisition module is used to acquire the video to be edited; A video input module, used for inputting the video to be edited into a video editing model, so as to output a target edited video through the video editing model; A second acquisition module is used to acquire target feedback data for the target clip video; The adjustment module is used to adjust the video editing model based on the target feedback data, and the adjusted video editing model is used to process the next video to be edited.

8. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 6 are implemented.

Images