Video editing optimization method and system based on artificial intelligence

Abstract

The invention relates to a video editing optimization method and system based on artificial intelligence, the system comprises a front-end module, an intelligent processing engine and a back-end service and data layer, and the method comprises the steps of intelligent lens selection and sorting, automatic transition and rhythm control, content perception editing, personalized style matching and man-machine collaborative verification. According to the video editing optimization method and system based on artificial intelligence, the processing efficiency is greatly improved, the processing speed is more than five times that of a traditional method, shot selection and editing quality are guaranteed through an algorithm, the editing possibility exceeding artificial imagination is provided, the method and system are adaptive to styles and content requirements of different users, and continuous expansion of functions is supported through modular design.

Description

Technical Field

[0001] The present invention relates to a method and system for optimizing video editing based on artificial intelligence. Background Art

[0002] Video editing refers to the process of selecting, trimming, combining, decorating video materials, and adding elements such as special effects, audio, and text to finally form a coherent, logical, and artistically expressive video work.

[0003] However, with the explosive growth of video content and the increasing demand for high-quality videos from users, traditional video editing methods are facing challenges such as low efficiency and limited creativity. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: In order to overcome the above technical problems, the present invention provides a method and system for optimizing video editing based on artificial intelligence.

[0005] The technical solution adopted by the present invention to solve its technical problems is: A video editing optimization system based on artificial intelligence, including a front-end module, an intelligent processing engine, a back-end service, and a data layer;

[0006] The front-end module includes a user interaction interface, a real-time preview engine, and a parameter adjustment panel;

[0007] User interaction interface: Supports drag-and-drop timeline editing, supports AI shortcut command input, and multi-view mode;

[0008] Real-time preview engine: 4K low-latency rendering based on WebGL, instant visualization of AI-generated effects, and multi-version comparison playback;

[0009] Parameter adjustment panel: Used for automated parameters, manual micro-control components, and intelligent recommendation sliders;

[0010] The intelligent processing engine includes a video analysis module, a decision generation module, and an effect rendering module;

[0011] Video analysis module: Adopts a multi-modal Transformer architecture to parallelly process visual features, audio, and text features, and outputs structured metadata;

[0012] Decision generation module: A clip strategy generator based on reinforcement learning, implementing a dual-path decision-making mechanism of a rule engine and generative AI;

[0013] Effect rendering module: Dynamically loads a stylized GAN model to achieve hardware acceleration of CUDA parallelized special effect rendering and NPU-accelerated real-time super-resolution;

[0014] The backend services include media asset management, a distributed processing framework, and a model training platform;

[0015] Media asset management: intelligent tagging system and version control;

[0016] Distributed processing framework: task scheduling based on Ray, feature extraction and lightweight inference for CPU clusters, model training and heavyweight rendering for GPU clusters, elastic scaling;

[0017] Model training platform: automated training pipeline, data augmentation, and federated learning;

[0018] The data layer includes a video feature database, a user behavior database, and a model parameter repository.

[0019] Preferably, the video feature database uses a FAISS vector database, with typical data being CLIP embedding vectors and optical flow features, mainly for fast retrieval of similar shots; the user behavior database uses a ClickHouse time-series database, with typical data being operation heatmaps and parameter adjustment histories, mainly for personalized recommendation model training; the model parameter repository uses Git-LFS and DVC version control, with typical data being PyTorch model checkpoints, mainly for model A / B testing and fast rollback.

[0020] Preferably, the AI quick command input of the user interface refers to describing the editing requirements through voice or text, and the multi-view mode includes a storyboard mode, a shot-splitting mode, and a waveform mode.

[0021] Preferably, the automated parameters of the parameter adjustment panel include an AI confidence threshold and a style intensity, the manual fine-tuning controls are used to adjust the keyframe curves and transition durations, and the intelligent recommendation sliders are dynamically sorted according to the user's historical behavior.

[0022] Preferably, in the model training platform, data augmentation is used for generating spatio-temporal domain adversarial samples, and federated learning is used to protect the privacy of the editing style. [[ID=z5]]

[0023] An optimization method for an AI-based video editing optimization system as described above, comprising the following steps:

[0024] a. Intelligent shot selection and sorting: a shot quality evaluation model based on deep learning, multi-dimensional feature analysis, sentiment analysis, and scene matching algorithms;

[0025] b. Automatic transition and rhythm control: music beat detection and time alignment, narrative rhythm prediction based on LSTM, and adaptive transition effect recommendation;

[0026] c. Content-aware editing: object recognition and tracking, semantic scene segmentation, and keyframe automatic extraction algorithm;

[0027] d. Personalized style matching: User preference learning, style transfer, multi-modal content matching;

[0028] e. Human-machine collaborative verification: Use a confidence-driven interaction design scheme to verify the results.

[0029] Preferably, the architecture of the deep learning-based lens quality assessment model includes heterogeneous assessment of technical quality, heterogeneous assessment of aesthetic scoring, and heterogeneous assessment of stability. The data augmentation scheme of the deep learning-based lens quality assessment model includes dynamic blur, illumination perturbation, compression artifacts, and style transfer. The training techniques of the deep learning-based lens quality assessment model include progressive training, hard example mining, and dynamic weighting. Visual quality assessment uses the NR-IQA model. Multi-dimensional feature analysis includes visual quality assessment, emotion-scene matching, and dynamic sorting optimization. Emotion-scene matching uses a visual and audio dual-channel emotion classifier.

[0030] Preferably, semantic scene segmentation uses the Mask2Former model to obtain pixel-level labels. The key frame extraction algorithm is change detection based on entropy value. Take 1s before and after the scene mutation point as the candidate key segment.

[0031] Preferably, the process of multi-modal content matching includes:

[0032] d1. The user uploads reference music and extracts Mel spectrogram features;

[0033] d2. Use FAISS to retrieve segments with similar rhythms in the video library;

[0034] d3. Use CLIP to screen the candidate set for semantic matching.

[0035] Preferably, in d1, the sampling rate for Mel spectrogram feature extraction is 22.05KHz, the number of Mel bands is 128, and the temporal alignment is fixed at 1300 frames / 30s.

[0036] The beneficial effects of the present invention are that the video editing optimization method and system based on artificial intelligence have greatly improved processing efficiency, which is more than 5 times the processing speed of traditional methods. The algorithm ensures the quality of lens selection and editing, provides editing possibilities beyond human imagination, adapts to different user styles and content requirements, and the modular design supports continuous function expansion. Detailed implementation manner

[0037] A video editing optimization system based on artificial intelligence includes a front-end module, an intelligent processing engine, a back-end service, and a data layer;

[0038] The front-end module includes a user interface, a real-time preview engine, and a parameter adjustment panel;

[0039] User interaction interface: Supports drag-and-drop timeline editing, AI shortcut command input, and multi-view mode;

[0040] Real-time preview engine: 4K low-latency rendering based on WebGL, instant visualization of AI-generated effects, and multi-version comparison playback;

[0041] Parameter adjustment panel: For automated parameters, manual micro-control components, and intelligent recommendation sliders;

[0042] The intelligent processing engine includes a video analysis module, a decision generation module, and an effect rendering module;

[0043] Video analysis module: Adopts a multi-modal Transformer architecture to process visual features, audio, and text features in parallel and outputs structured metadata;

[0044] Decision generation module: A clip strategy generator based on reinforcement learning, implementing a dual-path decision mechanism of a rule engine and generative AI;

[0045] Effect rendering module: Dynamically loads a stylized GAN model to achieve hardware acceleration of CUDA parallelized special effect rendering and NPU-accelerated real-time super-resolution;

[0046] The backend service includes media asset management, a distributed processing framework, and a model training platform;

[0047] Media asset management: Intelligent tag system and version control;

[0048] Distributed processing framework: Task scheduling based on Ray, performs feature extraction and lightweight inference on CPU clusters, performs model training and heavyweight rendering on GPU clusters, and has elastic scalability;

[0049] Model training platform: Automated training pipeline, data augmentation, and federated learning;

[0050] The data layer includes a video feature database, a user behavior database, and a model parameter repository.

[0051] Preferably, the video feature database uses a FAISS vector database, and typical data are CLIP embedding vectors and optical flow features, mainly used for fast retrieval of similar shots; the user behavior database uses a ClickHouse time series database, and typical data are operation heatmaps and parameter adjustment histories, mainly used for personalized recommendation model training; the model parameter repository uses Git-LFS and DVC version control, and typical data are PyTorch model checkpoints, mainly used for model A / B testing and fast rollback.

[0052] Preferably, the AI quick command input of the user interaction interface refers to describing the clip requirements through voice or text. The multi-view mode includes the storyboard mode, the shot-splitting mode, and the waveform mode.

[0053] Preferably, the automated parameters of the parameter adjustment panel include the AI confidence threshold and the style intensity. The manual fine-tuning control is used to adjust the key frame curve and the transition duration, and the intelligent recommendation slider is dynamically sorted according to the user's historical behavior.

[0054] Preferably, data augmentation in the model training platform is used for generating spatio-temporal domain adversarial samples, and federated learning is used to protect the privacy of the clip style.

[0055] An optimization method for an AI-based video clip optimization system as described above includes the following steps:

[0056] a. Intelligent shot selection and sorting: a shot quality evaluation model based on deep learning, multi-dimensional feature analysis, sentiment analysis, and scene matching algorithms;

[0057] b. Automatic transition and rhythm control: music beat detection and temporal alignment, narrative rhythm prediction based on LSTM, and adaptive transition effect recommendation;

[0058] c. Content-aware editing: object recognition and tracking, semantic scene segmentation, and automatic key frame extraction algorithm;

[0059] d. Personalized style matching: user preference learning, style transfer, and multi-modal content matching;

[0060] e. Human-machine collaborative verification: using a confidence-driven interaction design scheme to verify the results.

[0061] Preferably, the architecture of the shot quality evaluation model based on deep learning includes technical quality heterogeneous evaluation, aesthetic score heterogeneous evaluation, and stability heterogeneous evaluation. The data augmentation scheme of the shot quality evaluation model based on deep learning includes motion blur, illumination perturbation, compression artifacts, and style transfer. The training techniques of the shot quality evaluation model based on deep learning include progressive training, hard example mining, and dynamic weighting. The visual quality evaluation uses the NR-IQA model. The multi-dimensional feature analysis includes visual quality evaluation, emotion-scene matching, and dynamic sorting optimization. The emotion-scene matching uses a visual and audio dual-channel emotion classifier.

[0062] Preferably, semantic scene segmentation uses the Mask2Former model to obtain pixel-level labels. The key frame extraction algorithm is change detection based on entropy value. 1s before and after the scene mutation point is taken as the candidate key segment.

[0063] Preferably, the process of multi-modal content matching includes:

[0064] d1. The user uploads the reference music and extracts the Mel spectrogram features;

[0065] d2. Use FAISS to retrieve segments with similar rhythms in the video library;

[0066] d3. Use CLIP to screen the candidate set with semantic matching.

[0067] Preferably, in d1, the sampling rate for Mel spectrogram feature extraction is 22.05KHz, the number of Mel bands is 128, and the temporal alignment is fixed at 1300 frames / 30s.

[0068] WebGL, short for Web Graphics Library, is a 3D drawing protocol. This drawing technology standard allows combining JavaScript and OpenGLES2.0. By adding a JavaScript binding to OpenGLES2.0, WebGL can provide hardware-accelerated 3D rendering for HTML5 Canvas. In this way, web developers can use the system graphics card to more smoothly display 3D scenes and models in the browser, and can also create complex navigation and data visualization.

[0069] ClickHouse is an open-source columnar database used for processing large-scale time series data.

[0070] In ClickHouse, time series data processing is an important application scenario, which involves operations such as aggregating, filtering, transforming, and analyzing time series data. ClickHouse provides rich functions and usage methods for processing time series data, including time window aggregation, data sampling, data filtering, and data transformation. By reasonably using these functions, large-scale time series data can be processed efficiently.

[0071] Git-LFS is an open-source Git extension dedicated to version control of large files. When using Git to upload large files, if the file size exceeds 100MB, an error will be reported. Git-LFS replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git and stores the file content on remote servers such as GitHub.com or GitHub Enterprise.

[0072] DVC is a data version control tool that can help researchers manage data and models and run repeatable experiments. DVC mimics Git's commands and workflows and aims to be easily integrated into regular Git practices. Git is specifically used for storing and version controlling code, while DVC performs the same functions but focuses on data and model files.

[0073] The spatio-temporal domain adversarial sample generation adopts a three-dimensional adversarial attack algorithm, which ensures the naturalness of perturbations in the time series through optical flow constraints and only applies significant perturbations in the high-frequency region.

[0074] Its data augmentation strategies include temporal jitter, spatial deformation, and multi-modal attacks.

[0075] The implementation method of temporal jitter is the inter-frame temporal interpolation perturbation, and the defense objective is to improve the robustness of the model to frame rate changes;

[0076] The implementation method of spatial deformation is the elastic grid deformation, and the defense objective is to enhance the adaptability to lens distortion;

[0077] The implementation method of multi-modal attacks is to synchronously perturb visual and audio features, and the defense objective is to improve cross-modal consistency.

[0078] The steps of the federated learning protocol are as follows:

[0079] 1. Initialization: The server distributes the global model and transmits it through RSA-2048 encryption;

[0080] 2. Local training: The user device trains with private data without leaving the domain;

[0081] 3. Gradient upload: Upload the gradients of the style-related layers with differential privacy;

[0082] 4. Aggregation: Secure multi-party computation and private sharing;

[0083] 5. Update: Distribute the new model and verify the digital signature.

[0084] Example 1

[0085] Example of the system interaction process:

[0086] 1. The user uploads the original video, and the front end triggers an analysis request;

[0087] 2. The video analysis module generates metadata in JSON format, including key frames and timestamps;

[0088] 3. The decision-making module combines historical data to generate 3 editing schemes, and the historical data comes from the user behavior database;

[0089] 4. The effect rendering module calls the stylization model to generate a preview, and the stylization model parameters are obtained from the model repository;

[0090] 5. After the user selects a scheme, the distributed framework starts the final rendering and stores it in the media asset library.

[0091] Here, a hybrid decision-making mechanism is adopted. The rule engine ensures professionalism, the generative AI provides creativity, real-time federated learning enables real-time update of the edge model with the user's local behavior data, and multi-granularity version management supports version tracing from individual feature parameters to complete engineering files. With the above system architecture, the editing work efficiency can be increased by more than 2 times, and the adoption rate of AI-generated content can be increased by 30%.

[0092] Beat tolerance mechanism: When the detection confidence < 0.7, dynamic time warping alignment is adopted.

[0093] Attention LSTM: When predicting the editing point, both visual rhythm and audio rhythm are considered simultaneously.

[0094] Transition knowledge graph: A transition rule library built based on Film Grammar.

[0095] In the data augmentation scheme of the deep learning-based lens quality assessment model, the way of dynamic blur is to simulate camera shake with random motion kernel convolution, mainly used to improve stability and evaluation robustness; the implementation method of illumination perturbation is random Gamma correction, mainly used to enhance brightness adaptability; the implementation method of compression artifacts is WebP lossy compression, mainly used to improve noise tolerance; the implementation method of style transfer is fast AdaIN style mixing, mainly to prevent overfitting of aesthetic scores.

[0096] In the training techniques of the deep learning-based lens quality assessment model, progressive training is to first train the technical evaluation branch and then unfreeze the aesthetic branch. In hard example mining, the top 5% of samples with the largest prediction errors are retained in each epoch and added to the next round of training. In dynamic weighting, the weights of each loss term are automatically adjusted according to the performance on the validation set.

[0097] Example 2

[0098] Confidence-driven interaction design case: The user submits an editing request to the system, and the system AI automatically generates a plan with a confidence of 72%. The preset confidence of the system is 80%. That is, at this time, the confidence of the plan is less than the system preset confidence threshold, so the system displays a "low confidence warning", and the user manually adjusts the parameters until the confidence value of the plan generated by the system > 80%, and then the result is directly output.

[0099] Example 3

[0100] Adaptive optimization of the regional feature library:

[0101] 1. Cultural dimension: Color symbolism; Feature extraction method: Dominant color clustering and semantic analysis; Adaptation case: White represents mourning in the East and purity in the West.

[0102] 2. Cultural Dimension: Body Language; Feature Extraction Method: 3D Pose Estimation and Symbol Mapping; Adaptation Case: The OK gesture indicates agreement or confirmation in most regions, but is offensive in Brazil.

[0103] 3. Cultural Dimension: Religious Elements; Feature Extraction Method: Object Detection and Knowledge Graph Association; Adaptation Case: Sensitivity handling of the cross shape.

[0104] Example 4

[0105] Problem Handling:

[0106] AI misjudges "scattering petals at the wedding scene" as "litter scattered".

[0107] Solutions include: 1. Add more than 500 positive samples of wedding scenes.

[0108] 2. Rule Constraint: Disable the "litter" label when "wedding dress" and "arch" are detected.

[0109] 3. Feature Engineering: Increase the analysis of the petal falling trajectory. For example, if the trajectory of the petals is a parabola, it represents the scattering of petals at the wedding scene; if the trajectory of the petals is randomly scattered, it represents that the petals are litter.

[0110] Compared with the prior art, the video editing optimization method and system based on artificial intelligence have greatly improved the processing efficiency, which is more than 5 times the processing speed of traditional methods. The algorithm ensures the shot selection and editing quality, provides editing possibilities beyond human imagination, adapts to different user styles and content requirements, and the modular design supports continuous function expansion.

[0111] Enlightened by the ideal embodiments of the present invention described above, through the above description, relevant staff can make various changes and modifications without departing from the technical idea of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. An artificial intelligence-based video editing optimization system, characterized in that It includes a front-end module, an intelligent processing engine, a back-end service, and a data layer; The front-end module includes a user interaction interface, a real-time preview engine, and a parameter adjustment panel; User interaction interface: Supports drag-and-drop timeline editing, AI quick command input, and multi-view mode; Real-time preview engine: 4K low-latency rendering based on WebGL, instant visualization of AI-generated effects, and multi-version comparison playback; Parameter adjustment panel: For automated parameters, manual fine-tuning controls, and intelligent recommendation sliders; The intelligent processing engine includes a video analysis module, a decision generation module, and an effect rendering module; Video analysis module: Adopts a multi-modal Transformer architecture to parallel process visual features, audio, and text features, and outputs structured metadata; Decision generation module: A clip strategy generator based on reinforcement learning, implementing a dual-path decision-making mechanism of a rule engine and generative AI; Effect rendering module: Dynamically loads a stylized GAN model to achieve hardware acceleration of CUDA parallelized special effect rendering and NPU-accelerated real-time super-resolution; The back-end service includes media asset management, a distributed processing framework, and a model training platform; Media asset management: Intelligent tag system and version control; Distributed processing framework: Task scheduling based on Ray, performs feature extraction and lightweight inference on CPU clusters, conducts model training and heavy-load rendering on GPU clusters, with elastic scaling; Model training platform: Automated training pipeline, data augmentation, and federated learning; The data layer includes a video feature database, a user behavior database, and a model parameter repository.

2. The video clip optimization system based on artificial intelligence according to claim 1, characterized in that, The video feature database uses a FAISS vector database, with typical data being CLIP embedding vectors and optical flow features, mainly used for fast retrieval of similar shots; the user behavior database uses a ClickHouse time-series database, with typical data being operation heatmaps and parameter adjustment histories, mainly used for personalized recommendation model training; the model parameter repository uses Git-LFS and DVC version control, with typical data being PyTorch model checkpoints, mainly used for model A / B testing and quick rollback.

3. The video clip optimization system based on artificial intelligence according to claim 1, characterized in that The AI quick command input of the user interaction interface refers to describing the editing requirements through voice or text, and the multi-view mode includes a storyboard mode, a shot breakdown mode, and a waveform mode.

4. The video clip optimization system based on artificial intelligence according to claim 1, characterized in that The automated parameters of the parameter adjustment panel include AI confidence thresholds and style intensities, the manual fine-tuning controls are used to adjust keyframe curves and transition durations, and the intelligent recommendation sliders are dynamically sorted according to the user's historical behavior.

5. The video clip optimization system based on artificial intelligence according to claim 1, wherein In the model training platform, data augmentation is used for generating spatio-temporal domain adversarial samples, and federated learning is used to protect the privacy of the editing style.

6. An optimization method for an artificial intelligence-based video editing optimization system according to any one of claims 1-5, characterized in that, It includes the following steps: a. Intelligent shot selection and sorting: A shot quality assessment model based on deep learning, multi-dimensional feature analysis, sentiment analysis, and scene matching algorithms; b. Automatic transition and rhythm control: Music beat detection and temporal alignment, narrative rhythm prediction based on LSTM, and adaptive transition effect recommendation; c. Content-aware editing: Object recognition and tracking, semantic scene segmentation, and keyframe automatic extraction algorithms; d. Personalized style matching: User preference learning, style transfer, multi-modal content matching; e. Human-machine collaborative verification: Use a confidence-driven interaction design scheme for result verification.

7. The method for optimizing video editing based on artificial intelligence according to claim 6, characterized in that, The architecture of the deep learning-based lens quality assessment model includes heterogeneous assessment of technical quality, heterogeneous assessment of aesthetic scores, and heterogeneous assessment of stability. The data augmentation scheme of the deep learning-based lens quality assessment model includes motion blur, illumination perturbation, compression artifacts, and style transfer. The training techniques of the deep learning-based lens quality assessment model include progressive training, hard example mining, and dynamic weighting. Visual quality assessment uses the NR-IQA model. Multi-dimensional feature analysis includes visual quality assessment, emotion-scene matching, and dynamic sorting optimization. Emotion-scene matching uses a visual and audio dual-channel emotion classifier.

8. The video clip optimization method based on artificial intelligence according to claim 6, wherein, Semantic scene segmentation uses the Mask2Former model to obtain pixel-level labels. The key frame extraction algorithm is change detection based on entropy values. Take 1s before and after the scene mutation point as candidate key segments.

9. The method for optimizing video editing based on artificial intelligence according to claim 6, wherein, The process of multi-modal content matching includes: d1. The user uploads reference music and extracts Mel spectrogram features; d2. Use FAISS to retrieve segments with similar rhythms in the video library; d3. Use CLIP to screen the candidate set for semantic matching.

10. The method for optimizing video editing based on artificial intelligence according to claim 9, characterized in that, In d1, the sampling rate for Mel spectrogram feature extraction is 22.05KHz, the number of Mel bands is 128, and the temporal alignment is fixed at 1300 frames / 30s.