A video synthesis method and system

Abstract

The application discloses a video synthesis method and system, and relates to the technical field of audio and video processing.A video synthesis system comprises a video source acquisition and processing module, a cross-modal semantic understanding module, an attention tensor generation module, a hierarchical progressive fusion module and a quality evaluation and optimization module.Through a three-dimensional convolution network, the application extracts spatio-temporal joint features and constructs an audio-visual cross-modal attention mechanism, can quantize the dynamic correlation strength of audio events and video content and generate a subject space mask, thereby expanding traditional isolated visual processing into audio-visual semantic linkage understanding, so as to realize the deep guidance of multi-modal information to the synthesis process, and improve the synthesis effect of the video synthesis method and system.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, specifically to a video synthesis method and system. Background Technology

[0002] Video compositing is the process of seamlessly merging multiple video clips, images, graphics, special effects, and virtual elements into a single, continuous image using digital technology. It achieves precise overlay and synchronization of materials from different sources through techniques such as keying, masking, motion tracking, and 3D rendering, and adds effects such as transitions, filters, and animations to ultimately generate visual content with high realism or artistic expression. It is widely used in film and television production, advertising creativity, live streaming, virtual reality, and short video creation.

[0003] Traditional video compositing techniques mainly rely on basic algorithms such as linear overlay and transparency blending, which cannot deeply understand the semantics of video content. This results in fixed and rigid compositing parameters, making it difficult to adaptively adjust according to the subject, movement rhythm, and scene changes. At the same time, the compositing quality depends on repeated manual debugging, lacks automated evaluation and optimization mechanisms, and distributes computing resources evenly across all areas of the screen, resulting in low processing efficiency.

[0004] Based on the above, this invention proposes a video synthesis method and system with good synthesis effect. Summary of the Invention

[0005] To overcome the shortcomings of traditional video compositing techniques, which mainly rely on basic algorithms such as linear overlay and transparency blending, and cannot deeply understand the semantics of video content, resulting in fixed and rigid compositing parameters that are difficult to adaptively adjust according to the subject, movement rhythm and scene changes, as well as the fact that the compositing quality depends on repeated manual debugging, lacks an automated evaluation and optimization mechanism, and has low processing efficiency due to the even distribution of computing resources across all screen areas, this invention proposes a video compositing method and system with good compositing effect.

[0006] A video compositing method includes the following steps:

[0007] Receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata;

[0008] Feature extraction and subject-background separation are performed on the preprocessed video data to obtain the main features and binary space mask of the video data. The preprocessed video data is then segmented into multiple semantic segments by calculating the feature differences between adjacent frames. A cross-modal attention mechanism is constructed based on the preprocessed audio data, and the audio guidance weight matrix for each video frame is calculated.

[0009] Semantic keyframes are labeled for each semantic segment to obtain the distribution of semantic keyframes in each semantic segment. Based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes, a three-dimensional attention tensor is generated by calculation.

[0010] The preprocessed video data is fused in a layered and progressive manner to obtain the preliminary synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor.

[0011] A lightweight regression network is trained based on the 3D attention tensor and the initial contribution of each video source to predict the structural similarity value between the fused video frame and the reference video frame. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized based on the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data.

[0012] As a preferred aspect of the invention, the specific steps for preprocessing the video stream, audio stream, and metadata stream to obtain preprocessed video data, audio data, and metadata are as follows:

[0013] The video stream is subjected to bicubic interpolation to generate original resolution video data, half-resolution video data, and quarter-resolution video data. The audio stream is converted into a Mel spectrogram using short-time Fourier transform, and the Mel spectrogram is segmented into overlapping audio time-series blocks. Timestamps, frame rates, and shooting parameters are extracted from the metadata stream and encoded into metadata vectors. Using one video source as a reference, dynamic time warping is applied to the remaining video sources, and the time offset function with the reference video source is calculated. Based on the time offset function, the three resolution video data, audio time-series blocks, and metadata vectors are rearranged along a unified time axis to obtain preprocessed video data, audio data, and metadata.

[0014] As a preferred aspect of the invention, the specific steps of performing feature extraction and subject-background separation on the preprocessed video data to obtain the main features and binary space mask of the video data, segmenting the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames, constructing a cross-modal attention mechanism based on the preprocessed audio data, and calculating the audio-guided weight matrix for each video frame are as follows:

[0015] The original resolution video data in the preprocessed video data is input into the spatiotemporal 3D convolutional network. After processing through multiple residual blocks, the spatiotemporal feature tensor of the video data is output. A region proposal network is applied to the spatial dimension of the spatiotemporal feature tensor to generate subject candidate boxes. The RoI Align operation is performed on each subject candidate box to obtain the subject features of the video data and generate a binary space mask to distinguish the subject from the background.

[0016] The feature differences between adjacent frames of video data are calculated and scene boundaries are detected based on dynamic thresholds. The preprocessed video data is then segmented into multiple semantic segments based on scene boundaries. The dynamic threshold is the sum of the mean of the feature differences and twice the standard deviation of the feature differences. When the feature differences between adjacent frames of video data are greater than the dynamic threshold, they are determined to be scene boundaries.

[0017] The preprocessed audio data is linearly projected and converted into a key-value tensor. The spatiotemporal feature tensor of the video data is compressed into a visual query tensor through global average pooling. Based on the key-value tensor and the visual query tensor, the guided weight matrix of audio for each frame of video is calculated.

[0018] As a preferred aspect of the invention, the specific steps of marking each semantic segment with semantic keyframes to obtain the distribution of semantic keyframes in each semantic segment, and generating a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes, are as follows:

[0019] Frame difference is calculated for each time position in each semantic segment. The frame difference is composed of a weighted sum of the difference between the spatiotemporal feature tensor and the change in the binary space mask. Frames with a frame difference exceeding the dynamic threshold are marked as semantic key frames, and the rest are marked as transition frames. The distribution of semantic key frames in each semantic segment is obtained. The dynamic threshold needs to be adaptively adjusted according to the density of semantic key frames to ensure that the proportion of semantic key frames is within the preset range.

[0020] Gaussian filtering is applied to the binary spatial mask of the video data for smoothing, and further weighting is performed based on the saliency of the main features to obtain spatial attention. Temporal attention is initialized and the maximum value is processed in key frames, while the transition frames are decayed according to the cosine function to obtain temporal attention. Spatial attention and temporal attention are weighted and fused and normalized frame by frame to obtain the three-dimensional attention tensor.

[0021] As a preferred aspect of the invention, the specific steps of performing layered progressive fusion of the preprocessed video data to obtain initially synthesized video data, and dynamically calculating the initial contribution of each video source based on a binary space mask and a three-dimensional attention tensor during the video data fusion process, are as follows:

[0022] The underlying fusion takes quarter-resolution video data, downsampled 3D attention tensor and base contribution as input, calculates optical flow field and performs motion compensation alignment, separates static background and dynamic foreground through Gaussian mixture model, and outputs background layer and foreground residual components. The base contribution is pre-calculated by multiplying the 3D attention tensor and guiding weight matrix element by element.

[0023] The intermediate-layer fusion takes half-resolution video data, upsampled bottom-layer fusion results, downsampled 3D attention tensor, and binary space mask as input. The binary space mask is used to distinguish the subject and background processing paths. The background path is stabilized by weighted mixing, while the subject path is solved by Poisson fusion to smooth the boundaries. The intermediate-resolution fusion result is output.

[0024] Top-level fusion takes the original resolution video data, the upsampled mid-level fusion result, the 3D attention tensor, and the main features as input. It inherits the mid-level structure and retains the original details through guided filtering. The sharpness score of each video source is calculated in the overlapping area of ​​the main subject. The video source with the highest score is selected as the dominant one, and the transparency of the other video sources is reduced and superimposed. The output is the pre-synthesized video data.

[0025] Contribution calculation involves dynamically calculating the initial contribution of each video source for each frame during the video data fusion process. The initial contribution is based on the three-dimensional attention tensor, the guiding weight matrix, and the binary space mask, and is obtained through calculation.

[0026] As a preferred aspect of the invention, the specific steps of training a lightweight regression network based on the three-dimensional attention tensor and the initial contribution of each video source to predict the structural similarity value between the fused video frame and the reference video frame, adjusting the initial contribution based on the structural similarity value, evaluating the quality of video fusion through multiple video fusion metrics, iteratively optimizing the initial contribution based on the evaluation results, and re-fusing the video data accordingly to obtain the final contribution of each video source and the final synthesized video data are as follows:

[0027] A lightweight regression network is constructed. The three-dimensional attention tensor and the initial contribution of each video source are input into the trained lightweight regression network to obtain the predicted structural similarity value between the fused video frame and the reference video frame. When the predicted structural similarity value is lower than the preset threshold, the initial contribution is Gaussian smoothed.

[0028] The video source with the highest resolution is selected as a reference. Multiple video fusion indicators of the initially synthesized video data are calculated, and a video fusion quality score is obtained. When the video fusion quality score drops and falls below a preset threshold, the gradient of the video fusion quality score to the contribution is calculated, and the contribution of each video source is iteratively updated. Based on the updated contribution, the video data is re-fused until the video fusion quality score meets the preset requirements or reaches the maximum number of iterations, and the final contribution of each video source and the final synthesized video data are obtained.

[0029] A video compositing system, comprising:

[0030] The video source acquisition and processing module is used to receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata.

[0031] The cross-modal semantic understanding module is used to extract features and separate the subject from the background in the preprocessed video data to obtain the subject features and binary space mask of the video data. It also divides the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames. Based on the preprocessed audio data, a cross-modal attention mechanism is constructed and the audio guidance weight matrix for each frame of video is calculated.

[0032] The attention tensor generation module is used to mark semantic keyframes for each semantic segment, obtain the distribution of semantic keyframes in each semantic segment, and generate a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes.

[0033] The layered progressive fusion module is used to perform layered progressive fusion on the preprocessed video data to obtain the initially synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor.

[0034] The quality assessment and optimization module is used to train a lightweight regression network based on the three-dimensional attention tensor and the initial contribution of each video source, and predict the structural similarity value between the fused video frame and the reference video frame. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized according to the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data.

[0035] The present invention has the following advantages:

[0036] 1. This invention extracts spatiotemporal joint features through a three-dimensional convolutional network and constructs an audio-visual cross-modal attention mechanism, which can quantify the dynamic correlation strength between audio events and video content and generate a subject space mask. This expands the traditional isolated visual processing into an audio-visual semantic linkage understanding, so as to realize the deep guidance of multimodal information on the synthesis process and improve the synthesis effect of this video synthesis method and system.

[0037] 2. This invention adaptively labels semantic keyframes by calculating the weighted value of the visual differences between frames and the changes in the subject mask, and uses compressed parameters to represent transition frames instead of storing them completely. This can significantly reduce data redundancy while ensuring the quality of the main subject area, thereby constructing a non-uniform sampling time axis and generating a three-dimensional attention tensor. This enables intelligent tilting of computing resources toward important content and improves the rationality of resource allocation in this video synthesis method and system.

[0038] 3. This invention separates background modeling, subject contour optimization, and detail refining through a bottom-up three-level fusion pipeline. This not only addresses alignment, smoothing, and clarity issues at different resolution levels, but also introduces a competitive mechanism in overlapping subject areas to dynamically select the optimal source instead of simple weighting. This effectively avoids ghosting and artifacts, improves the naturalness and realism of the composite results, and enhances the compositing effect of this video compositing method and system.

[0039] 4. This invention establishes a dual-loop feedback system for pre-compositing quality prediction and post-compositing defect detection, which can proactively adjust high-risk parameters before fusion and automatically backtrack to optimize after fusion failure. This transforms traditional offline manual debugging into online closed-loop iteration, significantly improving the stability of output quality and reducing manual intervention costs, thereby enhancing the synthesis effect of this video synthesis method and system. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating a video synthesis method used in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the structure of a video synthesis system used in an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this invention.

[0043] Example 1: A video synthesis method, such as Figure 1 As shown, it includes the following steps:

[0044] Receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata;

[0045] Feature extraction and subject-background separation are performed on the preprocessed video data to obtain the main features and binary space mask of the video data. The preprocessed video data is then segmented into multiple semantic segments by calculating the feature differences between adjacent frames. A cross-modal attention mechanism is constructed based on the preprocessed audio data, and the audio guidance weight matrix for each video frame is calculated.

[0046] Semantic keyframes are labeled for each semantic segment to obtain the distribution of semantic keyframes in each semantic segment. Based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes, a three-dimensional attention tensor is generated by calculation.

[0047] The preprocessed video data is fused in a layered and progressive manner to obtain the preliminary synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor.

[0048] A lightweight regression network is trained based on the 3D attention tensor and the initial contribution of each video source to predict the structural similarity value between the fused video frame and the reference video frame. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized based on the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data.

[0049] The specific steps for preprocessing the video stream, audio stream, and metadata stream to obtain preprocessed video data, audio data, and metadata are as follows:

[0050] The video stream is subjected to bicubic interpolation to generate original resolution video data, half-resolution video data, and quarter-resolution video data. The audio stream is converted into a Mel spectrogram using short-time Fourier transform, and the Mel spectrogram is segmented into overlapping audio time-series blocks. Timestamps, frame rates, and shooting parameters are extracted from the metadata stream and encoded into metadata vectors. Using one video source as a reference, dynamic time warping is applied to the remaining video sources, and the time offset function with the reference video source is calculated. Based on the time offset function, the three resolution video data, audio time-series blocks, and metadata vectors are rearranged along a unified time axis to obtain preprocessed video data, audio data, and metadata.

[0051] The specific steps for performing feature extraction and subject-background separation on the preprocessed video data to obtain the main features and binary space mask of the video data, segmenting the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames, constructing a cross-modal attention mechanism based on the preprocessed audio data, and calculating the audio guidance weight matrix for each video frame are as follows:

[0052] The original resolution video data in the preprocessed video data is input into the spatiotemporal 3D convolutional network. After processing through multiple residual blocks, the spatiotemporal feature tensor of the video data is output. A region proposal network is applied to the spatial dimension of the spatiotemporal feature tensor to generate subject candidate boxes. The RoI Align operation is performed on each subject candidate box to obtain the subject features of the video data and generate a binary space mask to distinguish the subject from the background.

[0053] The feature differences between adjacent frames of video data are calculated and scene boundaries are detected based on dynamic thresholds. The preprocessed video data is then segmented into multiple semantic segments based on scene boundaries. The dynamic threshold is the sum of the mean of the feature differences and twice the standard deviation of the feature differences. When the feature differences between adjacent frames of video data are greater than the dynamic threshold, they are determined to be scene boundaries.

[0054] The preprocessed audio data is linearly projected and converted into a key-value tensor. The spatiotemporal feature tensor of the video data is compressed into a visual query tensor through global average pooling. Based on the key-value tensor and the visual query tensor, the guided weight matrix of audio for each frame of video is calculated.

[0055] The above steps extract spatiotemporal joint features through a 3D convolutional network and construct an audio-visual cross-modal attention mechanism, which can quantify the dynamic correlation strength between audio events and video content and generate a subject space mask. This expands the traditional isolated visual processing into an audio-visual semantic linkage understanding, so as to realize the deep guidance of multimodal information on the synthesis process and improve the synthesis effect of this video synthesis method and system.

[0056] The specific steps for generating a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes in each semantic segment are as follows:

[0057] Frame difference is calculated for each time position in each semantic segment. The frame difference is composed of a weighted sum of the difference between the spatiotemporal feature tensor and the change in the binary space mask. Frames with a frame difference exceeding the dynamic threshold are marked as semantic key frames, and the rest are marked as transition frames. The distribution of semantic key frames in each semantic segment is obtained. The dynamic threshold needs to be adaptively adjusted according to the density of semantic key frames to ensure that the proportion of semantic key frames is within the preset range.

[0058] Gaussian filtering is applied to the binary spatial mask of the video data for smoothing, and further weighting is performed based on the saliency of the main features to obtain spatial attention. Temporal attention is initialized and the maximum value is processed in key frames, while the transition frames are decayed according to the cosine function to obtain temporal attention. Spatial attention and temporal attention are weighted and fused and normalized frame by frame to obtain the three-dimensional attention tensor.

[0059] The above steps adaptively label semantic keyframes by calculating the weighted value of the visual differences between frames and the changes in the subject mask, and use compressed parameters to represent transition frames instead of storing them completely. This can significantly reduce data redundancy while ensuring the quality of the main subject area, thereby constructing a non-uniform sampling time axis and generating a three-dimensional attention tensor. This enables intelligent tilting of computing resources toward important content and improves the rationality of resource allocation in this video synthesis method and system.

[0060] The specific steps for performing layered progressive fusion of preprocessed video data to obtain initially synthesized video data, and dynamically calculating the initial contribution of each video source based on binary space mask and three-dimensional attention tensor during the video data fusion process, are as follows:

[0061] The underlying fusion takes quarter-resolution video data, downsampled 3D attention tensor and base contribution as input, calculates optical flow field and performs motion compensation alignment, separates static background and dynamic foreground through Gaussian mixture model, and outputs background layer and foreground residual components. The base contribution is pre-calculated by multiplying the 3D attention tensor and guiding weight matrix element by element.

[0062] The intermediate-layer fusion takes half-resolution video data, upsampled bottom-layer fusion results, downsampled 3D attention tensor, and binary space mask as input. The binary space mask is used to distinguish the subject and background processing paths. The background path is stabilized by weighted mixing, while the subject path is solved by Poisson fusion to smooth the boundaries. The intermediate-resolution fusion result is output.

[0063] Top-level fusion takes the original resolution video data, the upsampled mid-level fusion result, the 3D attention tensor, and the main features as input. It inherits the mid-level structure and retains the original details through guided filtering. The sharpness score of each video source is calculated in the overlapping area of ​​the main subject. The video source with the highest score is selected as the dominant one, and the transparency of the other video sources is reduced and superimposed. The output is the pre-synthesized video data.

[0064] Contribution calculation involves dynamically calculating the initial contribution of each video source for each frame during the video data fusion process. The initial contribution is based on the three-dimensional attention tensor, the guiding weight matrix, and the binary space mask, and is obtained through calculation.

[0065] The above steps separate background modeling, subject contour optimization, and detail refining through a bottom-up three-level fusion pipeline. This not only addresses alignment, smoothing, and clarity issues at different resolution levels but also introduces a competitive mechanism in overlapping subject areas to dynamically select the optimal source instead of simple weighting. This effectively avoids ghosting and artifacts, improves the naturalness and realism of the composite results, and enhances the compositing effect of this video compositing method and system.

[0066] The specific steps for training a lightweight regression network based on the 3D attention tensor and the initial contributions of each video source to predict the structural similarity value between the fused video frame and the reference video frame, adjusting the initial contribution based on the structural similarity value, evaluating the quality of video fusion through multiple video fusion metrics, iteratively optimizing the initial contribution based on the evaluation results, and re-fusing the video data to obtain the final contribution of each video source and the final synthesized video data are as follows:

[0067] A lightweight regression network is constructed. The three-dimensional attention tensor and the initial contribution of each video source are input into the trained lightweight regression network to obtain the predicted structural similarity value between the fused video frame and the reference video frame. When the predicted structural similarity value is lower than the preset threshold, the initial contribution is Gaussian smoothed.

[0068] The video source with the highest resolution is selected as a reference. Multiple video fusion indicators of the initially synthesized video data are calculated, and a video fusion quality score is obtained. When the video fusion quality score drops and falls below a preset threshold, the gradient of the video fusion quality score to the contribution is calculated, and the contribution of each video source is iteratively updated. Based on the updated contribution, the video data is re-fused until the video fusion quality score meets the preset requirements or reaches the maximum number of iterations, and the final contribution of each video source and the final synthesized video data are obtained.

[0069] The above steps establish a dual-loop feedback system for pre-compositing quality prediction and post-compositing defect detection, which can proactively adjust high-risk parameters before fusion and automatically backtrack to optimize after fusion failure. This transforms traditional offline manual debugging into online closed-loop iteration, significantly improving the stability of output quality and reducing manual intervention costs, thereby enhancing the synthesis effect of this video synthesis method and system.

[0070] Example 2, a video compositing system, such as Figure 2 As shown, it includes:

[0071] The video source acquisition and processing module is used to receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata.

[0072] The cross-modal semantic understanding module is used to extract features and separate the subject from the background in the preprocessed video data to obtain the subject features and binary space mask of the video data. It also divides the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames. Based on the preprocessed audio data, a cross-modal attention mechanism is constructed and the audio guidance weight matrix for each frame of video is calculated.

[0073] The attention tensor generation module is used to mark semantic keyframes for each semantic segment, obtain the distribution of semantic keyframes in each semantic segment, and generate a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes.

[0074] The layered progressive fusion module is used to perform layered progressive fusion on the preprocessed video data to obtain the initially synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor.

[0075] The quality assessment and optimization module is used to train a lightweight regression network based on the three-dimensional attention tensor and the initial contribution of each video source, and predict the structural similarity value between the fused video frame and the reference video frame. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized according to the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data.

[0076] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Parts not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A video synthesis method, characterized in that, Includes the following steps: Receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata; Feature extraction and subject-background separation are performed on the preprocessed video data to obtain the main features and binary space mask of the video data. The preprocessed video data is then segmented into multiple semantic segments by calculating the feature differences between adjacent frames. A cross-modal attention mechanism is constructed based on the preprocessed audio data, and the audio guidance weight matrix for each video frame is calculated. Semantic keyframes are labeled for each semantic segment to obtain the distribution of semantic keyframes in each semantic segment. Based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes, a three-dimensional attention tensor is generated by calculation. The preprocessed video data is fused in a layered and progressive manner to obtain the preliminary synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor. A lightweight regression network is trained based on the three-dimensional attention tensor and the initial contribution of each video source, and the structural similarity value between the fused video frame and the reference video frame is predicted accordingly. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized according to the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data. The specific steps for preprocessing the video stream, audio stream, and metadata stream to obtain preprocessed video data, audio data, and metadata are as follows: The video stream is subjected to bicubic interpolation to generate original resolution video data, half-resolution video data, and quarter-resolution video data. The audio stream is converted into a Mel spectrogram using short-time Fourier transform, and the Mel spectrogram is segmented into overlapping audio time-series blocks. Timestamps, frame rates, and shooting parameters are extracted from the metadata stream and encoded into metadata vectors. Using one video source as a reference, dynamic time warping is applied to the remaining video sources, and the time offset function with the reference video source is calculated. Based on the time offset function, the three resolution video data, audio time-series blocks, and metadata vectors are rearranged along a unified time axis to obtain preprocessed video data, audio data, and metadata. The specific steps for performing feature extraction and subject-background separation on the preprocessed video data to obtain the main features and binary space mask of the video data, segmenting the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames, constructing a cross-modal attention mechanism based on the preprocessed audio data, and calculating the audio guidance weight matrix for each video frame are as follows: The original resolution video data in the preprocessed video data is input into the spatiotemporal 3D convolutional network. After processing through multiple residual blocks, the spatiotemporal feature tensor of the video data is output. A region proposal network is applied to the spatial dimension of the spatiotemporal feature tensor to generate subject candidate boxes. The RoI Align operation is performed on each subject candidate box to obtain the subject features of the video data and generate a binary space mask to distinguish the subject from the background. The feature differences between adjacent frames of video data are calculated and scene boundaries are detected based on dynamic thresholds. The preprocessed video data is then segmented into multiple semantic segments based on scene boundaries. The dynamic threshold is the sum of the mean of the feature differences and twice the standard deviation of the feature differences. When the feature differences between adjacent frames of video data are greater than the dynamic threshold, they are determined to be scene boundaries. The preprocessed audio data is linearly projected and converted into a key-value tensor. The spatiotemporal feature tensor of the video data is compressed into a visual query tensor through global average pooling. Based on the key-value tensor and the visual query tensor, the audio guidance weight matrix for each frame of video is calculated. The specific steps for generating a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes in each semantic segment are as follows: Frame difference is calculated for each time position in each semantic segment. The frame difference is composed of a weighted sum of the difference between the spatiotemporal feature tensor and the change in the binary space mask. Frames with a frame difference exceeding the dynamic threshold are marked as semantic key frames, and the rest are marked as transition frames. The distribution of semantic key frames in each semantic segment is obtained. The dynamic threshold needs to be adaptively adjusted according to the density of semantic key frames to ensure that the proportion of semantic key frames is within the preset range. Gaussian filtering is applied to the binary space mask of the video data for smoothing, and further weighting is performed in combination with the saliency of the main features to obtain spatial attention. Temporal attention is initialized and the maximum value is processed in key frames, while the transition frames are decayed according to the cosine function to obtain temporal attention. Spatial attention and temporal attention are weighted and fused and normalized frame by frame to obtain the three-dimensional attention tensor. The specific steps for performing layered progressive fusion of preprocessed video data to obtain initially synthesized video data, and dynamically calculating the initial contribution of each video source based on binary space mask and three-dimensional attention tensor during the video data fusion process, are as follows: The underlying fusion takes quarter-resolution video data, downsampled 3D attention tensor and base contribution as input, calculates optical flow field and performs motion compensation alignment, separates static background and dynamic foreground through Gaussian mixture model, and outputs background layer and foreground residual components. The base contribution is pre-calculated by multiplying the 3D attention tensor and guiding weight matrix element by element. The intermediate-layer fusion takes half-resolution video data, upsampled bottom-layer fusion results, downsampled 3D attention tensor, and binary space mask as input. The binary space mask is used to distinguish the subject and background processing paths. The background path is stabilized by weighted mixing, while the subject path is solved by Poisson fusion to smooth the boundaries. The intermediate-resolution fusion result is output. Top-level fusion takes the original resolution video data, the upsampled mid-level fusion result, the 3D attention tensor, and the main features as input. It inherits the mid-level structure and retains the original details through guided filtering. The sharpness score of each video source is calculated in the overlapping area of ​​the main subject. The video source with the highest score is selected as the dominant one, and the transparency of the other video sources is reduced and superimposed. The output is the pre-synthesized video data. Contribution calculation involves dynamically calculating the initial contribution of each video source for each frame during the video data fusion process. The initial contribution is based on the three-dimensional attention tensor, the guiding weight matrix, and the binary space mask, and is obtained through calculation.

2. The video synthesis method according to claim 1, characterized in that, The specific steps for training a lightweight regression network based on the 3D attention tensor and the initial contributions of each video source to predict the structural similarity value between the fused video frame and the reference video frame, adjusting the initial contribution based on the structural similarity value, evaluating the quality of video fusion through multiple video fusion metrics, iteratively optimizing the initial contribution based on the evaluation results, and re-fusing the video data to obtain the final contribution of each video source and the final synthesized video data are as follows: A lightweight regression network is constructed. The three-dimensional attention tensor and the initial contribution of each video source are input into the trained lightweight regression network to obtain the predicted structural similarity value between the fused video frame and the reference video frame. When the predicted structural similarity value is lower than the preset threshold, the initial contribution is Gaussian smoothed. The video source with the highest resolution is selected as a reference. Multiple video fusion indicators of the initially synthesized video data are calculated, and a video fusion quality score is obtained. When the video fusion quality score drops and falls below a preset threshold, the gradient of the video fusion quality score to the contribution is calculated, and the contribution of each video source is iteratively updated. Based on the updated contribution, the video data is re-fused until the video fusion quality score meets the preset requirements or reaches the maximum number of iterations, and the final contribution of each video source and the final synthesized video data are obtained.

3. A video compositing system, applied to the video compositing method according to any one of claims 1-2, characterized in that, Including: The video source acquisition and processing module is used to receive multiple video sources, separate each video source into a video stream, an audio stream, and a metadata stream, and preprocess the video stream, audio stream, and metadata stream respectively to obtain preprocessed video data, audio data, and metadata. The cross-modal semantic understanding module is used to extract features and separate the subject from the background in the preprocessed video data to obtain the subject features and binary space mask of the video data. It also divides the preprocessed video data into multiple semantic segments by calculating the feature differences between adjacent frames. Based on the preprocessed audio data, a cross-modal attention mechanism is constructed and the audio guidance weight matrix for each frame of video is calculated. The attention tensor generation module is used to mark semantic keyframes for each semantic segment, obtain the distribution of semantic keyframes in each semantic segment, and generate a three-dimensional attention tensor based on the main features of the video data, the binary space mask, and the distribution of semantic keyframes. The layered progressive fusion module is used to perform layered progressive fusion on the preprocessed video data to obtain the initially synthesized video data. During the fusion process of the video data, the initial contribution of each video source is dynamically calculated based on the binary space mask and the three-dimensional attention tensor. The quality assessment and optimization module is used to train a lightweight regression network based on the three-dimensional attention tensor and the initial contribution of each video source, and predict the structural similarity value between the fused video frame and the reference video frame. The initial contribution is adjusted according to the structural similarity value, and the quality of video fusion is evaluated through multiple video fusion metrics. The initial contribution is iteratively optimized according to the evaluation results, and the video data is fused again accordingly to obtain the final contribution of each video source and the final synthesized video data.

Images