Automatic editing method of multi-angle video based on content defect detection

Abstract

The present invention discloses a multi-angle video automatic editing method based on content defect detection, which relates to the field of video processing technology. The method accurately synchronizes multi-track video materials to the same timeline through cross-correlation calculation, and uses dynamic threshold fuzzy detection, jitter detection and occlusion detection combined with rotation and displacement estimation to analyze and classify defects in video content frame by frame. According to the detection results, a dynamic programming algorithm optimized by state transfer is used to intelligently generate editing plans, select high-quality clips, automatically remove defective parts, and retain the coherence and multi-view diversity of the video content, and finally integrate and generate high-quality recorded videos. The present invention greatly improves the efficiency of video editing, reduces the need for manual intervention, and ensures that the output video quality meets professional application requirements. It is suitable for multiple scenarios such as conference records, education and training, and multi-angle video production, and has high practical value and promotion prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic video editing, and in particular to a multi-angle video automatic editing method based on content defect detection. Background Art

[0002] With the rapid development of multimedia technology, multi-angle video shooting technology has been widely used in scenes such as conference recording, education and training, and news reporting. This shooting method uses multiple cameras to record synchronously from different angles, which can fully capture scene information and provide users with a richer perspective. However, with the complexity of video materials and the increase in the amount of multi-track video data, synchronization, editing, and quality optimization issues in post-processing have become increasingly prominent, and more advanced technical support is urgently needed.

[0003] Although some automated video processing technologies have been applied to multi-angle video editing, the following key problems still exist: there are differences in sampling rate and start time for recording devices of multi-track videos. The existing cross-correlation algorithm based on fixed sampling rate is easily interfered by noise, resulting in insufficient synchronization accuracy, especially in high-precision scenes, the error often exceeds ±10 milliseconds; blur detection relies on fixed thresholds and is difficult to adapt to a variety of shooting equipment and environments; jitter detection cannot effectively distinguish between picture jitter and normal movement, and the error rate is high; occlusion detection has poor adaptability to dynamic scenes and is prone to missed and misjudgment in complex backgrounds; existing technologies usually ignore the advantages of multi-track materials and fail to consider both visual quality and perspective diversity, resulting in a lack of coherence and diversity in the generated editing solutions; existing editing algorithms find it difficult to balance the distribution of video defects and visual coherence, and the quality of the output video is unstable and cannot meet the needs of professional applications. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-angle video automatic editing method based on content defect detection, so as to solve the limitations of the prior art in terms of insufficient audio and video synchronization accuracy, high content defect detection limitations, single editing logic, and insufficient video quality optimization. By introducing dynamic programming algorithms, blur detection, jitter detection, and occlusion detection intelligent analysis methods, the present invention significantly improves editing efficiency and video quality, realizes the automatic integration of multi-view videos, ensures that the output video has higher clarity, coherence, and professionalism, meets the application requirements of multiple scenarios, and reduces manual intervention and post-processing costs.

[0005] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0006] The multi-angle video automatic editing method based on content defect detection includes the following steps:

[0007] S1. Input N video materials shot from multiple angles and reference audio A, where N is an integer greater than 1;

[0008] S2. Extract the corresponding audio from the video material, and synchronize the extracted audio with the reference audio A on the time axis by cross-correlation calculation;

[0009] S3. Perform content defect detection on each of the N video materials, including blur detection, jitter detection, and occlusion detection, and classify and evaluate the detected defective frames;

[0010] S4. According to the defect detection result, a video clip with the best visual quality is selected from the video track through a dynamic programming algorithm to generate an editing plan;

[0011] S5. According to the generated editing scheme, the corresponding video clips in the video material are integrated into a complete recorded video K, and the audio L of the corresponding length is obtained from the reference audio A to synthesize the final video R.

[0012] As a preferred solution of the present invention, the reference audio A defaults to a high-fidelity audio material recorded separately by a recording device and can cover the recording time of all video materials; if there is no separately recorded audio material, an audio track with the highest signal-to-noise ratio and complete coverage time is selected from the video material as the reference audio A.

[0013] As a preferred solution of the present invention, the extracted audio and the reference audio A are synchronized on the time axis by cross-correlation calculation, and the process is as follows:

[0014] Perform cross-correlation calculation on the extracted audio and the reference audio A at a preset sampling rate, and determine the time matching point based on the correlation threshold;

[0015] The offset between the extracted audio and the reference audio A is calculated based on the time matching point to complete audio-based video timeline synchronization.

[0016] As a preferred solution of the present invention, the preset sampling rate is dynamically adjusted according to the extracted audio quality. When the extracted audio signal-to-noise ratio is greater than 20dB, the preset sampling rate is 500Hz; when the extracted audio signal-to-noise ratio is less than 10dB, the preset sampling rate is 1000Hz.

[0017] As a preferred solution of the present invention, the blur detection is implemented in the following manner:

[0018] S31. Use the Sobel operator to calculate the edge gradient amplitude of each pixel in each frame of the video material;

[0019] S32. Calculate the average value of the edge gradient amplitude of all pixels in each frame of the image to obtain the average value of the edge gradient of the frame;

[0020] S33 records the average edge gradient of each frame in the video sequence to form an edge gradient data set of the video sequence;

[0021] S34. Calculate the overall edge gradient mean and standard deviation based on the edge gradient data set, and set the blur detection threshold to the overall edge gradient mean minus 50% of the standard deviation;

[0022] S35. Compare the edge gradient average value of each frame in the video sequence with the blur detection threshold. If it is lower than the blur detection threshold, the frame is determined to be a blurry frame.

[0023] As a preferred solution of the present invention, the jitter detection is implemented in the following manner:

[0024] a. In two consecutive frames, the SIFT feature point detection algorithm is used to extract the feature points of each frame;

[0025] b. Use the FLANN feature point matching algorithm to match the feature points of two consecutive frames to generate matching point pairs;

[0026] c. Calculate the displacement vector of each matching point pair and obtain the displacement components in the x and y directions;

[0027] d. Based on the displacement component, the displacement amplitude of all matching point pairs is counted, and the overall displacement from this frame to the next frame and the rotation angle between frames are calculated;

[0028] e. If the difference between the rotation angles of two consecutive frames is less than a preset angle threshold and the overall displacement is greater than a preset displacement threshold, it is determined that jitter occurs between the two frames.

[0029] As a preferred solution of the present invention, the occlusion detection is specifically to determine whether there is occlusion in the input video frame based on the improved binary classification deep neural network model, which is achieved in the following way:

[0030] The images of 8 consecutive video frames in the sliding window are input into the binary classification deep neural network model, and the tensor with the size of 8×171×128×3 is adjusted as the input data;

[0031] The structure of the convolution block of the binary classification deep neural network model is as follows: in each convolution block, the network is divided into two branches, the first branch uses 1D convolution to extract features of the time dimension, and the second branch uses 2D convolution to extract features of the space dimension;

[0032] The features of the two branches are added before the maximum pooling layer to fuse the temporal information and the spatial information. The binary classification deep neural network model is stacked with multiple convolution blocks, each of which contains convolution kernels of different sizes, including convolution kernels of 3×3×3, 1×1×1, 1×3×3, and 3×1×1.

[0033] Each convolution block is followed by a maximum pooling operation to reduce the feature size, and the binary classification result of occlusion detection is output through a fully connected layer to determine whether there is occlusion in each frame of the input video.

[0034] As a preferred solution of the present invention, the dynamic programming algorithm selects the video clips in the following manner:

[0035] S41. Calculate the start time and end time for each video track and extract the corresponding audio content;

[0036] S42. Mark the defective frame detected in the video and the frames within 3 seconds before and after the defective frame as defective areas;

[0037] S43. According to the position and defective area of ​​the defective frame, the video is divided into multiple continuous video segments, with the defective area as the boundary and the non-defective frame as the segment content. If the length of any segment is less than 2 seconds or there is a scene switch within the segment, it is merged with the adjacent segment;

[0038] S44. Calculate a defect value for each video clip, the lower the defect value, the better the quality of the video clip;

[0039] S45. Use dynamic programming method to optimize shot selection, calculate the optimal editing plan through global optimization in all video clips, define state dp[i] to represent the minimum sum of defect values ​​when selecting the i-th clip, and dp[j] to represent the minimum sum of defect values ​​when selecting the j-th clip; the state transfer equation is: dp[i]=min(dp[j]+cost[j][i]), where cost[j][i] is the defect value from clip j to clip i. If clip i and clip j are adjacent in time and have no defects, the transfer cost is low;

[0040] S46. In the case of multiple video tracks, maximize the selection of video clips from different tracks and perspectives to ensure the diversity of perspectives in the video.

[0041] As a preferred solution of the present invention, an intelligent selection mechanism based on content similarity is added to the dynamic programming algorithm, which specifically includes:

[0042] Extracting content features from each video clip by an image feature extraction algorithm and generating a corresponding content feature vector, wherein the content feature vector is used to characterize visual content features of the video clip;

[0043] The content feature vectors of adjacent or temporally continuous video clips are calculated using cosine similarity to obtain the content similarity between the video clips;

[0044] The transfer cost in dynamic programming is adjusted according to the similarity calculation result. If the content similarity between two adjacent video clips is higher than a preset threshold, the transfer cost between these clips is reduced.

[0045] As a preferred solution of the present invention, synthesizing the final video R also includes the following steps:

[0046] During the splicing of video clips, each splicing point is smoothed to reduce the visual jump;

[0047] Performing a quality check on the integrated recorded video K to ensure that the frame rate, resolution and bit rate of the recorded video K meet preset standards;

[0048] Double buffering technology is used when synthesizing and recording video K and audio L to ensure that the audio and video synchronization accuracy is within ±10 milliseconds;

[0049] Conduct a quality review of the final synthesized video R, including checking of picture stability, sound clarity, and audio and video synchronization.

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention greatly improves the efficiency of video editing through dynamic programming algorithms and automated processing technologies, significantly shortens the post-processing time, and reduces manual intervention and labor costs. By using technologies such as blur detection, jitter detection, and occlusion detection, low-quality frames are automatically identified and eliminated, ensuring the continuity and clarity of the video content. By intelligently selecting the optimal video clips, the output video achieves the best quality in visual effects and meets the needs of professional-level applications. At the same time, through precise audio and video synchronization and multi-view content fusion, the narrative ability and diversity of the video are enhanced, which is suitable for scenes such as conference records, education and training, and multi-view film and television production. In addition, by using cross-correlation calculation and double buffering technology, the audio and video synchronization accuracy is within ±10 milliseconds, which significantly improves the audio and video matching degree and optimizes the viewing experience. By combining dynamic programming and content similarity analysis, the connection of video clips is ensured to be natural and smooth, further improving the visual experience. The technology adopts custom dynamic thresholds and deep learning models, which reduces the difficulty of implementation while improving applicability, has a wide range of promotion value, and meets the needs of modern intelligent video production. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 is a schematic flow chart of an automatic editing method in an embodiment of the present invention;

[0054] Figure 3 This is a structural diagram of a binary classification deep neural network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0056] In the existing technology, conference video editing usually relies on manual screening and editing, which faces the problems of lengthy and repetitive video content, as well as quality issues such as blur, jitter, and occlusion. This leads to unstable video quality and requires a lot of post-processing, which increases the workload and time cost of editors.

[0057] The present invention handles these problems automatically. First, an audio synchronization algorithm is used to align different video tracks with reference audio. Then, a defect detection algorithm is used to identify blur, jitter, and occlusion problems in the video. Finally, based on the defect detection results, a dynamic programming algorithm is used to automatically select the best quality clips to generate a complete conference record video. This method can significantly improve the efficiency of video editing, reduce manual intervention, and ensure that the final video quality meets the core requirements of conference records.

[0058] like Figure 1 , Figure 2 FIG. 1 is an embodiment of the present invention, which provides a multi-angle video automatic editing method based on content defect detection, comprising the following steps:

[0059] S1. Input N video materials shot from multiple angles and reference audio A, where N is an integer greater than 1;

[0060] In large-scale conferences or activity recordings, multiple cameras may be used to synchronously shoot from multiple angles to capture more comprehensive and detailed scene information. In this case, the method of the present invention is still applicable, and all video materials obtained by shooting are only required to be used as input.

[0061] Preferably, reference audio A defaults to high-fidelity audio material recorded separately by a recording device and capable of covering the recording time of all video materials; if there is no separately recorded audio material, the audio track with the highest signal-to-noise ratio and complete coverage time is selected from the video material as reference audio A.

[0062] S2. Extract the corresponding audio from the video material, and synchronize the extracted audio with the reference audio A on the time axis by cross-correlation calculation;

[0063] In step S2, the extracted audio and the reference audio A are synchronized on the time axis by cross-correlation calculation, and the process is as follows:

[0064] Perform cross-correlation calculation on the extracted audio and the reference audio A at a preset sampling rate, and determine the time matching point based on the correlation threshold;

[0065] The preset sampling rate is dynamically adjusted according to the extracted audio quality. When the extracted audio signal-to-noise ratio is greater than 20dB, the preset sampling rate is 500Hz; when the extracted audio signal-to-noise ratio is less than 10dB, the preset sampling rate is 1000Hz.

[0066] In a specific embodiment, the audio VA1, VA2, VA3, etc. corresponding to the video materials V1, V2, V3, etc. are extracted, and then the time offset between the reference audio A and the audio VA1, VA2, VA3, etc. corresponding to all the video materials is calculated. The audio corresponding to each video is cross-correlated with the reference audio A, and the calculation formula is:

[0067] ;

[0068] in, and Respectively represent two audio signals (here, reference audio A and the audio corresponding to a certain video material); Indicates audio In time The complex conjugate at ; Indicates audio In time The value at Indicates time Perform the summation.

[0069] The value of the cross-correlation function can be regarded as the similarity of the two audios at a specific moment. The similarity calculation will adopt a sliding method, covering the entire time period from the beginning to the end of the reference audio A, and the sampling rate is set to 1000Hz, that is, the cross-correlation value is calculated every 1 millisecond.

[0070] The moment with the highest similarity is selected as the matching point of each video audio, and the offset between the audio VA1, VA2, VA3, etc. of the video material and the reference audio A is calculated based on the matching point. It should be noted that the threshold for audio and video synchronization detection recommended by the International Telecommunication Union (ITU) is -125 milliseconds to +45 milliseconds, so an accuracy of 1 millisecond can meet this requirement.

[0071] S3. Perform content defect detection on each of the N video materials, including blur detection, jitter detection, and occlusion detection, and classify and evaluate the detected defective frames;

[0072] In step S3, the blur detection is implemented in the following manner:

[0073] S31. Use the Sobel operator to calculate the edge gradient amplitude of each pixel in each frame of the video material;

[0074] S32. Calculate the average value of the edge gradient amplitude of all pixels in each frame of the image to obtain the average value of the edge gradient of the frame;

[0075] S33 records the average edge gradient of each frame in the video sequence to form an edge gradient data set of the video sequence;

[0076] S34. Calculate the overall edge gradient mean and standard deviation based on the edge gradient data set, and set the blur detection threshold to the overall edge gradient mean minus 50% of the standard deviation;

[0077] S35. Compare the edge gradient average value of each frame in the video sequence with the blur detection threshold. If it is lower than the blur detection threshold, the frame is determined to be a blurry frame.

[0078] In step S3, the jitter detection is implemented in the following manner:

[0079] a. In two consecutive frames, the SIFT feature point detection algorithm is used to extract the feature points of each frame;

[0080] b. Use the FLANN feature point matching algorithm to match the feature points of two consecutive frames to generate matching point pairs;

[0081] c. Calculate the displacement vector of each matching point pair and obtain the displacement components in the x and y directions;

[0082] d. Based on the displacement component, the displacement amplitude of all matching point pairs is counted, and the overall displacement from this frame to the next frame and the rotation angle between frames are calculated;

[0083] e. If the difference between the rotation angles of two consecutive frames is less than a preset angle threshold and the overall displacement is greater than a preset displacement threshold, it is determined that jitter occurs between the two frames.

[0084] In one embodiment, content defect detection is performed on video materials V1, V2, V3, etc. The process involves detecting the video frame by frame through a sliding window, mainly covering blur detection, jitter detection, and occlusion detection. During the detection process, we will classify and evaluate the defective frames found to determine their specific impact on the overall video quality.

[0085] Specifically, the out-of-focus phenomenon in the video usually manifests as a blurred image, and the notable feature of the blurred frame is the reduction of clear edge features in the image. Based on this feature, edge detection algorithms are widely used to identify blurred frames. In the prior art, most blur detection methods use a fixed threshold for judgment, but this method lacks adaptability in videos recorded by different shooting devices. For example, a video recorded by a low-quality shooting device may cause a clear frame to be misjudged as a blurred frame due to improper threshold setting. Therefore, in response to this problem, the present invention proposes a blur detection method based on edge gradient characteristics and dynamic threshold adjustment to improve the adaptability and accuracy of videos of different qualities.

[0086] In the implementation of the present invention, the edge gradient of each frame of the image is calculated by the Sobel operator. The Sobel operator is a commonly used edge detection operator that extracts edge information by calculating the gradient of pixels in the image. Specifically, for each frame of the image, the Sobel operator calculates the horizontal and vertical gradients of each pixel to obtain the edge gradient amplitude of the frame of the image. Then, by counting the average value of the edge gradient of each frame, the edge gradient distribution of the video frame sequence is generated.

[0087] The overall edge gradient mean and standard deviation of all video frames are calculated to set the dynamic threshold for blur detection. The standard deviation of the overall edge gradient reflects the consistency of clarity in the video. When the standard deviation is large, it means that the clarity of the frames in the video varies greatly, and there may be a large range of blur. In order to adapt to video content with different clarity, the present invention dynamically adjusts the threshold for blur detection. Specifically, the setting of the dynamic threshold is based on the overall edge gradient distribution of the video frame, and it is recommended to set it to the overall mean minus 50% of the standard deviation. The threshold adjustment method can be adapted according to the specific content and shooting conditions of the video, thereby ensuring that videos with different clarity and different scenes can perform accurate blur detection.

[0088] The average value of the edge gradient of each frame is compared with the dynamic threshold. If the average value of the edge gradient of a frame is lower than the dynamically set threshold, the frame is determined to be a blurred frame. This method can effectively identify the blurred area in the video and perform subsequent editing optimization. The advantage of this algorithm is that it avoids the limitation of relying on a fixed threshold, can adaptively adjust the detection standard according to the overall clarity of the video and the characteristics of the scene, and significantly improves the accuracy and reliability of blurred frame recognition.

[0089] In video shooting, camera shake is a common problem that causes unstable images, which significantly affects the viewing quality of the video. However, existing methods often lead to misjudgment due to the difficulty in effectively distinguishing normal camera motion from shake, especially in complex scenes, where all camera movements are easily misclassified as shake. To address this problem, the present invention proposes an improved shake detection algorithm based on feature point detection and motion estimation, the core of which is to accurately analyze the motion features between consecutive frames to accurately identify shake and eliminate the interference of normal motion.

[0090] Specifically, this method first uses the SIFT algorithm to extract key feature points in two consecutive frames. The SIFT algorithm is characterized by its robustness to scale, rotation and illumination changes, and can extract stable feature points from the image, thus laying the foundation for subsequent motion estimation. Subsequently, the feature points are quickly matched using the FLANN matching algorithm to generate corresponding matching point pairs from two consecutive frames. These matching point pairs contain the motion information of the corresponding pixels in the image and can fully reflect the changes between frames.

[0091] After obtaining the matching point pairs, this method calculates the displacement vector of each pair of matching feature points, including its displacement in the x-direction and y-direction, and further counts the displacement amplitudes of all matching points to calculate the overall motion from the current frame to the next frame. The overall motion reflects the global motion trend between video frames and is the core indicator for evaluating video stability. In order to deeply analyze the motion characteristics, this method also further processes the displacement between consecutive frames, calculates the translation vector (i.e., the total displacement in the x-direction and y-direction) and the rotation angle between frames, and thus obtains the motion direction and amplitude information between frames.

[0092] In order to accurately distinguish between jitter and normal camera motion, this method sets clear judgment rules. If the angular difference between the motion directions of two consecutive frames is lower than the set threshold (such as 30°, that is, the motion between the two frames is an acute angle change), and the displacement amplitude exceeds the set threshold (such as 10 pixels), it can be determined that jitter has occurred between the two frames. This judgment rule comprehensively considers the motion direction and displacement amplitude, avoiding the situation where misjudgment is caused by relying on only a single indicator. At the same time, the threshold setting of this method can be flexibly adjusted according to the specific application scenario to ensure the adaptability of the algorithm to different shooting conditions.

[0093] Through the above steps, the present invention realizes efficient and accurate shake detection. Compared with traditional methods, this algorithm shows higher robustness in complex shooting scenes, not only can it effectively identify camera shake, but also can avoid misjudging normal camera movement as shake, thereby significantly improving the quality and stability of the video.

[0094] Occlusion detection is a relatively complex task because there is no unified standard for the definition of occlusion in different research fields. In the context of conference recording videos, we consider objects that appear at inopportune times and block the subject (such as speakers, hosts, etc.) as occlusions. Since this task requires a deep semantic understanding of the video content, it is very appropriate to use algorithms based on deep learning.

[0095] The occlusion detection is specifically to determine whether there is occlusion in the input video frame based on the improved binary classification deep neural network model, which is achieved in the following way:

[0096] The images of 8 consecutive video frames in the sliding window are input into the binary classification deep neural network model, and the tensor with the size of 8×171×128×3 is adjusted as the input data;

[0097] The structure of the convolution block of the binary classification deep neural network model is as follows: in each convolution block, the network is divided into two branches, the first branch uses 1D convolution to extract features of the time dimension, and the second branch uses 2D convolution to extract features of the space dimension;

[0098] The features of the two branches are added before the maximum pooling layer to fuse the temporal information and the spatial information. The binary classification deep neural network model is stacked with multiple convolution blocks, each of which contains convolution kernels of different sizes, including convolution kernels of 3×3×3, 1×1×1, 1×3×3, and 3×1×1.

[0099] Each convolution block is followed by a maximum pooling operation to reduce the feature size, and the binary classification result of occlusion detection is output through a fully connected layer to determine whether there is occlusion in each frame of the input video.

[0100] The structure of the binary classification deep neural network model is as follows Figure 3 As shown in the figure. In the input layer, we use a sliding window to resize 8 consecutive frames of video to 8×171×128×3 and feed it into the model as an input tensor. A 3D convolutional neural network is used in the model to more effectively extract spatiotemporal features in the video. Before the maximum pooling layer, we use a spatial attention module to achieve the fusion of spatiotemporal information. With this design, the model can more accurately identify occlusions in the video, providing strong support for subsequent video processing and analysis.

[0101] After the video defect detection is completed, in order to select the best quality clips from multiple video materials V1, V2, V3, etc. to generate the final video clip, we designed an automatic video clip selection algorithm. The algorithm aims to ensure that the final video clip meets the following quality requirements:

[0102] Keep all key aspects of the meeting as intact as possible.

[0103] Try to avoid including defective clips unless the defect is unavoidable and occurs simultaneously in all video tracks.

[0104] Video switching should be natural and smooth, reducing frequent and meaningless track switching to optimize the audience's viewing experience.

[0105] To achieve these goals, our automatic video segment selection algorithm consists of the following key steps:

[0106] Time range and frame number calculation: After synchronizing the video tracks and completing defect detection, first determine the start and end time of each video track, and extract the corresponding audio content based on this. At the same time, calculate the total number of video frames to provide basic data for subsequent processing.

[0107] Defect marking: During the defect detection process, once a bad frame is identified, the algorithm will mark it as a defective area within a set time range before and after it to ensure that relevant defects are not missed.

[0108] Synchronization and data preparation: The synchronization results of all video tracks are converted into a set of frame data with defect marks as input for the selection process.

[0109] In the process of video editing, how to automatically select while ensuring the video quality is a key technology. This paper proposes a video clip selection algorithm based on dynamic programming. This method realizes the efficient selection and combination of video clips through a global optimization strategy, ensuring the quality of the final video and the viewing experience.

[0110] S4. According to the defect detection result, a video clip with the best visual quality is selected from the video track through a dynamic programming algorithm to generate an editing plan;

[0111] In this embodiment, in step S4, the dynamic programming algorithm selects the video clip in the following manner:

[0112] S41. Calculate the start time and end time for each video track and extract the corresponding audio content;

[0113] S42. Mark the defective frame detected in the video and the frames within 3 seconds before and after the defective frame as defective areas;

[0114] S43. According to the position and defective area of ​​the defective frame, the video is divided into multiple continuous video segments, with the defective area as the boundary and the non-defective frame as the segment content. If the length of any segment is less than 2 seconds or there is a scene switch within the segment, it is merged with the adjacent segment;

[0115] S44. Calculate a defect value for each video clip, the lower the defect value, the better the quality of the video clip;

[0116] S45. Use dynamic programming method to optimize shot selection, calculate the optimal editing plan through global optimization in all video clips, define state dp[i] to represent the minimum sum of defect values ​​when selecting the i-th clip, and dp[j] to represent the minimum sum of defect values ​​when selecting the j-th clip; the state transfer equation is: dp[i]=min(dp[j]+cost[j][i]), where cost[j][i] is the defect value from clip j to clip i. If clip i and clip j are adjacent in time and have no defects, the transfer cost is low;

[0117] S46. In the case of multiple video tracks, maximize the selection of video clips from different tracks and perspectives to ensure the diversity of perspectives in the video.

[0118] In one embodiment, a dynamic programming algorithm is added with an intelligent selection mechanism based on content similarity, specifically including:

[0119] Extracting content features from each video clip by an image feature extraction algorithm and generating a corresponding content feature vector, wherein the content feature vector is used to characterize visual content features of the video clip;

[0120] The content feature vectors of adjacent or temporally continuous video clips are calculated using cosine similarity to obtain the content similarity between the video clips;

[0121] The transfer cost in dynamic programming is adjusted according to the similarity calculation result. If the content similarity between two adjacent video clips is higher than a preset threshold, the transfer cost between these clips is reduced.

[0122] Specifically, this method first extracts the time range and audio content of each video track. By calculating the start time and end time of the video track, the corresponding audio content is extracted for subsequent audio and video synchronization analysis and quality optimization. At the same time, after the video defect detection is completed, the defective frame detected in the video and the frames within a certain time range before and after are marked as defective areas to accurately define the low-quality segments in the video.

[0123] Next, the video is divided into multiple continuous segments according to the position of the defective frame and the situation of the surrounding frames to ensure that there are fewer defects in each segment and the content of the segment is coherent. For each segment, the present invention evaluates the segment quality by calculating its "defect value". The defect value represents the overall quality of the segment. The higher the defect value, the more defects the segment contains and the worse the quality; conversely, the lower the defect value, the higher the segment quality.

[0124] After the segment division is completed, the present invention adopts a dynamic programming algorithm to perform global optimization selection of video segments. By recursively calculating the defect values ​​of all segments, the dynamic programming algorithm can ultimately select the lens combination with the smallest total defect value to achieve global optimization.

[0125] S5. According to the generated editing scheme, the corresponding video clips in the video material are integrated into a complete recorded video K, and the audio L of the corresponding length is obtained from the reference audio A to synthesize the final video R.

[0126] In one embodiment, the synthesis of video and audio further comprises the following steps:

[0127] During the splicing of video clips, each splicing point is smoothed to reduce the visual jump;

[0128] Performing a quality check on the integrated recorded video K to ensure that the frame rate, resolution and bit rate of the recorded video K meet preset standards;

[0129] Double buffering technology is used when synthesizing and recording video K and audio L to ensure that the audio and video synchronization accuracy is within ±10 milliseconds;

[0130] Conduct a quality review of the final synthesized video R, including checking of picture stability, sound clarity, and audio and video synchronization.

[0131] In addition, when multiple video tracks exist at the same time, the method further introduces a maximization selection strategy to give priority to video clips with different tracks and perspectives. By maximizing the selection of different tracks, the present invention ensures that the visual elements in the video are sufficiently diverse, avoids visual fatigue caused by repeated perspectives, and increases the richness of video content and viewing experience.

[0132] According to the previously generated video editing plan, we will select the corresponding high-quality video clips from the video materials V1, V2, and V3, and integrate them according to the time sequence and editing logic. This process is designed to ensure the coherence, integrity, and high quality of the conference record video K.

[0133] The specific steps are as follows:

[0134] Clip extraction: According to the time points marked in the editing plan, the required video clips are extracted from V1, V2, and V3 respectively.

[0135] Arrange in sequence: Arrange the clips in chronological order in the video editing software.

[0136] Add transition effects: Add appropriate transition effects between clips to ensure natural and smooth video switching.

[0137] Integration and export: After completing all editing operations, export the integrated video as a conference record video K.

[0138] Audio capture and synchronization: In order to match the video K, we need to capture the audio L of the corresponding length from the reference audio A. The key to this step is to ensure the time synchronization of audio and video.

[0139] The steps are as follows:

[0140] Time point positioning: refer to the starting time point and ending time point of video K in the reference editing plan, and locate the reference audio A in the audio editing software.

[0141] Audio capture: Based on the located time point, accurately capture the audio segment L with the same duration as the video K from the reference audio A.

[0142] Audio and video synchronization check: Perform a preliminary synthesis of the captured audio L and video K to check whether the audio and video are perfectly synchronized. If there is a deviation, make fine adjustments until they are completely synchronized.

[0143] Final synthesis and output:

[0144] After ensuring that the audio and video are synchronized correctly, we will perform the final synthesis operation to output the meeting record video R with audio.

[0145] The specific process is as follows:

[0146] Audio and video synthesis: In the video editing software, synthesize the video K and the audio L. Ensure that the quality of the audio and video is not lost during the synthesis process.

[0147] Quality check: Perform a comprehensive quality check on the synthesized conference record video R, including picture clarity, audio clarity, audio and video synchronization, etc.

[0148] Output settings: Set the output format, resolution, bit rate and other parameters as required to ensure that the output video R meets the expected playback and storage requirements.

[0149] Final output: After completing all settings, export and save the meeting record video R with audio. At this point, the entire video clipping and audio synchronization solution is completed.

[0150] In summary, the present invention uses the cross-correlation calculation of audio information to achieve accurate synchronization between video and reference audio. An efficient detection algorithm is designed for blur, jitter and occlusion defects in the video, which significantly improves the recognition rate. Through a dynamic programming algorithm and an intelligent selection mechanism based on content similarity, high-quality clips are automatically selected from multiple video tracks to ensure the coherence and consistency of the video content. In addition, this method does not rely on specific hardware equipment and has a wide range of applicability. It can be applied to multiple fields such as conference recording, education and training, effectively improving the efficiency and quality of video editing, and showing significant creativity and technical advantages.

[0151] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0152] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-angle video automatic editing method based on content defect detection, characterized in that: The following steps are involved: S1. Input N video materials shot from multiple angles and reference audio A, where N is an integer greater than 1; S2. Extract the corresponding audio from the video material, and synchronize the extracted audio with the reference audio A on the time axis by cross-correlation calculation; S3. Perform content defect detection on each of the N video materials, including blur detection, jitter detection, and occlusion detection, and classify and evaluate the detected defective frames; S4. According to the defect detection result, a video clip with the best visual quality is selected from the video track through a dynamic programming algorithm to generate an editing plan; S5. According to the generated editing scheme, the corresponding video clips in the video material are integrated into a complete recorded video K, and the audio L of the corresponding length is obtained from the reference audio A to synthesize the final video R; In step S4, the dynamic programming algorithm selects the video clips in the following manner: S41. Calculate the start time and end time for each video track and extract the corresponding audio content; S42. Mark the defective frame detected in the video and the frames within 3 seconds before and after the defective frame as defective areas; S43. According to the position and defective area of ​​the defective frame, the video is divided into multiple continuous video segments, with the defective area as the boundary and the non-defective frame as the segment content. If the length of any segment is less than 2 seconds or there is a scene switch within the segment, it is merged with the adjacent segment; S44. Calculate a defect value for each video clip, the lower the defect value, the better the quality of the video clip; S45. Use dynamic programming method to optimize shot selection, calculate the optimal editing plan through global optimization in all video clips, define state dp[i] to represent the minimum sum of defect values ​​when selecting the i-th clip, and dp[j] to represent the minimum sum of defect values ​​when selecting the j-th clip; the state transfer equation is: dp[i]=min(dp[j]+cost[j][i]), where cost[j][i] is the defect value from clip j to clip i. If clip i and clip j are adjacent in time and have no defects, the transfer cost is low; S46. In the case of multiple video tracks, maximize the selection of video clips from different tracks and perspectives to ensure the diversity of perspectives in the video.

2. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S1, the reference audio A is by default a high-fidelity audio material recorded separately by a recording device, and can cover the recording time of all video materials; If there is no separately recorded audio material, select the audio track with the highest signal-to-noise ratio and complete time coverage from the video material as reference audio A.

3. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S2, the extracted audio and the reference audio A are synchronized on the time axis by cross-correlation calculation, and the process is as follows: Perform cross-correlation calculation on the extracted audio and the reference audio A at a preset sampling rate, and determine the time matching point based on the correlation threshold; The offset between the extracted audio and the reference audio A is calculated based on the time matching point to complete audio-based video timeline synchronization.

4. The multi-angle video automatic editing method based on content defect detection according to claim 3 is characterized in that: The preset sampling rate is dynamically adjusted according to the extracted audio quality. When the extracted audio signal-to-noise ratio is greater than 20dB, the preset sampling rate is 500Hz; when the extracted audio signal-to-noise ratio is less than 10dB, the preset sampling rate is 1000Hz.

5. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S3, the blur detection is implemented in the following manner: S31. Use the Sobel operator to calculate the edge gradient amplitude of each pixel in each frame of the video material; S32. Calculate the average value of the edge gradient amplitude of all pixels in each frame of the image to obtain the average value of the edge gradient of the frame; S33 records the average edge gradient of each frame in the video sequence to form an edge gradient data set of the video sequence; S34. Calculate the overall edge gradient mean and standard deviation based on the edge gradient data set, and set the blur detection threshold to the overall edge gradient mean minus 50% of the standard deviation; S35. Compare the edge gradient average value of each frame in the video sequence with the blur detection threshold. If it is lower than the blur detection threshold, the frame is determined to be a blurry frame.

6. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S3, the jitter detection is implemented in the following manner: a. In two consecutive frames, the SIFT feature point detection algorithm is used to extract the feature points of each frame; b. Use the FLANN feature point matching algorithm to match the feature points of two consecutive frames to generate matching point pairs; c. Calculate the displacement vector of each matching point pair and obtain the displacement components in the x and y directions; d. Based on the displacement component, the displacement amplitude of all matching point pairs is counted, and the overall displacement from this frame to the next frame and the rotation angle between frames are calculated; e. If the difference between the rotation angles of two consecutive frames is less than a preset angle threshold and the overall displacement is greater than a preset displacement threshold, it is determined that jitter occurs between the two frames.

7. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S3, the occlusion detection is specifically to determine whether there is occlusion in the input video frame based on the improved binary classification deep neural network model, which is achieved by the following method: The images of 8 consecutive video frames in the sliding window are input into the binary classification deep neural network model, and the tensor with the size of 8×171×128×3 is adjusted as the input data; The structure of the convolution block of the binary classification deep neural network model is as follows: in each convolution block, the network is divided into two branches, the first branch uses 1D convolution to extract features of the time dimension, and the second branch uses 2D convolution to extract features of the space dimension; The features of the two branches are added before the maximum pooling layer to fuse the temporal information and the spatial information. The binary classification deep neural network model is stacked with multiple convolution blocks, each of which contains convolution kernels of different sizes, including convolution kernels of 3×3×3, 1×1×1, 1×3×3, and 3×1×1. Each convolution block is followed by a maximum pooling operation to reduce the feature size, and the binary classification result of occlusion detection is output through a fully connected layer to determine whether there is occlusion in each frame of the input video.

8. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: The dynamic programming algorithm incorporates an intelligent selection mechanism based on content similarity, specifically including: Extracting content features from each video clip by an image feature extraction algorithm and generating a corresponding content feature vector, wherein the content feature vector is used to characterize visual content features of the video clip; The content feature vectors of adjacent or temporally continuous video clips are calculated using cosine similarity to obtain the content similarity between the video clips; The transfer cost in dynamic programming is adjusted according to the similarity calculation result. If the content similarity between two adjacent video clips is higher than a preset threshold, the transfer cost between these clips is reduced.

9. The multi-angle video automatic editing method based on content defect detection according to claim 1 is characterized in that: In step S5, synthesizing the final video R further includes the following steps: During the splicing of video clips, each splicing point is smoothed to reduce the visual jump; Performing a quality check on the integrated recorded video K to ensure that the frame rate, resolution and bit rate of the recorded video K meet preset standards; Double buffering technology is used when synthesizing and recording video K and audio L to ensure that the audio and video synchronization accuracy is within ±10 milliseconds; Conduct a quality review of the final synthesized video R, including checking of picture stability, sound clarity, and audio and video synchronization.

Images