Video processing method and device, equipment and storage medium

Abstract

The invention provides a video processing method and device, equipment and a storage medium, and the method comprises the steps: segmenting a video material into a plurality of semantically coherent video clips, automatically extracting related video clips from the video material according to the text content provided by a user, and generating corresponding subtitles and voice broadcast ports matched with the related video clips, and finally synthesizing a section of finished video with clear logic and coherent content. According to the scheme, full-process automation from video material generation to final video file generation is realized, a large number of video materials can be quickly processed, the tedious manual editing process in traditional video production is reduced, the dependence on manual intervention is greatly reduced, the operation cost is reduced, the video production process is simplified, the video production efficiency is improved, and the user experience is improved. And the quality and the efficiency of finished products are also improved.

Description

Background Art

[0002] With the rapid development of the Internet, video has become one of the important ways to spread information. However, traditional video production relies on manual editing and processing, which is not only time-consuming and laborious, but also requires high professional skills of editors. In order to improve the video production efficiency and reduce the labor cost, automated video editing technology has gradually become a research hotspot in the industry.

[0003] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present invention, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] Aiming at the problems in the prior art, the purpose of the present invention is to provide a video processing method, device, equipment and storage medium, which overcomes the difficulties of the prior art and provides an automated video editing and splicing solution.

[0005] The first aspect of the present disclosure provides a video processing method, which includes:

[0006] Dividing the input video material into multiple semantically coherent video segments;

[0007] Generating corresponding subtitles and voiceovers according to the input text, and recording the start and end times of each voiceover;

[0008] Among the multiple video segments, selecting the video segments related to the input text based on semantic matching;

[0009] Adjusting the duration of the video segments related to the input text according to the start and end times of the voiceover to synchronize them with the corresponding subtitles and voiceovers;

[0010] Splicing the adjusted video segments in the order of text semantics, and synchronously synthesizing them with the matching corresponding subtitles and voiceovers to generate a finished video.

[0011] In some embodiments, the dividing the input video material into multiple semantically coherent video segments includes:

[0012] Extracting consecutive frames from the video material;

[0013] Calculating the color histogram for each frame image;

[0014] Using a segmentation algorithm based on shot boundary detection, determining the shot transition points by calculating the color histogram differences between the consecutive frames, and dividing the consecutive frames into multiple semantically coherent video segments based on the shot transition points.

[0015] In some embodiments, before selecting video segments related to the input text based on semantic matching, the video processing method further includes preprocessing each of the video segments, and the preprocessing includes:

[0016] Performing face detection on the video segment using a multi-task cascaded convolutional neural network, and removing the video segments containing target face information;

[0017] Wherein, the multi-task cascaded convolutional neural network includes a P-Net, an R-Net, and an O-Net, and generates a target face box through layer-by-layer screening by the P-Net, the R-Net, and the O-Net to determine the video segments to be removed.

[0018] In some embodiments, removing the video segments containing target face information includes:

[0019] Performing motion compensation on the detected target face box in the video segment using an optical flow-based motion estimation algorithm;

[0020] Removing the video segments containing the target face according to a preset face area and motion trajectory, in combination with the number of consecutive detections between frames.

[0021] In some embodiments, the preprocessing further includes using an image inpainting technique to remove the original captions and / or watermarks in the video segment.

[0022] In some embodiments, generating corresponding captions and voiceovers according to the input text includes:

[0023] Splitting the input text into sentences;

[0024] Generating corresponding captions for each split sentence, synchronously generating the voiceover through text-to-speech technology, and recording the start and end times of each voiceover.

[0025] In some embodiments, selecting video segments related to the input text based on semantic matching includes:

[0026] Extracting key frames for each video segment, and extracting text entities and their text feature vectors for each split sentence;

[0027] Extracting the image feature vectors of the key frames using a pre-trained cross-modal deep learning model;

[0028] Using a similarity calculation method to match the image feature vectors with the text feature vectors, and selecting the video segment with the highest similarity as the video segment related to the input text based on the matching score.

[0029] In some embodiments, adjusting the duration of the video segment related to the input text according to the start and end times of the voiceover includes:

[0030] Determine the actual duration and the target duration of the video segment with the highest similarity, where the target duration is obtained based on the start and end time intervals of the voiceover;

[0031] If the actual duration is greater than the target duration, trim the video segment with the highest similarity so that the actual duration is the same as the target duration;

[0032] If the actual duration is less than the target duration, for the video segment with the highest similarity, supplement the video segments with the second highest similarity and subsequent ones in descending order of the matching scores until the adjusted actual duration is less than the target duration.

[0033] In some embodiments, splicing the adjusted video segments in the order of text semantics and synchronously synthesizing them with the corresponding subtitles and voiceover that match includes:

[0034] Using a seamless splicing algorithm based on bilinear interpolation and deconvolution upsampling, splice the adjusted video segments in the order of text semantics, and synchronously superimpose the corresponding subtitles, voiceover, and background music that match.

[0035] A second aspect of the present disclosure provides a video processing apparatus, which includes:

[0036] A video segmentation module that divides the input video material into multiple semantically coherent video segments;

[0037] A text processing module that generates corresponding subtitles and voiceover according to the input text, and records the start and end times of each voiceover;

[0038] A semantic matching module that selects, based on semantic matching, video segments related to the input text from among the multiple video segments;

[0039] A duration synchronization module that adjusts the duration of the video segment related to the input text according to the start and end times of the voiceover to make it synchronous with the corresponding subtitles and voiceover;

[0040] A video splicing module that splices the adjusted video segments in the order of text semantics and synchronously synthesizes them with the corresponding subtitles and voiceover that match to generate a finished video.

[0041] A third aspect of the present disclosure provides an electronic device, which includes:

[0042] A processor;

[0043] A memory storing executable instructions of the processor;

[0044] Wherein, the processor is configured to execute the steps of the video processing method according to any of the above embodiments by executing the executable instructions.

[0045] The fourth aspect of the present disclosure provides a computer-readable storage medium for storing a program, which when executed implements the steps of the video processing method according to any of the embodiments.

[0046] The video processing method, apparatus, device and storage medium provided by the embodiments of the present disclosure have the following advantages:

[0047] According to the text content provided by the user, relevant video clips are automatically extracted from the video materials, and corresponding subtitles and voiceovers are generated to match them. Finally, a finished video with clear logic and coherent content is synthesized. This solution realizes the full-process automation from video materials to the generation of the final video file, can quickly process a large number of video materials, reduces the cumbersome manual editing process in traditional video production, greatly reduces the dependence on manual intervention, reduces the operation cost, simplifies the video production process, improves the video production efficiency, and also improves the quality and efficiency of the finished product.

[0048] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings.

[0050] Figure 1 A flowchart showing a video processing method provided by an embodiment of the present disclosure.

[0051] Figure 2 A flowchart showing a video processing method of another embodiment;

[0052] Figure 3 A flowchart showing the process architecture of the video processing solution provided by an embodiment of the present disclosure;

[0053] Figure 4 A schematic diagram showing the module structure of a video processing apparatus provided by an embodiment of the present disclosure;

[0054] Figure 5 A schematic diagram showing the structure of an electronic device provided by an embodiment of the present disclosure;

[0055] Figure 6 A schematic diagram of the structure of a computer program product according to an embodiment of the present disclosure. Detailed Implementation Manner

[0056] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. The features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments.

[0057] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0058] As Figure 1 shown, an embodiment of the present disclosure provides a video processing method, which is applied to a video processing system and includes but is not limited to the following steps:

[0059] Step 110: Split the input video material into multiple semantically coherent video segments;

[0060] Step 120: Generate corresponding subtitles and voiceovers according to the input text, and record the start and end times of each voiceover;

[0061] Step 130: In the multiple video segments, select the video segments related to the input text based on semantic matching;

[0062] Step 140: Adjust the duration of the video segments related to the input text according to the start and end times of the voiceovers to make them synchronized with the corresponding subtitles and voiceovers;

[0063] Step 150: Stitch the adjusted video segments in the order of text semantics, and synchronously synthesize them with the matching corresponding subtitles and voiceovers to generate a finished video.

[0064] The embodiments of the present disclosure can automatically extract relevant video clips from video materials according to the text content provided by the user, generate corresponding subtitles and voiceovers that match them, and finally synthesize a finished video with clear logic and coherent content. This solution realizes the full-process automation from video materials to the generation of the final video file, can quickly process a large number of video materials, reduces the cumbersome manual editing process in traditional video production, greatly reduces the dependence on manual intervention, reduces the operation cost, simplifies the video production process, improves the video production efficiency, and also improves the quality and efficiency of the finished product.

[0065] The following is a detailed description Figure 1 of each step in the video processing method shown.

[0066] When performing step 110, first obtain video materials from the video material library, and then perform scene segmentation on the video materials. Video scene segmentation is a video content analysis technology aimed at dividing a video into several semantically coherent video clips or scenes for subsequent processing and understanding. This technology is applied in fields such as movie editing, video summary generation, content recommendation, and video editing.

[0067] The basic principle of video is that a video usually consists of multiple shots, and a shot is the basic structural unit in the boundaries, production, and retrieval of a video sequence. Each shot is the picture taken by a camera under the same set of scenes within a continuous time period. There is quite a lot of repetitive information between the frame images in the same shot, usually representing the same event, theme, or location, and having consistent semantics. For example, the reporter's speech in a news segment is one shot, and a street interview is another shot. In an explanatory video, the teacher's explanation of a formula in front of a whiteboard is one shot, and switching to a PPT is another shot.

[0068] Therefore, the semantic consistency between shots makes the scene segmentation of the video easier and can effectively classify and analyze the video. The task of scene segmentation is to detect these logically related shot groups to help better understand the video content.

[0069] Therefore, different video clips are segmented through a video scene segmentation algorithm to ensure that each video clip has the same and consistent semantics. A complete shot usually describes a scene or a group of related actions. Through scene cutting, each video clip can only contain relatively consistent content, avoiding different themes being mixed in one video clip, which is convenient for subsequent video splicing.

[0070] Optionally, the video scene segmentation method includes algorithms based on color histogram changes, methods based on optical flow estimation, segmentation methods based on shot boundary detection (SBD), and methods based on spatio-temporal clustering. Among them, the segmentation algorithm based on shot boundary detection has a relatively small computational load and can be applied to process video data in real time. The following uses the segmentation algorithm based on shot boundary detection to illustrate the video scene segmentation scheme, which specifically includes the following steps:

[0071] Extract consecutive frames from the video material;

[0072] Calculate the color histogram for each frame of the image;

[0073] Using the segmentation algorithm based on shot boundary detection, determine the shot transition points by calculating the color histogram differences between the consecutive frames, and divide the consecutive frames into multiple semantically coherent video segments based on the shot transition points.

[0074] In this implementation step, first, the original video material can be imported in batches from a preset video material library. Using high-speed frame extraction technology, each video material is extracted frame by frame to generate a sequence of consecutive images, that is, consecutive frames. Subsequently, calculate the color histogram for each frame of the image. Optionally, use a normalization algorithm to standardize the histogram data for each frame.

[0075] Next, use the segmentation algorithm based on shot boundary detection to determine whether there is a significant change by calculating the color histogram differences (calculated using the L2 distance) between consecutive frames. When the detected histogram difference exceeds a preset threshold, it is considered that a shot transition has occurred, and then the original video material is segmented into independent video segments (scene segments). This scene segmentation process can not only ensure that each video segment has consistent semantic information internally, but also facilitate subsequent semantic matching and video segment retrieval.

[0076] Therefore, in "segmenting the input video material into multiple semantically coherent video segments", "semantically coherent" includes the following three points:

[0077] 1. The video content of the video segment is semantically related.

[0078] Each segmented video segment contains a complete and meaningful scene, rather than randomly intercepted irrelevant pictures. For example, in an interview video, a complete answer should be regarded as a semantically coherent video segment, rather than splitting the first half and the second half of the same sentence into two discontinuous video segments.

[0079] 2. The shot (scene) transitions are logical.

[0080] The transitions between video segments are natural and do not abruptly jump to completely different scenes. For example, in an instructional video, if a video segment of a corresponding subtitle shot shows a teacher writing on a blackboard, the next video segment shows the content on the blackboard or the teacher's explanation, rather than suddenly switching to an unrelated landscape scene.

[0081] This embodiment uses the color histogram difference to determine the shot transition point, and its specific principle is as follows:

[0082] ① Video frame extraction: First, extract adjacent frames F1 and F2 from the video material.

[0083] ② Calculation of color histograms:

[0084] Calculate the color histogram of each frame. We use H(i) to represent the count of color i in the color histogram.

[0085] Normalize the color histograms H1 and H2 calculated for each frame:

[0086]

[0087] ③ Histogram difference calculation: Calculate the difference between the two normalized histograms through the L2 distance:

[0088]

[0089] ④ Set a threshold T. If D>T, it indicates that the inter-frame difference increases significantly at this time, and the algorithm determines that a scene (shot) transition has occurred.

[0090] This embodiment uses a fixed threshold T. To ensure the accuracy of the segmentation result, a multi-level threshold judgment mechanism can also be adopted, and a dynamic threshold adjustment algorithm is introduced. Specifically, "dynamic threshold adjustment" is an adaptive shot transition detection method that dynamically adjusts the threshold for shot transition determination by analyzing the change trend of the video content in the video material, avoiding misjudging the transition point due to light changes and improving the detection ability for slow transition scenes.

[0091] In this embodiment, this method ensures the accuracy and robustness of video segmentation, laying a high-quality foundation for subsequent semantic matching, duration adjustment, and video splicing. Through experimental verification, this segmentation method can effectively identify shot transition points in the vast majority of video materials, ensuring that the segmentation result has a high accuracy and stability.

[0092] In another embodiment, it can also be to adopt a video segmentation model based on Transformer (such as TimeSformer), which captures long-term inter-frame dependencies through the self-attention mechanism to improve the segmentation accuracy of complex scenes.

[0093] In this embodiment, the video material library contains various types of videos, such as advertising shorts, meeting records, documentaries, etc. The system first uses a hierarchical scene segmentation algorithm for different types of videos. For videos with fast actions and frequent shot transitions, higher-frequency frame extraction and strict threshold judgment are adopted to ensure the accuracy of segmentation; for videos with slow rhythms and stable images, lower-frequency frame extraction is used to ensure that the overall content is not overly segmented.

[0094] As Figure 2 shown, the video processing method of another embodiment is also provided in the implementation manner of the present disclosure:

[0095] Step 210: Split the input video material into multiple semantically coherent video segments;

[0096] Step 220: Use a multi-task cascaded convolutional neural network MTCNN (Multi-task Cascaded Convolutional Networks) to perform face detection on the video segments, and remove the video segments containing target face information. Among them, the multi-task cascaded convolutional neural network includes P-Net, R-Net, and O-Net, and generates a target face box through layer-by-layer screening by the P-Net, R-Net, and O-Net to determine the video segments to be removed;

[0097] Step 230: Generate corresponding subtitles and voiceovers according to the input text, and record the start and end times of each voiceover;

[0098] Step 240: Among the multiple video segments, select the video segments related to the input text based on semantic matching;

[0099] Step 250: Adjust the duration of the video segments related to the input text according to the start and end times of the voiceover to make it synchronized with the corresponding subtitles and voiceover;

[0100] Step 260: Stitch the adjusted video segments in the order of text semantics, and synchronously synthesize them with the matching corresponding subtitles and voiceovers to generate a finished video.

[0101] Among them, step 220 is a preprocessing method for video segments, and its purpose is to improve the usability and legality of video content. This method not only ensures the efficiency and accuracy of face detection, but also lays a foundation for the subsequent video processing process.

[0102] Specifically, it is expanded as follows:

[0103] Generate an image pyramid for the video segment, input it into P-Net, and generate a face candidate box;

[0104] For the generated face candidate boxes, they are screened layer by layer through R-Net and O-Net, and the video segments containing the face information of the target person are eliminated.

[0105] Specifically, in step 220, the face detection process is as follows:

[0106] 1. Image pyramid generation: First, each frame image in the video segment is scaled multiple times (Resize) to generate image pyramids of different sizes. In this way, through the processing of P-Net at different scaling levels, faces of various scales can be effectively found, improving the robustness of detection. Each scaled frame image is input into P-Net to generate candidate face boxes.

[0107] 2. Preliminary candidate box screening: All the images in the image pyramid are input into P-Net to generate a large number of face candidate boxes. Subsequently, non-maximum suppression (NMS) is performed according to the intersection over union (IOU) value to screen out better-quality face candidate boxes.

[0108] 3. Refinement of candidate boxes: The face candidate boxes screened by P-Net will be sent to R-Net for further fine-tuning. In this process, non-maximum suppression is again performed using the IOU value to remove redundant face candidate boxes.

[0109] 4. Final face box output: After being processed by R-Net, the obtained face candidate boxes will be input into O-Net, which outputs the coordinates of the accurate bounding box (bbox) and the coordinates of the key points (Landmark), and finally generates accurate face boxes.

[0110] In the alternative method, eliminating the video segments containing the target face information includes:

[0111] Adopting an optical flow-based motion estimation algorithm to perform motion compensation on the detected target face box in the video segment to improve detection accuracy;

[0112] According to the preset face area and motion trajectory, combined with the number of consecutive detections between frames, the video segments containing the target face are eliminated.

[0113] In this embodiment, adopting motion compensation technology can accurately detect faces in motion in dynamic videos, avoiding false detections caused by motion blur or light changes, and solving the problem of false detections caused by motion blur and light changes in dynamic videos.

[0114] Among them, the Frame-wise Continuous Detection Count can refer to the number of times the same face target is continuously detected within a certain number of frames (such as a time window of N frames). At this time, it is determined that the face is valid and the elimination process is executed. This parameter is used to ensure that the detected target is not an incorrect recognition caused by instantaneous misdetection or noise.

[0115] In the portrait detection link, the system introduces an adaptive face recognition algorithm, which can not only detect static faces but also detect portraits in motion, greatly improving the recognition ability of portraits in complex videos. For the detected face information, the system makes judgments through preset rules. Only when the portrait area exceeds a certain proportion of the screen or appears continuously for multiple frames, is it recognized as an object to be eliminated.

[0116] Optionally, it can also be to use the Faster R-CNN model to replace the MTCNN, and improve the face detection speed and accuracy through the Region Proposal Network (RPN).

[0117] Optionally, the preprocessing of the video segment can also include using image restoration technology to remove the original captions and / or watermarks in the video segment.

[0118] For the original captions and watermarks, this embodiment designs a removal algorithm based on image edge detection and texture analysis. By analyzing the fixed positions and repetitive features in each video frame of the video segment, the original captions and watermarks are automatically located and removed, ensuring that the subsequent spliced video images are clean and tidy. After multiple experimental verifications, various types of watermarks and original caption information can be effectively deleted without affecting the main content of the video segment, while avoiding the problem of video information loss caused by accidental deletion.

[0119] Specifically, the image restoration technology includes the Inpainting algorithm of OpenCV or GAN (Generative Adversarial Network) technology.

[0120] In the implementation manner of the present disclosure, step 120 shown is executed Figure 1 to generate the current letter and voice broadcast according to the currently input text, which specifically includes the following steps:

[0121] Split the input text into sentences;

[0122] Generate corresponding subtitles for each split sentence, generate the voice broadcast through text-to-speech technology, and record the start and end times of each voice broadcast.

[0123] In this embodiment, the input text can be any business text such as promotional copy or product introduction copy. In this embodiment, natural language processing technology is first used to preprocess the input text. Exemplarily, for the input text, the text sentences are automatically divided using punctuation marks and language models to ensure that each sentence forms an independent paragraph.

[0124] Exemplarily, corresponding subtitle text is generated for each sentence, and a voiceover file is synchronously generated. During the generation of the voiceover, the system can automatically adjust the speaking speed, pauses, and emotional tone according to the sentence content to ensure that the generated voice is natural and fluent, and record the start and end times of each voiceover for easy subsequent synchronization and synthesis of the video and audio.

[0125] This embodiment can improve the matching degree between the corresponding subtitles and the content of subsequent video segments, enhancing the clarity of information transmission. And by automatically generating the voiceover, the cost of manual dubbing is reduced, and the video production efficiency is improved.

[0126] Optionally, DistilBERT is used to generate corresponding subtitles for each split sentence to reduce the computational amount; the VALL-E technology is adopted to generate personalized voiceovers.

[0127] In the embodiment of the present disclosure, step 130 is executed to select video segments related to the input text based on semantic matching, which specifically includes the following steps:

[0128] For each video segment, key frames are extracted, and for each split sentence, text entities and their text feature vectors are extracted;

[0129] The pre-trained cross-modal deep learning model CLIP (Contrastive Language–Image Pre-training) is used to extract the image feature vectors of the key frames;

[0130] A similarity calculation method is used to match the image feature vectors and the text feature vectors, and the video segment with the highest similarity is selected as the video segment related to the input text based on the matching score.

[0131] Relying on "selecting video clips related to the input text based on semantic matching", this implementation adopts a cross-modal deep learning model (such as CLIP) to ensure that the selected video clips are highly consistent with the text content by calculating the similarity between the image feature vector and the text feature vector. Compared with traditional random splicing or manual selection, the matching accuracy of this implementation can reach more than 90%, and the generated video content is more in line with user intentions. For example, in an educational video, the text explaining "variables" can accurately match the screen showing the definition of the variable. The function is to accurately match the text entities extracted from each sentence with the video clips obtained after the video scene segmentation processing, so as to achieve effective docking between the entities in the sentence and the video content. This process not only improves the accuracy of the information conveyed by the video content, but also provides higher flexibility and creative space for video production, and ultimately optimizes the effect of the overall video production.

[0132] In this embodiment, a number of key frames can be extracted from the segmented video segments at fixed time intervals, and the image feature vector of each frame can be extracted using a pre-trained CLIP model. The frame vectors in the same video segment are averaged and fused to form the overall image feature vector of the video segment.

[0133] The goal of the CLIP model is to learn a representation that can map text and images into a common, multimodal embedding space, so that similar text and images are closer in this space. For example, the key frame is 1 frame or other number of frames per second.

[0134] In an optional manner, the similarity calculation method adopts a cosine similarity calculation method.

[0135] Specifically, assuming that the speech time of each sentence is t, n text entities are extracted from it, and m video clips are extracted from all original video materials. For the n1th text entity, the CLIP model is used to calculate the similarity between the n1th text entity and the m video clips to select the video clip with the highest similarity to the text entity. The specific calculation logic is as follows:

[0136] (1) Extract s frames from the image in the i-th video clip and use the CLIP model to obtain the image feature vector f of each frame ij , where j = 1, 2, …, s;

[0137] (2) Calculate the average feature vector of each frame:

[0138]

[0139] (3) Calculate the text feature vector e of the n1th text entity using the CLIP model;

[0140] (4) Calculate the cosine similarity between the text feature vector and the image feature vector of the text entity:

[0141]

[0142] (5) Select the video segment with the highest similarity as the video segment related to the input text:

[0143]

[0144] Optionally, use the VATT (Vision-Audio-Text Transformer) model to replace the CLIP model, combine the multi-modal features of video, audio and text, and improve the matching accuracy.

[0145] In the embodiment of the present disclosure, step 140 is executed to adjust the duration of the selected video segment according to the start and end times of the voiceover, and precisely crop the relevant video segment to ensure that the video picture corresponds exactly to the text and voiceover time of the corresponding subtitle.

[0146] Among them, the specific process of step 140 is as follows:

[0147] Determine the actual duration and the target duration of the video segment with the highest similarity, where the target duration is obtained based on the start and end time intervals of the voiceover;

[0148] If the actual duration is greater than the target duration, crop the video segment with the highest similarity so that the actual duration is the same as the target duration;

[0149] If the actual duration is less than the target duration, for the video segment with the highest similarity, supplement the video segments with the second highest similarity and those after it in descending order of the matching score until the adjusted actual duration is less than the target duration.

[0150] In this embodiment, the target duration is obtained based on the start and end time intervals of the voiceover, which corresponds to the duration of the corresponding subtitle, and the target duration is t / n (where t is the duration of the entire video segment and n is the number of sentences obtained by splitting the input question).

[0151] If the actual duration t′ of the matched video segment is longer than the corresponding target duration t / n, it needs to be cropped. The purpose of cropping is to ensure that the playback time of this video segment is synchronized with the time of the corresponding subtitle and voiceover.

[0152] If t′ < t / n and the currently selected video clip is too short, then select the other video clips with the second-highest similarity to make up for it. At this time, the video of the remaining time period is t / n - t′, which is the calculated time to be supplemented to ensure that the final video duration is completely synchronized with the voiceover and the corresponding subtitles.

[0153] In the embodiment of the present disclosure, step 150 is executed. In the video splicing stage, the system arranges the video clips that have been cropped or compensated for duration in sequence according to the semantic order of the original text and the matching results, and uses seamless splicing technology to fuse the video clips to ensure a smooth transition between the video clips. To improve the visual experience of the finished video, the system automatically adds transition effects during the splicing process to eliminate the abruptness caused by the clip switching.

[0154] At the same time, the system performs audio-visual synchronization processing on the previously generated corresponding subtitles, voiceovers, and background music to ensure perfect matching of the video images, voiceovers, and corresponding subtitles on the timeline. For the background music part, the system can automatically select appropriate music according to the video style and adjust the balance between the volume and the voice to form a finished video with excellent audio-visual effects.

[0155] Finally, the finished video is encoded and compressed to output a standard digital video file, which can be directly used for network dissemination and marketing promotion.

[0156] The embodiment of the present disclosure provides an application example process of a video processing solution, as Figure 3 shown, and its technical solution process is as follows:

[0157] In the text processing stage, with the help of a large model, the input text is split into multiple paragraphs according to the topic words described in the paragraphs, and each paragraph is further split into sentences and the nouns of each sentence are found. The sentences are used to generate the corresponding subtitles, and a text and speech alignment model is used to generate the voiceovers and record the start and end times of the voiceovers.

[0158] In the video processing stage, video materials are extracted from the video material library, and the video materials are segmented based on the scene. The video detection portrait model is used to detect the face information of the segmented video clips and eliminate the video clips with face information.

[0159] In the semantic matching stage, based on the video semantic model, the nouns of each sentence are used to perform semantic matching with the remaining video clips after elimination to obtain relevant video clips.

[0160] In the video synthesis stage, the relevant video clips, corresponding subtitles, and voiceovers are synchronously synthesized based on the start and end times of the voiceovers to generate a finished video. Optionally, background music is also synchronously synthesized in the finished video.

[0161] Figure 4It is a schematic diagram of a module of an embodiment of a video processing device provided by an embodiment of the present disclosure. As Figure 4 shown, the video processing device 400 of the present disclosure includes, but is not limited to:

[0162] A video segmentation module 410 that divides the input video material into multiple semantically coherent video segments;

[0163] A text processing module 420 that generates corresponding subtitles and voiceovers according to the input text, and records the start and end times of each voiceover;

[0164] A semantic matching module 430 that selects video segments related to the input text based on semantic matching among the multiple video segments;

[0165] A duration synchronization module 440 that adjusts the duration of the video segments related to the input text according to the start and end times of the voiceover, so that it is synchronized with the corresponding subtitles and voiceover;

[0166] A video splicing module 450 that splices the adjusted video segments in the order of text semantics, and synchronously synthesizes them with the matching corresponding subtitles and voiceovers to generate a finished video.

[0167] In an alternative embodiment, the video segmentation module 410 is specifically configured to:

[0168] Extract consecutive frames from the video material;

[0169] Calculate the color histogram for each frame of the image;

[0170] Use a segmentation algorithm based on shot boundary detection to determine shot transition points by calculating the color histogram differences between the consecutive frames, and divide the consecutive frames into multiple semantically coherent video segments based on the shot transition points.

[0171] In an alternative embodiment, the video segmentation module 410 is specifically further configured to, before selecting video segments related to the input text based on semantic matching, perform preprocessing on each of the video segments, and the preprocessing includes:

[0172] Perform face detection on the video segments using a multi-task cascaded convolutional neural network, and exclude video segments containing target face information;

[0173] Among them, the multi-task cascaded convolutional neural network includes P-Net, R-Net, and O-Net, and generates target face frames layer by layer through P-Net, R-Net, and O-Net to determine the video segments to be excluded.

[0174] In an alternative embodiment, the video segmentation module 410 is specifically further configured to:

[0175] Adopt an optical flow - based motion estimation algorithm to perform motion compensation on the detected target face frame in the video clip;

[0176] Eliminate the video clip containing the target face according to the preset face area and motion trajectory, in combination with the number of consecutive detections between frames.

[0177] In an alternative embodiment, the pre - processing further includes using image inpainting technology to remove the original captions and / or watermarks in the video clip.

[0178] In an alternative embodiment, the text processing module 420 is specifically configured to:

[0179] Split the input text into sentences;

[0180] Generate corresponding subtitles for each split sentence, synchronously generate the voiceover through text - to - speech technology, and record the start and end times of each voiceover segment.

[0181] In an alternative embodiment, the semantic matching module 430 is specifically configured to:

[0182] Extract key frames for each video clip, and extract text entities and their text feature vectors for each split sentence;

[0183] Use a pre - trained cross - modal deep learning model to extract the image feature vectors of the key frames;

[0184] Adopt a similarity calculation method to match the image feature vectors and the text feature vectors, and select the video clip with the highest similarity as the video clip related to the input text based on the matching score.

[0185] In an alternative embodiment, the duration synchronization module 440 is specifically configured to:

[0186] Determine the actual duration and the target duration of the video clip with the highest similarity, where the target duration is obtained based on the start and end time intervals of the voiceover;

[0187] If the actual duration is greater than the target duration, crop the video clip with the highest similarity so that the actual duration is the same as the target duration;

[0188] If the actual duration is less than the target duration, for the video clip with the highest similarity, supplement the video clips with the second - highest and subsequent similarities in descending order of the matching score until the adjusted actual duration is less than the target duration.

[0189] In an alternative embodiment, the video splicing module 450 is specifically configured to:

[0190] Using a seamless splicing algorithm based on bilinear interpolation and deconvolution upsampling, the adjusted video segments are spliced in the order of text semantics, and the corresponding subtitles, voiceovers, and background music are synchronously superimposed.

[0191] This device can automatically extract relevant video segments from video materials according to the text content provided by the user, generate corresponding subtitles and voiceovers that match them, and finally synthesize a finished video with clear logic and coherent content. This solution realizes the full-process automation from video materials to the generation of the final video file, can quickly process a large number of video materials, reduces the cumbersome manual editing process in traditional video production, greatly reduces the dependence on manual intervention, reduces the operation cost, simplifies the video production process, improves the video production efficiency, and also improves the quality and efficiency of the finished product.

[0192] This device consists of a video segmentation module 410, a text processing module 420, a semantic matching module 430, a duration synchronization module 440, and a video splicing module 450. Each module can be interconnected through a data interface and cooperate to realize that the video processing device can be embedded into the front-end development tool as an independent software module, or can provide an interface as a cloud service for developers to remotely call.

[0193] An embodiment of the present invention also provides a video processing device, including a processor and a memory, in which executable instructions of the processor are stored. Among them, the processor is configured to execute the steps of a video processing method by executing the executable instructions.

[0194] As shown above, the video processing device of the present invention can automatically extract relevant video segments from video materials according to the text content provided by the user, generate corresponding subtitles and voiceovers that match them, and finally synthesize a finished video with clear logic and coherent content. This solution realizes the full-process automation from video materials to the generation of the final video file, can quickly process a large number of video materials, reduces the cumbersome manual editing process in traditional video production, greatly reduces the dependence on manual intervention, reduces the operation cost, simplifies the video production process, improves the video production efficiency, and also improves the quality and efficiency of the finished product.

[0195] Those skilled in the art of the technical field to which the present invention belongs can understand that various aspects of the present invention can be implemented as a system, a method, or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "platform" here.

[0196] Figure 5 is a schematic structural diagram of the video processing device of the present invention. The following refers toFigure 5 Describe the electronic device 500 according to this embodiment of the present invention. Figure 5 The displayed electronic device 500 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

[0197] As Figure 5 shown, the electronic device 500 is presented in the form of a general computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different platform components (including the storage unit 520 and the processing unit 510), a display unit 540, etc.

[0198] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 510, so that the processing unit 510 executes the steps according to various exemplary embodiments of the present invention described in the above video processing method part of this specification. For example, the processing unit 510 can execute steps as Figure 1 or Figure 2 shown.

[0199] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 521 and / or a cache storage unit 522, and may further include a read-only storage unit (ROM) 523.

[0200] The storage unit 520 may also include a program / utility 524 having a set (at least one) of program modules 525. Such program modules 525 include, but are not limited to: a processing system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples.

[0201] The bus 530 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any bus structure in a variety of bus structures.

[0202] The electronic device 500 may also communicate with one or more external devices 501 (such as a keyboard, a pointing device, a Bluetooth device, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be performed through the input / output (I / O) interface 550. Moreover, the electronic device 500 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 560. The network adapter 560 may communicate with other modules of the electronic device 500 through the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0203] Embodiments of the present disclosure also provide a computer-readable storage medium for storing a program, and the steps of the video processing method are implemented when the program is executed. In some possible implementation manners, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above video processing method section of this specification.

[0204] As Figure 6 shown, the computer program product 600 for implementing the above method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0205] A computer program product may employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0206] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium can also be any readable medium other than a readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted with any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0207] The program code for performing the processing of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or, can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0208] In summary, the object of the present invention is to provide a video processing method, device, equipment and storage medium, which can automatically extract relevant video clips from video materials according to the text content provided by the user, generate corresponding subtitles and voiceovers that match them, and finally synthesize a finished video with clear logic and coherent content. This solution realizes the full-process automation from video materials to the generation of the final video file, can quickly process a large number of video materials, reduces the cumbersome manual editing process in traditional video production, greatly reduces the dependence on manual intervention, reduces the operation cost, simplifies the video production process, improves the video production efficiency, and also improves the quality and efficiency of the finished product.

[0209] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A video processing method, characterized in that Including: Splitting the input video material into multiple semantically coherent video segments; Generating corresponding subtitles and voiceovers according to the input text, and recording the start and end times of each voiceover; Among the multiple video segments, selecting the video segments related to the input text based on semantic matching; Adjusting the duration of the video segments related to the input text according to the start and end times of the voiceovers to synchronize them with the corresponding subtitles and voiceovers; Splicing the adjusted video segments in the order of text semantics, and synchronously synthesizing them with the matching corresponding subtitles and voiceovers to generate a finished video.

2. The video processing method according to claim 1, wherein The splitting of the input video material into multiple semantically coherent video segments includes: Extracting consecutive frames from the video material; Calculating the color histogram for each frame image; Using a segmentation algorithm based on shot boundary detection, determining shot transition points by calculating the color histogram differences between the consecutive frames, and dividing the consecutive frames into multiple semantically coherent video segments based on the shot transition points.

3. The video processing method according to claim 1, characterized in that Before selecting the video segments related to the input text based on semantic matching, the video processing method further includes preprocessing each video segment, and the preprocessing includes: Performing face detection on the video segment using a multi-task cascaded convolutional neural network, and removing the video segments containing target face information; Among them, the multi-task cascaded convolutional neural network includes P-Net, R-Net, and O-Net, and generates target face boxes layer by layer through the P-Net, R-Net, and O-Net to determine the video segments to be removed.

4. The video processing method according to claim 3, wherein The removing of the video segments containing target face information includes: Performing motion compensation on the detected target face boxes in the video segment using an optical flow-based motion estimation algorithm; Removing the video segments containing the target face according to the preset face area and motion trajectory, in combination with the number of consecutive detections between frames.

5. The video processing method according to claim 3, wherein The preprocessing further includes using image restoration technology to remove the original subtitles and / or watermarks in the video segment.

6. The video processing method according to claim 1, characterized in that The generating of the corresponding subtitles and voiceovers according to the input text includes: Splitting the input text into sentences; Generating corresponding subtitles for each split sentence, and synchronously generating the voiceover through text-to-speech technology, and recording the start and end times of each voiceover.

7. The video processing method according to claim 6, wherein The selecting of the video segments related to the input text based on semantic matching includes: Extracting key frames for each video segment, and extracting text entities and their text feature vectors for each split sentence; Using a pre-trained cross-modal deep learning model to extract the image feature vectors of the key frames; Adopting a similarity calculation method to match the image feature vectors with the text feature vectors, and selecting the video segment with the highest similarity score as the video segment related to the input text.

8. The video processing method according to claim 6, wherein The adjusting of the duration of the video segments related to the input text according to the start and end times of the voiceovers includes: Determine the actual duration and the target duration of the video segment with the highest similarity, where the target duration is obtained based on the start and end time intervals of the voiceover; If the actual duration is greater than the target duration, crop the video segment with the highest similarity so that the actual duration is the same as the target duration; If the actual duration is less than the target duration, for the video segment with the highest similarity, supplement the video segments with the second highest similarity and subsequent ones in descending order of the matching score until the adjusted actual duration is less than the target duration.

9. The video processing method according to claim 1, wherein The splicing of the adjusted video segments in the order of text semantics and the synchronous synthesis with the corresponding subtitles and voiceovers includes: Using a seamless splicing algorithm based on bilinear interpolation and deconvolution upsampling, splice the adjusted video segments in the order of text semantics, and synchronously overlay the corresponding subtitles, voiceovers, and background music.

10. A video processing device, characterized in that, Including: A video segmentation module that divides the input video material into multiple semantically coherent video segments; A text processing module that generates corresponding subtitles and voiceovers according to the input text and records the start and end times of each voiceover; A semantic matching module that selects video segments related to the input text based on semantic matching among multiple video segments; A duration synchronization module that adjusts the duration of the video segments related to the input text according to the start and end times of the voiceover to make it synchronized with the corresponding subtitles and voiceovers; A video splicing module that splices the adjusted video segments in the order of text semantics and synchronously synthesizes them with the corresponding subtitles and voiceovers to generate a finished video.

11. An electronic device, characterized in that, Including: A processor; A memory that stores executable instructions of the processor; Wherein, the processor is configured to execute the steps of the video processing method according to any one of claims 1 to 9 by executing the executable instructions.

12. A computer-readable storage medium for storing a program, characterized in that, When the program is executed, the steps of the video processing method according to any one of claims 1 to 9 are implemented.

Images