Video mixing and cutting method, device and equipment and storage medium

Abstract

The invention discloses a video mixing method and device, equipment and a storage medium, and the method comprises the steps: dividing an obtained original video into a plurality of video clips, and determining a subtitle region of each video clip; obtaining text description information corresponding to each video clip, and screening a preset number of first target video clips from each video clip according to each text description information and the user prompt text; generating script information of each first target video clip based on the user prompt text, screening second target video clips from the first target video clips according to the verified script information, and generating subtitles and dubbles of each second target video clip; and according to the video content, the subtitles and the dubbing of each second target video clip, splicing each second target video clip to form the mixed video of the original video, so that deep understanding of the video content is realized, the demand of a user for the personalized mixed video is met, and the rich and vivid mixed video can be generated.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a video mixing method, device, equipment and storage medium. Background Art

[0002] At present, the mainstream solution for automatic video mixing actually only includes audio generation and video splicing technology. Users first upload the script or generate the script with a large model, and then manually group the videos to match the script fragments, generate the voice within the fragments and splice the fragments together.

[0003] However, with the explosive growth of video content, users have an increasing demand for personalized and customized videos. However, existing video remixing algorithms and products have the following problems: Requires user intervention throughout the process: Some products require users to understand the content of all video groups and reasonably match them to the corresponding script segments. Lack of in-depth understanding of video content: Algorithm systems and products do not perform video analysis or only perform visual-text black-box matching for creation. Remix scripts are generated based on thin information such as video titles and tags, and the semantics and emotions of the video cannot be accurately grasped, resulting in poor remixing results for simple categories of videos such as non-e-commerce promotions, lack of coherence and logic. Difficult to generate high-quality scripts: Existing script generation methods mostly rely on templates or simple keyword matching, which makes it difficult to generate scripts that are creative and attractive and conform to the original video content. Lack of fusion of multimodal information: Most existing algorithms and products only use one of text, audio, and visual information for video analysis or creation, which makes it difficult to make full use of multimodal information such as images and audio of the video, resulting in uneven remixing quality and lack of richness and vividness. Summary of the invention

[0004] The present invention provides a video mixing method, device, equipment and storage medium to solve the problems in the prior art of video mixing such as lack of in-depth understanding of video content and poor quality.

[0005] In a first aspect, an embodiment of the present invention provides a video mixing method, comprising:

[0006] Dividing the acquired original video into a plurality of video segments, and determining a subtitle area of ​​each of the video segments;

[0007] Acquire text description information corresponding to each of the video clips, and select a preset number of first target video clips from each of the video clips according to the text description information and user prompt text;

[0008] Generate script information of each first target video segment based on the user prompt text, select second target video segments from the first target video segments according to the verified script information, and generate subtitles and dubbing of each second target video segment;

[0009] According to the video content of each of the second target video clips and the subtitles and dubbing, each of the second target video clips is spliced ​​to form a mixed video of the original video.

[0010] Optionally, the acquired original video is divided into multiple video segments, including:

[0011] Performing audio transcription on the preprocessed original video to determine the audio transcription text corresponding to the preprocessed original video;

[0012] Identifying key frames in the original video and matching the key frames with the audio transcript;

[0013] If there is a match, dividing the acquired original video into a plurality of video segments based on the audio transcription text;

[0014] If there is no match, the acquired original video is divided into multiple video segments based on the key frames.

[0015] Optionally, determining the subtitle area of ​​each of the video clips includes:

[0016] Performing optical character recognition on the original video to obtain a recognition result, matching the recognition result with the audio transcription text corresponding to each of the video clips, and determining a text box corresponding to each of the video clips;

[0017] Anomalies are filtered and bounding rectangles are calculated for each of the text boxes to determine the subtitle area corresponding to each of the video clips.

[0018] Optionally, obtaining text description information corresponding to each of the video clips, and filtering a preset number of first target video clips from each of the video clips according to each of the text description information and a user prompt text, includes:

[0019] Acquire text description information corresponding to each of the video clips based on the large model;

[0020] Calculating the similarity between the text summary of each of the text description information and the user prompt text;

[0021] A preset number of video segments are selected in descending order of similarity as first target video segments.

[0022] Optionally, generating script information of each first target video segment based on the user prompt text, selecting second target video segments from the first target video segments according to the verified script information, and generating subtitles and dubbing of each second target video segment according to the script information, includes:

[0023] Generate script information corresponding to each of the first target video clips based on the large model according to each of the first target video clips and the user prompt text, wherein the script information includes: a script storyboard description and a script storyboard dubbing content;

[0024] Verifying the script information, and selecting a second target video segment from the first target video segment according to the verified script information;

[0025] Translating the script storyboard dubbing content corresponding to the second target video into a target language to generate subtitles for each of the second target video clips;

[0026] The storyboard dubbing content script corresponding to the second target video is converted into audio in the target language to generate dubbing for each of the second target video clips.

[0027] Optionally, converting the storyboard dubbing content script corresponding to the second target video into audio in the target language to generate dubbing for each second target video segment includes:

[0028] Reproduce the sound based on the pre-uploaded voice clips;

[0029] Based on the timbre, the script storyboard dubbing content corresponding to the second target video is converted into the target language audio of the timbre to generate the dubbing of each of the second target video clips.

[0030] Optionally, the splicing of the second target video segments according to the video content of the second target video segments and the subtitles and dubbing to form a mixed video of the original video includes:

[0031] Determine a mixed-cut position corresponding to each of the second target video segments, and determine the video content of each of the second target video segments and a splicing order of the subtitles and dubbing based on the mixed-cut position;

[0032] The second target video segments are spliced ​​together based on the splicing order to form a mixed video of the original video.

[0033] In a second aspect, an embodiment of the present invention provides a video mixing device, comprising:

[0034] A determination module, used to divide the acquired original video into a plurality of video segments, and determine the subtitle area of ​​each of the video segments;

[0035] An acquisition module, used for acquiring text description information corresponding to each of the video clips, and screening a preset number of target video clips from each of the video clips according to each of the text description information;

[0036] A generating module, used to determine the script information corresponding to each of the target video segments, and generate subtitles and dubbing for each of the target video segments according to the script information;

[0037] The creation and synthesis module is used to splice each of the target video clips according to the video content of each of the target video clips as well as the subtitles and dubbing to form a mixed video of the original video.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0039] at least one processor; and

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video mixing method described in any embodiment of the present invention.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video mixing method described in any embodiment of the present invention when executed.

[0043] The technical solution of the embodiment of the present invention is to divide the acquired original video into multiple video clips and determine the subtitle area of ​​each video clip; obtain text description information corresponding to each video clip, and screen a preset number of first target video clips from each video clip according to each text description information and user prompt text; generate script information of each first target video clip based on the user prompt text, screen second target video clips from the first target video clip according to the verified script information, and generate subtitles and dubbing of each second target video clip; splice each second target video clip according to the video content of each second target video clip and the subtitles and dubbing to form a mixed-cut video of the original video, thereby solving the problems of lack of deep understanding of video content and poor quality in video mixing in the prior art, realizing deep understanding of video content, determining script information through a large model, meeting users' needs for personalized videos, and enhancing the attractiveness of mixed-cut creation; generating dubbing and new subtitles based on the script information corresponding to each target video clip, thereby generating a mixed-cut result with richness and vividness, and enhancing the expressiveness of the video.

[0044] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a video mixing method provided in Embodiment 1 of the present invention;

[0047] Figure 2 A flow chart of original video analysis provided in Embodiment 1 of the present invention;

[0048] Figure 3 A flowchart of a video mixing method provided in Embodiment 2 of the present invention;

[0049] Figure 4 A specific flow chart of a video mixing method provided in Embodiment 2 of the present invention;

[0050] Figure 5 A flowchart of a video remix script generation provided in the second embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the structure of a video mixing device provided in Embodiment 3 of the present invention;

[0052] Figure 7 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0054] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] Embodiment 1

[0056] Figure 1 This is a flow chart of a video mixing method provided in the first embodiment of the present invention. This embodiment is applicable to the case of performing video mixing. The method can be performed by a video mixing device. The mixing device can be implemented in the form of hardware and / or software. The mixing device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0057] S110: Divide the acquired original video into a plurality of video segments, and determine a subtitle area of ​​each of the video segments.

[0058] The original video may refer to the original video to be spliced. The video clip may refer to the entire original video divided into several smaller, continuous video parts. The subtitle area may refer to the area in the video screen for displaying text.

[0059] Specifically, the original video may be divided according to the key frames of the original video to obtain multiple video segments. Alternatively, the original video may be transcribed for audio, the audio transcribed text corresponding to the original video may be determined, and the original video may be divided according to the semantic information of the text to obtain multiple video segments. Optical character recognition may be performed on the original video, and the subtitle area of ​​each video segment may be determined by matching the recognition result with the audio transcribed text corresponding to each video segment.

[0060] In this embodiment, through in-depth understanding of the video content, a suitable division method is selected to divide the acquired original video to obtain multiple video segments; and optical character recognition is performed on the original video, and the recognition result is matched with the audio transcription text corresponding to each video segment, so that the subtitle area of ​​each video segment can be determined, which not only improves the efficiency of information transmission of the mixed video, but also enhances the readability of the mixed video.

[0061] S120: Obtain text description information corresponding to each of the video segments, and select a preset number of first target video segments from each of the video segments according to the text description information and user prompt text.

[0062] Among them, the text description information may refer to the text information corresponding to the content of the video clip. The preset number may refer to the number of pre-set target video clips, which can be specifically determined according to the summary length of the text description information corresponding to each video clip, and the maximum input length of the big model, wherein the big model is a big model for extracting summary of the text description information, which may be a pre-trained big model, or may be a big model called through an external API. User prompt text may refer to the prompt text entered by the user, which is used to guide the big model to generate script information that meets the personalized needs of the user, for example, "(help me generate) a Nanjing food introduction video (script)", etc. The first target video clip may refer to the candidate video clip screened out for generating a mixed-cut video.

[0063] Specifically, the visual big model can be used to obtain all or sampled visual (frame) content in each video clip, and generate text description information corresponding to each video clip; based on the maximum input length of a pre-trained big model or a big model called by an external API, and the length of a text summary of the text description information extracted by the big model, the number of target video clips is pre-set, and based on the similarity of the text summary of each of the text description information and the user prompt text, a preset number of first target video clips are screened from each video clip from the preset number.

[0064] In this embodiment, the text description information corresponding to each video clip is obtained through the visual big model, and the number of target video clips is determined based on the summary length and the maximum input length of the big model through the summary of the text description information extracted by the big model; a preset number of target video clips are filtered from each video clip based on the preset number.

[0065] S130, generating script information of each of the first target video segments based on the user prompt text, selecting second target video segments from the first target video segments according to the verified script information, and generating subtitles and dubbing of each of the second target video segments.

[0066] Among them, script information may refer to the content of an executable file used to describe and control a software application. Verification may refer to determining whether the personalized needs of the user are met. Verified script information may refer to verified script information. The second target video clip may refer to a video clip for generating a mashup video. Subtitles may refer to descriptive subtitles that describe the action, scene or background information in the video to help the audience better understand the content of the picture; dubbing may refer to narration dubbing that provides explanations for the video to help the audience understand the content of the video.

[0067] Specifically, the script information corresponding to each first target video clip can be generated according to the user prompt text by using a pre-trained large model or calling the large model through an external API. The script information corresponding to each first target video clip can be judged according to the user's personalized needs, to determine whether the script information corresponding to each first target video clip meets the user's personalized needs, and to screen out the script information that meets the user's personalized needs, that is, the script information that has passed the verification. The second target video clip corresponding to the verified script information is screened from each first target video clip. Based on the script information corresponding to each second target video clip, use TTS and other technologies to generate dubbing and new subtitles, that is, based on text-to-speech synthesis technology, convert the script into speech, and generate new subtitles based on the speech information and script information.

[0068] In this embodiment, the script information corresponding to each first target video clip is generated according to the user prompt text through the large model, and the script information corresponding to each first target video clip is verified based on the user's personalized needs. The second target video clip is selected from the first target video clip according to the verified script information, which not only makes the created mixed-cut video more vivid and more content-intensive, but also meets the user's demand for personalized video, and effectively enhances the attractiveness of mixed-cut creation; based on the script information corresponding to each second target video clip, TTS and other technologies are used to generate dubbing and new subtitles, that is, based on text-to-speech synthesis technology, the script is converted into speech, and new subtitles are generated based on the speech information and the script information; based on the subtitles and dubbing of each target video clip, a mixed-cut result with richness and vividness is generated, thereby enhancing the expressiveness of the video.

[0069] S140: splicing the second target video segments according to the video contents of the second target video segments and the subtitles and dubbing to form a mixed video of the original video.

[0070] The splicing process may refer to a process of merging multiple video clips into a coherent video file.

[0071] Specifically, after determining the subtitles and dubbing of each second target video clip, multimodal information can be fully integrated according to the video content of each second target video clip and the subtitles and dubbing, and multimodal information such as images and audio of the video can be fully utilized to transcode, splice and package each second target video clip, thereby finally generating a rich and vivid mixed-cut video, which effectively improves the expressiveness of the mixed-cut video.

[0072] In this embodiment, the acquired original video is divided into a plurality of video segments, and the subtitle area of ​​each video segment is determined; the text description information corresponding to each video segment is obtained, and a preset number of first target video segments are screened from each video segment according to each text description information and a user prompt text; the script information of each first target video segment is generated based on the user prompt text, and the second target video segment is screened from the first target video segment according to the verified script information, and subtitles and dubbing of each second target video segment are generated; according to the video content of each second target video segment and the subtitles and dubbing, each second target video segment is spliced ​​to form a mixed-cut video of the original video, thereby solving the problems of the lack of deep understanding of video content and poor quality in video mixing in the prior art, achieving deep understanding of video content, determining script information through a large model, meeting user needs for personalized videos, and enhancing the attractiveness of mixed-cut creation; generating dubbing and new subtitles based on the script information corresponding to each target video segment, thereby generating a mixed-cut result with richness and vividness, and enhancing the expressiveness of the video.

[0073] Optionally, the acquired original video is divided into multiple video segments, including:

[0074] Performing audio transcription on the preprocessed original video to determine the audio transcription text corresponding to the preprocessed original video;

[0075] Preprocessing may refer to audio noise reduction and voice extraction, specifically using audio processing technology to remove background noise and extract voice information. Audio transcription may refer to using speech recognition technology to convert audio into text, which is the audio transcription text.

[0076] Specifically, the acquired original video is first subjected to audio noise reduction and voice extraction to obtain a preprocessed original video; and the preprocessed original video is subjected to audio transcription using speech recognition technology to obtain an audio transcription text corresponding to the original video.

[0077] Identifying key frames in the original video and matching the key frames with the audio transcript;

[0078] Among them, the key frame may refer to a video frame where a scene switch or a character action occurs.

[0079] Specifically, image processing technology is used to identify key frames in the video. After the key frames are identified, they are matched with the audio transcription text corresponding to the original video to determine the text content corresponding to each key frame. Specifically, by visually analyzing the key frames, the consistency of the audio transcription text corresponding to the original video and the visual content of the key frames is determined through the visual big model. If they are consistent, they match; if not, they do not match.

[0080] If there is a match, dividing the acquired original video into a plurality of video segments based on the audio transcription text;

[0081] Specifically, if the audio transcript corresponding to the original video matches the key frame visual content, the transcript is segmented into individual sentences based on the audio transcript according to the end marks of the sentences in the audio transcript (such as a period, question mark, exclamation mark, etc.). The segmented sentences are aligned with the timeline of the original video. According to the time period corresponding to the sentence, multiple corresponding video clips are extracted from the original video. The extracted video clips are saved as independent video files for subsequent use or analysis.

[0082] If there is no match, the acquired original video is divided into multiple video segments based on the key frames.

[0083] Specifically, if the audio transcription text corresponding to the original video matches the keyframe visual content, the original video is divided into different segments based on the identified keyframes; during the segmentation process, the length of the segment can be determined based on the time interval between the keyframes. For example, if the time interval between two keyframes is long, then the video content between them may be more complex or change greatly, and this period of time can be divided into a separate segment.

[0084] In this embodiment, the audio transcription of the preprocessed original video is performed to determine the audio transcription text corresponding to the preprocessed original video; the key frames in the original video are identified and the key frames are matched with the audio transcription text; the specific division method is determined based on the matching result, if it matches, the obtained original video is divided into multiple video segments based on the audio transcription text; if it does not match, the obtained original video is divided into multiple video segments based on the key frames. Through a flexible division method, the obtained original video is divided into multiple video segments, which ensures the rationality and effectiveness of video division.

[0085] Optionally, determining the subtitle area of ​​each of the video clips includes:

[0086] Performing optical character recognition on the original video to obtain a recognition result, matching the recognition result with the audio transcription text corresponding to each of the video clips, and determining a text box corresponding to each of the video clips;

[0087] Optical Character Recognition (OCR) refers to the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then uses character recognition methods to translate the shapes into computer text. A text box can refer to a movable and resizable text or graphic container, which is mainly used to insert and manage text content in a video.

[0088] Specifically, the text information in the video clips can be obtained by performing OCR on the original video segments and converted into an editable text format; the text information is matched with the audio transcription text corresponding to each video clip, and through text matching, it can be determined which part of the audio transcription text the text box in each video clip corresponds to. If the text extracted by OCR appears in the audio transcription text and the context is consistent, then it can be considered that this part of the text box corresponds to the audio clip, that is, the text box corresponding to each video clip is obtained.

[0089] Anomalies are filtered and bounding rectangles are calculated for each of the text boxes to determine the subtitle area corresponding to each of the video clips.

[0090] Among them, abnormal filtering may refer to abnormal value filtering. Specifically, a quartile abnormal value filtering method may be used to remove abnormal text boxes, such as text boxes with excessive position deviations. Enclosing rectangle calculation may refer to calculating the enclosing rectangle of each text box, and then determining the position and size of the subtitle area. The subtitle area may refer to a specific area used to display text annotations when a video is played.

[0091] In this embodiment, optical character recognition is performed on the original video to obtain a recognition result, and the recognition result is matched with the audio transcription text corresponding to each video clip to determine the text box corresponding to each video clip; each text box is subjected to abnormal filtering and external rectangle calculation to determine the subtitle area corresponding to each video clip, which not only improves the comprehensibility of the video content, but also enhances the audience's viewing experience.

[0092] For example, Figure 2 This is a flow chart of original video analysis provided by Embodiment 1 of the present invention. Figure 2 As shown, dividing the acquired original video into multiple video segments includes:

[0093] 1. Create a video analysis task.

[0094] 2. Perform audio denoising on the original video, remove background noise, and extract human voice information.

[0095] 3. Convert the processed audio of the original video into text.

[0096] 4. Identify the key frames of the processed original video and save the key frames.

[0097] 5. Match the audio transcription results with the key frames to determine the text content corresponding to each key frame.

[0098] 6. If there is a match, the original video is divided into multiple video segments according to the end marks of the sentences in the audio transcription text (such as period, question mark, exclamation mark, etc.); if there is no match, the obtained original video is divided into multiple video segments based on the key frames.

[0099] 7. If the audio transcription content and visual content of each video clip are consistent, the comprehensive title, the audio transcription content of each video clip and the sampling frames in each video clip are analyzed to generate a text report about the video clip (i.e., the text description information corresponding to each video clip); if the audio transcription content and visual content of each video clip do not match, the comprehensive title and the sampling frames in each video clip are analyzed to generate a text report about the video clip (i.e., the text description information corresponding to each video clip).

[0100] 8. Use a pre-trained big model, or call the big model through an external API, to extract the text content of the video clips (i.e., the text summary of the text description information corresponding to each target video clip), and determine the corresponding labels, etc.

[0101] 9. Summarize, label and keyframe the text content of the extracted video clips, generate script information for each video clip according to the user prompt text, and display it on the user interface for users to view and modify according to their personalized needs.

[0102] Embodiment 2

[0103] Figure 3 This is a flow chart of a video mixing method provided by Embodiment 2 of the present invention. The technical solution of this embodiment is further refined on the basis of the above embodiment. Figure 3 As shown, the method includes:

[0104] S210: Divide the acquired original video into a plurality of video segments, and determine a subtitle area of ​​each of the video segments.

[0105] S220: Obtain text description information corresponding to each of the video clips based on the large model.

[0106] The big model may refer to a visual big model; and the text description information may refer to text information corresponding to the content of the video clip.

[0107] Specifically, the video clips may be visually analyzed first, and then the consistency of the audio transcription content and the visual content of each video clip may be determined through the visual big model. If the audio transcription content and the visual content of each video clip are consistent, the audio transcription content of each video clip is integrated, and the corresponding text description information is generated for all or sampled visual (frame) content in the video clip through the visual big model. If the audio transcription content and the visual content of each video clip are inconsistent, or there is no human voice, that is, no audio transcription content, the audio transcription content of each video clip is not integrated, and only the corresponding text description information is generated for all or sampled visual (frame) content in the video clip through the visual big model.

[0108] In this embodiment, based on the actual situation and through a deep understanding of the video content, it can be determined whether to integrate the audio transcription content of each video clip, and then the corresponding text description information can be generated for all or sampled visual (frame) content in the video clip through the visual big model, so as to facilitate the subsequent generation of a mixed-cut video that is highly related to the original video content and has coherence and logic.

[0109] S230: Calculate the similarity between the text summary of each text description information and the user prompt text.

[0110] The text summary of the text description information may refer to a summary of text information corresponding to the content of the video clip and extracted keywords.

[0111] Specifically, the text summary and the user prompt text can be converted into vector representations by using a pre-trained text embedding model in the form of sentence vector embedding, wherein the pre-trained text embedding model can be Word2Vec, BERT, etc.; sentence vector embedding can refer to converting the entire sentence or paragraph into a vector of a fixed length. After the text embedding is completed, the similarity between the text summary vector and the user prompt text vector can be calculated by a numerical similarity calculation method; wherein the numerical similarity calculation method can be cosine similarity; cosine similarity measures their similarity by calculating the cosine value of the angle between two vectors. The closer the value is to 1, the more similar the vectors are; the closer the value is to 0, the greater the difference between the vectors.

[0112] In this embodiment, the text summary of the video clip and the user prompt text can be embedded through a pre-trained text embedding model to obtain their vector representations; the similarity between the text summary of each text description information and the user prompt text is calculated through cosine similarity.

[0113] S240 . Select a preset number of video segments in descending order of similarity as first target video segments.

[0114] The preset number may refer to the number of first target video segments that are preset. Specifically, it may be determined according to the summary length of the text description information corresponding to each video segment and the maximum input length of the large model, wherein the large model is a large model for extracting a summary of the text description information, which may be a pre-trained large model, or a large model called through an external API.

[0115] In this embodiment, the video segments may be sorted in descending order according to the similarity, and then a preset number of video segments may be selected in sequence as the first target video segments.

[0116] S250, generating script information for each of the first target video segments based on the user prompt text, selecting second target video segments from the first target video segments according to the verified script information, and generating subtitles and dubbing for each of the second target video segments.

[0117] S260: splicing the second target video segments according to the video content of the second target video segments and the subtitles and dubbing to form a mixed video of the original video.

[0118] In this embodiment, the obtained original video is divided into multiple video clips, and the subtitle area of ​​each video clip is determined; the text description information corresponding to each video clip is obtained based on the large model; the similarity between the text summary of each text description information and the user prompt text is calculated; a preset number of video clips are selected in descending order of similarity as first target video clips; script information of each first target video clip is generated based on the user prompt text, second target video clips are screened from the first target video clips according to the verified script information, and subtitles and dubbing of each second target video clip are generated; each second target video clip is spliced ​​according to the video content of each second target video clip and the subtitles and dubbing to form a mixed video of the original video; not only a deep understanding of the video content is achieved, the user's demand for personalized mixed videos is met, and the attractiveness of mixed creation is enhanced, but also a mixed video with richness and vividness can be generated.

[0119] For example, Figure 5 A flowchart of a video remix script generation provided in the second embodiment of the present invention is shown in FIG. Figure 5 As shown, determining the script information corresponding to each of the target video clips includes:

[0120] 1. Generate a unified vector representation through text embedding based on the text summary of the text description information corresponding to each target video clip and the user prompt text.

[0121] 2. Calculate the similarity between the text summary of each text description information and the user prompt text through cosine similarity.

[0122] 3. Sort the video clips in descending order of similarity, and then select K (i.e., a preset number) video clips in turn as the first target video clips.

[0123] 4. According to the splicing prompt words (i.e., user prompt text), the script information corresponding to each first target video segment is generated through the large model, and the second target video segment is selected from the first target video segment according to the verified script information.

[0124] 5. The reading time of the script corresponding to the second target video clip can be estimated at a speed of 6 words per second, thereby controlling the total duration of the mixed video.

[0125] 6. You can also intercept part of the script information according to user requirements, or continue to write the script information through a large model.

[0126] Optionally, determining script information corresponding to each of the target video segments, and generating subtitles and dubbing for each of the target video segments according to the script information, includes:

[0127] Generate script information corresponding to each of the first target video clips based on the large model according to each of the first target video clips and the user prompt text, wherein the script information includes: a script storyboard description and a script storyboard dubbing content;

[0128] Among them, the script storyboard description can refer to the detailed description of each shot in the video; the script storyboard dubbing content can refer to the text content prepared for the dubbing, dialogue or narration in the video.

[0129] Specifically, by using a pre-trained big model, or a big model called by an external API, script information corresponding to each first target video segment is generated according to each first target video segment and the user prompt text. The script information effectively distinguishes the storyboard description from the voice text.

[0130] Verifying the script information, and selecting a second target video segment from the first target video segment according to the verified script information;

[0131] Among them, verification can refer to judging whether the personalized needs of users are met. Specifically, the script information corresponding to each first target video clip can be judged according to the personalized needs of users. If the script information meets the personalized needs of users, it is determined that the script information has passed the verification, that is, the verified script information is obtained; then the first target video clip corresponding to the verified script information is screened out from the first target video clips to be determined as the second target video clip. Screening the second target video clip from the first target video clip according to the verified script information can not only make the created mixed-cut video more vivid and more tense in content, but also meet the user's needs for personalized videos, effectively enhancing the attractiveness of mixed-cut creation.

[0132] Translating the script storyboard dubbing content corresponding to the second target video into a target language to generate subtitles for each of the second target video clips;

[0133] The target language may be a language pre-specified by the user, such as Chinese, English, etc.

[0134] Specifically, the script storyboard dubbing content corresponding to the second target video can be translated into a pre-specified language according to the user's requirements to obtain the subtitles of each second target video segment.

[0135] The script storyboard dubbing content corresponding to the second target video is converted into audio in the target language to generate dubbing for each of the second target video clips.

[0136] Specifically, the script storyboard dubbing content corresponding to the second target video can be converted into voice in a language pre-specified by the user through TTS to obtain the dubbing of each target video clip.

[0137] In this embodiment, according to each of the first target video clips and the user prompt text, script information corresponding to each of the first target video clips is generated based on the large model, wherein the script information includes: a script storyboard description and a script storyboard dubbing content; the script information is verified, and second target video clips are screened from the first target video clips according to the verified script information; the script storyboard dubbing content corresponding to the second target video is translated into a target language to generate subtitles for each of the second target video clips; the script storyboard dubbing content script corresponding to the second target video is converted into audio in the target language to generate dubbing for each of the second target video clips; not only is the storyboard description distinguished from the voice text, but the script information of each target video clip can also be customized according to the user prompt text, so that the creation result is more vivid and has more content tension, while being able to meet the user's demand for personalized videos.

[0138] Optionally, converting the storyboard dubbing content script corresponding to the second target video into audio in the target language to generate dubbing for each second target video segment includes:

[0139] Reproduce the sound based on the pre-uploaded voice clips;

[0140] Among them, timbre replication can refer to a speech synthesis method based on deep learning technology. By recording a small amount of audio data, a speech model that is highly similar to the original timbre can be quickly generated.

[0141] Based on the timbre, the script storyboard dubbing content corresponding to the second target video is converted into the target language audio of the timbre to generate the dubbing of each of the second target video clips.

[0142] In this embodiment, the timbre of the pre-uploaded voice clip can be replicated, and based on the timbre of the replicated pre-uploaded voice clip, the script storyboard dubbing content corresponding to the second target video can be converted into the target language audio of the timbre through TTS, so as to obtain the customized timbre dubbing of each second target video clip, thereby meeting the user's demand for personalized video.

[0143] Optionally, after the dubbing of each of the second target video clips is completed, music and music sequels can be generated based on the macro model through text prompts and / or reference audio provided by the user. Furthermore, the generated music and the music uploaded by the user can be used as background music for the generated mashup video.

[0144] Optionally, the splicing of the second target video segments according to the video content of the second target video segments and the subtitles and dubbing to form a mixed video of the original video includes:

[0145] Determine a mixed-cut position corresponding to each of the second target video segments, and determine the video content of each of the second target video segments and a splicing order of the subtitles and dubbing based on the mixed-cut position;

[0146] The mixing position may refer to the joining point of different video clips in the video mixing, or may refer to the specific position of the audio and video clips on the timeline.

[0147] Specifically, the mixing positions corresponding to the target video clips may be determined by a greedy algorithm, and the video contents of the second target video clips and the splicing order of the subtitles and dubbing on the timeline may be determined according to the mixing positions.

[0148] The second target video segments are spliced ​​together based on the splicing order to form a mixed video of the original video.

[0149] In this embodiment, the remixed positions corresponding to the second target video clips can be determined by a greedy algorithm, and the splicing order of the video content, subtitles and dubbing of each second target video clip on the timeline can be determined according to the remixed positions. The video content, subtitles and dubbing of each second target video clip are spliced ​​based on the splicing order, making full use of the multimodal information of the video, such as images and audio, to generate a rich and vivid remixed video.

[0150] Optionally, the determining the mixing position corresponding to each of the second target video segments includes:

[0151] Performing multimodal embedding on the text summary of the text description information corresponding to each of the second target video clips and the key frames in the original video to obtain multimodal embedding features of each of the second target video clips;

[0152] Among them, multimodal embedding can refer to the realization of cross-modal information interaction and fusion by converting different data modalities (such as text, images, audio and video) into a unified vector representation; multimodal embedding features can refer to quantitative features.

[0153] Specifically, through multimodal embedding technology, the text summary of the text description information corresponding to each second target video clip and the key frames in the original video are multimodally embedded to obtain the first vector features of the text summary and the vector features of the key frames, that is, the multimodal embedding features include the first vector features of the text summary and the vector features of the key frames.

[0154] Performing text embedding on the script information and the text summary of the text description information corresponding to each of the second target video segments to obtain the script content embedding features of each of the second target video segments;

[0155] Among them, text embedding can refer to a technology that converts text into a dense vector representation of fixed length. The script content embedding feature also refers to the quantitative feature.

[0156] Specifically, through text embedding technology, the script storyboard description in the script information corresponding to each second target video clip is text embedded, and the text summary of each text description information can also be text embedded to obtain the vector features of the script storyboard description and the second vector features of the text summary, that is, the script content embedding features can include the vector features of the script storyboard description and the second vector features of the text summary.

[0157] Calculating the similarity between the multimodal embedding feature of each of the second target video segments and the script content embedding feature;

[0158] Specifically, the similarity between the multimodal embedding features of each second target video segment and the embedding features of the script content is calculated by a numerical similarity calculation method, such as cosine similarity. Figure 4 A specific flow chart of a video mixing method provided in Embodiment 2 of the present invention; Figure 4 As shown, the description of each script storyboard (i.e., the script storyboard description), the text summary of each video clip (i.e., the text summary of the text description information corresponding to each video clip), and the scene key frames (i.e., the key frames in the original video) are respectively text-embedded or multimodally embedded, and converted into a unified vector representation. The cosine similarity formula is used to calculate the similarity between the two multimodal embedding features of each second target video clip and the similarity between the two script content embedding features, and all the calculated similarity values ​​are organized into a matrix, that is, similarity matrix A and similarity matrix B are obtained respectively.

[0159] Based on the maximum similarity value, a non-repetitive mixed-cut position allocation is performed on each of the second target video segments through a greedy algorithm to determine the mixed-cut position corresponding to each of the second target video segments.

[0160] Specifically, a greedy algorithm can be used to determine the mixing position of each second target video segment with the maximum similarity as the goal, so as to ensure that a second target video segment only appears in one mixing position. Figure 4 As shown, x second target video clips are assigned to n mixed-cut positions without duplication, and each mixed-cut position can be assigned k second target video clips, which requires x≥n*k to ensure that each second target video clip appears in only one mixed-cut position.

[0161] Furthermore, if the number of target video clips is small, that is, x is small, the number of second target video clips allocated to each storyboard (i.e., each mixed-cut position) is less than or equal to k, and the requirement of the number of video clips at the mixed-cut position cannot be met. Then, the target video clips can be directly sorted from large to small according to the similarity between the multimodal embedding features and the script content embedding features, and allocated to each mixed-cut position in sequence, and duplication is allowed. That is, only when the mixed-cut position determined according to the script storyboard description is less than or equal to the number of target video clips, can the allocation be performed based on the greedy algorithm with as little duplication as possible.

[0162] For example, the similarity matrix A and the similarity matrix B (a normal matrix with the same variance and mean) can be directly combined |A|B|. The specific allocation schemes of the second target video segments include:

[0163] Input check: If there is no mix position (matrix row number is 0) or no second target video segment (column number is 0), an error is reported.

[0164] Special case handling: If the number of mixed-cut positions exceeds the number of second target video clips, the similarity ranking of all second target video clips at each mixed-cut position is directly returned (from high to low). Since it is not guaranteed that each mixed-cut position can be assigned a unique second target video clip, only a recommended order is provided.

[0165] Initialization preparation: Create a list (assigned_choices) to record the second target video segment assigned to each mixed-cut position. The initial list is empty.

[0166] Create a flag array (assigned_positions) to record whether each second target video segment has been assigned.

[0167] Greedy allocation process: Perform at most k rounds of allocation: Try to allocate a second target video segment to each mixed-cut position in each round.

[0168] Traverse each mixed-cut position: if all second target video clips have been allocated, end early. Specifically, find the second target video clip with the highest similarity among the options that are not allocated to the current mixed-cut position; and assign the second target video clip to the current mixed-cut position and mark it as allocated. Repeat the above search, allocation and marking process k times to ensure that each mixed-cut position can obtain at most k second target video clips, or until the second target video clips are used up.

[0169] Output result: Return the list of options assigned to each mixed-cut position, the length of the list does not exceed k, and the second target video clips are arranged in the order of allocation. This allocation algorithm can be similar to the "snatching seats" game: each student (i.e., mixed-cut position) takes turns to choose his favorite seat (i.e., the second target video clip). Once a seat is taken, no one else can choose it. In each round, each mixed-cut position selects the current best (i.e., the most similar) remaining second target video clip, and a total of k rounds are performed to ensure that each mixed-cut position grabs at most k second target video clips.

[0170] In a specific embodiment, assuming there are 2 mixed-cut positions, 3 second target video segments, k=2: First round: Mixed-cut position 1 selects the most similar second target video segment A, and mixed-cut position 2 selects the second most similar second target video segment B. Second round: Mixed-cut position 1 selects the remaining second target video segment C, and position 2 has no second target video segment to choose from. Result: Mixed-cut position 1 is assigned to the second target video segment [A, C], and position 2 is assigned to the second target video segment [B].

[0171] like Figure 4As shown, after greedy allocation, if the duration of the second target video segment is much shorter than the corresponding dubbing duration, the end time point of the video segment can be extended backwards with reference to the key frame time point or another second target video segment that can be matched can be spliced ​​to expand the video duration. Combining the video content, subtitles and dubbing of each second target video segment, as well as the determined mixed-cut positions of each second target video segment, a list of newly edited video segments (i.e., each second target video segment) is obtained, and then a mixed-cut video of the original video is obtained by splicing and synthesis.

[0172] In this embodiment, the text summary of the text description information corresponding to each second target video clip and the key frames in the original video are multimodally embedded to obtain the multimodal embedding features of each second target video clip; the script information corresponding to each second target video clip is text-embedded to obtain the script content embedding features of each second target video clip; the similarity between the multimodal embedding features and the script content embedding features of each second target video clip is calculated; based on the maximum similarity value, the second target video clips are assigned non-repetitive mixed-cut positions through a greedy algorithm, the mixed-cut positions corresponding to each second target video clip are determined, and a mixed-cut result with richness and vividness is generated, which effectively improves the expressiveness of the video.

[0173] Optionally, after determining the remixed positions corresponding to each second target video clip, the remixed positions of each second target video clip can also be adaptively adjusted according to the dubbing time corresponding to each second target video clip to ensure accurate matching of the dubbing with the video screen and enhance the expressive effect of the remixed video.

[0174] In one embodiment, it is not necessary to perform video parsing and video segment matching. Instead, a script containing only voice information is generated directly based on the user prompt text through a large model, and translation and TTS are performed. The user selects the video clips corresponding to each script and directly generates a mixed cut. The generation speed is fast and the cost is low, and the specific selection can be made according to the actual situation.

[0175] Embodiment 3

[0176] Figure 6 This is a schematic diagram of the structure of a video mixing device provided in Embodiment 3 of the present invention.

[0177] like Figure 6 As shown, the device comprises:

[0178] A determination module 310, configured to divide the acquired original video into a plurality of video segments, and determine a subtitle area of ​​each of the video segments;

[0179] An acquisition module 320 is used to acquire text description information corresponding to each of the video clips, and select a preset number of first target video clips from each of the video clips according to the text description information and a user prompt text;

[0180] A generating module 330, configured to generate script information of each of the first target video segments based on the user prompt text, select second target video segments from the first target video segments according to the verified script information, and generate subtitles and dubbing of each of the second target video segments;

[0181] The creation and synthesis module 340 is used to splice each of the second target video clips according to the video content of each of the second target video clips and the subtitles and dubbing to form a mixed video of the original video.

[0182] Optionally, the determination module 310 is specifically configured to:

[0183] Performing audio transcription on the preprocessed original video to determine the audio transcription text corresponding to the preprocessed original video;

[0184] Identifying key frames in the original video and matching the key frames with the audio transcript;

[0185] If there is a match, dividing the acquired original video into a plurality of video segments based on the audio transcription text;

[0186] If there is no match, the acquired original video is divided into multiple video segments based on the key frames.

[0187] Optionally, the determination module 310 is further configured to:

[0188] Performing optical character recognition on the original video to obtain a recognition result, matching the recognition result with the audio transcription text corresponding to each of the video clips, and determining a text box corresponding to each of the video clips;

[0189] Anomalies are filtered and bounding rectangles are calculated for each of the text boxes to determine the subtitle area corresponding to each of the video clips.

[0190] Optionally, the acquisition module 320 is specifically configured to:

[0191] Acquire text description information corresponding to each of the video clips based on the large model;

[0192] Calculating the similarity between the text summary of each of the text description information and the user prompt text;

[0193] A preset number of video segments are selected in descending order of similarity as first target video segments.

[0194] Optionally, the generating module 330 includes:

[0195] A first generating unit is configured to generate script information corresponding to each of the first target video segments based on a large model according to each of the first target video segments and the user prompt text, wherein the script information includes: a script storyboard description and a script storyboard dubbing content;

[0196] Verifying the script information, and selecting a second target video segment from the first target video segment according to the verified script information;

[0197] A second generating unit, configured to translate the script storyboard dubbing content corresponding to the second target video into a target language, and generate subtitles for each of the second target video segments;

[0198] The third generating unit is used to convert the script storyboard dubbing content corresponding to the second target video into the audio of the target language, and generate the dubbing of each second target video clip.

[0199] Optionally, the third generating unit is specifically used for:

[0200] Reproduce the sound based on the pre-uploaded voice clips;

[0201] Based on the timbre, the script storyboard dubbing content corresponding to the second target video is converted into the target language audio of the timbre to generate the dubbing of each of the second target video clips.

[0202] Optionally, the splicing module 340 includes:

[0203] a determining unit, configured to determine a mixed-cut position corresponding to each of the second target video segments, and determine the video content of each of the second target video segments and a splicing order of the subtitles and dubbing based on the mixed-cut position;

[0204] A splicing unit is used to splice each of the second target video segments based on the splicing order to form a mixed video of the original video.

[0205] The video remixing device provided in the embodiment of the present invention can execute the video remixing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0206] Embodiment 4

[0207] Figure 7A schematic diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0208] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 so that the at least one processor 11 can perform the method provided by the present invention. Preferably, the electronic device 10 can be a heterogeneous computing device, and the processor 11 can be a multi-threaded general-purpose processor or a computing acceleration processor. Exemplarily, the heterogeneous computing device may include: at least one multi-threaded general-purpose processor and one computing acceleration processor.

[0209] The processor 11 can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 12 or a computer program loaded from a storage unit 18 into a random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0210] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0211] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a video mixing method.

[0212] In some embodiments, the video mixing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the video mixing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the video mixing method in any other appropriate manner (e.g., by means of firmware).

[0213] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0214] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0215] In the context of the present invention, a computer-readable storage medium stores computer instructions, which are used to implement the video mixing method provided by the present invention when the processor executes. A computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0216] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD monitor)) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user may provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0217] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN), blockchain network, and the Internet.

[0218] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0220] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video mixing method, characterized in that: include: Dividing the acquired original video into a plurality of video segments, and determining a subtitle area of ​​each of the video segments; Acquire text description information corresponding to each of the video clips, and select a preset number of first target video clips from each of the video clips according to the text description information and user prompt text; Generate script information of each first target video segment based on the user prompt text, select second target video segments from the first target video segments according to the verified script information, and generate subtitles and dubbing of each second target video segment; According to the video content of each of the second target video clips and the subtitles and dubbing, each of the second target video clips is spliced ​​to form a mixed video of the original video.

2. The method according to claim 1, characterized in that Divide the acquired original video into multiple video segments, including: Performing audio transcription on the preprocessed original video to determine the audio transcription text corresponding to the preprocessed original video; Identifying key frames in the original video and matching the key frames with the audio transcript; If there is a match, dividing the acquired original video into a plurality of video segments based on the audio transcription text; If there is no match, the acquired original video is divided into multiple video segments based on the key frames.

3. The method according to claim 1, characterized in that Determining the subtitle area of ​​each of the video clips includes: Performing optical character recognition on the original video to obtain a recognition result, matching the recognition result with the audio transcription text corresponding to each of the video clips, and determining a text box corresponding to each of the video clips; Anomalies are filtered and bounding rectangles are calculated for each of the text boxes to determine the subtitle area corresponding to each of the video clips.

4. The method according to claim 1, characterized in that: Acquiring text description information corresponding to each of the video clips, and filtering a preset number of first target video clips from each of the video clips according to the text description information and the user prompt text, including: Acquire text description information corresponding to each of the video clips based on the large model; Calculating the similarity between the text summary of each of the text description information and the user prompt text; A preset number of video segments are selected in descending order of similarity as first target video segments.

5. The method according to claim 4, characterized in that Generating script information of each first target video segment based on the user prompt text, selecting second target video segments from the first target video segments according to the verified script information, and generating subtitles and dubbing of each second target video segment according to the script information, including: Generate script information corresponding to each of the first target video clips based on the large model according to each of the first target video clips and the user prompt text, wherein the script information includes: a script storyboard description and a script storyboard dubbing content; Verifying the script information, and selecting a second target video segment from the first target video segment according to the verified script information; Translating the script storyboard dubbing content corresponding to the second target video into a target language to generate subtitles for each of the second target video clips; The storyboard dubbing content script corresponding to the second target video is converted into audio in the target language to generate dubbing for each of the second target video clips.

6. The method according to claim 5, characterized in that Converting the storyboard dubbing content script corresponding to the second target video into the audio of the target language to generate the dubbing of each second target video segment includes: Reproduce the sound based on the pre-uploaded voice clips; Based on the timbre, the script storyboard dubbing content corresponding to the second target video is converted into the target language audio of the timbre to generate the dubbing of each of the second target video clips.

7. The method according to claim 1, characterized in that The step of splicing the second target video segments according to the video contents of the second target video segments and the subtitles and dubbing to form a mixed video of the original video includes: Determine a mixed-cut position corresponding to each of the second target video segments, and determine the video content of each of the second target video segments and a splicing order of the subtitles and dubbing based on the mixed-cut position; The second target video segments are spliced ​​together based on the splicing order to form a mixed video of the original video.

8. A video mixing device, characterized in that: include: A determination module, used to divide the acquired original video into a plurality of video segments, and determine the subtitle area of ​​each of the video segments; An acquisition module, configured to acquire text description information corresponding to each of the video clips, and select a preset number of first target video clips from each of the video clips according to the text description information and a user prompt text; A generating module, configured to generate script information of each of the first target video segments based on the user prompt text, select second target video segments from the first target video segments according to the verified script information, and generate subtitles and dubbing of each of the second target video segments; The creation and synthesis module is used to splice each of the second target video clips according to the video content of each of the second target video clips and the subtitles and dubbing to form a mixed video of the original video.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video mixing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video mixing method according to any one of claims 1 to 7 when executed.

Images