Video mixing and cutting method and related device
Through automated video storyboarding, segmentation and splicing technology, the problem of low efficiency of artificial video mixing and cutting is solved, efficient video mixing and cutting processing is achieved, and mixed-cut videos that are in line with the video script are generated.
Patent Information
- Application Number
- CN202411159990.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-01
AI Technical Summary
The existing artificial video mixing and cutting methods are relatively inefficient, especially when the video material is large, it requires a lot of manpower and time.
By obtaining video scripts and materials, performing storyboarding and segmenting, filtering semantic-related video clips, and automatically splicing to form mixed-cut videos, and using speech recognition, picture difference and vectorization technologies to achieve automated processing.
No manual participation is required, which significantly saves manpower and time, improves the efficiency of video mixed clipping, and the generated mixed clipping video is in line with the original script.
Smart Images

Figure CN118890499A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video processing technology, and in particular to a video mixing method and related devices. Background Art
[0002] Video remix, also known as mixed editing, is a creative technique that combines multiple video clips together through editing, splicing, etc. to form a new video work. Through video remix, you can create a video with a unique artistic effect, so that you can realize the creative expression of the creator. In addition, combining videos at different time points, different scenes, and different angles can also increase the diversity, layering, and viewing experience of the video.
[0003] The current video remixing method is mainly based on manual video remixing, that is, the video clips in the video material are edited and spliced by humans. Although the manual video remixing method can create new video works, the efficiency of this method is low, and when the scale of the video material is large, the disadvantages of this method will be doubled. Summary of the invention
[0004] In view of this, the present application provides a video mixing method and related devices, which are used to solve the problem of low efficiency of the existing manual video mixing methods. The technical solution is as follows:
[0005] The first aspect of the present application provides a video mixing method, comprising:
[0006] Obtaining a video script and video materials related to the video script;
[0007] Storyboarding the video script to obtain a plurality of storyboard scripts;
[0008] Segmenting the video material to obtain a plurality of video clips;
[0009] Selecting video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively;
[0010] splicing the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script;
[0011] The videos corresponding to the several storyboard scripts are spliced together to obtain a mixed video corresponding to the video scripts.
[0012] In a possible implementation, the video material is segmented to obtain a plurality of video clips, including:
[0013] Performing speech recognition on the commentary audio of the video material to obtain speech recognition text; segmenting the speech recognition text to obtain a plurality of text segments; acquiring a video segment corresponding to each text segment from the video material to obtain a plurality of video segments;
[0014] or,
[0015] Determine the picture difference between adjacent video frames in the video material; determine the segmentation position according to the picture difference between adjacent video frames in the video material; segment the video material according to the segmentation position to obtain a plurality of video clips.
[0016] In a possible implementation, selecting video segments semantically related to each storyboard script from the plurality of video segments to obtain video segment sets corresponding to the plurality of storyboard scripts respectively includes:
[0017] Obtaining the storyboard descriptions corresponding to the plurality of storyboard scripts, and obtaining the video descriptions corresponding to the plurality of video clips;
[0018] Determining the semantic relevance of each storyboard script and each video clip according to the storyboard descriptions corresponding to the plurality of storyboard scripts and the video descriptions corresponding to the plurality of video clips;
[0019] According to the semantic relevance between each storyboard script and each video clip, video clips semantically related to each storyboard script are screened from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively.
[0020] In a possible implementation manner, obtaining video descriptions corresponding to the plurality of video clips respectively includes:
[0021] For each video clip:
[0022] Extracting key video frames from the video clip, wherein the key video frames are sudden changes in the video picture;
[0023] A picture description of the key video frame is obtained as a video description corresponding to the video clip.
[0024] In a possible implementation, determining the semantic relevance between each storyboard script and each video clip according to the storyboard descriptions respectively corresponding to the plurality of storyboard scripts and the video descriptions respectively corresponding to the plurality of video clips includes:
[0025] Vectorizing the storyboard descriptions corresponding to the plurality of storyboard scripts to obtain storyboard description vectors corresponding to the plurality of storyboard scripts, and vectorizing the video descriptions corresponding to the plurality of video clips to obtain video description vectors corresponding to the plurality of video clips;
[0026] For each storyboard script: calculate the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to each video clip, and obtain the semantic relevance between the storyboard script and each video clip, wherein the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to a video clip is used as the semantic relevance between the storyboard script and the video clip.
[0027] In a possible implementation, the step of selecting video segments semantically related to each storyboard script from the plurality of video segments according to the semantic relevance between each storyboard script and each video segment, and obtaining video segment sets corresponding to the plurality of storyboard scripts respectively, includes:
[0028] For each storyboard:
[0029] Determine the video length corresponding to the storyboard script;
[0030] According to the semantic relevance between the storyboard script and each video clip, and in combination with the video length corresponding to the storyboard script, video clips semantically related to the storyboard script are screened from the plurality of video clips to obtain a set of video clips corresponding to the storyboard script.
[0031] In a possible implementation, determining the video duration corresponding to the storyboard script includes:
[0032] Calculate the ratio of the text-to-speech duration of the storyboard script to the text-to-speech duration of the video script to obtain the video duration ratio corresponding to the storyboard script;
[0033] The video duration corresponding to the storyboard script is determined according to the proportion of the video duration corresponding to the storyboard script and the target video duration.
[0034] In a possible implementation, the step of selecting video clips semantically related to the storyboard script from the plurality of video clips based on the semantic relevance between the storyboard script and each video clip and combining the video duration corresponding to the storyboard script to obtain a set of video clips corresponding to the storyboard script includes:
[0035] Video clips are selected from the plurality of video clips in order of the semantic relevance between the storyboard script and each video clip from high to low, until the total duration of all the selected video clips is equal to or greater than the duration of the video corresponding to the storyboard script, and the selected video clips constitute a video clip set corresponding to the storyboard script.
[0036] In a possible implementation, the step of splicing the video segments in the video segment set corresponding to each storyboard script includes:
[0037] For each storyboard:
[0038] Determining a splicing order of the video clips in the video clip set corresponding to the storyboard script according to the semantic relevance of each video clip in the video clip set corresponding to the storyboard script and the storyboard script;
[0039] According to the determined splicing order, each video clip in the video clip set corresponding to the storyboard script is spliced.
[0040] In a possible implementation, the video mixing method further includes:
[0041] The mixed video corresponding to the video script is post-processed, wherein the post-processing includes one or more of the following processes: adding commentary audio, adding subtitles, adding background music, and adding transition effects.
[0042] A second aspect of the present application provides a video mixing and editing device, comprising: a data acquisition module, a video script storyboard module, a video material segmentation module, a video clip screening module, a video clip splicing module and a video splicing module;
[0043] The data acquisition module is used to acquire the video script and video materials related to the video script;
[0044] The video script storyboard module is used to storyboard the video script to obtain a plurality of storyboard scripts;
[0045] The video material segmentation module is used to segment the video material to obtain a plurality of video clips;
[0046] The video clip screening module is used to screen video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively;
[0047] The video clip splicing module is used to splice the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script;
[0048] The video splicing module is used to splice the videos corresponding to the several storyboard scripts to obtain a mixed video corresponding to the video scripts.
[0049] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0050] The memory is used to store computer programs;
[0051] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned video mixing methods.
[0052] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned video mixing methods.
[0053] A fifth aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of any one of the above-mentioned video mixing methods.
[0054] By means of the above technical scheme, the video remixing method provided by the present application first obtains the video script and the video material related to the video script, then storyboards the video script to obtain several storyboards, and divides the video material to obtain several video clips, then selects the video clips semantically related to each storyboard from the several video clips to obtain the video clip sets corresponding to the several storyboards, and then splices the video clips in the video clip set corresponding to each storyboard to obtain the videos corresponding to the several storyboards, and finally splices the videos corresponding to the several storyboards to obtain the mixed video corresponding to the video script. The video remixing method provided by the present application can automatically storyboard the video script and divide the video material, can automatically select the video clips semantically related to each storyboard, can automatically splice the video clips in the video clip set corresponding to each storyboard, and splice the videos corresponding to each storyboard, and finally obtain a mixed video that matches the video script. The video remixing method provided by the present application does not require manual participation, and compared with the manual video remixing method, it saves manpower and time, and greatly improves the efficiency of video remixing. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0056] Figure 1 A schematic diagram of a system architecture involved in this application;
[0057] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;
[0058] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;
[0059] Figure 4 A schematic diagram of a video mixing method provided in an embodiment of the present application;
[0060] Figure 5 A schematic diagram of a process for selecting video clips semantically related to each storyboard script from a plurality of video clips to obtain a set of video clips corresponding to the plurality of storyboard scripts, provided in an embodiment of the present application;
[0061] Figure 6 A schematic diagram of the structure of a video mixing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0062] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0063] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0064] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0065] Before introducing the video mixing method provided by the present application, the system architecture involved in the present application is first described.
[0066] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal device 101 and a server 102, and the terminal device 101 may interact with the server 102 through a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The example includes a server. The terminal device 101 can obtain the video script and the video material related to the video script, and transmit the video script and the video material to the server 102 through the network. The server 102 processes the video script and the video material using the video remixing method provided in the present application to obtain the remixed video corresponding to the video script, and then sends the remixed video corresponding to the video script to the terminal, and the terminal outputs the remixed video corresponding to the video script.
[0067] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal may use the video remixing method provided in the present application to process the video script and the video material to obtain a remixed video corresponding to the video script, and then output the remixed video corresponding to the video script.
[0068] Next, the product form of the above-mentioned terminal is described.
[0069] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.
[0070] Figure 2 An optional hardware structure diagram of a terminal is shown.
[0071] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.
[0072] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0073] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.
[0074] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.
[0075] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.
[0076] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.
[0077] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.
[0078] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.
[0079] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.
[0080] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.
[0081] Next, the product form of the above server is described.
[0082] Figure 3 A structural diagram of the above server is provided, such as Figure 3 As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.
[0083] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0084] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0085] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).
[0086] The memory 304 may be used to store software codes related to the video mixing method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.
[0087] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
[0088] The current video remixing method is mainly a manual video remixing method, that is, a manual video editing tool is used to perform operations such as editing and splicing on video materials according to a video script to obtain a new video work. However, manually using video editing tools to perform operations such as editing and splicing on video materials consumes a relatively large amount of human resources and time. In view of the problems existing in the manual video remixing method, the inventor of this case attempted to propose a method that can automatically realize video remixing to overcome the defects of the existing video remixing method. To this end, research was conducted. Through continuous research, a video remixing method was finally proposed. The video remixing method does not require manual participation. Compared with the manual video remixing method, it can save human resources and time and improve the efficiency of video remixing.
[0089] Next, the video mixing method provided by the present application is introduced through the following embodiments.
[0090] See also Figure 4, shows a schematic flow chart of a video mixing method provided in an embodiment of the present application, which may include:
[0091] Step S401: Obtain a video script and video materials related to the video script.
[0092] The video script may be a manually written video script or an automatically generated video script.
[0093] In a possible implementation, the process of automatically generating a video script may include: generating a video script based on a pre-trained large language model and based on the video script generation requirements. Specifically, a prompt containing the video script generation requirements (such as a prompt containing information such as objects involved in the video script, the style of the video script, and the number of words in the video script) is input into the pre-trained large language model, and the large language model generates and outputs the video script based on the input prompt.
[0094] In this embodiment, the video material related to the video script can be a video material manually shot according to the video script, or can be a video material collected according to the video script. The video material can include one or more video resources, usually including multiple video resources.
[0095] Step S402: storyboard the video script to obtain a plurality of storyboard scripts.
[0096] Among them, each storyboard script is a storyboard video script.
[0097] The process of storyboarding a video script to obtain a plurality of storyboard scripts may include: structurally splitting the video script to obtain a plurality of video script segments, and using each video script segment as a storyboard script to obtain a plurality of storyboard scripts.
[0098] Exemplarily, the video script is structured and split to obtain three video script segments. Assuming that the three video script segments are video script segment 1, video script segment 2, and video script segment 3, video script segment 1 is used as storyboard script 1 (video script for storyboard 1), video script segment 2 is used as storyboard script 2 (video script for storyboard 2), and video script segment 3 is used as storyboard script 3 (video script for storyboard 3), thereby obtaining three storyboard scripts, i.e., three storyboard video scripts.
[0099] In a possible implementation, the video script obtained in step S401 can be structurally split based on the pre-trained large language model (i.e., the video script is structurally split using the text understanding ability of the large language model) to obtain a number of storyboards. Specifically, the prompt instruction prompt containing the video script is input into the pre-trained large language model, and the large language model structurally splits the video script according to the prompt instruction prompt, and outputs a number of video script segments obtained by the splitting.
[0100] It should be noted that this embodiment is not limited to the structured splitting of video scripts based on a pre-trained large language model, and other solutions that can achieve the structured splitting of video scripts are also applicable to this application.
[0101] Step S403: Segment the video material to obtain a number of video clips.
[0102] There are many ways to segment the video material to obtain a number of video clips. This embodiment provides two ways to achieve this.
[0103] The first implementation method of segmenting the video material to obtain a plurality of video clips is as follows: performing speech recognition on the commentary audio of the video material to obtain speech recognition text; segmenting the speech recognition text to obtain a plurality of text clips; and obtaining the video clip corresponding to each text clip from the video material, thereby obtaining a plurality of video clips.
[0104] Optionally, after obtaining the speech recognition text, the speech recognition text may be structurally split based on a pre-trained large language model to obtain a plurality of text segments. Specifically, the prompt instruction prompt containing the speech recognition text is input into the pre-trained large language model, and the pre-trained large language model structurally splits the speech recognition text according to the prompt instruction prompt, and outputs a plurality of text segments obtained by the splitting.
[0105] The second implementation method of segmenting the video material to obtain a plurality of video clips is as follows: determining the picture difference between adjacent video frames in the video material; determining the segmentation position according to the picture difference between adjacent video frames in the video material; and segmenting the video material according to the segmentation position to obtain a plurality of video clips. When determining the segmentation position according to the picture difference between adjacent video frames in the video material, the position between two video frames whose picture difference is greater than a preset picture difference threshold can be determined as the segmentation position.
[0106] Exemplarily, a video resource includes 20 video frames. By determining the picture differences between adjacent video frames, 19 picture differences can be obtained. Assuming that the picture difference between the 10th video frame and the 11th video frame is greater than a preset picture difference threshold, the video frame is split between the 10th video frame and the 11th video frame.
[0107] Step S404: Filter video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts.
[0108] For each storyboard script, video clips semantically related to the storyboard script are screened from a number of video clips, and the screened video clips form a video clip set corresponding to the storyboard script, thereby obtaining video clip sets corresponding to the several storyboard scripts.
[0109] Step S405: splicing the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script.
[0110] For each storyboard script, the video clips in the video clip set corresponding to the storyboard script are spliced together to obtain the video corresponding to the storyboard script.
[0111] Exemplarily, a video script is storyboarded to obtain three storyboards, namely storyboard 1, storyboard 2, and storyboard 3, wherein the set of video clips corresponding to storyboard 1 includes three video clips, the set of video clips corresponding to storyboard 2 includes five video clips, and the set of video clips corresponding to storyboard 3 includes four video clips. Then, the three video clips contained in the set of video clips corresponding to storyboard 1 are spliced together to obtain the video corresponding to storyboard 1, the five video clips contained in the set of video clips corresponding to storyboard 2 are spliced together to obtain the video corresponding to storyboard 2, and the four video clips contained in the set of video clips corresponding to storyboard 3 are spliced together to obtain the video corresponding to storyboard 3.
[0112] Through step S405, the videos corresponding to the several storyboard scripts can be obtained.
[0113] Step S406: splice the videos corresponding to the several storyboard scripts to obtain a mixed video corresponding to the video scripts.
[0114] After obtaining the videos corresponding to several storyboard scripts, the videos corresponding to the several storyboard scripts can be spliced together to obtain a mixed video corresponding to the entire video script.
[0115] Exemplarily, there are three storyboards, namely storyboard 1, storyboard 2, and storyboard 3. After obtaining the videos corresponding to storyboard 1, storyboard 2, and storyboard 3, respectively, the videos corresponding to storyboard 1, storyboard 2, and storyboard 3 are spliced in the order of video corresponding to storyboard 1, video corresponding to storyboard 2, and video corresponding to storyboard 3, that is, the tail of the video corresponding to storyboard 1 is spliced with the head of the video corresponding to storyboard 2, and the tail of the video corresponding to storyboard 2 is spliced with the head of the video corresponding to storyboard 3, so as to obtain the mixed video corresponding to the entire video script.
[0116] Optionally, after obtaining the mixed-cut video corresponding to the entire video script through steps S401 to S406, the mixed-cut video corresponding to the entire video script can be post-processed, wherein the post-processing may include but is not limited to part or all of the following processing: adding commentary audio, adding subtitles, adding background music, adding transition special effects, etc.
[0117] The video remixing method provided in the embodiment of the present application first obtains a video script and video materials related to the video script, then storyboards the video script to obtain several storyboards, and divides the video materials to obtain several video clips, then screens the video clips semantically related to each storyboard from the several video clips to obtain a set of video clips corresponding to several storyboards, and then splices the video clips in the video clip set corresponding to each storyboard to obtain videos corresponding to several storyboards, and finally splices the videos corresponding to several storyboards to obtain a remixed video corresponding to the video script. The video remixing method provided in the embodiment of the present application can automatically storyboard the video script and divide the video materials, automatically screen the video clips semantically related to each storyboard, automatically splice the video clips in the video clip set corresponding to each storyboard, and splice the videos corresponding to each storyboard, and finally obtain a remixed video that matches the video script. The video remixing method provided in the embodiment of the present application does not require manual participation, which saves manpower and time compared to the manual video remixing method, and improves the efficiency of video remixing.
[0118] In another embodiment of the present application, the specific implementation process of "Step S404: screening video clips semantically related to each storyboard script from a plurality of video clips to obtain a set of video clips corresponding to a plurality of storyboard scripts" in the above embodiment is introduced.
[0119] See also Figure 5 , showing a flow chart of selecting video clips semantically related to each storyboard script from a plurality of video clips to obtain a set of video clips corresponding to the plurality of storyboard scripts, which may include:
[0120] Step S501: Obtain storyboard descriptions corresponding to a plurality of storyboard scripts, and obtain video descriptions corresponding to a plurality of video clips.
[0121] Among them, the storyboard description corresponding to a storyboard script is a description of the storyboard corresponding to the storyboard script, and the video description corresponding to a video clip is a description of the picture of the video clip.
[0122] In a possible implementation, the storyboard descriptions corresponding to the several storyboard scripts can be obtained based on the pre-trained large language model (the storyboard descriptions corresponding to the several storyboard scripts can be generated using the text generation capability of the large language model). Specifically, the prompt instruction prompt containing each storyboard script can be input into the large language model, and the large language model generates the storyboard description according to the prompt instruction prompt.
[0123] It should be noted that the process of obtaining the storyboard description corresponding to the storyboard script is the process of visualizing the text content of the storyboard script. For example, the storyboard script is "Wow, this bowl of noodles is so fragrant", and the storyboard description corresponding to the storyboard script is "The camera focuses on a bowl of steaming noodles, emitting a mouth-watering aroma".
[0124] In addition to obtaining the storyboard descriptions corresponding to several storyboard scripts, it is also necessary to obtain the video descriptions corresponding to several video clips. In one possible implementation, the process of obtaining the video descriptions corresponding to several video clips may include: for each video clip, extracting a key video frame (may be one or more) from the video clip, obtaining the screen description of the key video frame as the video description corresponding to the video clip, so as to obtain the video descriptions corresponding to several video clips. Among them, the key video frame is a sudden change frame of the video screen. It should be noted that this embodiment is not limited to obtaining the screen description of the key video frame in the video clip as the video description corresponding to the video clip, and the screen description of all video frames in the video clip may also be obtained as the video description corresponding to the video clip.
[0125] In one possible implementation, a pre-trained image-text alignment multimodal model may be used to obtain a screen description of a key video frame. Specifically, the key video frame is input into a pre-trained image-text alignment multimodal model, and the image-text alignment multimodal model generates and outputs a screen description of the key video frame. It should be noted that this embodiment is not limited to using a pre-trained image-text alignment multimodal model to obtain a screen description of a key video frame, and other solutions that can obtain a screen description of a key video frame are also applicable to this application.
[0126] Step S502: Determine the semantic relevance between each storyboard script and each video clip according to the storyboard descriptions corresponding to the plurality of storyboard scripts and the video descriptions corresponding to the plurality of video clips.
[0127] In a possible implementation, according to the storyboard descriptions corresponding to the plurality of storyboard scripts and the video descriptions corresponding to the plurality of video clips, the process of determining the semantic relevance between each storyboard script and each video clip may include:
[0128] Step S5021, vectorize the storyboard descriptions corresponding to the several storyboard scripts to obtain the storyboard description vectors corresponding to the several storyboard scripts, and vectorize the video descriptions corresponding to the several video clips to obtain the video description vectors corresponding to the several video clips. In a possible implementation, the storyboard descriptions corresponding to the several storyboard scripts can be vectorized based on the text embedding model (the storyboard descriptions corresponding to the several storyboard scripts can be vectorized using the text vectorization capability of the text embedding model) to obtain the storyboard description vectors corresponding to the several storyboard scripts. Similarly, the video descriptions corresponding to the several video clips can also be vectorized based on the text embedding model (the video descriptions corresponding to the several video clips can be vectorized using the text vectorization capability of the text embedding model) to obtain the video description vectors corresponding to the several video clips. Specifically, the storyboard descriptions corresponding to each storyboard script are respectively input into the text embedding model, and the text embedding model vectorizes the input storyboard descriptions and outputs the storyboard description vectors. Similarly, the video descriptions corresponding to each video clip are respectively input into the text embedding model, and the text embedding model vectorizes the input video descriptions and outputs the video description vectors.
[0129] Step S5022: for each storyboard script: calculate the similarity (such as cosine similarity) between the storyboard description vectorization corresponding to the storyboard script and the video description vector corresponding to each video clip, and obtain the semantic relevance between the storyboard script and each video clip.
[0130] Among them, the similarity between the storyboard description vector corresponding to a storyboard script and the video description vector corresponding to a video clip is used as the semantic relevance between the storyboard script and the video clip.
[0131] Step S503: According to the semantic relevance between each storyboard script and each video clip, video clips semantically related to each storyboard script are selected from a plurality of video clips to obtain a set of video clips corresponding to the plurality of storyboard scripts.
[0132] Specifically, according to the semantic relevance between each storyboard and each video segment, the process of selecting video segments semantically related to each storyboard from a plurality of video segments to obtain a set of video segments corresponding to the plurality of storyboards may include:
[0133] Step S5031: For each storyboard script, determine the video duration corresponding to the storyboard script.
[0134] Among them, the video duration corresponding to the storyboard script is the duration that needs to be allocated to the storyboard corresponding to the storyboard script.
[0135] In one possible implementation, the process of determining the video duration corresponding to the storyboard script may include: calculating the ratio of the text-to-speech duration of the storyboard script to the text-to-speech duration of the entire video script as the video duration ratio corresponding to the storyboard script; and determining the video duration corresponding to the storyboard script based on the video duration ratio corresponding to the storyboard script and the target video duration.
[0136] Specifically, the video duration corresponding to a storyboard script can be calculated as follows:
[0137] (1)
[0138] Among them, duration TTS(script of shot) Indicates the duration of the text-to-speech conversion of the storyboard (the duration of converting the storyboard into speech). TTS(script) , which indicates the duration of the text-to-speech conversion of the entire video script (the duration of converting the entire video script into speech). shot Indicates the duration of the video corresponding to the storyboard script (that is, the duration of the storyboard corresponding to the storyboard script). video Indicates the target video length (the length of the final mixed-cut video).
[0139] Step S5032: based on the semantic relevance between the storyboard script and each video clip and in combination with the video duration corresponding to the storyboard script, video clips semantically related to the storyboard script are screened from a plurality of video clips to obtain a set of video clips corresponding to the storyboard script.
[0140] According to the semantic relevance between the storyboard script and each video clip, and in combination with the video duration corresponding to the storyboard script, video clips semantically related to the storyboard script are screened from a plurality of video clips to obtain a video clip set corresponding to the storyboard script, which process may include: selecting video clips from a plurality of video clips in descending order of the semantic relevance between the storyboard script and each video clip, until the total duration of all the selected video clips is equal to or greater than the video duration corresponding to the storyboard script, and the selected video clips constitute the video clip set corresponding to the storyboard script.
[0141] For example, there are three storyboards, namely storyboard 1, storyboard 2, and storyboard 3, and there are six video clips, namely video clip 1, video clip 2, video clip 3, ..., video clip 6. For storyboard 1, its semantic relevance to video clip 1 is s11, its semantic relevance to video clip 2 is s12, its semantic relevance to video clip 3 is s3, ..., its semantic relevance to video clip 6 is s6. Assume that s2>s5>s6>s3>s4> s1, first select video clip 2, assuming that the duration of video clip 2 is less than the duration of the video corresponding to storyboard script 1, then select video clip 5, assuming that the total duration of video clip 2 and video clip 5 is less than the duration of the video corresponding to storyboard script 1, then select video clip 6, assuming that the total duration of video clip 2, video clip 5, and video clip 6 is greater than the duration of the video corresponding to storyboard script 1, then end the selection, and combine video clip 2, video clip 5, and video clip 6 to form a video clip set corresponding to storyboard script 1. The video clip set corresponding to storyboard script 2 and the video clip set corresponding to storyboard script 3 can be obtained in a similar manner.
[0142] In another embodiment of the present application, the specific implementation process of "step S405: splicing the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script" is introduced.
[0143] The specific implementation process of splicing the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script may include:
[0144] Step a1: for each storyboard script, determine the splicing order of each video segment in the video segment set corresponding to the storyboard script according to the semantic relevance between each video segment in the video segment set corresponding to the storyboard script and the storyboard script.
[0145] Specifically, the video clip with the greatest semantic relevance to the storyboard script in the set of video clips corresponding to the storyboard script is placed in the front, the video clip with the second greatest semantic relevance to the storyboard script in the set of video clips corresponding to the storyboard script is placed in the second place, and so on.
[0146] Step a2: splicing the video clips in the video clip set corresponding to the storyboard script according to the determined splicing order to obtain the video corresponding to the storyboard script.
[0147] Exemplarily, there are two storyboards, storyboard 1 and storyboard 2, the set of video clips corresponding to storyboard 1 is {video clip 11, video clip 12, video clip 13}, and the set of video clips corresponding to storyboard 2 is {video clip 21, video clip 22, video clip 23, video clip 24}. Assuming that the semantic relevance between video clip 12 and storyboard 1 is greater than the semantic relevance between video clip 13 and storyboard 1, and the semantic relevance between video clip 13 and storyboard 1 is greater than the semantic relevance between video clip 11 and storyboard 1, it can be determined that the splicing order of the three video clips in the video clip set corresponding to storyboard 1 is video clip 12, video clip 13, and video clip 11, that is, the tail of video clip 12 is spliced with the head of video clip 13, and the tail of video clip 13 is spliced with the head of video clip 11. , so as to obtain the video corresponding to storyboard script 1. Assuming that the semantic relevance between video clip 24 and storyboard script 2 is greater than the semantic relevance between video clip 22 and storyboard script 2, the semantic relevance between video clip 22 and storyboard script 2 is greater than the semantic relevance between video clip 21 and storyboard script 2, and the semantic relevance between video clip 21 and storyboard script 2 is greater than the semantic relevance between video clip 23 and storyboard script 2, then the splicing order of the four video clips in the video clip set corresponding to storyboard script 2 is determined to be video clip 24, video clip 22, video clip 21, and video clip 23, that is, the tail of video clip 24 is spliced with the head of video clip 22, the tail of video clip 22 is spliced with the head of video clip 21, and the tail of video clip 21 is spliced with the head of video clip 23, so as to obtain the video corresponding to storyboard script 2.
[0148] For a storyboard script, the total duration of each video clip in the video clip set corresponding to the storyboard script is usually greater than the duration of the video corresponding to the storyboard script. Therefore, the duration of the video obtained by splicing the video clips in the video clip set corresponding to the storyboard script is greater than the duration of the video corresponding to the storyboard script. In this case, a part of the last video clip (the video clip with the lowest similarity to the storyboard script) can be cut off so that the duration of the video obtained after splicing is equal to the duration of the video corresponding to the storyboard script. It should be noted that cutting can be performed before splicing or after splicing.
[0149] Considering that the video material may contain multiple video resources with different resolutions, and thus, the resolutions of the video clips in the video clip set corresponding to each storyboard script may be different, in view of this, before splicing the video clips in the video clip set corresponding to each storyboard script, the video clips in the video clip set corresponding to each storyboard script may be first processed with a unified resolution (unified to a specified resolution), that is, the resolutions of the video clips in the video clip set corresponding to each storyboard script are made the same, and then the video clips with the same resolution are spliced. It should be noted that for video clips with a small difference in aspect ratio with the target video (the aspect ratio of the final mixed-cut video to be obtained), image interpolation or super-resolution methods can be used for processing, and for video clips with a large difference in aspect ratio with the target video (for example, the final mixed-cut video to be obtained is a horizontal version video, while the video clip is a vertical version video), edge filling can be used for processing, such as black edge filling or Gaussian edge blur filling. Before splicing the video clips in the video clip set corresponding to the storyboard script, in addition to unified resolution processing, unified frame rate processing (unified to a specified frame rate), subtitle removal processing, audio removal processing, etc. can also be performed.
[0150] After obtaining the videos corresponding to each storyboard script, the videos corresponding to each storyboard script can be spliced in the order of each storyboard script, so as to obtain a mixed video corresponding to the entire video script.
[0151] After obtaining the mixed-cut video corresponding to the entire video script, it can be post-processed, for example, adding commentary audio, adding subtitles, adding background music, adding transition effects, etc., to obtain the final mixed-cut video.
[0152] The above describes the video mixing method provided by the embodiment of the present application, and the following describes the device corresponding to the above video mixing method.
[0153] See also Figure 6 , Figure 6 A structural schematic diagram of a video remixing device provided in an embodiment of the present application, the video remixing device may include: a data acquisition module 601, a video script storyboard module 602, a video material segmentation module 603, a video clip screening module 604, a video clip splicing module 605 and a video splicing module 606.
[0154] The data acquisition module 601 is used to acquire the video script and video materials related to the video script.
[0155] The video script storyboard module 602 is used to storyboard the video script to obtain a plurality of storyboard scripts.
[0156] The video material segmentation module 603 is used to segment the video material to obtain a plurality of video clips.
[0157] The video clip screening module 604 is used to screen video clips semantically related to each storyboard script from a plurality of video clips, and obtain a set of video clips corresponding to a plurality of storyboard scripts.
[0158] The video clip splicing module 605 is used to splice the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script.
[0159] The video splicing module 606 is used to splice the videos corresponding to several storyboard scripts to obtain a mixed video corresponding to the video scripts.
[0160] In a possible implementation, when the video material segmentation module 603 segments the video material to obtain a plurality of video segments, it is specifically used to:
[0161] Performing speech recognition on the commentary audio of the video material to obtain speech recognition text; segmenting the speech recognition text to obtain a plurality of text segments; and acquiring a video segment corresponding to each text segment from the video material to obtain a plurality of video segments.
[0162] In another possible implementation, when the video material segmentation module 603 segments the video material to obtain a plurality of video segments, it is specifically used to:
[0163] Determine the picture difference between adjacent video frames in the video material; determine the segmentation position according to the picture difference between adjacent video frames in the video material; segment the video material according to the segmentation position to obtain a plurality of video clips.
[0164] In a possible implementation, the video segment screening module 604 may include: a storyboard description acquisition submodule, a video description acquisition submodule, a semantic relevance determination submodule and a video segment screening submodule.
[0165] The storyboard description acquisition submodule is used to obtain the storyboard descriptions corresponding to several storyboard scripts.
[0166] The video description acquisition submodule is used to acquire video descriptions corresponding to a plurality of video clips.
[0167] The semantic relevance determination submodule is used to determine the semantic relevance between each storyboard script and each video clip according to the storyboard descriptions corresponding to the several storyboard scripts and the video descriptions corresponding to the several video clips.
[0168] The video clip screening submodule is used to screen video clips semantically related to each storyboard script from a number of video clips according to the semantic relevance between each storyboard script and each video clip, so as to obtain a set of video clips corresponding to the several storyboard scripts.
[0169] In a possible implementation, when the video description acquisition submodule acquires the video descriptions corresponding to the plurality of video clips, it is specifically used to:
[0170] For each video clip:
[0171] Extracting key video frames from the video clip, wherein the key video frames are sudden changes in the video picture;
[0172] A picture description of the key video frame is obtained as a video description corresponding to the video clip.
[0173] In a possible implementation, the semantic relevance determination submodule is specifically used to determine the semantic relevance of each storyboard script and each video clip according to the storyboard descriptions corresponding to the plurality of storyboard scripts and the video descriptions corresponding to the plurality of video clips:
[0174] Vectorizing the storyboard descriptions corresponding to the plurality of storyboard scripts to obtain storyboard description vectors corresponding to the plurality of storyboard scripts, and vectorizing the video descriptions corresponding to the plurality of video clips to obtain video description vectors corresponding to the plurality of video clips;
[0175] For each storyboard script: calculate the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to each video clip, and obtain the semantic relevance between the storyboard script and each video clip, wherein the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to a video clip is used as the semantic relevance between the storyboard script and the video clip.
[0176] In a possible implementation, the video segment screening submodule is specifically used to screen video segments semantically related to each storyboard script from a plurality of video segments according to the semantic relevance between each storyboard script and each video segment, and obtain video segment sets corresponding to the plurality of storyboard scripts respectively:
[0177] For each storyboard:
[0178] Determine the video length corresponding to the storyboard script;
[0179] According to the semantic relevance between the storyboard script and each video clip, and in combination with the video duration corresponding to the storyboard script, video clips semantically related to the storyboard script are screened from a number of video clips to obtain a set of video clips corresponding to the storyboard script.
[0180] In a possible implementation, when determining the video duration corresponding to the storyboard script, the video clip screening submodule is specifically used to:
[0181] Calculate the ratio of the text-to-speech duration of the storyboard script to the text-to-speech duration of the video script to obtain the video duration ratio corresponding to the storyboard script;
[0182] The video duration corresponding to the storyboard script is determined according to the proportion of the video duration corresponding to the storyboard script and the target video duration.
[0183] In a possible implementation, the video segment screening submodule screens video segments semantically related to the storyboard script from a plurality of video segments according to the semantic relevance between the storyboard script and each video segment and in combination with the video duration corresponding to the storyboard script to obtain a video segment set corresponding to the storyboard script, specifically for:
[0184] Video clips are selected from the plurality of video clips in order of the semantic relevance between the storyboard script and each video clip from high to low, until the total duration of all the selected video clips is equal to or greater than the duration of the video corresponding to the storyboard script, and the selected video clips constitute a video clip set corresponding to the storyboard script.
[0185] In a possible implementation, when the video segment splicing module 605 splices the video segments in the video segment set corresponding to each storyboard script, it is specifically used to:
[0186] For each storyboard:
[0187] Determining a splicing order of the video clips in the video clip set corresponding to the storyboard script according to the semantic relevance of each video clip in the video clip set corresponding to the storyboard script and the storyboard script;
[0188] According to the determined splicing order, each video clip in the video clip set corresponding to the storyboard script is spliced.
[0189] In a possible implementation, the video mixing device may further include: a mixed video post-processing module.
[0190] The mixed video post-processing module is used to post-process the mixed video corresponding to the video script, wherein the post-processing includes one or more of the following processes: adding commentary audio, adding subtitles, adding background music, and adding transition effects.
[0191] The video mixing device provided in the embodiment of the present application first obtains a video script and video materials related to the video script, then storyboards the video script to obtain several storyboards, and divides the video materials to obtain several video clips, then screens the video clips semantically related to each storyboard script from the several video clips to obtain a video clip set corresponding to several storyboards, and then splices the video clips in the video clip set corresponding to each storyboard script to obtain videos corresponding to several storyboards, and finally splices the videos corresponding to several storyboards to obtain a mixed video corresponding to the video script. The video mixing device provided in the embodiment of the present application can automatically storyboard the video script and divide the video materials, can automatically screen the video clips semantically related to each storyboard script, can automatically splice the video clips in the video clip set corresponding to each storyboard script, and splice the videos corresponding to each storyboard script, and finally obtain a mixed video that matches the video script. The video mixing device provided in the embodiment of the present application does not require manual participation in the process of realizing video mixing, which saves manpower and time compared to the manual video mixing scheme, and improves the efficiency of video mixing.
[0192] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.
[0193] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0194] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;
[0195] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;
[0196] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the video mixing method provided in the above embodiment.
[0197] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the video mixing method provided in the above embodiment.
[0198] The embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the video mixing method provided in the above embodiment.
[0199] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0200] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0201] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0202] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A video mixing method, characterized in that: include: Obtaining a video script and video materials related to the video script; Storyboarding the video script to obtain a plurality of storyboard scripts; Segmenting the video material to obtain a plurality of video clips; Selecting video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively; splicing the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script; The videos corresponding to the several storyboard scripts are spliced together to obtain a mixed video corresponding to the video scripts.
2. The video mixing method according to claim 1, characterized in that: The video material is segmented to obtain a plurality of video clips, including: Performing speech recognition on the commentary audio of the video material to obtain speech recognition text; segmenting the speech recognition text to obtain a plurality of text segments; acquiring a video segment corresponding to each text segment from the video material to obtain a plurality of video segments; or, Determine the picture difference between adjacent video frames in the video material; determine the segmentation position according to the picture difference between adjacent video frames in the video material; segment the video material according to the segmentation position to obtain a plurality of video clips.
3. The video mixing method according to claim 1, characterized in that: The step of selecting video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively includes: Obtaining the storyboard descriptions corresponding to the plurality of storyboard scripts, and obtaining the video descriptions corresponding to the plurality of video clips; Determining the semantic relevance of each storyboard script and each video clip according to the storyboard descriptions corresponding to the plurality of storyboard scripts and the video descriptions corresponding to the plurality of video clips; According to the semantic relevance between each storyboard script and each video clip, video clips semantically related to each storyboard script are screened from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively.
4. The video mixing method according to claim 3, characterized in that: The obtaining of video descriptions corresponding to the plurality of video clips respectively includes: For each video clip: Extracting key video frames from the video clip, wherein the key video frames are sudden changes in the video picture; A picture description of the key video frame is obtained as a video description corresponding to the video clip.
5. The video mixing method according to claim 3, characterized in that: The determining the semantic relevance between each storyboard script and each video clip according to the storyboard descriptions respectively corresponding to the plurality of storyboard scripts and the video descriptions respectively corresponding to the plurality of video clips comprises: Vectorizing the storyboard descriptions corresponding to the plurality of storyboard scripts to obtain storyboard description vectors corresponding to the plurality of storyboard scripts, and vectorizing the video descriptions corresponding to the plurality of video clips to obtain video description vectors corresponding to the plurality of video clips; For each storyboard script: calculate the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to each video clip, and obtain the semantic relevance between the storyboard script and each video clip, wherein the similarity between the storyboard description vector corresponding to the storyboard script and the video description vector corresponding to a video clip is used as the semantic relevance between the storyboard script and the video clip.
6. The video mixing method according to claim 3, characterized in that: The step of selecting video clips semantically related to each storyboard script from the plurality of video clips according to the semantic relevance between each storyboard script and each video clip, and obtaining video clip sets corresponding to the plurality of storyboard scripts respectively, comprises: For each storyboard: Determine the video length corresponding to the storyboard script; According to the semantic relevance between the storyboard script and each video clip, and in combination with the video length corresponding to the storyboard script, video clips semantically related to the storyboard script are screened from the plurality of video clips to obtain a set of video clips corresponding to the storyboard script.
7. The video mixing method according to claim 6, characterized in that: Determining the video duration corresponding to the storyboard script includes: Calculate the ratio of the text-to-speech duration of the storyboard script to the text-to-speech duration of the video script to obtain the video duration ratio corresponding to the storyboard script; The video duration corresponding to the storyboard script is determined according to the proportion of the video duration corresponding to the storyboard script and the target video duration.
8. The video mixing method according to claim 6, characterized in that: The method further comprises: selecting video clips semantically related to the storyboard script from the plurality of video clips according to the semantic relevance between the storyboard script and each video clip and combining the video duration corresponding to the storyboard script to obtain a set of video clips corresponding to the storyboard script, including: Video clips are selected from the plurality of video clips in order of the semantic relevance between the storyboard script and each video clip from high to low, until the total duration of all the selected video clips is equal to or greater than the duration of the video corresponding to the storyboard script, and the selected video clips constitute a video clip set corresponding to the storyboard script.
9. The video mixing method according to claim 1, characterized in that: The step of splicing the video clips in the video clip set corresponding to each storyboard script comprises: For each storyboard: Determining a splicing order of the video clips in the video clip set corresponding to the storyboard script according to the semantic relevance of each video clip in the video clip set corresponding to the storyboard script and the storyboard script; According to the determined splicing order, each video clip in the video clip set corresponding to the storyboard script is spliced.
10. The video mixing method according to any one of claims 1 to 9, characterized in that: Also includes: The mixed video corresponding to the video script is post-processed, wherein the post-processing includes one or more of the following processes: adding commentary audio, adding subtitles, adding background music, and adding transition effects.
11. A video mixing device, characterized in that: include: Data acquisition module, video script storyboard module, video material segmentation module, video clip screening module, video clip splicing module and video splicing module; The data acquisition module is used to acquire the video script and video materials related to the video script; The video script storyboard module is used to storyboard the video script to obtain a plurality of storyboard scripts; The video material segmentation module is used to segment the video material to obtain a plurality of video clips; The video clip screening module is used to screen video clips semantically related to each storyboard script from the plurality of video clips to obtain video clip sets corresponding to the plurality of storyboard scripts respectively; The video clip splicing module is used to splice the video clips in the video clip set corresponding to each storyboard script to obtain the video corresponding to each storyboard script; The video splicing module is used to splice the videos corresponding to the several storyboard scripts to obtain a mixed video corresponding to the video scripts.
12. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the video mixing method as described in any one of claims 1 to 10.
13. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the video mixing method as described in any one of claims 1 to 10.
14. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the steps of the video mixing method as described in any one of claims 1 to 10.
Citation Information
Cited By
Video mixing and cutting method and device, electronic equipment and storage medium
CN120151574A
Automatic video editing method based on AI split mirror generation
CN120416583A
Multi-modal material-oriented intelligent video editing method and system
CN120416584A
Intelligent video clip method and system for multi-modal material
CN120416584B
