Video generation method and device, and storage medium

Abstract

Embodiments of the present disclosure provide a video generation method and device, and a storage medium. The method comprises: acquiring a video script, and acquiring text features of the video script; acquiring video features of each video material among a plurality of video materials; acquiring a similarity matrix between the text features and the video features; matching the video material with the video script on the basis of the similarity matrix to obtain a video material sequence; and generating a target video file on the basis of the video material sequence. In the embodiments of the present disclosure, retrieval and sorting of the video materials are implemented by means of the matching between the video script and the video materials, and then video packaging is carried out on the basis of the video material sequence, thereby better achieving automated editing and production of videos.

Description

Video generation method, device and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202311346772.6 filed on October 17, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby cited in their entirety as part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a video generation method, device, and storage medium. Background Art

[0003] With the development of network technology and the improvement of terminal device hardware performance, the demand for video creation and video editing on terminal devices is becoming increasingly strong. In order to lower the threshold for video creation and facilitate users to quickly produce video works, the function of automatic video generation has come into being.

[0004] Some existing automatic video generation functions usually require users to upload video materials, and then automatically perform packaging processes such as adding music, special effects, transitions, etc. to the video materials. However, the existing automatic video generation functions can only implement the above simple packaging process and cannot achieve more complex automated editing and production.

[0005] Summary of the Invention

[0006] The embodiments of the present disclosure provide a video generation method, device, and storage medium to better implement automated video editing and production.

[0007] In a first aspect, an embodiment of the present disclosure provides a video generation method, comprising:

[0008] Obtaining a video script and obtaining text features of the video script;

[0009] Obtaining video features of each video material from a plurality of video materials;

[0010] Obtaining a similarity matrix between the text features and the video features;

[0011] Matching the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0012] A target video file is generated according to the video material sequence.

[0013] In a second aspect, an embodiment of the present disclosure provides a video generation device, including:

[0014] A script acquisition unit, used to acquire a video script;

[0015] A text encoding unit, configured to obtain text features of the video script;

[0016] A video encoding unit, configured to obtain video features of each of the plurality of video materials;

[0017] A matching unit, configured to obtain a similarity matrix between the text features and the video features; and match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0018] The video packaging unit is used to generate a target video file according to the video material sequence.

[0019] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0020] The memory stores computer-executable instructions;

[0021] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video generation method described in the first aspect and various possible designs of the first aspect.

[0022] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0023] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video generation method as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] FIG1 is a scene example diagram of a video generation method provided by an embodiment of the present disclosure;

[0026] FIG2 is a flow chart of a video generation method according to an embodiment of the present disclosure;

[0027] FIG3 is a schematic diagram of a script library provided by an embodiment of the present disclosure;

[0028] FIG4 is a structural block diagram of a video generating device provided by an embodiment of the present disclosure; and

[0029] FIG5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0031] Some existing automatic video generation functions usually require users to upload video materials, and then automatically perform packaging processes such as adding music, special effects, transitions, etc. to the video materials. However, the existing automatic video generation functions can only achieve the above-mentioned simple packaging process, and cannot achieve more complex automated editing and production, especially the retrieval and sorting of video materials. Usually, manual retrieval and sorting of video materials are required, which is a huge workload and time-consuming when there are a lot of video materials.

[0032] To address the above technical issues, the present disclosure provides a video generation method, which comprises obtaining a video script and text features of the video script; obtaining video features of each of a plurality of video materials; obtaining a similarity matrix between the text features and the video features; matching the video materials with the video script based on the similarity matrix to obtain a video material sequence; and generating a target video file based on the video material sequence. In this embodiment, the matching between the video script and the video material enables retrieval and sorting of the video materials, and then video packaging is performed based on the video material sequence, thereby better realizing automated video editing and production.

[0033] The video generation method provided by the present invention is applicable to electronic devices such as terminal devices or servers. Taking the terminal device as an example as shown in Figure 1, the terminal device can obtain a video script and video material, and then obtain text features of the video script and video features of the video material, and obtain a similarity matrix between the text features and the video features; match the video material and the video script based on the similarity matrix to obtain a video material sequence; and generate a target video file according to the video material sequence.

[0034] The video generation method disclosed herein will be described in detail below with reference to specific embodiments.

[0035] Referring to FIG2 , FIG2 is a flow chart of a video generation method provided by an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices or servers, and the video generation method includes:

[0036] S201. Obtain a video script and obtain text features of the video script.

[0037] In this embodiment, the video script may be composed of a set of plots or actions (for example, a food-related video script may include adding oil, pouring ingredients, cooking process, etc., and a travel-related video script may include travel events, etc.). More specifically, it may include shots, camera movements, lines, etc.

[0038] Optionally, the user may input text data of a video script of an expected editing effect, and the electronic device receives the text data of the video script input by the user and obtains the video script according to the text data of the video script.

[0039] Optionally, the user may input one or more reference videos, and the electronic device receives the reference videos input by the user, obtains description information of the reference videos, and obtains the video script based on the description information of the reference videos. Optionally, when obtaining the description information of the reference videos, the reference videos may be sliced ​​to obtain multiple video clips, each of which may be a plot or an action, etc., and then a video description model is used to obtain the description information of each video clip. The video description model may be a machine learning model or any other model that can implement a video description function. The reference videos may also be sliced ​​using a machine learning model or any other model that can implement a video slicing function.

[0040] Of course, you can also combine the above two solutions, or use other methods to obtain video scripts, which are not limited here.

[0041] Among them, the above-mentioned video scripts can be structured video scripts, that is, each video script adopts the same structure, for example, including shots, camera movements, lines, etc.

[0042] After obtaining the video script, the video script can be encoded to obtain text features of the video script. Optionally, a text encoder can be used to encode the video script to obtain text features (feature vectors) of the video script. The text encoder can use any model or algorithm that can extract text features, such as a text encoder (Text Encoder) of a Transformer model, or a text encoder (Text Encoder) in a CLIP model, etc., which are not limited here.

[0043] S202: Obtain video features of each video material in a plurality of video materials.

[0044] In this embodiment, the user may input multiple video materials, and for each of the multiple video materials, video features (feature vectors) may be extracted.

[0045] Optionally, when obtaining the video features of any video material, multiple video frames may be extracted from the video material, such as key frames or frames at equal intervals, and then the video frames may be encoded using a video encoder to obtain the features of each video frame. The features of each video frame are then average-pooled to obtain the video features of the video material. The video encoder (Video Encoder) may use any model or algorithm capable of extracting video features, such as the video encoder (Vision Transformer) in the CLIP model, and is not limited here.

[0046] Optionally, the text features of the video script and the video features of the video material can be obtained by using the text encoder (Text Encoder) and video encoder (Video Encoder) of the same ClIP model respectively, so that only one model is needed to implement the above-mentioned text encoding and video encoding, and the text features of the video script and the video features of the video material can be obtained more efficiently.

[0047] In addition, it should be noted that there is no restriction on the order in which S201 and S202 are executed. They can be executed simultaneously or one after another.

[0048] S203: Obtain a similarity matrix between the text features and the video features.

[0049] In this embodiment, the similarity between the text features of each video script and the video features of each video material can be obtained to construct a similarity matrix. Assuming there are M video scripts and N video materials, the similarity matrix is ​​an M×N matrix. The similarity can use cosine similarity or other similarity metrics.

[0050] Optionally, in order to better match the video script with the video material, in this embodiment, at least two of the following similarities are obtained between the text features of any video script and the video features of any video material:

[0051] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0052] The weighted sum of the at least two similarities mentioned above is performed to obtain the similarity between the text features of any video script and the video features of any video material. After obtaining the similarity between the text features of each video script and the video features of each video material, a similarity matrix can be constructed.

[0053] Among them, the average value of the maximum similarity between each video frame of the video material and each word in the video script is, for any video material, the similarity between a certain video frame of the video material and each word in a certain video script is obtained, and the maximum similarity is taken out as the maximum similarity between the video frame and the words in the video script. Similarly, the maximum similarity between each other frame of the video material and the words in the video script is also obtained, and then the average value of these maximum similarities is taken; and the average value of the maximum similarity between each word in the video script and each video frame of the video material is, for any video script, the similarity between a certain word in the video script and each video frame of a certain video material is obtained, and the maximum similarity is taken out as the maximum similarity between the word and each video frame of the video material. Similarly, the maximum similarity between other words in the video script and each video frame of the video material is also obtained, and then the average value of these maximum similarities is taken.

[0054] S204: Match the video material with the video script based on the similarity matrix to obtain a video material sequence.

[0055] In this embodiment, the video material and the video script can be matched based on the similarity matrix. The matching process can adopt a beam search algorithm (Beam Search) or a greedy algorithm to obtain a video material sequence that matches each video script one by one.

[0056] Taking the beam search algorithm as an example, a beam size (replaced by bs in this embodiment) can be pre-specified. When searching for video materials matching any video script, the total score (total similarity) of the new video sequence after each candidate video material is added to the current video material sequence is obtained, and the bs new video material sequences with the highest total scores are retained. Next, when searching for video materials matching the next video script, based on the bs new video material sequences retained in the previous step, further searches are performed, and the bs new video material sequences with the highest total scores are retained. This step is repeated until the matching is completed. Finally, bs video material sequences are obtained, and the video material sequence with the highest total score is selected as the final video material sequence.

[0057] S205: Generate a target video file according to the video material sequence.

[0058] In this embodiment, after obtaining a video material sequence that matches each video script one by one, the video material sequence can be packaged, especially including but not limited to adding music, special effects, transitions, animations, text, etc., and then generating a target video file that has been edited and rendered, thereby realizing automatic video editing (AutoCut).

[0059] Optionally, a preset video packaging model may be used to add at least one of the following to the video material sequence: music, special effects, transitions, animations or texts to generate the target video file, wherein the preset video packaging model may be any model or algorithm that can realize the addition of music, special effects, transitions, animations and texts; or, a video template may be obtained for the video material sequence, and the video material sequence may be packaged according to the video template to generate the target video file, wherein the video template may be a combination of a series of effects, which may include adding music, special effects, transitions, animations, texts, etc., and by adapting the video template to the video material sequence, a series of effects in the video template may be applied to the video material sequence.

[0060] The video generation method provided in this embodiment obtains a video script and text features of the video script; obtains video features of each video material from a plurality of video materials; obtains a similarity matrix between the text features and the video features; matches the video materials with the video script based on the similarity matrix to obtain a video material sequence; and generates a target video file based on the video material sequence. In this embodiment, the matching between the video script and the video material is used to retrieve and sort the video materials, and then the video is packaged based on the video material sequence, thereby better realizing automated video editing and production.

[0061] Based on any of the above embodiments, when obtaining a video script based on the text data of the video script and / or the description information of the reference video, the following methods may be specifically included:

[0062] Method 1: Segmenting the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determining each of the sentences as the video script;

[0063] Among them, the text data of the video script and / or the description information of the reference video can be first divided into sentences to obtain multiple sentences. Each sentence can include a complete action or a part of the complete content. Considering that each sentence may be relatively detailed, each sentence can be determined as a video script.

[0064] Method 2: Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script;

[0065] The multiple sentences obtained by the sentence segmentation may not be detailed or complete enough, so a preset script library may be used to obtain a video script, wherein the preset script library may include multiple preset video scripts. In this method, each sentence obtained by the sentence segmentation may be matched with the preset script library to retrieve the preset video script that matches each sentence from the preset script library, and these preset video scripts are determined as the video script;

[0066] Method 3: Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences. If the number of the sentences is less than a preset value, retrieve the preset video script associated with each of the sentences from a preset script library, and determine the sentences and the preset video script as the video script.

[0067] Among them, the amount of sentence data obtained by sentence segmentation may be relatively small. If the number of sentences is less than the preset value, the preset video script related to each sentence can be retrieved from the preset script library, and the sentence and these preset video scripts can be determined as video scripts, thereby realizing the expansion of the sentence, or only these preset video scripts can be determined as video scripts.

[0068] Based on the above embodiments, the preset script library may include a first-level preset video script and a second-level preset video script, wherein the first-level preset video script is associated with at least one second-level preset video script, and the second-level preset video script is a finer-grained video script based on the first-level preset video script. For example, some preset video scripts for videos related to "raising cats and buying things" in the preset script library shown in Figure 3, the first-level preset video script is for human-pet interaction, and the second-level preset video script includes fine-grained video scripts of various types of human-pet interaction actions. Of course, the preset script library may also include finer-grained third-level preset video scripts, fourth-level preset video scripts, etc., which are not limited here.

[0069] Furthermore, in the above embodiment, when retrieving the preset video script associated with each sentence from the preset script library, the target first-level preset video script associated with each sentence can be retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script can be obtained, and the target first-level preset video script and the target second-level preset video script can be determined as the preset video script associated with each sentence. Of course, the target second-level preset video script can also be directly determined as the preset video script associated with each sentence (when encoding the text, only the target second-level preset video script is encoded).

[0070] Optionally, when retrieving the preset video script related to each sentence from the preset script library, the following steps are included:

[0071] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0072] Encoding the preset video script in the preset script library using the text encoder to obtain text features of the preset video script;

[0073] The preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0074] In this embodiment, a text encoder may be used to encode each sentence to obtain the text features of each sentence. Similarly, a text encoder may be used to encode the preset scripts in the preset script library to obtain the text features of the preset video scripts. The text features of the sentences and the text features of the preset video scripts are then similarly matched to obtain the preset video script with the highest similarity for each sentence. In the case where the preset script library includes a first-level preset video script and a second-level preset video script, only the first-level preset video script may be encoded to obtain the text features of the first-level preset video script. Then, only the text features of the sentences and the text features of the first-level preset video script need to be similarly matched.

[0075] Based on any of the above embodiments, when using the beam search algorithm to obtain a video material sequence that matches the video script, for some special scenes, such as travel and cooking, there is a time sequence. Therefore, a time constraint can be added during the execution of the beam search algorithm, as follows:

[0076] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0077] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0078] In this embodiment, when searching for video materials matching any video script through the beam search algorithm, the total score (total similarity) of the new video sequence after any candidate video material is added to the current video material sequence is obtained. If the shooting time of the candidate video material added to the current video material sequence is before the shooting time of the previous video material in the video material sequence, it is necessary to reduce the total score corresponding to the candidate video material, so as to minimize the occurrence of reverse shooting time in the video material sequence.

[0079] Optionally, after obtaining the video material sequence, the video material sequence may be deduplicated to avoid the situation where the same video material appears multiple times in the video material sequence.

[0080] In addition, based on the video material sequence, a video description model can be used to obtain the video description of the target video file, which will not be described in detail here.

[0081] Corresponding to the video generation method described in the above embodiment, FIG4 is a block diagram of the structure of a video generation device provided by an embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to FIG4 , the video generation device 400 includes: a script acquisition unit 401, a text encoding unit 402, a video encoding unit 403, a matching unit 404, and a video packaging unit 405.

[0082] The script acquisition unit 401 is used to acquire the video script;

[0083] A text encoding unit 402, configured to obtain text features of the video script;

[0084] The video encoding unit 403 is used to obtain video features of each video material in the plurality of video materials;

[0085] A matching unit 404 is configured to obtain a similarity matrix between the text features and the video features; match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0086] The video packaging unit 405 is configured to generate a target video file according to the video material sequence.

[0087] In one or more embodiments of the present disclosure, when acquiring the text features of the video script, the text encoding unit 402 is configured to:

[0088] The video script is encoded using a text encoder to obtain text features of the video script.

[0089] In one or more embodiments of the present disclosure, when acquiring a video script, the script acquisition unit 401 is configured to:

[0090] Receiving text data of a video script input by a user, and obtaining the video script according to the text data of the video script; and / or

[0091] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script based on the description information of the reference video.

[0092] In one or more embodiments of the present disclosure, when acquiring the description information of the reference video, the script acquisition unit 401 is configured to:

[0093] The reference video is sliced ​​to obtain multiple video segments, and description information of each video segment is obtained using a video description model.

[0094] In one or more embodiments of the present disclosure, when obtaining the video script according to the method, the script obtaining unit 401 is configured to:

[0095] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0096] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0097] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0098] In one or more embodiments of the present disclosure, the preset script library includes a first-level preset video script and a second-level preset video script, wherein the first-level preset video script is associated with at least one second-level preset video script. When the script acquisition unit 401 retrieves the preset video script associated with each of the sentences from the preset script library, it is configured to:

[0099] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0100] In one or more embodiments of the present disclosure, when retrieving the preset video script related to each of the sentences from the preset script library, the script acquisition unit 401 is configured to:

[0101] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0102] Encoding the preset video script in the preset script library using the text encoder to obtain text features of the preset video script;

[0103] The preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0104] In one or more embodiments of the present disclosure, when acquiring the video features of each video material in the plurality of video materials, the video encoding unit 403 is configured to:

[0105] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame. The features of each video frame are average-pooled to obtain video features of the video material.

[0106] In one or more embodiments of the present disclosure, when obtaining the similarity matrix between the text features and the video features, the matching unit 404 is configured to:

[0107] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0108] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0109] The similarity between the text feature of any video script and the video feature of any video material is obtained by performing a weighted summation on the at least two similarities to construct the similarity matrix.

[0110] In one or more embodiments of the present disclosure, when matching the video material with the video script based on the similarity matrix to obtain a video material sequence, the matching unit 404 is configured to:

[0111] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence that matches the video script.

[0112] In one or more embodiments of the present disclosure, when the matching unit 404 adopts the beam search algorithm to obtain a video material sequence matching the video script, it is configured to:

[0113] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0114] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0115] In one or more embodiments of the present disclosure, when generating a target video file according to the video material sequence, the video packaging unit 405 is configured to:

[0116] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations, or text to generate the target video file; or

[0117] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0118] The video generation device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0119] Referring to FIG5 , there is shown a schematic diagram of the structure of an electronic device 500 suitable for implementing an embodiment of the present disclosure. The electronic device 500 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG5 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0120] As shown in FIG5 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0121] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows an electronic device 500 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0122] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0123] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0124] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0125] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0126] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0129] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0130] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, comprising:

[0132] Obtaining a video script and obtaining text features of the video script;

[0133] Obtaining video features of each video material from a plurality of video materials;

[0134] Obtaining a similarity matrix between the text features and the video features;

[0135] Matching the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0136] A target video file is generated according to the video material sequence.

[0137] According to one or more embodiments of the present disclosure, obtaining text features of the video script includes:

[0138] The video script is encoded using a text encoder to obtain text features of the video script.

[0139] According to one or more embodiments of the present disclosure, obtaining the video script includes:

[0140] Receiving text data of a video script input by a user, and obtaining the video script according to the text data of the video script; and / or

[0141] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script based on the description information of the reference video.

[0142] According to one or more embodiments of the present disclosure, obtaining description information of the reference video includes:

[0143] The reference video is sliced ​​to obtain multiple video segments, and description information of each video segment is obtained using a video description model.

[0144] According to one or more embodiments of the present disclosure, the obtaining of the video script includes:

[0145] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0146] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0147] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0148] According to one or more embodiments of the present disclosure, the preset script library includes a first-level preset video script and a second-level preset video script, wherein the first-level preset video script is associated with at least one second-level preset video script, and retrieving the preset video script associated with each of the sentences from the preset script library includes:

[0149] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0150] According to one or more embodiments of the present disclosure, the step of retrieving a preset video script related to each of the sentences from a preset script library includes:

[0151] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0152] Encoding the preset video script in the preset script library using the text encoder to obtain text features of the preset video script;

[0153] The preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0154] According to one or more embodiments of the present disclosure, obtaining a video feature of each video material from a plurality of video materials includes:

[0155] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame. The features of each video frame are average-pooled to obtain video features of the video material.

[0156] According to one or more embodiments of the present disclosure, obtaining a similarity matrix between the text features and the video features includes:

[0157] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0158] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0159] The similarity between the text feature of any video script and the video feature of any video material is obtained by performing a weighted summation on the at least two similarities to construct the similarity matrix.

[0160] According to one or more embodiments of the present disclosure, matching the video material with the video script based on the similarity matrix to obtain a video material sequence includes:

[0161] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence that matches the video script.

[0162] According to one or more embodiments of the present disclosure, the step of using a beam search algorithm to obtain a video material sequence matching the video script includes:

[0163] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0164] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0165] According to one or more embodiments of the present disclosure, generating a target video file according to the video material sequence includes:

[0166] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations, or text to generate the target video file; or

[0167] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0168] In a second aspect, according to one or more embodiments of the present disclosure, a video generating device is provided, including:

[0169] A script acquisition unit, used to acquire a video script;

[0170] A text encoding unit, configured to obtain text features of the video script;

[0171] A video encoding unit, configured to obtain video features of each of the plurality of video materials;

[0172] A matching unit, configured to obtain a similarity matrix between the text features and the video features; and match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0173] The video packaging unit is used to generate a target video file according to the video material sequence.

[0174] According to one or more embodiments of the present disclosure, when acquiring the text features of the video script, the text encoding unit is configured to:

[0175] The video script is encoded using a text encoder to obtain text features of the video script.

[0176] According to one or more embodiments of the present disclosure, when acquiring a video script, the script acquisition unit is configured to:

[0177] Receiving text data of a video script input by a user, and obtaining the video script according to the text data of the video script; and / or

[0178] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script based on the description information of the reference video.

[0179] According to one or more embodiments of the present disclosure, when acquiring the description information of the reference video, the script acquisition unit is configured to:

[0180] The reference video is sliced ​​to obtain multiple video segments, and description information of each video segment is obtained using a video description model.

[0181] According to one or more embodiments of the present disclosure, when obtaining the video script according to the method, the script obtaining unit is configured to:

[0182] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0183] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0184] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0185] According to one or more embodiments of the present disclosure, the preset script library includes a first-level preset video script and a second-level preset video script, wherein the first-level preset video script is associated with at least one second-level preset video script. When the script acquisition unit retrieves the preset video script related to each of the sentences from the preset script library, it is configured to:

[0186] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0187] According to one or more embodiments of the present disclosure, when the script acquisition unit retrieves the preset video script related to each of the sentences from the preset script library, it is configured to:

[0188] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0189] Encoding the preset video script in the preset script library using the text encoder to obtain text features of the preset video script;

[0190] The preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0191] According to one or more embodiments of the present disclosure, when acquiring the video features of each video material in the plurality of video materials, the video encoding unit is configured to:

[0192] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame. The features of each video frame are average-pooled to obtain video features of the video material.

[0193] According to one or more embodiments of the present disclosure, when obtaining the similarity matrix between the text features and the video features, the matching unit is configured to:

[0194] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0195] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0196] The similarity between the text feature of any video script and the video feature of any video material is obtained by performing a weighted summation on the at least two similarities to construct the similarity matrix.

[0197] According to one or more embodiments of the present disclosure, when the matching unit matches the video material with the video script based on the similarity matrix to obtain a video material sequence, it is configured to:

[0198] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence that matches the video script.

[0199] According to one or more embodiments of the present disclosure, when the matching unit adopts the beam search algorithm to obtain a video material sequence matching the video script, it is configured to:

[0200] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0201] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0202] According to one or more embodiments of the present disclosure, when generating a target video file according to the video material sequence, the video packaging unit is configured to:

[0203] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations, or text to generate the target video file; or

[0204] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0205] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0206] The memory stores computer-executable instructions;

[0207] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video generation method described in the first aspect and various possible designs of the first aspect.

[0208] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0209] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the video generation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0210] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0211] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0212] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A video generation method, comprising: Obtaining a video script and obtaining text features of the video script; Obtaining video features of each video material among a plurality of video materials; Obtaining a similarity matrix between the text features and the video features; Matching the video material with the video script based on the similarity matrix to obtain a video material sequence; A target video file is generated according to the video material sequence.

2. The method according to claim 1, wherein: The obtaining of text features of the video script includes: The video script is encoded using a text encoder to obtain text features of the video script.

3. The method according to claim 1 or 2, wherein: The step of obtaining the video script includes: Receiving text data of a video script input by a user, and acquiring the video script according to the text data of the video script; and / or Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script according to the description information of the reference video.

4. The method according to claim 3, wherein: The obtaining the description information of the reference video includes: The reference video is sliced ​​to obtain multiple video segments, and the description information of each video segment is obtained using a video description model.

5. The method according to claim 3 or 4, wherein: The step of obtaining the video script according to the method comprises: Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

6. The method according to claim 5, wherein: The preset script library includes a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and the retrieving the preset video script related to each of the sentences from the preset script library includes: The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

7. The method according to claim 5 or 6, wherein: The retrieving a preset video script related to each of the sentences from a preset script library includes: Encoding the sentence using a text encoder to obtain text features of the sentence; Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script; A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

8. The method according to any one of claims 1 to 7, wherein: The obtaining of the video feature of each video material in the plurality of video materials comprises: For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame, and the features of each video frame are average-pooled to obtain video features of the video material.

9. The method according to claim 8, wherein: The obtaining of a similarity matrix between the text features and the video features includes: Obtain at least two of the following similarities between the text features of any video script and the video features of any video material: Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material; The similarity matrix is ​​constructed by performing a weighted summation on the at least two similarities to obtain the similarity between the text feature of any video script and the video feature of any video material.

10. The method according to any one of claims 1 to 9, wherein: The matching of the video material with the video script based on the similarity matrix to obtain a video material sequence includes: Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence matching the video script.

11. The method according to claim 10, wherein: The step of using a beam search algorithm to obtain a video material sequence matching the video script includes: When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence; If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

12. The method according to any one of claims 1 to 11, wherein: The step of generating a target video file according to the video material sequence comprises: Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations or texts to generate the target video file; or Obtain a video template for the video material sequence, and perform video processing on the video material sequence according to the video template. Packaging, generating target video files.

13. A video generating device, comprising: A script acquisition unit, used for acquiring a video script; A text encoding unit, used to obtain text features of the video script; A video encoding unit, used for obtaining video features of each video material among a plurality of video materials; A matching unit, used to obtain a similarity matrix between the text features and the video features; Matching the video material with the video script based on the similarity matrix to obtain a video material sequence; The video packaging unit is used to generate a target video file according to the video material sequence.

14. An electronic device comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, wherein: The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

Images