Video generation method and device and storage medium

Abstract

The embodiment of the invention provides a video generation method and device and a storage medium, and the method comprises the steps: obtaining a video script, and obtaining a text feature of the video script; acquiring a video feature of each video material in the plurality of video materials; obtaining a similarity matrix between the text features and the video features; matching the video material with the video script based on the similarity matrix to obtain a video material sequence; and generating a target video file according to the video material sequence. According to the embodiment of the invention, through matching between the video script and the video material, retrieval and sorting of the video material are realized, so that video packaging is carried out based on the video material sequence, and automatic editing and production of the video are better realized.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a video generation method, device, and storage medium. Background Art

[0002] With the development of network technology and the improvement of terminal equipment hardware performance, the demand for video creation and video editing on terminal equipment is becoming more and more vigorous. In order to lower the threshold of video creation and facilitate users to quickly produce video works, the function of automatically generating videos came into being.

[0003] Some existing automatic video generation functions usually require users to upload video materials, and then automatically perform packaging processes such as adding music, special effects, transitions, etc. to the video materials. However, the existing automatic video generation functions can only achieve the above-mentioned simple packaging process and cannot achieve more complex automated editing and production. Summary of the invention

[0004] The embodiments of the present disclosure provide a video generation method, device and storage medium to better realize the automatic editing and production of videos.

[0005] In a first aspect, an embodiment of the present disclosure provides a video generation method, including:

[0006] Obtaining a video script and obtaining text features of the video script;

[0007] Obtaining video features of each video material among a plurality of video materials;

[0008] Obtaining a similarity matrix between the text features and the video features;

[0009] Matching the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0010] A target video file is generated according to the video material sequence.

[0011] In a second aspect, an embodiment of the present disclosure provides a video generating device, including:

[0012] A script acquisition unit, used for acquiring a video script;

[0013] A text encoding unit, used to obtain text features of the video script;

[0014] A video encoding unit, used for obtaining video features of each video material among a plurality of video materials;

[0015] A matching unit, configured to obtain a similarity matrix between the text features and the video features; and to match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0016] The video packaging unit is used to generate a target video file according to the video material sequence.

[0017] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0018] The memory stores computer-executable instructions;

[0019] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video generation method described in the first aspect and various possible designs of the first aspect.

[0020] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0022] The video generation method, device and storage medium provided in the disclosed embodiment obtain a video script and obtain text features of the video script; obtain video features of each video material in multiple video materials; obtain a similarity matrix between the text features and the video features; match the video material with the video script based on the similarity matrix to obtain a video material sequence; and generate a target video file based on the video material sequence. In the disclosed embodiment, the video material is retrieved and sorted by matching the video script with the video material, and then the video is packaged based on the video material sequence, so as to better realize the automatic editing and production of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A scene example diagram of a video generation method provided by an embodiment of the present disclosure;

[0025] Figure 2 A schematic diagram of a video generation method flow chart provided by an embodiment of the present disclosure;

[0026] Figure 3 A schematic diagram of a script library provided for an embodiment of the present disclosure;

[0027] Figure 4 A structural block diagram of a video generation device provided by an embodiment of the present disclosure;

[0028] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Some existing automatic video generation functions usually require users to upload video materials, and then automatically perform packaging processes such as adding music, special effects, transitions, etc. to the video materials. However, the existing automatic video generation functions can only achieve the above-mentioned simple packaging process, and cannot achieve more complex automated editing and production, especially the retrieval and sorting of video materials. Usually, manual retrieval and sorting of video materials are required, which is a huge workload and time-consuming when there are a lot of video materials.

[0031] In order to solve the above technical problems, the present disclosure provides a video generation method, which obtains a video script and a text feature of the video script; obtains the video feature of each video material in a plurality of video materials; obtains a similarity matrix between the text feature and the video feature; matches the video material with the video script based on the similarity matrix to obtain a video material sequence; and generates a target video file according to the video material sequence. In this embodiment, the video material is retrieved and sorted by matching the video script with the video material, and then the video is packaged based on the video material sequence, so as to better realize the automatic editing and production of the video.

[0032] The video generation method provided by the present disclosure is applicable to electronic devices such as terminal devices or servers, such as Figure 1Taking the terminal device as an example, the terminal device can obtain the video script and the video material, and then obtain the text features of the video script and the video features of the video material, and obtain the similarity matrix between the text features and the video features; match the video material and the video script based on the similarity matrix to obtain a video material sequence; and generate a target video file according to the video material sequence.

[0033] The video generation method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0034] refer to Figure 2 , Figure 2 The method of this embodiment can be applied in electronic devices such as terminal devices or servers, and the video generation method includes:

[0035] S201, obtaining a video script, and obtaining text features of the video script.

[0036] In this embodiment, the video script may be composed of a set of plots or actions (for example, a food-related video script may include adding oil, pouring in ingredients, cooking process, etc., and a travel-related video script may include travel events, etc.), and more specifically, it may include shots, camera movements, lines, etc.

[0037] Optionally, the user may input text data of a video script of an expected editing effect, and the electronic device receives the text data of the video script input by the user and obtains the video script according to the text data of the video script.

[0038] Optionally, the user may input one or more reference videos, and the electronic device receives the reference videos input by the user, obtains the description information of the reference videos, and obtains the video script according to the description information of the reference videos. Optionally, when obtaining the description information of the reference video, the reference video may be sliced ​​to obtain multiple video clips, each of which may be a plot or an action, etc., and then the description information of each video clip may be obtained using a video description model, which may be a machine learning model or any other model capable of realizing the video description function. Slicing the reference video may also use a machine learning model or any other model capable of realizing the video slicing function.

[0039] Of course, you can also combine the above two solutions, or use other methods to obtain the video script, which is not limited here.

[0040] Among them, the above-mentioned video script can be a structured video script, that is, each video script adopts the same structure, for example, all include shots, camera movements, lines, etc.

[0041] After obtaining the video script, the video script can be encoded to obtain the text features of the video script. Optionally, a text encoder can be used to encode the video script to obtain the text features (feature vectors) of the video script. The text encoder can use any model or algorithm that can realize text feature extraction, such as a text encoder (Text Encoder) of a Transformer model, or a text encoder (Text Encoder) in a CLIP model, etc., which is not limited here.

[0042] S202: Obtain video features of each video material among a plurality of video materials.

[0043] In this embodiment, the user may input a plurality of video materials, and for each of the plurality of video materials, a video feature (feature vector) may be extracted.

[0044] Optionally, when obtaining the video features of any video material, multiple video frames may be extracted from any video material, such as extracting key frames or extracting frames at equal intervals, and then a video encoder is used to encode the video frames to obtain the features of each video frame, and the features of each video frame are averaged and pooled to obtain the video features of the video material. The video encoder (Video Encoder) may use any model or algorithm that can realize video feature extraction, such as the video encoder (Vision Transformer) in the CLIP model, etc., which is not limited here.

[0045] Optionally, the text features of the video script and the video features of the video material can be obtained by respectively using the text encoder (Text Encoder) and video encoder (Video Encoder) of the same ClIP model, so that only one model is needed to implement the above-mentioned text encoding and video encoding, and the text features of the video script and the video features of the video material can be obtained more efficiently.

[0046] In addition, it should be noted that there is no restriction on the execution order of S201 and S202 , and they can be executed simultaneously or one after another.

[0047] S203: Obtain a similarity matrix between the text features and the video features.

[0048] In this embodiment, the similarity between the text features of each video script and the video features of each video material can be obtained to construct a similarity matrix. Assuming that there are M video scripts and N video materials, the similarity matrix is ​​an M×N matrix. The similarity can be cosine similarity, or other similarities can be used.

[0049] Optionally, in order to better match the video script with the video material, in this embodiment, at least two similarities between the text features of any video script and the video features of any video material are obtained:

[0050] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0051] The weighted sum of the at least two similarities mentioned above is performed to obtain the similarity between the text features of any video script and the video features of any video material. After obtaining the similarity between the text features of each video script and the video features of each video material, a similarity matrix can be constructed.

[0052] Among them, the average value of the maximum similarity between each video frame of the video material and each word in the video script is, for any video material, the similarity between a certain video frame of the video material and each word in a certain video script is obtained respectively, and the maximum similarity is taken out as the maximum similarity between the video frame and the words in the video script. Similarly, the maximum similarity between the words in the video script and each other frame of the video material is also obtained, and then the average value of these maximum similarities is taken; and the average value of the maximum similarity between each word in the video script and each video frame of the video material is, for any video script, the similarity between a certain word of the video script and each video frame of the video material is obtained respectively, and the maximum similarity is taken out as the maximum similarity between the word and each video frame of the video material. Similarly, the maximum similarity between the other words in the video script and each video frame of the video material is also obtained, and then the average value of these maximum similarities is taken.

[0053] S204: Match the video material with the video script based on the similarity matrix to obtain a video material sequence.

[0054] In this embodiment, the video material and the video script can be matched based on the similarity matrix. The matching process can use a beam search algorithm (Beam Search) or a greedy algorithm to obtain a video material sequence that matches each video script one by one.

[0055] Among them, taking the beam search algorithm as an example, a beam size (replaced by bs in this embodiment) can be pre-specified. When searching for video materials matching any video script, the total score (total similarity) of the new video sequence after each candidate video material is added to the current existing video material sequence is obtained, and the bs new video material sequences with the highest total scores are retained; in the next step, when searching for video materials matching the next video script, on the basis of retaining the bs new video material sequences in the previous step, searches are performed separately, and the bs new video material sequences with the highest total scores are retained, and this step is repeated until the matching is completed. Finally, bs video material sequences are obtained, and the video material sequence with the highest total score is taken as the final video material sequence.

[0056] S205: Generate a target video file according to the video material sequence.

[0057] In this embodiment, after obtaining the video material sequence that matches each video script one by one, the video material sequence can be packaged, especially including but not limited to adding music, special effects, transitions, animations, text, etc., and then generating a target video file that has been edited and rendered, thereby realizing automatic video editing (AutoCut).

[0058] Optionally, a preset video packaging model may be used to add at least one of the following to the video material sequence: music, special effects, transitions, animations or texts to generate the target video file, wherein the preset video packaging model may be any model or algorithm that can realize the addition of music, special effects, transitions, animations and texts; or, a video template may be obtained for the video material sequence, and the video material sequence may be packaged according to the video template to generate the target video file, wherein the video template may be a combination of a series of effects, which may include adding music, special effects, transitions, animations, texts, etc., and the video template is adapted to the video material sequence so that a series of effects in the video template are applied to the video material sequence.

[0059] The video generation method provided in this embodiment obtains a video script and a text feature of the video script; obtains the video feature of each video material in a plurality of video materials; obtains a similarity matrix between the text feature and the video feature; matches the video material with the video script based on the similarity matrix to obtain a video material sequence; and generates a target video file based on the video material sequence. In this embodiment, the video material is retrieved and sorted by matching the video script with the video material, and then the video is packaged based on the video material sequence, so as to better realize the automatic editing and production of the video.

[0060] On the basis of any of the above embodiments, when obtaining the video script according to the text data of the video script and / or the description information of the reference video, the following methods may be specifically included:

[0061] Method 1: Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script;

[0062] Among them, the text data of the video script and / or the description information of the reference video can be first divided into sentences to obtain multiple sentences, each of which can include a complete action or a part of the complete content. Considering that each sentence may be relatively detailed, each sentence can be determined as a video script.

[0063] Method 2: Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each sentence from a preset script library, and determine the preset video script as the video script;

[0064] Among them, the multiple sentences obtained by sentence segmentation may not be detailed or complete enough, so the video script can be obtained in combination with a preset script library, wherein the preset script library may include multiple preset video scripts. In this method, each sentence obtained by sentence segmentation can be matched with the preset script library to retrieve the preset video script matched by each sentence from the preset script library, and these preset video scripts are determined as the video script;

[0065] Method three: Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences. If the number of the sentences is less than a preset value, retrieve the preset video script related to each sentence from a preset script library, and determine the sentence and the preset video script as the video script;

[0066] Among them, the amount of sentence data obtained by sentence segmentation may be relatively small. If the number of sentences is less than the preset value, the preset video script related to each sentence can be retrieved from the preset script library, and the sentence and these preset video scripts can be determined as video scripts, thereby realizing the expansion of the sentence, or only these preset video scripts can be determined as video scripts.

[0067] Based on the above embodiment, the preset script library may include a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and the secondary preset video script is a more fine-grained video script based on the primary preset video script, for example Figure 3The preset script library shown includes some preset video scripts for videos related to "raising cats and buying things". The first-level preset video script is for human-pet interaction, and the second-level preset video script includes fine-grained video scripts of various types of human-pet interaction actions. Of course, the preset script library can also include more fine-grained third-level preset video scripts, fourth-level preset video scripts, etc., which are not limited here.

[0068] Furthermore, in the above embodiment, when retrieving the preset video script related to each sentence from the preset script library, the target first-level preset video script related to each sentence can be retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video script related to each sentence. Of course, the target second-level preset video script can also be directly determined as the preset video script related to each sentence (when encoding the text, only the target second-level preset video script is encoded).

[0069] Optionally, when retrieving the preset video script related to each sentence from the preset script library, it includes:

[0070] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0071] Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script;

[0072] A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0073] In this embodiment, a text encoder may be used to encode each sentence to obtain the text features of each sentence. Similarly, a text encoder may be used to encode the preset scripts in the preset script library to obtain the text features of the preset video scripts, and then the text features of the sentences and the text features of the preset video scripts are matched for similarity to obtain the preset video script with the highest similarity for each sentence. In the case where the preset script library includes a first-level preset video script and a second-level preset video script, only the first-level preset video script may be encoded to obtain the text features of the first-level preset video script, and then only the text features of the sentences and the text features of the first-level preset video script need to be matched for similarity.

[0074] On the basis of any of the above embodiments, when the beam search algorithm is used to obtain a video material sequence matching a video script, for some special scenes, such as travel, cooking, etc., there is a time sequence, so a time constraint can be added during the execution of the beam search algorithm, as follows:

[0075] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0076] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0077] In this embodiment, when searching for video materials matching any video script through the beam search algorithm, the total score (total similarity) of the new video sequence after any candidate video material is added to the current existing video material sequence is obtained. If the shooting time of the candidate video material added to the current existing video material sequence is before the shooting time of the previous video material in the video material sequence, it is necessary to reduce the total score corresponding to the candidate video material, so as to minimize the occurrence of reverse shooting time in the video material sequence.

[0078] Optionally, after obtaining the video material sequence, the video material sequence may be deduplicated to avoid the situation where the same video material appears multiple times in the video material sequence.

[0079] In addition, based on the video material sequence, a video description model can be used to obtain the video description of the target video file, which will not be elaborated here.

[0080] Corresponding to the video generation method of the above embodiment, Figure 4 The structure block diagram of the video generation device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 4 The video generating device 400 includes: a script acquiring unit 401, a text encoding unit 402, a video encoding unit 403, a matching unit 404, and a video packaging unit 405.

[0081] The script acquisition unit 401 is used to acquire the video script;

[0082] A text encoding unit 402, used to obtain text features of the video script;

[0083] The video encoding unit 403 is used to obtain the video feature of each video material in the plurality of video materials;

[0084] A matching unit 404 is used to obtain a similarity matrix between the text features and the video features; match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0085] The video packaging unit 405 is used to generate a target video file according to the video material sequence.

[0086] In one or more embodiments of the present disclosure, when acquiring the text features of the video script, the text encoding unit 402 is used to:

[0087] The video script is encoded using a text encoder to obtain text features of the video script.

[0088] In one or more embodiments of the present disclosure, when acquiring the video script, the script acquisition unit 401 is used to:

[0089] Receiving text data of a video script input by a user, and acquiring the video script according to the text data of the video script; and / or

[0090] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script according to the description information of the reference video.

[0091] In one or more embodiments of the present disclosure, when acquiring the description information of the reference video, the script acquisition unit 401 is used to:

[0092] The reference video is sliced ​​to obtain multiple video segments, and the description information of each video segment is obtained using a video description model.

[0093] In one or more embodiments of the present disclosure, when obtaining the video script according to the method, the script obtaining unit 401 is used to:

[0094] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0095] Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0096] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0097] In one or more embodiments of the present disclosure, the preset script library includes a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and the script acquisition unit 401 is used to retrieve the preset video script related to each of the sentences from the preset script library:

[0098] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0099] In one or more embodiments of the present disclosure, when retrieving the preset video script related to each of the sentences from the preset script library, the script acquisition unit 401 is used to:

[0100] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0101] Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script;

[0102] A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0103] In one or more embodiments of the present disclosure, when acquiring the video feature of each video material in the plurality of video materials, the video encoding unit 403 is configured to:

[0104] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame, and the features of each video frame are average-pooled to obtain video features of the video material.

[0105] In one or more embodiments of the present disclosure, when acquiring the similarity matrix between the text feature and the video feature, the matching unit 404 is used to:

[0106] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0107] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0108] The similarity matrix is ​​constructed by performing a weighted summation on the at least two similarities to obtain the similarity between the text feature of any video script and the video feature of any video material.

[0109] In one or more embodiments of the present disclosure, when the matching unit 404 matches the video material with the video script based on the similarity matrix to obtain a video material sequence, it is configured to:

[0110] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence matching the video script.

[0111] In one or more embodiments of the present disclosure, when the matching unit 404 adopts the beam search algorithm to obtain a video material sequence matching the video script, it is configured to:

[0112] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0113] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0114] In one or more embodiments of the present disclosure, when generating a target video file according to the video material sequence, the video packaging unit 405 is used to:

[0115] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations or texts to generate the target video file; or

[0116] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0117] The video generating device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0118] refer to Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure, and the electronic device 500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 5 As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0120] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0121] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0122] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0123] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0124] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0125] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, including:

[0131] Obtaining a video script and obtaining text features of the video script;

[0132] Obtaining video features of each video material among a plurality of video materials;

[0133] Obtaining a similarity matrix between the text features and the video features;

[0134] Matching the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0135] A target video file is generated according to the video material sequence.

[0136] According to one or more embodiments of the present disclosure, obtaining text features of the video script includes:

[0137] The video script is encoded using a text encoder to obtain text features of the video script.

[0138] According to one or more embodiments of the present disclosure, obtaining a video script includes:

[0139] Receiving text data of a video script input by a user, and acquiring the video script according to the text data of the video script; and / or

[0140] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script according to the description information of the reference video.

[0141] According to one or more embodiments of the present disclosure, obtaining description information of the reference video includes:

[0142] The reference video is sliced ​​to obtain multiple video segments, and the description information of each video segment is obtained using a video description model.

[0143] According to one or more embodiments of the present disclosure, the obtaining the video script according to the method includes:

[0144] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0145] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0146] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0147] According to one or more embodiments of the present disclosure, the preset script library includes a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and the retrieving the preset video script related to each of the sentences from the preset script library includes:

[0148] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0149] According to one or more embodiments of the present disclosure, the retrieving a preset video script related to each of the sentences from a preset script library includes:

[0150] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0151] Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script;

[0152] A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0153] According to one or more embodiments of the present disclosure, the step of obtaining a video feature of each video material in a plurality of video materials includes:

[0154] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame, and the features of each video frame are average-pooled to obtain video features of the video material.

[0155] According to one or more embodiments of the present disclosure, the obtaining a similarity matrix between the text features and the video features includes:

[0156] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0157] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0158] The similarity matrix is ​​constructed by performing a weighted summation on the at least two similarities to obtain the similarity between the text feature of any video script and the video feature of any video material.

[0159] According to one or more embodiments of the present disclosure, matching the video material with the video script based on the similarity matrix to obtain a video material sequence includes:

[0160] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence matching the video script.

[0161] According to one or more embodiments of the present disclosure, the step of using a beam search algorithm to obtain a video material sequence matching the video script includes:

[0162] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0163] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0164] According to one or more embodiments of the present disclosure, generating a target video file according to the video material sequence includes:

[0165] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations or texts to generate the target video file; or

[0166] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0167] In a second aspect, according to one or more embodiments of the present disclosure, a video generating device is provided, including:

[0168] A script acquisition unit, used for acquiring a video script;

[0169] A text encoding unit, used to obtain text features of the video script;

[0170] A video encoding unit, used for obtaining video features of each video material among a plurality of video materials;

[0171] A matching unit, configured to obtain a similarity matrix between the text features and the video features; and to match the video material with the video script based on the similarity matrix to obtain a video material sequence;

[0172] The video packaging unit is used to generate a target video file according to the video material sequence.

[0173] According to one or more embodiments of the present disclosure, when acquiring the text features of the video script, the text encoding unit is used to:

[0174] The video script is encoded using a text encoder to obtain text features of the video script.

[0175] According to one or more embodiments of the present disclosure, when acquiring a video script, the script acquisition unit is used to:

[0176] Receiving text data of a video script input by a user, and acquiring the video script according to the text data of the video script; and / or

[0177] Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script according to the description information of the reference video.

[0178] According to one or more embodiments of the present disclosure, when acquiring the description information of the reference video, the script acquisition unit is used to:

[0179] The reference video is sliced ​​to obtain multiple video segments, and the description information of each video segment is obtained using a video description model.

[0180] According to one or more embodiments of the present disclosure, when the script acquisition unit acquires the video script according to the method, it is configured to:

[0181] Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or

[0182] Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or

[0183] The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

[0184] According to one or more embodiments of the present disclosure, the preset script library includes a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and when the script acquisition unit retrieves the preset video script related to each of the sentences from the preset script library, it is used to:

[0185] The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

[0186] According to one or more embodiments of the present disclosure, when the script acquisition unit retrieves the preset video script related to each of the sentences from the preset script library, it is used to:

[0187] Encoding the sentence using a text encoder to obtain text features of the sentence;

[0188] Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script;

[0189] A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

[0190] According to one or more embodiments of the present disclosure, when acquiring the video feature of each video material in the plurality of video materials, the video encoding unit is configured to:

[0191] For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame, and the features of each video frame are average-pooled to obtain video features of the video material.

[0192] According to one or more embodiments of the present disclosure, when acquiring the similarity matrix between the text feature and the video feature, the matching unit is used to:

[0193] Obtain at least two of the following similarities between the text features of any video script and the video features of any video material:

[0194] Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material;

[0195] The similarity matrix is ​​constructed by performing a weighted summation on the at least two similarities to obtain the similarity between the text feature of any video script and the video feature of any video material.

[0196] According to one or more embodiments of the present disclosure, when the matching unit matches the video material with the video script based on the similarity matrix to obtain a video material sequence, it is configured to:

[0197] Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence matching the video script.

[0198] According to one or more embodiments of the present disclosure, when the matching unit adopts a beam search algorithm to obtain a video material sequence matching the video script, it is configured to:

[0199] When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence;

[0200] If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

[0201] According to one or more embodiments of the present disclosure, when the video packaging unit generates a target video file according to the video material sequence, it is configured to:

[0202] Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations or texts to generate the target video file; or

[0203] A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

[0204] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0205] The memory stores computer-executable instructions;

[0206] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the video generation method described in the first aspect and various possible designs of the first aspect.

[0207] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0208] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the video generation method as described in the first aspect and various possible designs of the first aspect is implemented.

[0209] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0210] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0211] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video generation method, characterized in that: include: Obtaining a video script and obtaining text features of the video script; Obtaining video features of each video material among a plurality of video materials; Obtaining a similarity matrix between the text features and the video features; Matching the video material with the video script based on the similarity matrix to obtain a video material sequence; A target video file is generated according to the video material sequence.

2. The method according to claim 1, characterized in that: The obtaining of text features of the video script includes: The video script is encoded using a text encoder to obtain text features of the video script.

3. The method according to claim 1, characterized in that The step of obtaining the video script includes: Receiving text data of a video script input by a user, and acquiring the video script according to the text data of the video script; and / or Receive a reference video input by a user, obtain description information of the reference video, and obtain the video script according to the description information of the reference video.

4. The method according to claim 3, characterized in that The obtaining the description information of the reference video includes: The reference video is sliced ​​to obtain multiple video segments, and the description information of each video segment is obtained using a video description model.

5. The method according to claim 3 or 4, characterized in that: The step of obtaining the video script according to the method comprises: Sentence the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, and determine each of the sentences as the video script; or Segment the text data of the video script and / or the description information of the reference video to obtain a plurality of sentences, retrieve a preset video script related to each of the sentences from a preset script library, and determine the preset video script as the video script; or The text data of the video script and / or the description information of the reference video are divided into sentences to obtain multiple sentences. If the number of the sentences is less than a preset value, the preset video script related to each sentence is retrieved from the preset script library, and the sentence and the preset video script are determined as the video script.

6. The method according to claim 5, characterized in that The preset script library includes a primary preset video script and a secondary preset video script, wherein the primary preset video script is associated with at least one secondary preset video script, and the retrieving the preset video script related to each of the sentences from the preset script library includes: The target first-level preset video script related to each of the sentences is retrieved from the preset script library, and the target second-level preset video script associated with the target first-level preset video script is obtained, and the target first-level preset video script and the target second-level preset video script are determined as the preset video scripts related to each of the sentences.

7. The method according to claim 5, characterized in that The retrieving a preset video script related to each of the sentences from a preset script library includes: Encoding the sentence using a text encoder to obtain text features of the sentence; Using the text encoder to encode the preset video script in the preset script library to obtain text features of the preset video script; A preset video script related to the sentence is determined according to the text features of the sentence and the text features of the preset video script.

8. The method according to claim 1, characterized in that The obtaining of the video feature of each video material in the plurality of video materials comprises: For any video material, multiple video frames are extracted, and the video frames are encoded using a video encoder to obtain features of each video frame, and the features of each video frame are average-pooled to obtain video features of the video material.

9. The method according to claim 8, characterized in that The obtaining of a similarity matrix between the text features and the video features includes: Obtain at least two of the following similarities between the text features of any video script and the video features of any video material: Similarity between the text features of the video script and the video features of the video material, similarity between the video features of the video material and each word of the video script, similarity between the text features of the video script and the features of each video frame of the video material, average value of the maximum similarity between each video frame of the video material and each word of the video script, average value of the maximum similarity between each word of the video script and each video frame of the video material; The similarity matrix is ​​constructed by performing a weighted summation on the at least two similarities to obtain the similarity between the text feature of any video script and the video feature of any video material.

10. The method according to claim 1, characterized in that The matching of the video material with the video script based on the similarity matrix to obtain a video material sequence includes: Based on the similarity matrix, a beam search algorithm is used to obtain a video material sequence matching the video script.

11. The method according to claim 10, characterized in that The step of using a beam search algorithm to obtain a video material sequence matching the video script includes: When determining each video material in the video material sequence by the beam search algorithm, the shooting time of the candidate video material is compared with the shooting time of the previous video material in the video material sequence; If the shooting time of any candidate video material is before the shooting time of the previous video material in the video material sequence, the score of the candidate video material is reduced.

12. The method according to claim 1, characterized in that The step of generating a target video file according to the video material sequence comprises: Adding at least one of the following to the video material sequence using a preset video packaging model: music, special effects, transitions, animations or texts to generate the target video file; or A video template is obtained for the video material sequence, and video packaging is performed on the video material sequence according to the video template to generate a target video file.

13. A video generating device, characterized in that: include: A script acquisition unit, used for acquiring a video script; A text encoding unit, used to obtain text features of the video script; A video encoding unit, used for obtaining video features of each video material among a plurality of video materials; A matching unit, used to obtain a similarity matrix between the text features and the video features; Matching the video material with the video script based on the similarity matrix to obtain a video material sequence; The video packaging unit is used to generate a target video file according to the video material sequence.

14. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.

Images