Video generation and orchestration model acquisition method, apparatus, device, and storage medium

Abstract

This disclosure provides a method, apparatus, device, and storage medium for acquiring video generation and arrangement models, relating to artificial intelligence fields such as deep learning, large-scale models, natural language processing, and computer vision. The video generation method may include: acquiring text information to be processed and segmenting it into sentences to obtain target sentences; selecting video shots matching each target sentence from video shots in a media library as candidate materials for each target sentence; for each target sentence, determining a filtering method based on its position in the text information to be processed, and selecting target materials for generating the desired target video from the candidate materials corresponding to the target sentences according to the filtering method; and generating a target video based on the target materials corresponding to each target sentence. Applying the solution described in this disclosure can save costs and improve processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, apparatus, devices and storage media for obtaining video generation and arrangement models in the fields of deep learning, large models, natural language processing and computer vision. Background Technology

[0002] As a form of information medium, video is becoming increasingly popular due to its rich media format and ability to provide immersive experiences. This, in turn, raises the question of how to generate videos. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for obtaining video generation and arrangement models.

[0004] A video generation method, comprising: The text information to be processed is obtained, and the text information to be processed is segmented into sentences to obtain each target sentence; Each video shot in the material library that matches a target sentence is selected as a candidate material for that target sentence. For each target sentence, the following processing is performed: based on the position of the target sentence in the text information to be processed, the filtering method corresponding to the target sentence is determined, and target materials for generating the required target video are selected from the candidate materials corresponding to the target sentence according to the filtering method; the target video is generated based on the target materials corresponding to each target sentence.

[0005] A method for obtaining an orchestration model, comprising: In any training batch, training samples corresponding to the relevance category task and training samples corresponding to the consistency category task are obtained respectively, wherein the relevance category task and the consistency category task are both tasks corresponding to the orchestration model; The arrangement model is trained using the training samples. The trained arrangement model is used to select target materials from the candidate materials corresponding to each target sentence when generating the required target video. The target sentence is obtained by segmenting the text information to be processed into sentences. The candidate materials corresponding to any target sentence are video shots selected from each video shot in the material library that match the target sentence.

[0006] A video generation device includes: a text processing module, a material retrieval module, and a video generation module; The text processing module is used to acquire the text information to be processed and to segment the text information to be processed into sentences to obtain each target sentence; The material retrieval module is used to select video shots that match each target sentence from each video shot in the material library, as candidate materials corresponding to each target sentence; The video generation module is used to perform the following processing on each target sentence: determine the filtering method corresponding to the target sentence according to the position of the target sentence in the text information to be processed, select the target material for generating the required target video from the candidate material corresponding to the target sentence according to the filtering method, and generate the target video according to the target material corresponding to each target sentence.

[0007] An orchestration model acquisition device includes: a sample construction module and a model training module; The sample construction module is used to obtain training samples corresponding to the relevance category task and the consistency category task in any training batch, wherein the relevance category task and the consistency category task are both tasks corresponding to the orchestration model. The model training module is used to train the arrangement model using the training samples. The trained arrangement model is used to select target materials for generating the target video from the candidate materials corresponding to each target sentence when generating the required target video. The target sentence is obtained by segmenting the text information to be processed into sentences. The candidate materials corresponding to any target sentence are video shots selected from each video shot in the material library that match the target sentence.

[0008] An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0010] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart of an embodiment of the video generation method described in this disclosure; Figure 2 This is a schematic diagram illustrating the overall implementation process of the video generation method described in this disclosure; Figure 3 This is a flowchart of an embodiment of the orchestration model acquisition method described in this disclosure; Figure 4 This is a schematic diagram of the structure of the orchestration model described in this disclosure; Figure 5 This is a schematic diagram of the composition structure of Embodiment 500 of the video generation apparatus described in this disclosure; Figure 6 This is a schematic diagram of the composition structure of Embodiment 600 of the arrangement model acquisition device described in this disclosure; Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0015] Figure 1 This is a flowchart illustrating an embodiment of the video generation method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0016] In step 101, the text information to be processed is obtained and segmented into sentences to obtain each target sentence.

[0017] In step 102, video shots that match each target sentence are selected from each video shot in the material library and used as candidate materials for each target sentence.

[0018] In step 103, for each target sentence, the following processing is performed: the filtering method corresponding to the target sentence is determined according to the position of the target sentence in the text information to be processed, and the target material for generating the required target video is selected from the candidate material corresponding to the target sentence according to the filtering method; the target video is generated according to the target material corresponding to each target sentence.

[0019] Traditional video generation methods mainly include: 1) Manual editing and creation, which involves manually indexing video clips based on the script and performing manual synthesis and rendering based on the index results. However, this method involves a large amount of manual operation, which requires a large amount of manpower and time, and is inefficient; 2) Script-based live-action shooting, but this method is more expensive, less efficient, and not easy to replicate.

[0020] The solution described in the above embodiments can automatically generate the required target video based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection, and video generation. This saves manpower and time costs, improves processing efficiency, and is applicable to any text information to be processed, thus having wide applicability.

[0021] Preferably, the article selected by the user can be used as the article to be processed, and the text information of the article to be processed can be extracted, and the extracted text information can be used as the text information to be processed.

[0022] For example, the article selected by the user could be one published or reposted on a self-media platform, which may include text, images, and other information. Accordingly, text information can be extracted from the article to obtain the required text information to be processed. This allows for convenient and quick acquisition of the necessary text information based on the user-selected article.

[0023] Furthermore, preferably, before extracting text information from the article to be processed, a pre-trained article classification model can be used to classify the article. In response to determining, based on the classification result, that the article to be processed does not belong to a predetermined article type, text information extraction is performed on the article to be processed. And / or, after extracting text information, invalid information filtering can be applied to the extracted text information, and the filtered text information can be used as the text information to be processed. The article classification model can be pre-trained using constructed training samples.

[0024] In other words, for an article to be processed, its type can be determined first. If the article belongs to a predetermined type, processing can be terminated directly; otherwise, subsequent processing can continue, such as extracting text information from the article. The specific article types mentioned are not limited; they typically refer to article types unsuitable for generating corresponding videos, such as simple photo captions. This process can improve the quality of the subsequently generated video.

[0025] The text information extracted from the article to be processed can be filtered for invalid information, and then the filtered text information can be used as the text information to be processed. Invalid information usually refers to information that is not helpful to the subsequently generated video content. Filtering out this information in advance can reduce the workload of subsequent processing, save resources, and further improve processing efficiency. For example, if the article to be processed is a user-reposted article, then information such as "Article source..." can be filtered out. The specific types / aspects of invalid information can be determined according to actual needs.

[0026] The text information to be processed can be segmented into sentences, and for easy differentiation, each segmented sentence can be referred to as a target sentence. For example, the text information to be processed can be segmented according to the punctuation marks used to indicate the end of a sentence, and correspondingly, each target sentence is a complete sentence.

[0027] Then, video shots that match each target sentence can be selected from each video shot in the material library as candidate materials for each target sentence.

[0028] The media library can be pre-built and may include multiple video shots, i.e., multiple short videos, each corresponding to a complete shot. For example, different long videos can be divided using a Shot Boundary Detection (SBD) algorithm to obtain multiple video shots of the same or different durations. Low-quality filtering can be applied to the segmented video shots, and then the filtered video shots can be added to the media library.

[0029] Preferably, for each target sentence, the following processing can be performed: obtain the text encoding of the target sentence, obtain the similarity between the video encoding of each video shot in the material library and the text encoding, and select the Q video shots with the highest similarity as candidate materials corresponding to the target sentence, where Q is a positive integer greater than one.

[0030] The specific value of Q can be determined according to actual needs; for example, it can be 20.

[0031] Assuming the media library contains 1000 (the number is for illustrative purposes only) video shots, the video codes of each video shot can be obtained and stored in advance. For any target sentence, such as target sentence a, the text code of target sentence a can be obtained, and the similarity between the text code of target sentence a and the video codes of the 1000 video shots can be obtained. Then, the 20 similarity values ​​with the highest values ​​can be selected from the 1000 similarity values, and the video shots corresponding to these 20 similarity values ​​can be used as candidate media for target sentence a.

[0032] Through the above processing, candidate materials corresponding to each target sentence can be retrieved, thus laying a good foundation for subsequent processing.

[0033] Preferably, for any target sentence, the text encoding of the target sentence can be determined using a pre-trained Cross-Modal Contrastive Language-Image Pre-Training (CLIP) model. The target sentence can be used as input to the CLIP model to obtain the output text encoding. Additionally, the CLIP model can be used to determine the video encoding of each video shot in the resource library. The CLIP model is a mature and existing model, thus ensuring the accuracy of the obtained encoding results.

[0034] After obtaining the candidate materials corresponding to each target sentence, the target materials for generating the target video can be further selected from the candidate materials.

[0035] Preferably, the target sentences can be sorted according to their position in the text information to be processed, from first to last. Then, the target sentences can be traversed sequentially from front to back. For each target sentence traversed, the following processing can be performed: the traversed target sentence is taken as the target sentence to be processed. In response to the target sentence to be processed being the first target sentence after sorting, the target material corresponding to the target sentence to be processed is directly selected from the candidate material corresponding to the target sentence to be processed. In response to the target sentence to be processed being a target sentence that is not the first target sentence after sorting, the target material corresponding to the target sentence to be processed is selected from the candidate material corresponding to the target sentence to be processed, in combination with the target sentences preceding the target sentence to be processed and the target material corresponding to the target sentences preceding the target sentence to be processed.

[0036] That is, depending on whether the target sentence to be processed is the first target sentence after sorting, different processing methods can be used to determine the corresponding target material, thereby making the processing more targeted and improving the accuracy of the determined target material.

[0037] Preferably, if the target sentence to be processed is the first target sentence after sorting, then the candidate materials corresponding to the target sentence to be processed can be sorted in descending order of similarity between the corresponding video code and the text code of the target sentence to be processed, and the target material corresponding to the target sentence to be processed can be determined based on the sorting result.

[0038] Preferably, if the target sentence to be processed is not the first target sentence after sorting, then the following processing can be performed on each candidate material corresponding to the target sentence to be processed: Based on the latest obtained first sequence and second sequence, a third sequence and a fourth sequence corresponding to the candidate material are generated respectively. The first sequence includes: the video codes of the target materials corresponding to each target sentence before the target sentence to be processed. The second sequence includes: the text codes of each target sentence before the target sentence to be processed. The third sequence is obtained by adding the video code of the candidate material to the end of the first sequence. The fourth sequence is obtained by adding the text code of the target sentence to the end of the second sequence. The comprehensive score of the candidate material is determined based on the third sequence and the fourth sequence. The candidate materials are sorted in descending order of comprehensive score. The target material corresponding to the target sentence to be processed is determined based on the sorting result. The video code of the target material corresponding to the target sentence to be processed is added to the end of the first sequence to obtain the updated first sequence. The text code of the target sentence to be processed is added to the end of the second sequence to obtain the updated second sequence.

[0039] For example, assuming there are 10 target sentences, numbered from Target Sentence 1 to Target Sentence 10 in sorting order, the target material corresponding to Target Sentence 1 can be determined first. For instance, the 20 candidate materials corresponding to Target Sentence 1 can be sorted according to the similarity between the corresponding video encoding and the text encoding of Target Sentence 1, from highest to lowest. The target material corresponding to Target Sentence 1 can then be determined based on the sorting results. Next, the target materials corresponding to Target Sentences 2 through 10 can be determined sequentially. Taking Target Sentence 5 as an example, for each of the 20 candidate materials corresponding to it, the following processing can be performed: Taking Candidate Material 1 as an example, the video encoding of Candidate Material 1 can be added to the end of the first sequence to obtain the third sequence corresponding to Candidate Material 1. The text encoding of Target Sentence 5 can be added to the end of the second sequence to obtain the candidate material. The fourth sequence corresponding to Material 1 includes the video codes of the target materials corresponding to Target Sentence 1, Target Sentence 2, Target Sentence 3, and Target Sentence 4 in the first sequence and the text codes of Target Sentence 1, Target Sentence 2, Target Sentence 3, and Target Sentence 4 in the second sequence. The comprehensive score of Candidate Material 1 can be determined based on the third and fourth sequences. Similarly, the comprehensive scores of Candidate Material 2 to Candidate Material 20 can be obtained. The 20 candidate materials corresponding to Target Sentence 5 can be sorted in descending order of comprehensive score, and the target material corresponding to Target Sentence 5 can be determined based on the sorting result. After that, the first and second sequences can be updated. The video code of the target material corresponding to Target Sentence 5 can be added to the end of the first sequence, and the text code of Target Sentence 5 can be added to the end of the second sequence.

[0040] As can be seen, since there are no other target sentences before the target sentence that is first after sorting, the target material can be directly selected from the corresponding candidate material, thereby improving processing efficiency. For each target sentence that is not first after sorting, the target material can be selected from the corresponding candidate material by combining the target sentences before it and the target material corresponding to each of the target sentences before it, thereby improving the accuracy of the determined target material.

[0041] Preferably, for any candidate material corresponding to any target sentence (the target sentence to be processed) that is not in the first position after sorting, the similarity between the video encoding of the candidate material and the text encoding of the target sentence to be processed can be obtained, and the sequence relevance score between the third sequence and the fourth sequence corresponding to the candidate material and the sequence consistency score of the third sequence can be obtained. Then, the comprehensive score of the candidate material can be determined by combining the similarity, the sequence relevance score and the sequence consistency score.

[0042] For example, the similarity score, the sequence relevance score, and the sequence consistency score can be multiplied by their respective weights, and the three products can be summed to obtain the overall score for the candidate material. The specific values ​​of each weight can be determined according to actual needs.

[0043] Preferably, the sequence relevance score and the sequence consistency score can be determined using a pre-trained orchestration model based on the third and fourth sequences. For example, the third and fourth sequences can be used as inputs to the orchestration model to obtain the output sequence relevance score and the sequence consistency score.

[0044] With the help of a pre-trained orchestration model, the required score information can be obtained efficiently and accurately.

[0045] As mentioned earlier, after sorting the candidate materials corresponding to the target sentence to be processed, the target material corresponding to the target sentence can be determined based on the sorting results. Preferably, the candidate materials ranked in the top L positions after sorting can be used as the target material corresponding to the target sentence to be processed, where L is a positive integer and the value of L is a fixed value. Alternatively, the value of L can be determined according to the principle of matching the total duration of the selected target materials with the duration of the text-to-speech (TTS) speech of the target sentence to be processed.

[0046] In other words, the value of L can be a fixed value, and the specific value can be determined according to actual needs, such as 1. Alternatively, the value of L can also be dynamically determined, which is very flexible and convenient.

[0047] For the latter approach, the TTS audio duration of the target sentence to be processed can be determined according to existing methods. That is, when configuring TTS audio for the subsequently generated target video, the audio duration of the target sentence to be processed is determined. Then, the value of L can be determined according to the principle that the total duration of the selected target materials equals the TTS audio duration of the target sentence to be processed. For example, if only the candidate material ranked first after sorting is selected as the target material, the total duration is equal to the TTS audio duration of the target sentence to be processed, then the value of L is 1. If only the candidate material ranked first and second after sorting are selected, the total duration is equal to the TTS audio duration of the target sentence to be processed, then the value of L is 1. The candidate material ranked 2nd is used as the target material. The total duration is equal to the TTS speech duration of the target sentence to be processed. Then the value of L is 2, and so on. In special case, for example, if the value of L is 2, the total duration will be less than the TTS speech duration of the target sentence to be processed. However, if the value of L is 3, the total duration will be greater than the TTS speech duration of the target sentence to be processed. Therefore, the candidate material ranked 3rd after sorting can be trimmed (to shorten its duration) so that when the value of L is 3, the total duration is equal to the TTS speech duration of the target sentence to be processed.

[0048] After obtaining the target materials corresponding to each target sentence, a target video can be generated based on the target materials. Preferably, the target video can be rendered using Fast-Forward-Moving-PictureExpert Group (FFMPEG) technology based on the target materials.

[0049] It can synthesize videos of each target material according to their playback time from first to last. In addition, it can also perform the following operations: add subtitles to the synthesized video based on each target sentence, configure TTS voice, add bullet comments, add intro and outro, etc., to improve the video presentation effect.

[0050] Based on the above introduction, Figure 2 This is a schematic diagram of the overall implementation process of the video generation method described in this disclosure. For specific implementation, please refer to the aforementioned relevant descriptions, which will not be repeated here. In particular, the target material can be selected from the candidate materials by means of a pre-trained choreography model. The following is a further explanation of how to obtain the choreography model.

[0051] Figure 3 This is a flowchart illustrating an embodiment of the orchestration model acquisition method described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0052] In step 301, in any training batch, training samples corresponding to the relevance category task and the consistency category task are obtained respectively. Both the relevance category task and the consistency category task are tasks corresponding to the orchestration model.

[0053] In step 302, the arrangement model is trained using the training samples. The trained arrangement model is used to select target materials from the candidate materials corresponding to each target sentence when generating the required target video. The target sentence is obtained by segmenting the text information to be processed into sentences. The candidate materials corresponding to any target sentence are video shots selected from each video shot in the material library that match the target sentence.

[0054] The above-described method can automatically generate the required target video based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection with the help of an arrangement model, and video generation, thereby saving manpower and time costs and improving processing efficiency.

[0055] Preferably, P text-video pairs corresponding to the batch can be obtained, where P is a positive integer greater than one, and the specific value can be determined according to actual needs. Each text-video pair can include: a text set consisting of at least two sentences and a video set consisting of at least two video shots. The at least two sentences come from the same text information and are arranged in order of their appearance in the text information. Each sentence in the text set corresponds to at least one video shot in the video set, and the video shots in the video set are arranged in order of their playback time. Furthermore, training samples corresponding to the relevance category task and the consistency category task can be constructed based on the obtained text-video pairs.

[0056] For example, an article and its corresponding video can be obtained through some means, that is, a video that matches the article. Further, by extracting text information from the article and segmenting it by sentence, n sentences can be obtained, and these n sentences can be used to form a text set. In addition, the video corresponding to the article can be divided using the SBD algorithm to obtain m video shots, and these m video shots can be used to form a video set. Then, the text set and the video set can be used to form a text-video pair, where n and m are both positive integers greater than one.

[0057] In practical applications, the orchestration model can be trained using multiple batches. Within each batch, training samples for the relevance category task and the consistency category task can be constructed based on the obtained P text-video pairs. In other words, the training process of the orchestration model is constrained by two main tasks: the relevance category task and the consistency category task. Relevance refers to whether the text and video are related, and consistency refers to whether a set of consecutive video shots have a consistent style.

[0058] Preferably, the method for constructing training samples corresponding to the relevance category task may include: using each acquired text-video pair as a positive sample; for any acquired text-video pair, randomly selecting k1 video shots from other text-video pairs, where k1 is a positive integer and less than the number of video shots in the video set of the text-video pair, inserting the selected k1 video shots into the video set of the text-video pair to obtain a modified video set, and using the text set of the text-video pair and the modified video set to form a negative sample; or, for any text-video pair, randomly selecting k2 sentences from other text-video pairs, where k2 is a positive integer and less than the number of sentences in the text set of the text-video pair, inserting the selected k2 sentences into the text set of the text-video pair to obtain a modified text set, and using the modified text set and the video set of the text-video pair to form a negative sample; using the positive sample and the negative sample as training samples corresponding to the relevance category task.

[0059] The specific values ​​of k1 and k2 can be determined according to actual needs. In addition, the specific number of negative samples generated can also be determined according to actual needs.

[0060] For example, for any text-video pair 'a', k1 video shots can be randomly selected from other text-video pairs in the batch. These k1 video shots can come from the same text-video pair or from multiple different text-video pairs. These k1 video shots can be randomly inserted into the video set of text-video pair 'a' to obtain a modified video set. Then, the text set of text-video pair 'a' and the modified video set can be used to form a negative sample. Alternatively, the video set can be modified instead of the text set. That is, k2 sentences can be randomly selected from other text-video pairs in the batch. These k2 sentences can come from the same text-video pair or from multiple different text-video pairs. These k2 sentences can be randomly inserted into the text set of text-video pair 'a' to obtain a modified text set. Then, the modified text set and the video set of text-video pair 'a' can be used to form a negative sample.

[0061] Preferably, the method for constructing training samples corresponding to the consistency category task may include: for each obtained text-video pair, randomly selecting one of the following three predetermined methods to process the video set: keeping the video set unchanged; randomly selecting k3 video shots from other text-video pairs, where k3 is a positive integer and less than the number of video shots in the video set, and replacing k3 video shots in the video set with the selected k3 video shots; randomly adjusting the arrangement order of some video shots in the video set; and using each text-video pair processed according to the predetermined method as the training samples corresponding to the consistency category task. The specific value of k3 can be determined according to actual needs.

[0062] Taking text-video pair b as an example, for the video set within it, processing can be performed using method one, method two, or method three. In method one, the video set can remain unchanged. In method two, k3 video shots can be randomly selected from other text-video pairs in the batch. These k3 video shots can come from the same text-video pair or from multiple different text-video pairs, and can then be used to randomly replace k3 video shots in the video set. In method three, the arrangement order of some video shots in the video set can be randomly adjusted, thereby updating the random positions of the video set.

[0063] As can be seen, through the above processing, training samples corresponding to different tasks can be easily and quickly constructed based on the obtained text-video pairs. Then, the training samples can be used to train the orchestration model, thereby improving the training effect of the model.

[0064] As one possible implementation method, Figure 4 This is a schematic diagram of the structure of the orchestration model described in this disclosure. Figure 4As shown, the text encoding represents the text encoding of sentences in the training samples, and the video encoding represents the video encoding of video shots in the training samples. Both text and video encodings can be obtained through the CLIP model. Based on the text and video encodings, information encoding, type encoding, position encoding, transformer, and multilayer perceptron (MLP) can be performed sequentially. The diagram illustrates processing methods such as Perceptron. It assumes the number of sentences is n and the number of video shots is m. The relevance category task outputs a binary classification result, such as 0 or 1, indicating whether the text and video (sentences and video shots) are related, and calculates the first loss (loss1) accordingly. It also outputs the insertion position of the sentence or video shot and calculates the second loss (loss2). The consistency category task outputs a method prediction result, such as 0, 1, or 2, indicating whether the video set is processed using method one, method two, or method three, and calculates the third loss (loss3) accordingly. The model can then be updated by combining loss1, loss2, and loss3. This process can be repeated until the model converges. The trained model can then be applied in practice, such as selecting target materials from candidate materials corresponding to each target sentence when generating the desired target video.

[0065] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0066] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0067] Figure 5 This is a schematic diagram of the structural composition of embodiment 500 of the video generation apparatus described in this disclosure. Figure 5 As shown, it includes: a text processing module 501, a material retrieval module 502, and a video generation module 503.

[0068] The text processing module 501 is used to acquire the text information to be processed and to segment the text information into sentences to obtain each target sentence.

[0069] The material retrieval module 502 is used to select video shots that match each target sentence from each video shot in the material library, and use them as candidate materials for each target sentence.

[0070] The video generation module 503 is used to perform the following processing on each target sentence: determine the filtering method corresponding to the target sentence according to the position of the target sentence in the text information to be processed, select the target material for generating the required target video from the candidate material corresponding to the target sentence according to the filtering method, and generate the target video according to the target material corresponding to each target sentence.

[0071] The solution described in the above-described device embodiment can automatically generate the required target video based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection, and video generation. This saves manpower and time costs, improves processing efficiency, and is applicable to any text information to be processed, thus having wide applicability.

[0072] Preferably, the text processing module 501 can use the article selected by the user as the article to be processed, and can extract text information from the article to be processed, and then use the extracted text information as the text information to be processed.

[0073] Furthermore, preferably, before extracting text information from the article to be processed, the text processing module 501 can also classify the article to be processed using a pre-trained article classification model. In response to determining, based on the classification result, that the article to be processed does not belong to a predetermined article type, text information extraction can be performed on the article to be processed. And / or, after extracting text information from the article to be processed, invalid information filtering can be performed on the extracted text information, and the filtered text information can be used as the text information to be processed. The article classification model can be pre-trained using constructed training samples.

[0074] For the text information to be processed, the text processing module 501 can segment it into sentences, and each segmented sentence can be used as a target sentence. For example, the text information to be processed can be segmented according to the punctuation marks used to indicate the end of a sentence, and correspondingly, each target sentence is a complete sentence.

[0075] Subsequently, the material retrieval module 502 can select video shots that match each target sentence from the video shots in the material library, as candidate materials corresponding to each target sentence. The material library can be pre-built and may include multiple video shots, that is, multiple short videos, each short video corresponding to a complete shot.

[0076] Preferably, the material retrieval module 502 can perform the following processing for each target sentence: obtain the text code of the target sentence, obtain the similarity between the video code of each video shot in the material library and the text code, and take the Q video shots with the highest similarity as candidate materials corresponding to the target sentence, where Q is a positive integer greater than one.

[0077] Preferably, the material retrieval module 502 can determine the text encoding of any target sentence using the pre-trained CLIP model. In addition, it can also determine the video encoding of each video shot in the material library using the CLIP model.

[0078] After obtaining the candidate materials corresponding to each target sentence, the video generation module 503 can further select the target materials for generating the target video from the candidate materials.

[0079] Preferably, the video generation module 503 can sort each target sentence according to its position in the text information to be processed, and then iterate through each target sentence in the order of appearance from front to back. For each target sentence encountered, the following processing can be performed: the encountered target sentence is taken as the target sentence to be processed; if the target sentence to be processed is the first target sentence after sorting, the target material corresponding to the target sentence to be processed is directly selected from the candidate material corresponding to the target sentence to be processed; if the target sentence to be processed is not the first target sentence after sorting, the target material corresponding to the target sentence to be processed is selected from the candidate material corresponding to the target sentence to be processed, in combination with the target sentences preceding the target sentence to be processed and the target material corresponding to the target sentences preceding the target sentence to be processed.

[0080] Preferably, if the target sentence to be processed is the first target sentence after sorting, then the video generation module 503 can sort the candidate materials corresponding to the target sentence to be processed in descending order of similarity between the corresponding video code and the text code of the target sentence to be processed, and determine the target material corresponding to the target sentence to be processed based on the sorting result.

[0081] Preferably, if the target sentence to be processed is not the first target sentence after sorting, the video generation module 503 can perform the following processing on each candidate material corresponding to the target sentence to be processed: Based on the latest obtained first sequence and second sequence, generate a third sequence and a fourth sequence corresponding to the candidate material respectively. The first sequence includes: the video codes of the target materials corresponding to each target sentence before the target sentence to be processed. The second sequence includes: the text codes of each target sentence before the target sentence to be processed. The third sequence is obtained by adding the video code of the candidate material to the end of the first sequence. The fourth sequence is obtained by adding the text code of the target sentence to the end of the second sequence. The comprehensive score of the candidate material is determined based on the third sequence and the fourth sequence. The candidate materials are sorted in descending order of comprehensive score. The target material corresponding to the target sentence to be processed is determined based on the sorting result. The video code of the target material corresponding to the target sentence to be processed is added to the end of the first sequence to obtain the updated first sequence. The text code of the target sentence to be processed is added to the end of the second sequence to obtain the updated second sequence.

[0082] Preferably, for any candidate material corresponding to any target sentence (the target sentence to be processed) that is not in the first position after sorting, the video generation module 503 can obtain the similarity between the video encoding of the candidate material and the text encoding of the target sentence to be processed, and can obtain the sequence relevance score between the third sequence and the fourth sequence corresponding to the candidate material and the sequence consistency score of the third sequence. Then, the comprehensive score of the candidate material can be determined by combining the similarity, the sequence relevance score and the sequence consistency score.

[0083] For example, the similarity score, the sequence relevance score, and the sequence consistency score can be multiplied by their respective weights, and the three products can be added together to obtain the overall score of the candidate material.

[0084] Preferably, the video generation module 503 can determine the sequence relevance score and the sequence consistency score based on the third sequence and the fourth sequence using a pre-trained arrangement model.

[0085] As mentioned earlier, for the target sentence to be processed, after sorting its corresponding candidate materials, the target material corresponding to the target sentence to be processed can be determined according to the sorting result. Preferably, the video generation module 503 can use the candidate materials in the top L positions after sorting as the target material corresponding to the target sentence to be processed, where L is a positive integer and the value of L is a fixed value, or the value of L is determined according to the principle of matching the total duration of the selected target materials with the TTS speech duration of the target sentence to be processed.

[0086] After obtaining the target materials corresponding to each target sentence, the video generation module 503 can generate a target video based on the target materials. Preferably, the video generation module 503 can render the target video using FFMPEG technology based on the target materials.

[0087] Figure 6 This is a schematic diagram of the structural composition of Embodiment 600 of the arrangement model acquisition device described in this disclosure. Figure 6 As shown, it includes: a sample construction module 601 and a model training module 602.

[0088] The sample construction module 601 is used to obtain the training samples corresponding to the correlation category task and the consistency category task in any training batch. Both the correlation category task and the consistency category task are tasks corresponding to the orchestration model.

[0089] The model training module 602 is used to train the choreography model using the training samples. The trained choreography model is used to select target materials from the candidate materials corresponding to each target sentence when generating the required target video. The target sentence is obtained by segmenting the text information to be processed into sentences. The candidate materials corresponding to any target sentence are video shots selected from each video shot in the material library that match the target sentence.

[0090] By employing the solution described in the above-described device embodiment, the required target video can be automatically generated based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection with the aid of an arrangement model, and video generation, thereby saving manpower and time costs and improving processing efficiency.

[0091] Preferably, the sample construction module 601 can obtain P text-video pairs corresponding to the batch, where P is a positive integer greater than one, and the specific value can be determined according to actual needs. Each text-video pair can include: a text set consisting of at least two sentences and a video set consisting of at least two video shots. The at least two sentences come from the same text information and are arranged in order of their appearance in the text information. Each sentence in the text set corresponds to at least one video shot in the video set, and the video shots in the video set are arranged in order of their playback time. Furthermore, training samples corresponding to the relevance category task and the consistency category task can be constructed based on the obtained text-video pairs.

[0092] Preferably, the sample construction module 601 constructs training samples corresponding to the relevance category task in the following ways: Each acquired text-video pair is used as a positive sample; for any acquired text-video pair, k1 video shots are randomly selected from other text-video pairs, where k1 is a positive integer and less than the number of video shots in the video set of the text-video pair; the selected k1 video shots are inserted into the video set of the text-video pair to obtain a modified video set; a negative sample is formed using the text set of the text-video pair and the modified video set; or, for any text-video pair, k2 sentences are randomly selected from other text-video pairs, where k2 is a positive integer and less than the number of sentences in the text set of the text-video pair; the selected k2 sentences are inserted into the text set of the text-video pair to obtain a modified text set; a negative sample is formed using the modified text set and the video set of the text-video pair; the positive samples and the negative samples are used as training samples corresponding to the relevance category task.

[0093] In addition, preferably, the method by which the sample construction module 601 constructs training samples corresponding to the consistency category task may include: for each text-video pair obtained, randomly selecting one of the following three predetermined methods to process the video set: keeping the video set unchanged; randomly selecting k3 video shots from other text-video pairs, where k3 is a positive integer and less than the number of video shots in the video set, and replacing k3 video shots in the video set with the selected k3 video shots; randomly adjusting the arrangement order of some video shots in the video set; and using each text-video pair processed according to the predetermined method as the training samples corresponding to the consistency category task.

[0094] Figure 5 and Figure 6 The specific workflow of the device embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0095] In summary, the solution described in this disclosure can automatically generate the required target video, thereby saving manpower and time costs, improving processing efficiency, and ensuring the video quality of the generated target video, thus effectively promoting the development of the video ecosystem.

[0096] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly deep learning, large-scale models, natural language processing, and computer vision. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0097] Furthermore, the text information and video footage in the embodiments described in this disclosure are not targeted at any specific user and do not reflect the personal information of any particular user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0098] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0099] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0100] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0101] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the methods described in this disclosure by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0108] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: The text information to be processed is obtained, and the text information to be processed is segmented into sentences to obtain each target sentence; Each video shot in the material library that matches a target sentence is selected as a candidate material for that target sentence. For each target sentence, the following processing is performed: The target sentences are sorted according to their position in the text information to be processed, from first to last. Each target sentence is then traversed sequentially from front to back, and for each traversed target sentence, the following processing is performed: The traversed target sentence is taken as the target sentence to be processed. If the target sentence to be processed is the first target sentence after sorting, the target material corresponding to the target sentence to be processed is directly selected from the candidate materials corresponding to the target sentence to be processed. If the target sentence to be processed is not the first target sentence after sorting, the following processing is performed on each candidate material corresponding to the target sentence to be processed: Based on the latest obtained first and second sequences, a third and fourth sequence corresponding to the candidate materials are generated respectively. The first sequence includes: each target sentence preceding the target sentence to be processed corresponds to... The target material video encoding, the second sequence including: the text encoding of each target sentence preceding the target sentence to be processed, the third sequence obtained by adding the video encoding of the candidate material to the end of the first sequence, the fourth sequence obtained by adding the text encoding of the target sentence to be processed to the end of the second sequence, the comprehensive score of the candidate material determined according to the third sequence and the fourth sequence, the candidate materials sorted in descending order of comprehensive score, the target material corresponding to the target sentence to be processed determined according to the sorting result, the video encoding of the target material corresponding to the target sentence to be processed is added to the end of the first sequence to obtain an updated first sequence, and the text encoding of the target sentence to be processed is added to the end of the second sequence to obtain an updated second sequence; the target video is generated according to the target material corresponding to each target sentence.

2. The method according to claim 1, wherein, The process of obtaining the text information to be processed includes: taking the article selected by the user as the article to be processed, extracting text information from the article to be processed, and using the extracted text information as the text information to be processed.

3. The method according to claim 2, further comprising one or all of the following: Before extracting text information from the article to be processed, the article to be processed is classified using a pre-trained article classification model. In response to determining that the article to be processed does not belong to a predetermined article type based on the classification result, text information is extracted from the article to be processed. After extracting text information from the article to be processed, the extracted text information is filtered for invalid information, and the filtered text information is used as the text information to be processed.

4. The method according to claim 1, wherein, The step of selecting video shots that match each target sentence from each video shot in the material library as candidate materials for each target sentence includes: For any target sentence, the following processing is performed: obtain the text encoding of the target sentence, obtain the similarity between the video encoding of each video shot in the material library and the text encoding, and select the Q video shots with the highest similarity as candidate materials corresponding to the target sentence, where Q is a positive integer greater than one.

5. The method according to claim 4, wherein, The step of obtaining the text encoding of the target sentence includes: determining the text encoding of the target sentence using a pre-trained cross-modal contrastive learning model; It also includes: using the cross-modal contrastive learning model to determine the video encoding of each video shot in the material library.

6. The method according to claim 1, wherein, The step of directly selecting the target material corresponding to the target sentence from the candidate material corresponding to the target sentence after sorting, in response to the target sentence being the first target sentence after sorting, includes: The candidate materials corresponding to the target sentence to be processed are sorted in descending order of similarity between the corresponding video encoding and the text encoding of the target sentence to be processed, and the target material corresponding to the target sentence to be processed is determined based on the sorting result.

7. The method according to claim 1, wherein, The determination of the comprehensive score of the candidate material based on the third sequence and the fourth sequence includes: The similarity between the video encoding of the candidate material and the text encoding of the target sentence to be processed is obtained, and the sequence relevance score between the third sequence and the fourth sequence and the sequence consistency score of the third sequence are obtained. The comprehensive score of the candidate material is determined by combining the similarity score, the sequence relevance score, and the sequence consistency score.

8. The method according to claim 7, wherein, The process of obtaining the sequence relevance score between the third sequence and the fourth sequence, and the sequence consistency score of the third sequence, includes: Based on the third sequence and the fourth sequence, the sequence relevance score and the sequence consistency score are determined using a pre-trained orchestration model.

9. The method according to claim 1, wherein, The step of determining the target material corresponding to the target sentence to be processed based on the sorting result includes: taking the candidate material in the top L positions after sorting as the target material corresponding to the target sentence to be processed, where L is a positive integer and the value of L is a fixed value, or the value of L is determined according to the principle of matching the total duration of the selected target material with the text-to-speech (TTS) duration of the target sentence to be processed.

10. The method according to any one of claims 1 to 9, wherein, The step of generating the target video based on the target material corresponding to each target sentence includes: rendering the target video using fast image expert group technology based on the target material.

11. A method for obtaining an orchestration model, the method being used to generate the orchestration model as described in claim 8, comprising: In any training batch, training samples corresponding to the relevance category task and training samples corresponding to the consistency category task are obtained respectively, wherein the relevance category task and the consistency category task are both tasks corresponding to the orchestration model; The arrangement model is trained using the training samples.

12. The method according to claim 11, wherein, The step of obtaining training samples corresponding to relevance category tasks and consistency type tasks respectively includes: Obtain P text-video pairs corresponding to the batch, where P is a positive integer greater than one. Each text-video pair includes: a text set consisting of at least two sentences and a video set consisting of at least two video shots. The at least two sentences come from the same text information and are arranged in order of their appearance in the text information. Each sentence in the text set corresponds to at least one video shot in the video set, and the video shots in the video set are arranged in order of their playback time. Based on the acquired text-video pairs, training samples corresponding to the relevance category task and the consistency category task are constructed.

13. The method according to claim 12, wherein, The training samples constructed for the relevance category task include: Each text-video pair obtained is used as a positive sample; For any text-video pair obtained, randomly select k1 video shots from other text-video pairs, where k1 is a positive integer and less than the number of video shots in the video set of the text-video pair. Insert the selected k1 video shots into the video set of the text-video pair to obtain a modified video set. Use the text set of the text-video pair and the modified video set to form a negative sample. Alternatively, for any text-video pair, randomly select k2 sentences from other text-video pairs, where k2 is a positive integer and less than the number of sentences in the text set of the text-video pair. Insert the selected k2 sentences into the text set of the text-video pair to obtain a modified text set. Use the modified text set and the video set of the text-video pair to form a negative sample. The positive and negative samples are used as training samples for the correlation category task.

14. The method according to claim 12 or 13, wherein, The training samples corresponding to the consistency category task are constructed as follows: For each text-video pair obtained, the video set within it is randomly selected and processed using one of the following three predetermined methods: keeping the video set unchanged; randomly selecting k3 video shots from other text-video pairs, where k3 is a positive integer and less than the number of video shots in the video set, and replacing the k3 video shots in the video set with the selected k3 video shots; or randomly adjusting the arrangement order of some video shots in the video set. Each text-video pair processed according to the predetermined method will be used as a training sample for the consistency category task.

15. A video generation apparatus comprising: Text processing module, material retrieval module, and video generation module; The text processing module is used to acquire the text information to be processed and to segment the text information to be processed into sentences to obtain each target sentence; The material retrieval module is used to select video shots that match each target sentence from each video shot in the material library, as candidate materials corresponding to each target sentence; The video generation module is configured to perform the following processing on each target sentence: sort each target sentence according to its position in the text information to be processed; traverse each target sentence sequentially according to the sorting order, and perform the following processing on each target sentence encountered: take the encountered target sentence as the target sentence to be processed; in response to the target sentence to be processed being the first target sentence after sorting, directly select the target material corresponding to the target sentence to be processed from the candidate material corresponding to the target sentence to be processed; in response to the target sentence to be processed being a target sentence not the first target sentence after sorting, perform the following processing on each candidate material corresponding to the target sentence to be processed: generate a third sequence and a fourth sequence corresponding to the candidate material based on the latest obtained first sequence and second sequence, respectively, wherein the first sequence includes: each target sentence preceding the target sentence to be processed. The video encoding of the target material corresponding to each sentence is as follows: the second sequence includes the text encoding of each target sentence preceding the target sentence to be processed; the third sequence is obtained by adding the video encoding of the candidate material to the end of the first sequence; the fourth sequence is obtained by adding the text encoding of the target sentence to be processed to the end of the second sequence; the comprehensive score of the candidate material is determined based on the third and fourth sequences; the candidate materials are sorted in descending order of comprehensive score; the target material corresponding to the target sentence to be processed is determined based on the sorting result; the video encoding of the target material corresponding to the target sentence to be processed is added to the end of the first sequence to obtain an updated first sequence; and the text encoding of the target sentence to be processed is added to the end of the second sequence to obtain an updated second sequence; a target video is generated based on the target material corresponding to each target sentence.

16. The apparatus according to claim 15, wherein, The text processing module takes the article selected by the user as the article to be processed, extracts text information from the article to be processed, and uses the extracted text information as the text information to be processed.

17. The apparatus according to claim 16, wherein, The text processing module is further configured to: classify the article to be processed using a pre-trained article classification model before extracting text information from the article to be processed; extract text information from the article to be processed in response to determining, based on the classification result, that the article to be processed does not belong to a predetermined article type; and / or, after extracting text information from the article to be processed, filter out invalid information from the extracted text information and use the filtered text information as the text information to be processed.

18. The apparatus according to claim 15, wherein, For any target sentence, the material retrieval module performs the following processing: obtains the text encoding of the target sentence, obtains the similarity between the video encoding of each video shot in the material library and the text encoding, and selects the Q video shots with the highest similarity as candidate materials corresponding to the target sentence, where Q is a positive integer greater than one.

19. The apparatus according to claim 18, wherein, The material retrieval module uses a pre-trained cross-modal contrastive learning model to determine the text encoding of the target sentence; The material retrieval module is further used to determine the video encoding of each video shot in the material library using the cross-modal contrastive learning model.

20. The apparatus according to claim 15, wherein, In response to the fact that the target sentence to be processed is the first target sentence after sorting, the video generation module sorts the candidate materials corresponding to the target sentence to be processed in descending order of similarity between the corresponding video encoding and the text encoding of the target sentence to be processed, and determines the target material corresponding to the target sentence to be processed based on the sorting result.

21. The apparatus according to claim 15, wherein, The video generation module obtains the similarity between the video encoding of the candidate material and the text encoding of the target sentence to be processed, and obtains the sequence relevance score between the third sequence and the fourth sequence and the sequence consistency score of the third sequence. The module combines the similarity, the sequence relevance score and the sequence consistency score to determine the comprehensive score of the candidate material.

22. The apparatus according to claim 21, wherein, The video generation module determines the sequence relevance score and the sequence consistency score based on the third sequence and the fourth sequence using a pre-trained arrangement model.

23. The apparatus according to claim 15, wherein, When the video generation module determines the target material corresponding to the target sentence to be processed based on the sorting result, it takes the candidate material in the top L positions after sorting as the target material corresponding to the target sentence to be processed. L is a positive integer and the value of L is a fixed value. Alternatively, the value of L is determined according to the principle that the total duration of the selected target material matches the text-to-speech (TTS) duration of the target sentence to be processed.

24. The apparatus according to any one of claims 15 to 23, wherein, The video generation module renders the target video using rapid image expert group technology based on the target material.

25. An apparatus for acquiring an choreography model, the apparatus being used to generate the choreography model as described in claim 22, comprising: Sample construction module and model training module; The sample construction module is used to obtain training samples corresponding to the relevance category task and the consistency category task in any training batch, wherein the relevance category task and the consistency category task are both tasks corresponding to the orchestration model. The model training module is used to train the orchestration model using the training samples.

26. The apparatus according to claim 25, wherein, The sample construction module obtains P text-video pairs corresponding to the batch, where P is a positive integer greater than one. Each text-video pair includes: a text set consisting of at least two sentences and a video set consisting of at least two video shots. The at least two sentences come from the same text information and are arranged in order of their appearance in the text information. Each sentence in the text set corresponds to at least one video shot in the video set, and the video shots in the video set are arranged in order of their playback time. Based on the obtained text-video pairs, training samples corresponding to the relevance category task and training samples corresponding to the consistency category task are constructed.

27. The apparatus according to claim 26, wherein, The sample construction module uses each acquired text-video pair as a positive sample. For any acquired text-video pair, it randomly selects k1 video shots from other text-video pairs, where k1 is a positive integer and less than the number of video shots in the video set of the text-video pair. The selected k1 video shots are inserted into the video set of the text-video pair to obtain a modified video set. A negative sample is formed using the text set of the text-video pair and the modified video set. Alternatively, for any text-video pair, it randomly selects k2 sentences from other text-video pairs, where k2 is a positive integer and less than the number of sentences in the text set of the text-video pair. The selected k2 sentences are inserted into the text set of the text-video pair to obtain a modified text set. A negative sample is formed using the modified text set and the video set of the text-video pair. The positive sample and the negative sample are used as training samples for the relevance category task.

28. The apparatus according to claim 26 or 27, wherein, For each text-video pair obtained, the sample construction module randomly selects one of the following three predetermined methods to process the video set: keep the video set unchanged; randomly select k3 video shots from other text-video pairs, where k3 is a positive integer and less than the number of video shots in the video set, and replace the k3 video shots in the video set with the selected k3 video shots; randomly adjust the arrangement order of some video shots in the video set; and use each text-video pair processed according to the predetermined method as the training sample corresponding to the consistency category task.

29. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

31. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-14.

Images