Video generation and organizational model obtaining method, apparatus, device, and storage medium

Abstract

To provide a video generation method and an organizational model obtaining method, an apparatus, a device, and a storage medium.SOLUTION: A video generation method comprises: obtaining processing waiting text information and dividing it into phrases to obtain each target phrase; selecting, as candidate materials corresponding to each target phrase, each video shot that matches each target phrase from among video shots in the material library; determining, for the target phrase, a screening scheme corresponding to the target phrase according to an appearance position of the target phrase in the processing waiting text information, and selecting target materials for generating a desired target video from among the candidate materials corresponding to the target phrase according to the screening scheme; and generating the target video from the target materials corresponding to each target phrase.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for obtaining a video generation and organization model in the fields of deep learning, big model, natural language processing, computer vision and the like. [Background technology]

[0002] Video, as a type of information medium, is becoming increasingly popular due to its characteristics of rich media format and its ability to provide an immersive experience, etc. Accordingly, the question of how to generate the video is raised. Summary of the Invention [Problem to be solved by the invention]

[0003] The present disclosure provides video generation and organization model acquisition methods, apparatus, devices, and storage media. [Means for solving the problem]

[0004] A video generation method is provided, the method including: obtaining text information to be processed; dividing the text information to be processed by phrases to obtain target phrases; selecting, as candidate material corresponding to each target phrase, video shots matching each target phrase from among video shots in a material library; determining, for each target phrase, a screening scheme corresponding to the target phrase according to a position at which the target phrase appears in the text information to be processed; selecting target material for generating a desired target video from among the candidate materials corresponding to the target phrase according to the screening scheme; and generating the target video based on the target material corresponding to each target phrase.

[0005] The present invention provides a method for obtaining an organizational model, the method including: obtaining, in any one of the training batches, training samples corresponding to a correlation category task, which is a task corresponding to the organizational model; and obtaining training samples corresponding to a consistency category task, which is a task corresponding to the organizational model; training the organizational model using the training samples; and, when generating a desired target video, the trained organizational model is configured to select target material for generating the target video from candidate material corresponding to each target phrase, the target phrase being obtained by dividing text information to be processed by phrase, and the candidate material corresponding to any one of the target phrases being video shots that match the target phrases, each selected from each video shot in a material library.

[0006] We provide a video generation device comprising: a text processing module that acquires text information to be processed and divides the text information to be processed by phrases to obtain each target phrase; a material search module that selects, as candidate material corresponding to each target phrase, video shots that match each target phrase from among video shots in a material library; and a video generation module that determines, for each target phrase, a screening method corresponding to the target phrase according to a position at which the target phrase appears in the text information to be processed, and performs a process of selecting target material for generating a desired target video from among candidate materials corresponding to the target phrase according to the screening method, and generates the target video based on the target material corresponding to each target phrase.

[0007] The present invention provides an apparatus for acquiring an organizational model, the apparatus comprising: a sample construction module for acquiring, in any one of the training batches, training samples corresponding to a correlation category task, which is a task corresponding to the organizational model, and training samples corresponding to a consistency category task, which is a task corresponding to the organizational model; and a model training module for training the organizational model using the training samples, the trained organizational model being configured to select target material for generating a desired target video from candidate material corresponding to each target phrase, the target phrases being obtained by dividing text information to be processed by phrase, and the candidate material corresponding to any one of the target phrases being video shots that are matched to the target phrases selected from each video shot in a material library.

[0008] The present invention provides an electronic device comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the above method.

[0009] A non-transitory computer readable storage medium is provided having stored thereon computer instructions for causing a computer to perform the above method.

[0010] A computer program product is provided which comprises computer programs / instructions which, when executed by a processor, implement the above method.

[0011] It should be understood that the contents described in this section are not intended to identify key or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood from the following specification. [Brief description of the drawings]

[0012] The drawings are for better understanding of the present technical solution and are not intended to limit the present application. [Figure 1] 1 is a flow chart of an embodiment of a video generation method according to the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of the overall implementation process of the video generation method according to the present disclosure. [Diagram 3] 1 is a flowchart of an embodiment of a method for obtaining an organization model according to the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of the structure of an organizational model according to the present disclosure. [Diagram 5] FIG. 5 is a schematic diagram of a configuration of an embodiment 500 of a video generation device according to the present disclosure. [Figure 6] 6 is a schematic diagram of a configuration of an embodiment 600 of an organization model acquisition device according to the present disclosure. [Figure 7] 7 is a schematic block diagram of an electronic device 700 that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, exemplary embodiments of the present application will be described based on the drawings. For ease of understanding, various details of the embodiments of the present application are included, and they should be regarded as merely examples. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of simplicity, the following description omits the description of well-known functions and structures.

[0014] Furthermore, the term "and / or" in this specification is simply a relational relationship describing related objects, and means that three relations can exist. For example, A and / or B can mean three situations: A exists alone, A and B exist simultaneously, and B exists alone. Also, the phrase " / " in this specification generally means that the related objects before and after it are in an "or" relationship.

[0015] 1 is a flowchart of an embodiment of a video generating method according to the present disclosure. As shown in FIG. 1, the following specific embodiments are included:

[0016] In step 101, the waiting text information is obtained, and the waiting text information is divided into phrases to obtain each target phrase.

[0017] In step 102, video shots that match each target phrase are selected from the video shots in the material library as candidate materials corresponding to each target phrase.

[0018] In step 103, for each target phrase, a screening method corresponding to the target phrase is determined according to the position of the target phrase in the text information to be processed, and a target material for generating a desired target video is selected from the candidate materials corresponding to the target phrase in accordance with the screening method, and a target video is generated based on the target material corresponding to each target phrase.

[0019] Traditional video creation methods mainly include: 1) artificial editing and creation, that is, artificially indexing video clips based on manuscripts, and artificially synthesizing and rendering based on the indexing results, etc., which involves a large amount of artificial operations, so it requires relatively large labor and time costs and is inefficient; 2) live-action shooting based on scenarios, which is expensive, inefficient and difficult to replicate.

[0020] Meanwhile, by adopting the technical solution described in the above embodiment of the method, a desired target video can be automatically generated based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection, video generation, etc., which can save labor costs and time costs, improve processing efficiency, and can be applied to any text information to be processed, and has a wide applicability, etc.

[0021] Preferably, the sentence selected by the user is set as a waiting sentence, text information extraction is performed on the waiting sentence, and the extracted text information is set as the waiting text information.

[0022] For example, the text selected by the user may refer to a text that the user has published or reprinted on a self-media platform and that may include various information such as text and pictures. This allows text information extraction to be performed on the text to obtain desired waiting text information. In other words, the desired waiting text information can be easily and quickly obtained based on the text selected by the user.

[0023] Also, preferably, before extracting text information from the waiting text, the waiting text is classified using a pre-trained text classification model, and when it is determined based on the classification result that the waiting text does not belong to a predetermined text type, text information is extracted from the waiting text, and / or after extracting text information from the waiting text, invalid information filtering is performed on the extracted text information, and the filtered text information is used as the waiting text information. The text classification model can be obtained by training in advance using a constructed training sample.

[0024] That is, for a text waiting to be processed, the text type to which it belongs can first be identified. If the text type is a predetermined text type, the process can be terminated. If not, the subsequent process, i.e., extraction of text information from the text waiting to be processed, can be performed. The specific text type of the predetermined text type is not limited, and usually refers to a text type that is not suitable for generating a corresponding video, such as a simple photo commentary type. This process can improve the quality of the video to be generated later.

[0025] Furthermore, invalid information filtering may be performed on the text information extracted from the waiting text, and the filtered text information may be the waiting text information. Since invalid information usually refers to information that is not useful for the video content to be generated later, filtering such information in advance may reduce the amount of work in the later processing, save resource consumption, and further improve processing efficiency. For example, if the waiting text is a text that has been copied by a user, information such as "the source of the text..." may be filtered. The specific information / how much information is included in the invalid information may be determined according to actual needs.

[0026] The to-be-processed text information may be divided into phrases. For ease of distinction, each of the divided phrases may be referred to as a target phrase. For example, the to-be-processed text information may be divided according to punctuation marks that indicate the end of a phrase. Each target phrase is then a complete phrase.

[0027] Thereafter, video shots that match each target phrase may be selected from the video shots in the material library as candidate materials corresponding to each target phrase.

[0028] The material library is pre-built and can contain multiple video shots, i.e., multiple short videos, each of which corresponds to one complete shot. For example, different collected long videos can be divided using a shot boundary detection (SBD) algorithm to obtain multiple video shots of the same or different durations, and the divided video shots can be subjected to low-quality filtering, etc., before the filtered video shots are added to the material library.

[0029] Preferably, for each target phrase, a text code of the target phrase is obtained, and a similarity between the video code of each video shot in the material library and the text code is obtained, and Q video shots with the highest similarity are selected as candidate materials corresponding to the target phrase, where Q is a positive integer greater than 1.

[0030] The specific value of Q can be set according to actual needs, and may be, for example, 20.

[0031] If the material library contains 1,000 (the number is just an example) video shots, the video code of each video shot can be acquired and stored in advance, and for an arbitrary target phrase such as target phrase a, the text code of target phrase a can be acquired, and the similarity between the text code of target phrase a and the video codes of the 1,000 video shots can be acquired. Furthermore, from the acquired 1,000 similarities, the 20 similarities with the largest values ​​can be selected, and the video shots corresponding to the 20 similarities can be used as candidate materials corresponding to target phrase a.

[0032] The above process allows candidate material for each target phrase to be retrieved individually, providing a good basis for subsequent processing.

[0033] Preferably, for any target phrase, a pre-trained cross-modal contrastive learning (CLIP, Contrastive Language-Image Pre-Training) model may be used to determine the text code of the target phrase. That is, the target phrase can be used as an input to the CLIP model to obtain a text code that is output. Also, the CLIP model can be used to determine the video code of each video shot in the material library. Since the CLIP model is a mature existing model, the accuracy of the obtained coding results is ensured.

[0034] After obtaining candidate materials corresponding to each target phrase, target materials for generating a target video may be further selected from the candidate materials.

[0035] Preferably, the target phrases are sorted in the order of their appearance in the awaiting text information, and then the target phrases are traversed in order from front to back, and each time a traversed target phrase is made a awaiting target phrase. If the awaiting target phrase is the top target phrase after sorting, target material corresponding to the awaiting target phrase is directly selected from the candidate materials corresponding to the awaiting target phrase, and if the awaiting target phrase is not the top target phrase after sorting, target material corresponding to the awaiting target phrase is selected from the candidate materials corresponding to the awaiting target phrase in combination with each target phrase preceding the awaiting target phrase and its corresponding target material.

[0036] In other words, depending on whether the target phrase to be processed is the top target phrase after sorting or not, different processing methods can be adopted to determine the corresponding target material, which can make the processing more accurate and improve the accuracy of the determined target material.

[0037] Preferably, if the waiting target phrase is the top target phrase after sorting, the candidate materials corresponding to the waiting target phrase are sorted in descending order of similarity between the corresponding video code and the text code of the waiting target phrase, and the target material corresponding to the waiting target phrase is determined based on the sorting result.

[0038] Preferably, when the to-be-processed target phrase is not the top target phrase after sorting, the following steps may be performed: for each candidate material corresponding to the to-be-processed target phrase, based on a most recently obtained first sequence including video codes of target materials corresponding to each target phrase preceding the to-be-processed target phrase and a most recently obtained second sequence including text codes of each target phrase preceding the to-be-processed target phrase, a third sequence obtained by adding the video codes of the candidate material to the end of the first sequence and a fourth sequence obtained by adding the text codes of the to-be-processed target phrase to the end of the second sequence, respectively, corresponding to the candidate material; a total score of the candidate material is determined based on the third and fourth sequences; the candidate materials are sorted in descending order of total score; the target material corresponding to the to-be-processed target phrase is determined based on the sorting result; the video codes of the target material corresponding to the to-be-processed target phrase are added to the end of the first sequence to obtain an updated first sequence; and the text codes of the to-be-processed target phrase are added to the end of the second sequence to obtain an updated second sequence.

[0039] For example, if the text includes 10 target phrases numbered as target phrase 1 to target phrase 10 in sorting order, the target material corresponding to target phrase 1 may be determined first. For example, 20 candidate materials corresponding to target phrase 1 may be sorted in descending order of similarity between the corresponding video code and the text code of target phrase 1, and the target material corresponding to target phrase 1 may be determined based on the sorting result, and then the target materials corresponding to target phrases 2 to 10 may be determined in sequence. Taking target phrase 5 as an example, the following processing may be performed for each of the 20 candidate materials corresponding to it. Taking candidate material 1 as an example, the video code of candidate material 1 may be added to the end of a first sequence including the video codes of target materials corresponding to target phrase 1, target phrase 2, target phrase 3, and target phrase 4 to obtain a third sequence corresponding to candidate material 1, the text code of target phrase 5 may be added to the end of a second sequence including the text codes of target phrase 1, target phrase 2, target phrase 3, and target phrase 4 to obtain a fourth sequence corresponding to candidate material 1, and further an overall score for candidate material 1 may be determined based on the third sequence, the fourth sequence, and so on. Similarly, the total scores of candidate material 2 to candidate material 20 may be obtained, and the 20 candidate materials corresponding to target phrase 5 may be sorted in descending order of total score, and the target material corresponding to target phrase 5 may be determined based on the sorting result. After that, the first sequence and the second sequence may be further updated, i.e., the video code of the target material corresponding to target phrase 5 may be added to the end of the first sequence, and the text code of target phrase 5 may be added to the end of the second sequence.

[0040] In this way, since there are no other target phrases before the top target phrase after sorting, processing efficiency is improved by selecting target material directly from the corresponding candidate material, while for each target phrase that does not become the top after sorting, the previous target phrase and the target material corresponding to each previous target phrase are combined to select target material from the corresponding candidate material, thereby improving the accuracy of the determined target material, etc.

[0041] Preferably, for any candidate material corresponding to any target phrase (awaiting processing target phrase) that is not ranked first after sorting, a similarity between the video code of the candidate material and the text code of the waiting target phrase is obtained, a sequence correlation score between the third and fourth sequences corresponding to the candidate material and a sequence consistency score of the third sequence are obtained, and further, an overall score for the candidate material is determined by combining the similarity, the sequence correlation score and the sequence consistency score.

[0042] For example, the similarity, the sequence correlation score, and the sequence consistency score can be multiplied by a corresponding weight, and the three products can be added together to obtain the total score of the candidate material. The specific value of each weight can be determined according to actual needs.

[0043] Preferably, the sequence correlation score and the sequence consistency score may be determined using a pre-trained organizational model based on the third sequence and the fourth sequence, for example, the third sequence and the fourth sequence may be used as inputs of the organizational model to obtain the output sequence correlation score and the sequence consistency score.

[0044] That is, the pre-trained orchestration model allows the desired score information to be obtained efficiently and accurately.

[0045] As described above, for a pending target phrase, the candidate materials corresponding to the pending target phrase are sorted, and then the target materials corresponding to the pending target phrase can be determined based on the sorting result. Preferably, the top L candidate materials after sorting can be determined as the target materials corresponding to the pending target phrase. L is a positive integer. The value of L is a fixed value or a value determined based on the principle that the total duration of the selected target materials matches the Text-to-Speech (TTS) speech duration of the pending target phrase.

[0046] That is, the value of L may be a fixed value, and the specific value may be determined according to actual needs, such as 1. Alternatively, the value of L may be a dynamically determined value, which is very flexible and convenient.

[0047] In the latter method, the TTS voice time of the target phrase to be processed, i.e., the voice time of the target phrase when the TTS voice is arranged in the target video to be subsequently generated, is determined by the existing method, and then the value of L may be determined according to the principle that the total time of the selected target materials is equal to the TTS voice time of the target phrase to be processed. For example, when only the first candidate material after sorting is selected as the target material and the total time is equal to the TTS voice time of the target phrase to be processed, the value of L is 1. When the first and second candidate materials after sorting are selected as the target materials and the total time is equal to the TTS voice time of the target phrase to be processed, the value of L is 2. The rest is similar. In a special case, for example, when the value of L is 2, the total time is smaller than the TTS voice time of the target phrase to be processed, while when the value of L is 3, the total time is larger than the TTS voice time of the target phrase to be processed, the third candidate material after sorting may be edited to a certain extent (to shorten its time) so that when the value of L is 3, the total time is equal to the TTS voice time of the target phrase to be processed.

[0048] After obtaining the target material corresponding to each target phrase respectively, a target video can be generated from the target material, and preferably, the target video can be obtained by rendering according to the target material using Fast-Forward-Moving-Picture Expert Group (FFMPEG) technology.

[0049] In other words, video synthesis can be performed for each target material in the order of the playback time of each target material. Furthermore, the presentation effect of the video can be improved by adding subtitles to the synthesized video based on each target phrase, arranging TTS voice, adding barrages, adding openings and endings, etc.

[0050] In conjunction with the above description, Figure 2 is a schematic diagram of the overall implementation process of the video generation method according to the present disclosure, and the specific implementation can refer to the above related description, and will not be described in detail here. Here, a pre-trained organization model can be used to select a target material from the candidate materials. The method of obtaining the organization model will be further described below.

[0051] 3 is a flowchart of an embodiment of a method for obtaining a formation model according to the present disclosure. As shown in FIG. 3, the following specific embodiments are included:

[0052] In step 301, in any one of the training batches, a training sample corresponding to a correlation category task, which is a task corresponding to an organizational model, and a training sample corresponding to a consistency category task, which is a task corresponding to an organizational model, are obtained.

[0053] In step 302, the training samples are used to train an organization model, and the trained organization model is configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and the candidate material corresponding to any target phrase being a video shot that matches the target phrase, selected from among video shots in a material library.

[0054] By adopting the technical proposal of the above-mentioned method embodiment, a desired target video can be automatically generated based on the waiting text information through a series of operations such as text segmentation, candidate material selection, target material selection according to an organizational model, and video generation, thereby saving labor costs and time costs and improving processing efficiency, etc.

[0055] Preferably, P text-video pairs corresponding to the batch can be obtained, where P is a positive integer greater than 1, and a specific value can be set according to actual needs. Each text-video pair can include a text set of at least two phrases and a video set of at least two video shots. The at least two phrases are obtained from the same text information and sorted in the order of their appearance in the text information, each phrase in the text set corresponds to at least one video shot in the video set, and each video shot in the video set is sorted in the order of their playing time. Furthermore, a training sample corresponding to a correlation category task and a training sample corresponding to a consistency category task can be constructed based on the obtained text-video pairs.

[0056] For example, a certain sentence and its corresponding video, i.e., a video that matches the sentence, can be obtained by some method. Furthermore, the sentence can be subjected to text information extraction and phrase-by-phrase division to obtain n phrases, and a text set can be constructed using these n phrases. Furthermore, a video corresponding to the sentence can be divided using an SBD algorithm to obtain m video shots, and a video set can be constructed using these m video shots. Furthermore, a text video pair can be constructed using the text set and the video set. Both n and m are positive integers greater than 1.

[0057] In practical applications, the organization model can be trained by multiple batches, where in each batch, based on the P text-video pairs obtained, a training sample corresponding to the correlation category task and a training sample corresponding to the consistency category task can be constructed respectively. That is, in the training process of the organization model, there are mainly two major tasks that are constrained, one is the correlation category task and the other is the consistency category task. Correlation refers to whether the text and the video are correlated, and consistency refers to whether the style of a series of consecutive video shots is consistent.

[0058] Preferably, the method for constructing training samples corresponding to the correlation category task may include: setting each acquired text video pair as a positive sample; for any acquired text video pair, randomly selecting k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs; inserting the selected k1 video shots into the video set in the text video pair to obtain a corrected video set; constructing a negative sample using the text set and the corrected video set in the text video pair; or, for any text video pair, randomly selecting k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs; inserting the selected k2 phrases into the text set in the text video pair to obtain a corrected text set; constructing a negative sample using the corrected text set and the video set in the text video pair; and setting the positive sample and the negative sample as training samples corresponding to the correlation category task.

[0059] The specific values ​​of k1 and k2 can be determined according to actual needs, and the specific number of negative samples to be generated can also be determined according to actual needs.

[0060] For example, for any text video pair a, k1 video shots may be randomly selected from other text video pairs in the batch. The k1 video shots may be obtained from the same text video pair or from multiple different text video pairs. Furthermore, the k1 video shots may be randomly inserted into the video set in the text video pair a to obtain a corrected video set, and the text set in the text video pair a and the corrected video set may be used to construct one negative sample. Alternatively, the text set may be corrected without correcting the video set. That is, k2 phrases may be randomly selected from other text video pairs in the batch. The k2 phrases may be obtained from the same text video pair or from multiple different text video pairs. Furthermore, the k2 phrases may be randomly inserted into the text set in the text video pair a to obtain a corrected text set, and the corrected text set and the video set in the text video pair a may be used to construct one negative sample.

[0061] Preferably, the method for constructing training samples corresponding to the consistency category task may include: for each text video pair obtained, for the video set in the text video pair, randomly select and process one of three predetermined methods: keeping the video set unchanged; randomly selecting k3 (k3 is a positive integer, smaller than the number of video shots in the video set) video shots from other text video pairs, and replacing the selected k3 video shots with the k3 video shots in the video set; and randomly adjusting the sorting order of some video shots in the video set, and each text video pair processed by the predetermined method is a training sample corresponding to the consistency category task. The specific value of k3 can be determined according to actual needs.

[0062] Taking text video pair b as an example, the video set therein can be processed by randomly selecting method 1, method 2, or method 3. Here, in method 1, the video set can be left unchanged. In method 2, k3 video shots can be randomly selected from other text video pairs in the batch. The k3 video shots can be obtained from the same text video pair or from multiple different text video pairs. Furthermore, the k3 video shots can randomly replace k3 video shots in the video set. In method 3, the sorting order of some video shots in the video set can be randomly adjusted, i.e., random position updates can be performed on the video set.

[0063] As can be seen from the above description, through the above processing, training samples corresponding to different tasks can be easily and quickly constructed based on the acquired text-video pairs, and the training samples can be used to train an organizational model, thereby improving the model training effect, etc.

[0064] FIG. 4 is a schematic structural diagram of an organization model according to the present disclosure as one possible embodiment. As shown in FIG. 4, the text code represents the text code of a phrase in a training sample, and the video code represents the video code of a video shot in a training sample. Both the text code and the video code can be obtained by a CLIP model. The text code and the video code can be sequentially processed by information coding, type coding, position coding, Transformer, Multilayer Perceptron (MLP), etc. In the figure, the number of phrases is n, and the number of video shots is m. In addition, the correlation category task outputs a binary classification result such as 0 or 1, which indicates whether the text and the video (phrase and the video shot) are correlated or not, and accordingly calculates a first loss (loss1), and outputs an insertion position of the phrase or the video shot, and accordingly calculates a second loss (loss2). The consistency category task outputs a method prediction result such as 0, 1, or 2, which indicates whether the video set is processed using the method 1, method 2, or method 3, and accordingly calculates a third loss (loss3). Furthermore, the model can be updated by combining loss 1, loss 2 and loss 3. The above process is repeated until the model converges, and the trained model can be applied in practical applications, such as generating a desired target video, selecting target material for generating the target video from candidate materials corresponding to each target phrase, etc.

[0065] In addition, for the above-mentioned embodiment of the method, the example is expressed as a combination of a series of operations for the sake of simplicity of description, but as a person skilled in the art will understand, some steps according to the present application can be performed in other orders or simultaneously, so the present application is not limited to the order of operations described. Next, a person skilled in the art should understand that all the embodiments described in the specification are preferred embodiments, and related operations and modules are not necessarily required by the present application. In addition, for parts not described in detail in one embodiment, reference can be made to related descriptions in other embodiments.

[0066] The above describes the method embodiment, and the following uses an apparatus embodiment to further describe the technical solution described in this disclosure.

[0067] 5 is a schematic diagram of a configuration of a video generation device according to an embodiment 500 of the present disclosure. As shown in FIG. 5, the device includes a text processing module 501, a material search module 502, and a video generation module 503.

[0068] The text processing module 501 obtains the pending text information and divides the pending text information into phrases to obtain each target phrase.

[0069] The material search module 502 selects, as candidate material corresponding to each target phrase, video shots that match each target phrase from among the video shots in the material library.

[0070] The video generation module 503 determines a screening method corresponding to each target phrase according to the position where the target phrase appears in the text information to be processed, selects target materials for generating a desired target video from candidate materials corresponding to the target phrase according to the screening method, and generates a target video based on the target materials corresponding to each target phrase.

[0071] By adopting the technical solution described in the embodiment of the device, a desired target video can be automatically generated based on the text information to be processed through a series of operations such as text segmentation, candidate material selection, target material selection, video generation, etc., which can save labor costs and time costs, improve processing efficiency, and can be applied to any text information to be processed, and has a wide range of applicability.

[0072] Preferably, the text processing module 501 may set the sentence selected by the user as a waiting sentence, extract text information from the waiting sentence, and set the extracted text information as the waiting text information.

[0073] Preferably, the text processing module 501 classifies the waiting text using a pre-trained text classification model before extracting text information from the waiting text, and extracts text information from the waiting text in response to determining that the waiting text does not belong to a predetermined text type based on the classification result, and / or performs invalid information filtering on the extracted text information after extracting text information from the waiting text, and sets the filtered text information as the waiting text information. The text classification model can be obtained by training in advance using a constructed training sample.

[0074] The text processing module 501 may divide the to-be-processed text information into phrases and treat each divided phrase as a target phrase. For example, the to-be-processed text information may be divided according to punctuation marks that indicate the end of a phrase, and each target phrase may become a complete phrase accordingly.

[0075] The material search module 502 may then select a video shot that matches each target phrase from among the video shots in the material library as a candidate material corresponding to each target phrase. The material library is pre-built and may include multiple video shots, i.e., multiple short videos, each corresponding to a complete shot.

[0076] Preferably, the material search module 502 may obtain a text code of the target phrase for each target phrase, obtain a similarity between the video code of each video shot in the material library and the text code, and select Q video shots with the highest similarity as candidate materials corresponding to the target phrase, where Q is a positive integer greater than 1.

[0077] Preferably, for any target phrase, the material retrieval module 502 determines a text code for that target phrase using a pre-trained CLIP model, and may also use the CLIP model to determine a video code for each video shot in the material library.

[0078] After the candidate materials corresponding to each target phrase are obtained, the video generation module 503 may further select target materials for generating a target video from among the candidate materials.

[0079] Preferably, the video generation module 503 may sort each target phrase in the order of its appearance in the to-be-processed text information, then traverse each target phrase in front-to-back order, and for each traversed target phrase, set it as a to-be-processed target phrase. If the to-be-processed target phrase is the top target phrase after sorting, the video generation module 503 may directly select target material corresponding to the to-be-processed target phrase from among the candidate materials corresponding to the to-be-processed target phrase, or if the to-be-processed target phrase is not the top target phrase after sorting, select target material corresponding to the to-be-processed target phrase from among the candidate materials corresponding to the to-be-processed target phrase, together with each target phrase preceding the to-be-processed target phrase and its corresponding target material.

[0080] Preferably, if the pending target phrase is the top target phrase after sorting, the video generation module 503 may sort each candidate material corresponding to the pending target phrase in descending order of similarity between the corresponding video code and the text code of the pending target phrase, and determine the target material corresponding to the pending target phrase based on the sorting result.

[0081] Preferably, when the waiting target phrase is not the top target phrase after sorting, the video generation module 503 may generate, for each candidate material corresponding to the waiting target phrase, a third sequence obtained by adding the video code of the candidate material to the end of the first sequence and a fourth sequence obtained by adding the text code of the waiting target phrase to the end of the second sequence, based on a most recently obtained first sequence including video codes of target materials corresponding to each target phrase located before the waiting target phrase and a most recently obtained second sequence including text codes of each target phrase located before the waiting target phrase, respectively, corresponding to the candidate material, determine a total score of the candidate material based on the third and fourth sequences, sort the candidate materials in descending order of total score, determine the target material corresponding to the waiting target phrase based on the sorting result, add the video code of the target material corresponding to the waiting target phrase to the end of the first sequence to obtain an updated first sequence, and add the text code of the waiting target phrase to the end of the second sequence to obtain an updated second sequence.

[0082] Preferably, for any candidate material corresponding to any target phrase (awaiting processing target phrase) that is not ranked first after sorting, the video generation module 503 obtains a similarity between the video code of the candidate material and the text code of the waiting target phrase, obtains a sequence correlation score between the third and fourth sequences corresponding to the candidate material and a sequence consistency score of the third sequence, and further determines an overall score for the candidate material by combining the similarity, the sequence correlation score and the sequence consistency score.

[0083] For example, the similarity score, the sequence correlation score, and the sequence consistency score may each be multiplied by a corresponding weight, and the three products may be added together to form an overall score for the candidate material.

[0084] Preferably, the video generation module 503 may determine the sequence correlation score and the sequence consistency score using a pre-trained organizational model based on the third sequence and the fourth sequence.

[0085] As described above, for a pending target phrase, after sorting the corresponding candidate materials, target materials corresponding to the pending target phrase can be determined based on the sorting result. Preferably, the video generation module 503 may select the top L candidate materials after sorting as target materials corresponding to the pending target phrase. L is a positive integer. The value of L is a fixed value or a value determined based on the principle that the total duration of the selected target materials matches the TTS voice duration of the pending target phrase.

[0086] After the target material corresponding to each target phrase is respectively obtained, the video generation module 503 may generate a target video based on the target material. Preferably, the video generation module 503 can obtain the target video by rendering according to the target material with FFMPEG technology.

[0087] 6 is a schematic diagram of the configuration of an embodiment 600 of an apparatus for obtaining a composition model according to the present disclosure. As shown in FIG. 6, the apparatus includes a sample construction module 601 and a model training module 602.

[0088] The sample construction module 601 acquires, in any one of the training batches, training samples corresponding to a correlation category task, which is a task corresponding to the organizational model, and training samples corresponding to a consistency category task, which is a task corresponding to the organizational model.

[0089] The model training module 602 uses the training samples to train an organization model, and the trained organization model is configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and the candidate material corresponding to any target phrase being a video shot that matches the target phrase, each of which is selected from among video shots in a material library.

[0090] By adopting the technical solution of the embodiment of the device described above, it is possible to automatically generate a desired target video based on the text information waiting to be processed through a series of operations such as text segmentation, candidate material selection, target material selection according to an editing model, and video generation, thereby saving labor costs and time costs and improving processing efficiency, etc.

[0091] Preferably, the sample construction module 601 can obtain P text-video pairs corresponding to the batch, where P is a positive integer greater than 1, and the specific value can be set according to actual needs. Each text-video pair may include a text set of at least two phrases and a video set of at least two video shots. The at least two phrases are obtained from the same text information and sorted in order of appearance position in the text information, each phrase in the text set corresponds to at least one video shot in the video set, and each video shot in the video set is sorted in order of playing time. Furthermore, a training sample corresponding to a correlation category task and a training sample corresponding to a consistency category task can be constructed based on the obtained text-video pairs.

[0092] Preferably, the method of the sample construction module 601 constructing the training sample corresponding to the correlation category task may include: taking each acquired text video pair as a positive sample; for any acquired text video pair, randomly selecting k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs; inserting the selected k1 video shots into the video set in the text video pair to obtain a corrected video set; constructing a negative sample using the text set and the corrected video set in the text video pair; or, for any text video pair, randomly selecting k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs; inserting the selected k2 phrases into the text set in the text video pair to obtain a corrected text set; constructing a negative sample using the corrected text set and the video set in the text video pair; and using the positive sample and the negative sample as the training sample corresponding to the correlation category task;

[0093] In addition, preferably, the method in which the sample construction module 601 constructs training samples corresponding to the consistency category task may include randomly selecting and processing one of three predetermined methods for the video set in each acquired text video pair, including leaving the video set unchanged, randomly selecting k3 (k3 is a positive integer and is smaller than the number of video shots in the video set) video shots from other text video pairs and replacing the selected k3 video shots with k3 video shots in the video set, and randomly adjusting the order of some of the video shots in the video set, and setting each text video pair processed by the predetermined method as a training sample corresponding to the consistency category task.

[0094] The specific workflow of the apparatus embodiment shown in FIG. 5 and FIG. 6 can refer to the relevant description in the above-mentioned method embodiment, so it will not be described in detail here.

[0095] In short, by adopting the technical solution described in this disclosure, the desired target video can be automatically generated, saving labor and time costs, improving processing efficiency, and ensuring the video effects, etc. of the generated target video, thereby effectively promoting the development of video eco.

[0096] The technical proposal described in this application can be applied to the field of artificial intelligence, especially to such fields as deep learning, large-scale model, natural language processing and computer vision. Artificial intelligence is a discipline that studies how computers simulate human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, etc. Artificial intelligence software technologies mainly include computer vision technology, voice recognition technology, natural language processing technology, and several directions such as machine learning / deep learning, big data processing technology, knowledge map technology, etc.

[0097] The text information and video shots in the embodiments described in this disclosure are not targeted to any specific user, and do not reflect any specific user's personal information. In the technical solution disclosed in this disclosure, the acquisition, storage, application, processing, transmission, provision and distribution of the personal information of the users involved all comply with the provisions of relevant laws and regulations and are not contrary to public order and morals.

[0098] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0099] 7 is a schematic block diagram of an electronic device 700 that may be used to implement embodiments of the present disclosure. The electronic device represents various forms of digital computers, such as laptops, desktop computers, workbenches, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as PDAs, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present disclosure as described and / or claimed herein.

[0100] 7, the device 700 includes a computing means 701 that can perform various appropriate operations and processes according to a computer program stored in a read only memory (ROM) 702 or loaded from a storage means 708 into a random access memory (RAM) 703. The RAM 703 may store various programs and data necessary for the operation of the device 700. The computing means 701, the ROM 702, and the RAM 703 are connected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0101] Several components of the device 700 are connected to the I / O interface 705, including input means 706, e.g. a keyboard, mouse, etc., output means 707, e.g. various types of displays, speakers, etc., storage means 708, e.g. magnetic disks, optical disks, etc., and communication means 709, e.g. a network card, modem, wireless communication transceiver, etc. The communication means 709 enables the device 700 to exchange information / data with other devices via computer networks, e.g. the Internet, and / or various telecommunication networks.

[0102] The computing means 701 may be various general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing means 701 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing means 701 executes various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program physically embodied in a machine-readable medium, such as the storage means 708. In some embodiments, some or all of the computer program may be loaded and / or installed in the device 700 via the ROM 702 and / or the communication means 709. When the computer program is loaded into the RAM 703 and executed by the computing means 701, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing means 701 may be configured in any other suitable manner (eg, via firmware) to perform the methods described herein.

[0103] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), field programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include being embodied in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor. The programmable processor may be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transfer data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Program codes for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that, when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes can be executed entirely on the machine, partially on the machine, as a stand-alone package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, a machine-readable medium is a tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media include one or more line-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) for providing input to the computer by the user. Other types of devices may also be used to provide interaction with a user. For example, feedback provided to the user may be any form of sensing feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form (including sound input, speech input, or tactile input).

[0107] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and an internetwork.

[0108] The computer system may include clients and servers. The clients and servers are generally remote from each other and typically interact through a communication network. The relationship of clients and servers arises by virtue of computer programs running on the corresponding computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server in combination with a blockchain.

[0109] It should be understood that steps can be rearranged, added, or removed using the various types of flows shown above. For example, each step described in this application can be performed in a parallel order or a sequential order, or can be performed in a different order, and is not limited thereto, as long as the desired results of the technical solution disclosed in this application can be achieved.

[0110] The above specific embodiments do not constitute limitations on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, partial combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the scope of protection of this application. (Other possible items) (Item 1) 1. A method for generating video, comprising: obtaining text information to be processed, and dividing the text information to be processed by phrases to obtain target phrases; Selecting video shots that match each target phrase from among the video shots in the material library as candidate materials corresponding to each target phrase; For each target phrase, a screening method corresponding to the target phrase is determined according to the position of the target phrase in the text information to be processed, and a process is performed to select target materials for generating a desired target video from among candidate materials corresponding to the target phrase according to the screening method, and the target video is generated based on the target materials corresponding to each target phrase. The method includes: (Item 2) The obtaining of the to-be-processed text information includes: The text selected by the user is treated as a waiting text, text information is extracted from the waiting text, and the extracted text information is treated as the waiting text. The method according to item 1. (Item 3) classifying the waiting text using a pre-trained text classification model before extracting text information from the waiting text, and extracting text information from the waiting text when it is determined based on the classification result that the waiting text does not belong to a predetermined text type; extracting text information from the waiting document, and then filtering the extracted text information with invalid information, and setting the filtered text information as the waiting text information; Further includes any or all of The method according to item 2. (Item 4) selecting, as candidate materials corresponding to each target phrase, video shots that match each target phrase from among the video shots in a material library; For each target phrase, a text code of the target phrase is obtained, a similarity between the video code of each video shot in the material library and the text code is obtained, and Q video shots (Q is a positive integer greater than 1) with the highest similarity are selected as candidate materials corresponding to the target phrase. 2. The method according to item 1, comprising: (Item 5) obtaining a textual code of the target phrase includes determining a textual code of the target phrase using a pre-trained cross-modal contrastive learning model; 5. The method of claim 4, further comprising using the cross-modal contrastive learning model to determine a video code for each video shot in the material library. (Item 6) determining a screening method corresponding to the target phrase according to an appearance position of the target phrase in the to-be-processed text information, and selecting target material for generating a desired target video from among candidate materials corresponding to the target phrase according to the screening method; sorting each target phrase in the order of its occurrence in the to-be-processed text information, and traversing each target phrase in order from front to back; for each traversed target phrase, the process selects target material directly corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase if the pending target phrase is the top target phrase after sorting, and selects target material corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase in combination with each target phrase preceding the pending target phrase and its corresponding target material if the pending target phrase is not the top target phrase after sorting; 5. The method according to item 4, comprising: (Item 7) selecting target material directly corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase in response to the pending target phrase being the top target phrase after sorting; sorting each candidate material corresponding to the pending target phrase in descending order of similarity between the corresponding video code and the text code of the pending target phrase, and determining a target material corresponding to the pending target phrase based on the sorting result; 7. The method according to item 6, comprising: (Item 8) selecting target material corresponding to the pending target phrase from among candidate material corresponding to the pending target phrase, in conjunction with each target phrase preceding the pending target phrase and its corresponding target material, in response to the pending target phrase not being the top target phrase after sorting; for each candidate material corresponding to the pending target phrase, generate a third sequence obtained by adding the video code of the candidate material to the end of the first sequence and a fourth sequence obtained by adding the text code of the target phrase to the end of the second sequence, based on a most recently obtained first sequence including video codes of target materials corresponding to each target phrase located before the pending target phrase and a most recently obtained second sequence including text codes of each target phrase located before the pending target phrase, and determine an overall score for the candidate material based on the third sequence and the fourth sequence; sorting each candidate material in descending order of overall score, determining a target material corresponding to said pending target phrase based on the sorting result, adding a video code of the target material corresponding to said pending target phrase to the end of said first sequence to obtain an updated first sequence, and adding a text code of said pending target phrase to the end of said second sequence to obtain an updated second sequence; 7. The method according to item 6, comprising: (Item 9) determining an overall score for the candidate material based on the third sequence and the fourth sequence, obtaining a similarity between the video code of the candidate material and the text code of the target phrase to be processed; obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence agreement score for the third sequence; combining said similarity scores, said sequence correlation scores, and said sequence identity scores to determine an overall score for said candidate material; 9. The method according to item 8, comprising: (Item 10) Obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence agreement score for the third sequence, determining the sequence correlation score and the sequence agreement score based on the third sequence and the fourth sequence using a pre-trained organizational model; 10. The method according to item 9, comprising: (Item 11) determining target material corresponding to the pending target phrase based on the sorting result includes determining top L candidate materials after sorting as target material corresponding to the pending target phrase; L is a positive integer, and the value of L is a fixed value or a value determined based on the principle that the total duration of the selected target material matches the text-to-speech TTS speech duration of the pending target phrase; Item 9. The method according to item 7 or 8. (Item 12) generating the target video based on target material corresponding to each target phrase includes obtaining the target video by rendering the target material with a quick picture expert technique; The method according to any one of items 1 to 10. (Item 13) A method for obtaining an organization model, comprising the steps of: In any one of the training batches, a training sample corresponding to a correlation category task, which is a task corresponding to the organizational model, and a training sample corresponding to a consistency category task, which is a task corresponding to the organizational model, are obtained; training the organization model using the training samples, the trained organization model being configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and the candidate material corresponding to any target phrase being a video shot that matches the target phrase, the video shot being selected from among the video shots in a material library; The method includes: (Item 14) Obtaining training samples corresponding to the correlation category task and training samples corresponding to the consistency category task, respectively, Obtain P text-video pairs (P is a positive integer greater than 1) corresponding to the training batch, each text-video pair including a text set of at least two phrases and a video set of at least two video shots, the at least two phrases being obtained from the same text information and sorted in order of their appearance positions in the text information, each phrase in the text set corresponding to at least one video shot in the video set, and the video shots in the video set being sorted in order of their playing time; Based on the obtained text-video pairs, construct a training sample corresponding to the correlation category task and a training sample corresponding to the agreement category task. 14. The method according to item 13, comprising: (Item 15) Constructing training samples corresponding to the correlation category task includes: Each obtained text-video pair is taken as a positive sample, For any of the obtained text video pairs, randomly select k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs, insert the selected k1 video shots into the video set in the text video pair to obtain a corrected video set, and use the text set in the text video pair and the corrected video set to construct a negative sample; or, for any of the text video pairs, randomly select k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs, insert the selected k2 phrases into the text set in the text video pair to obtain a corrected text set, and use the corrected text set and the video set in the text video pair to construct a negative sample; The positive samples and the negative samples are training samples corresponding to the correlation category task. 15. The method according to item 14, comprising: (Item 16) Constructing training samples corresponding to the matching category task includes: For each text video pair obtained, the video set in the text video pair is randomly selected and processed in one of three predetermined ways: keeping the video set unchanged; randomly selecting k3 (k3 is a positive integer and is smaller than the number of video shots in the video set) video shots from other text video pairs and replacing the selected k3 video shots with the k3 video shots in the video set; or randomly adjusting the order of some video shots in the video set; Each of the text-video pairs processed in the predetermined manner is used as a training sample corresponding to the matching category task. 16. The method according to item 14 or 15, comprising: (Item 17) 1. A video generation device, comprising: a text processing module that obtains pending text information and divides the pending text information into phrases to obtain target phrases; a material retrieval module for selecting, as candidate material corresponding to each target phrase, video shots that match each target phrase from among the video shots in the material library; a video generation module for determining a screening method for each target phrase according to a position of the target phrase in the to-be-processed text information, selecting target materials for generating a desired target video from among candidate materials for the target phrase according to the screening method, and generating the target video based on the target materials for each target phrase; An apparatus comprising: (Item 18) Item 18. The device according to item 17, wherein the text processing module sets a sentence selected by the user as a waiting sentence, performs text information extraction on the waiting sentence, and sets the extracted text information as the waiting text information. (Item 19) The text processing module further comprises: Before extracting text information from the pending text, classifying the pending text using a pre-trained text classification model, and extracting text information from the pending text when it is determined based on the classification result that the pending text does not belong to a predetermined text type; and / or extracting text information from the waiting text, and then filtering the extracted text information with invalid information, and setting the filtered text information as the waiting text information; Item 19. The apparatus according to item 18. (Item 20) The material search module includes: The device described in item 17, for any target phrase, obtains a text code of the target phrase, obtains a similarity between the video code of each video shot in the material library and the text code, and performs a process of selecting Q video shots (Q is a positive integer greater than 1) with the greatest similarity as candidate materials corresponding to the target phrase. (Item 21) the material retrieval module determines a text code of the target phrase using a pre-trained cross-modal contrastive learning model; 21. The apparatus of claim 20, wherein the material retrieval module further uses the cross-modal contrastive learning model to determine a video code for each video shot in the material library. (Item 22) The video generation module includes: sorting each target phrase in the order of appearance position in the to-be-processed text information, traversing each target phrase in front-to-back order, setting each traversed target phrase as a to-be-processed target phrase, and if the to-be-processed target phrase is the top target phrase after sorting, selecting target material corresponding to the to-be-processed target phrase directly from among candidate materials corresponding to the to-be-processed target phrase, and if the to-be-processed target phrase is not the top target phrase after sorting, selecting target material corresponding to the to-be-processed target phrase from among candidate materials corresponding to the to-be-processed target phrase, in combination with each target phrase preceding the to-be-processed target phrase and its corresponding target material; 21. The apparatus according to item 20. (Item 23) The video generation module includes: if the pending target phrase is the top target phrase after sorting, sorting the candidate materials corresponding to the pending target phrase in descending order of similarity between the corresponding video code and the text code of the pending target phrase, and determining the target material corresponding to the pending target phrase based on the sorting result; 23. The apparatus according to item 22. (Item 24) The video generation module includes: if the pending target phrase is not the top target phrase after sorting, generate, for each candidate material corresponding to the pending target phrase, a third sequence obtained by adding a video code of the candidate material to the end of the first sequence and a fourth sequence obtained by adding a text code of the pending target phrase to the end of the second sequence, based on a most recently obtained first sequence including a video code of a target material corresponding to each target phrase located before the pending target phrase and a most recently obtained second sequence including a text code of each target phrase located before the pending target phrase, and determine an overall score for the candidate material based on the third sequence and the fourth sequence; sorting each candidate material in descending order of overall score, determining a target material corresponding to said pending target phrase based on the sorting result, adding a video code of the target material corresponding to said pending target phrase to the end of said first sequence to obtain an updated first sequence, and adding a text code of said pending target phrase to the end of said second sequence to obtain an updated second sequence; 23. The apparatus according to item 22. (Item 25) The video generation module includes: obtaining a similarity between a video code of the candidate material and a text code of the target phrase to be processed, obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence consistency score for the third sequence, and determining an overall score for the candidate material by combining the similarity, the sequence correlation score, and the sequence consistency score; 25. The apparatus according to item 24. (Item 26) The video generation module includes: determining the sequence correlation score and the sequence agreement score based on the third sequence and the fourth sequence using a pre-trained organizational model; 26. The apparatus according to item 25. (Item 27) when determining target material corresponding to the target phrase to be processed based on the sorting result, the video generation module determines top L candidate materials after sorting as target materials corresponding to the target phrase to be processed; 25. The apparatus of claim 23 or 24, wherein L is a positive integer, and the value of L is a fixed value or a value determined based on the principle that the total duration of the selected target material matches the text-to-speech TTS speech duration of the to-be-processed target phrase. (Item 28) 27. The device according to any one of items 17 to 26, wherein the video generation module obtains the target video by rendering according to a quick picture expert technique based on the target material. (Item 29) An apparatus for acquiring a composition model, comprising: A sample construction module for obtaining, in each training batch, a training sample corresponding to a correlation category task, which is a task corresponding to the organizational model, and a training sample corresponding to a consistency category task, which is a task corresponding to the organizational model; a model training module that uses the training samples to train the organization model, the trained organization model being configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and each candidate material corresponding to a target phrase being a video shot that is selected from among video shots in a material library to match the target phrase; An apparatus comprising: (Item 30) The sample construction module includes: Obtain P text-video pairs (P is a positive integer greater than 1) corresponding to the training batch, each text-video pair including a text set of at least two phrases and a video set of at least two video shots, the at least two phrases being obtained from the same text information and sorted in order of their appearance positions in the text information, each phrase in the text set corresponding to at least one video shot in the video set, and the video shots in the video set being sorted in order of their playing time; Based on the obtained text-video pairs, construct a training sample corresponding to the correlation category task and a training sample corresponding to the agreement category task. 30. The apparatus according to item 29. (Item 31) The sample construction module includes: Each obtained text-video pair is taken as a positive sample, For any of the obtained text video pairs, randomly select k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs, insert the selected k1 video shots into the video set in the text video pair to obtain a corrected video set, and use the text set in the text video pair and the corrected video set to construct a negative sample; or, for any of the text video pairs, randomly select k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs, insert the selected k2 phrases into the text set in the text video pair to obtain a corrected text set, and use the corrected text set and the video set in the text video pair to construct a negative sample; The positive samples and the negative samples are training samples corresponding to the correlation category task. Item 31. The apparatus according to item 30. (Item 32) The sample construction module includes: For each of the obtained text video pairs, the video set in the text video pair is randomly selected and processed in one of three predetermined methods: keeping the video set unchanged; randomly selecting k3 (k3 is a positive integer and is smaller than the number of video shots in the video set) video shots from other text video pairs and replacing the selected k3 video shots with the k3 video shots in the video set; or randomly adjusting the arrangement order of some of the video shots in the video set; Each of the text-video pairs processed in the predetermined manner is used as a training sample corresponding to the matching category task. Item 32. The device according to item 30 or 31. (Item 33) At least one processor; a memory in communication with the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of items 1 to 16. An electronic device. (Item 34) A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the method according to any one of items 1 to 16. (Item 35) A computer program which, when executed by a processor, implements the method according to any one of items 1 to 16.

Claims

1. 1. A method for generating video, comprising: obtaining a textual information to be processed, and dividing the textual information to be processed by phrases to obtain target phrases; Selecting video shots that match each target phrase from among the video shots in the material library as candidate materials corresponding to each target phrase; For each target phrase, a screening method corresponding to the target phrase is determined according to the position of the target phrase in the text information to be processed, and a process is performed to select target materials for generating a desired target video from among candidate materials corresponding to the target phrase according to the screening method, and the target video is generated based on the target materials corresponding to each target phrase. The method includes:

2. The obtaining of the to-be-processed text information includes: The text selected by the user is treated as a waiting text, text information is extracted from the waiting text, and the extracted text information is treated as the waiting text. The method of claim 1.

3. classifying the waiting text using a pre-trained text classification model before extracting text information from the waiting text, and extracting text information from the waiting text when it is determined based on the classification result that the waiting text does not belong to a predetermined text type; extracting text information from the waiting document, and then filtering the extracted text information with invalid information, and setting the filtered text information as the waiting text information; Further includes any or all of The method of claim 2.

4. selecting, as candidate materials corresponding to each target phrase, video shots that match each target phrase from among the video shots in a material library; For each target phrase, a text code of the target phrase is obtained, a similarity between the video code of each video shot in the material library and the text code is obtained, and Q video shots (Q is a positive integer greater than 1) with the highest similarity are selected as candidate materials corresponding to the target phrase.

2. The method of claim 1, comprising:

5. obtaining a textual code of the target phrase includes determining a textual code of the target phrase using a pre-trained cross-modal contrastive learning model; The method of claim 4 , further comprising: using the cross-modal contrastive learning model to determine a video code for each video shot in the material library, respectively.

6. determining a screening method corresponding to the target phrase according to an appearance position of the target phrase in the to-be-processed text information, and selecting target material for generating a desired target video from among candidate materials corresponding to the target phrase according to the screening method; sorting each target phrase in the order of its occurrence in the to-be-processed text information, and traversing each target phrase in order from front to back; for each traversed target phrase, the process selects target material directly corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase if the pending target phrase is the top target phrase after sorting, and selects target material corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase in combination with each target phrase preceding the pending target phrase and its corresponding target material if the pending target phrase is not the top target phrase after sorting; 5. The method of claim 4, comprising:

7. selecting target material directly corresponding to the pending target phrase from among candidate materials corresponding to the pending target phrase in response to the pending target phrase being the top target phrase after sorting; sorting each candidate material corresponding to the pending target phrase in descending order of similarity between the corresponding video code and the text code of the pending target phrase, and determining a target material corresponding to the pending target phrase based on the sorting result; 7. The method of claim 6, comprising:

8. selecting target material corresponding to the pending target phrase from among candidate material corresponding to the pending target phrase, in conjunction with each target phrase preceding the pending target phrase and its corresponding target material, in response to the pending target phrase not being the top target phrase after sorting; for each candidate material corresponding to the pending target phrase, generate a third sequence obtained by adding the video code of the candidate material to the end of the first sequence and a fourth sequence obtained by adding the text code of the target phrase to the end of the second sequence, based on a most recently obtained first sequence including video codes of target materials corresponding to each target phrase located before the pending target phrase and a most recently obtained second sequence including text codes of each target phrase located before the pending target phrase, and determine an overall score for the candidate material based on the third sequence and the fourth sequence; sorting each candidate material in descending order of overall score, determining a target material corresponding to said pending target phrase based on the sorting result, adding a video code of the target material corresponding to said pending target phrase to the end of said first sequence to obtain an updated first sequence, and adding a text code of said pending target phrase to the end of said second sequence to obtain an updated second sequence; 7. The method of claim 6, comprising:

9. determining an overall score for the candidate material based on the third sequence and the fourth sequence, obtaining a similarity between the video code of the candidate material and the text code of the target phrase to be processed; obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence agreement score for the third sequence; combining said similarity scores, said sequence correlation scores, and said sequence identity scores to determine an overall score for said candidate material; 9. The method of claim 8, comprising:

10. Obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence agreement score for the third sequence, determining the sequence correlation score and the sequence agreement score based on the third sequence and the fourth sequence using a pre-trained organizational model; 10. The method of claim 9, comprising:

11. determining target material corresponding to the pending target phrase based on the sorting result includes determining top L candidate materials after sorting as target material corresponding to the pending target phrase; L is a positive integer, and the value of L is a fixed value or a value determined based on the principle that the total duration of the selected target material matches the text-to-speech TTS speech duration of the pending target phrase; The method of claim 7.

12. generating the target video based on target material corresponding to each target phrase includes obtaining the target video by rendering the target material with a quick picture expert technique; The method of claim 1.

13. A method for obtaining an organization model, comprising the steps of: In one of the training batches, a training sample corresponding to a correlation category task, which is a task corresponding to the organizational model, and a training sample corresponding to a consistency category task, which is a task corresponding to the organizational model, are obtained; training the organization model using the training samples, the trained organization model being configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and the candidate material corresponding to any target phrase being a video shot that matches the target phrase, the video shot being selected from among the video shots in a material library; The method includes:

14. Obtaining training samples corresponding to the correlation category task and training samples corresponding to the consistency category task, respectively, Obtain P text-video pairs (P is a positive integer greater than 1) corresponding to the training batch, each text-video pair including a text set of at least two phrases and a video set of at least two video shots, the at least two phrases being obtained from the same text information and sorted in order of their appearance positions in the text information, each phrase in the text set corresponding to at least one video shot in the video set, and the video shots in the video set being sorted in order of their playing time; Based on the obtained text-video pairs, construct a training sample corresponding to the correlation category task and a training sample corresponding to the agreement category task.

14. The method of claim 13, comprising:

15. Constructing training samples corresponding to the correlation category task includes: Each obtained text-video pair is taken as a positive sample, For any of the obtained text video pairs, randomly select k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs, insert the selected k1 video shots into the video set in the text video pair to obtain a corrected video set, and use the text set in the text video pair and the corrected video set to construct a negative sample; or, for any of the text video pairs, randomly select k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs, insert the selected k2 phrases into the text set in the text video pair to obtain a corrected text set, and use the corrected text set and the video set in the text video pair to construct a negative sample; S The positive samples and the negative samples are training samples corresponding to the correlation category task.

15. The method of claim 14, comprising:

16. Constructing training samples corresponding to the matching category task includes: For each text video pair obtained, the video set in the text video pair is randomly selected and processed in one of three predetermined ways: keeping the video set unchanged; randomly selecting k3 (k3 is a positive integer and is smaller than the number of video shots in the video set) video shots from other text video pairs and replacing the selected k3 video shots with the k3 video shots in the video set; or randomly adjusting the order of some video shots in the video set; Each of the text-video pairs processed in the predetermined manner is used as a training sample corresponding to the matching category task.

15. The method of claim 14, comprising:

17. 1. A video generation device, comprising: a text processing module that obtains pending text information and divides the pending text information into phrases to obtain target phrases; a material retrieval module for selecting, as candidate material corresponding to each target phrase, video shots that match each target phrase from among the video shots in the material library; a video generation module for determining a screening method for each target phrase according to a position of the target phrase in the to-be-processed text information, selecting target materials for generating a desired target video from among candidate materials for the target phrase according to the screening method, and generating the target video based on the target materials for each target phrase; An apparatus comprising:

18. 18. The apparatus according to claim 17, wherein the text processing module sets a sentence selected by a user as a waiting sentence, performs text information extraction on the waiting sentence, and sets the extracted text information as the waiting text information.

19. The text processing module further comprises: Before extracting text information from the pending text, classifying the pending text using a pre-trained text classification model, and extracting text information from the pending text when it is determined based on the classification result that the pending text does not belong to a predetermined text type; and / or extracting text information from the waiting text, and then filtering the extracted text information with invalid information, and setting the filtered text information as the waiting text information; 20. The apparatus of claim 18.

20. The material search module includes: The device of claim 17, further comprising: a text code for each target phrase; a similarity between the text code and a video code of each video shot in the material library; and a process of selecting Q video shots (Q is a positive integer greater than 1) having the highest similarity as candidate materials corresponding to the target phrase.

21. the material retrieval module determines a text code of the target phrase using a pre-trained cross-modal contrastive learning model; 21. The apparatus of claim 20, wherein the material retrieval module is further configured to use the cross-modal contrastive learning model to determine a video code for each video shot in the material library.

22. The video generation module includes: sorting each target phrase in the order of appearance position in the to-be-processed text information, traversing each target phrase in front-to-back order, setting each traversed target phrase as a to-be-processed target phrase, and if the to-be-processed target phrase is the top target phrase after sorting, selecting target material corresponding to the to-be-processed target phrase directly from among candidate materials corresponding to the to-be-processed target phrase, and if the to-be-processed target phrase is not the top target phrase after sorting, selecting target material corresponding to the to-be-processed target phrase from among candidate materials corresponding to the to-be-processed target phrase, in combination with each target phrase preceding the to-be-processed target phrase and its corresponding target material; 21. The apparatus of claim 20.

23. The video generation module includes: if the pending target phrase is the top target phrase after sorting, sorting the candidate materials corresponding to the pending target phrase in descending order of similarity between the corresponding video code and the text code of the pending target phrase, and determining the target material corresponding to the pending target phrase based on the sorting result; 23. The apparatus of claim 22.

24. The video generation module includes: if the pending target phrase is not the top target phrase after sorting, generate, for each candidate material corresponding to the pending target phrase, a third sequence obtained by adding a video code of the candidate material to the end of the first sequence and a fourth sequence obtained by adding a text code of the pending target phrase to the end of the second sequence, based on a most recently obtained first sequence including a video code of a target material corresponding to each target phrase located before the pending target phrase and a most recently obtained second sequence including a text code of each target phrase located before the pending target phrase, and determine an overall score for the candidate material based on the third sequence and the fourth sequence; sorting each candidate material in descending order of overall score, determining a target material corresponding to said pending target phrase based on the sorting result, adding a video code of the target material corresponding to said pending target phrase to the end of said first sequence to obtain an updated first sequence, and adding a text code of said pending target phrase to the end of said second sequence to obtain an updated second sequence; 23. The apparatus of claim 22.

25. The video generation module includes: obtaining a similarity between a video code of the candidate material and a text code of the target phrase to be processed, obtaining a sequence correlation score between the third sequence and the fourth sequence and a sequence consistency score for the third sequence, and determining an overall score for the candidate material by combining the similarity, the sequence correlation score, and the sequence consistency score; 25. The apparatus of claim 24.

26. The video generation module includes: determining the sequence correlation score and the sequence agreement score based on the third sequence and the fourth sequence using a pre-trained organizational model; 26. The apparatus of claim 25.

27. when determining target material corresponding to the target phrase to be processed based on the sorting result, the video generation module determines top L candidate materials after sorting as target materials corresponding to the target phrase to be processed; 24. The apparatus of claim 23, wherein L is a positive integer, and the value of L is a fixed value or a value determined based on the principle that a total duration of selected target material matches a text-to-speech TTS speech duration of the pending target phrase.

28. The apparatus according to any one of claims 17 to 26, wherein the video generation module obtains the target video by rendering according to a quick picture expert technique based on the target material.

29. An apparatus for acquiring a composition model, comprising: A sample construction module for obtaining, in each training batch, a training sample corresponding to a correlation category task, which is a task corresponding to the organizational model, and a training sample corresponding to a consistency category task, which is a task corresponding to the organizational model; a model training module that uses the training samples to train the organization model, the trained organization model being configured to select target material for generating a desired target video from among candidate material corresponding to each target phrase, the target phrases being obtained by dividing the to-be-processed text information into phrases, and each candidate material corresponding to a target phrase being a video shot that is selected from among video shots in a material library to match the target phrase; An apparatus comprising:

30. The sample construction module includes: Obtain P text-video pairs (P is a positive integer greater than 1) corresponding to the training batch, each text-video pair including a text set of at least two phrases and a video set of at least two video shots, the at least two phrases being obtained from the same text information and sorted in order of their appearance positions in the text information, each phrase in the text set corresponding to at least one video shot in the video set, and the video shots in the video set being sorted in order of their playing time; Based on the obtained text-video pairs, construct a training sample corresponding to the correlation category task and a training sample corresponding to the agreement category task.

30. The apparatus of claim 29.

31. The sample construction module includes: Each obtained text-video pair is taken as a positive sample, For any of the obtained text video pairs, randomly select k1 (k1 is a positive integer and is smaller than the number of video shots in the video set in the text video pair) video shots from other text video pairs, insert the selected k1 video shots into the video set in the text video pair to obtain a corrected video set, and use the text set in the text video pair and the corrected video set to construct a negative sample; or, for any of the text video pairs, randomly select k2 (k2 is a positive integer and is smaller than the number of phrases in the text set in the text video pair) phrases from other text video pairs, insert the selected k2 phrases into the text set in the text video pair to obtain a corrected text set, and use the corrected text set and the video set in the text video pair to construct a negative sample; The positive samples and the negative samples are training samples corresponding to the correlation category task.

31. The apparatus of claim 30.

32. The sample construction module includes: For each of the obtained text video pairs, the video set in the text video pair is randomly selected and processed in one of three predetermined methods: keeping the video set unchanged; randomly selecting k3 (k3 is a positive integer and is smaller than the number of video shots in the video set) video shots from other text video pairs and replacing the selected k3 video shots with the k3 video shots in the video set; or randomly adjusting the order of some video shots in the video set; Each of the text-video pairs processed in the predetermined manner is used as a training sample corresponding to the matching category task.

32. Apparatus according to claim 30 or 31.

33. At least one processor; a memory in communication with the at least one processor; 17. An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions, when executed by the at least one processor, causing the at least one processor to perform the method of any one of claims 1 to 16.

34. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to carry out the method according to any one of claims 1 to 16.

35. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images