Video generation method and device, electronic device, and computer-readable storage medium

Abstract

The present disclosure provides a video generation method and device, which relates to the field of computer technology, specifically to technical fields such as large models, deep learning, image processing, and speech recognition. The specific implementation scheme is as follows: semantically segmenting and position-marking the material to be processed to obtain a semantic text block set including at least one semantic text block; classifying the semantic text blocks in the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the material to be processed, and a video material library; generating a non-key image sequence based on the non-key text block set and the video material library; and generating a target video based on the key image sequence and the non-key image sequence, thereby improving the video generation effect.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, specifically to technical fields such as large models, deep learning, image processing, and speech recognition, and in particular to a video generation method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As a kind of information medium, video can provide an immersive experience due to its rich media form. For this reason, video is becoming more and more popular.

[0003] Current text-generated videos primarily use an editing-based video generation method. This method's main process involves understanding the shot language based on the copywriting and generating shot text. The method then searches the source library to retrieve a large amount of footage. Finally, an editing algorithm is used to pair the shot text with the video's imagery to form a video timeline. While this method can generate some imagery, it ignores the lack of effective footage during video generation, resulting in a gradual decline in the relevance of imagery as the video progresses. Summary of the Invention

[0004] The present disclosure provides a video generation method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a video generation method is provided, which includes: semantically segmenting and position-marking a material to be processed to obtain a semantic text block set including at least one semantic text block; classifying the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the material to be processed, and a video material library; generating a non-key image sequence based on the non-key text block set and the video material library; and generating a target video based on the key image sequence and the non-key image sequence.

[0006] According to a second aspect, a video generation device is provided, which includes: a segmentation unit, configured to semantically segment and position-mark a material to be processed, and obtain a semantic text block set including at least one semantic text block; a classification unit, configured to classify the semantic text block set based on key event information, and obtain a key text block set and a non-key text block set; a key generation unit, configured to generate a key image sequence based on the key text block set, the material to be processed, and a video material library; a non-key generation unit, configured to generate a non-key image sequence based on the non-key text block set and the video material library; and a video generation unit, configured to generate a target video based on the key image sequence and the non-key image sequence.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described in any implementation of the first aspect.

[0009] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0010] The video generation method and device provided by the embodiments of the present disclosure firstly perform semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; secondly, classify the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; thirdly, generate a key image sequence based on the key text block set, the material to be processed and the video material library; thirdly, generate a non-key image sequence based on the non-key text block set and the video material library; finally, generate a target video based on the key image sequence and the non-key image sequence. Thus, by performing semantic segmentation on the material to be processed to obtain a semantic text block set, the material belonging to the same subject is effectively distinguished; based on the classification of the semantic text block set, a non-key image sequence and a key image sequence are generated, which can strongly correlate key images and weakly correlate non-key images, thereby improving the rationality of target video generation; generating a target video through the material to be processed saves the video generation cost and improves the generation effect of the target video.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a flow chart of an embodiment of a video generation method according to the present disclosure;

[0014] Figure 2 This is a schematic diagram of a structure for generating a target video in the present disclosure;

[0015] Figure 3is a structural diagram of an embodiment of a video generating device according to the present disclosure;

[0016] Figure 4 It is a block diagram of an electronic device used to implement the video generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] Some existing methods for generating videos from text are based on greedy algorithms, which select the image material with the highest relevance score for each sentence in the text to generate a video. This algorithm ignores the situation of insufficient effective material during video generation, resulting in the relevance of the video material gradually decreasing.

[0019] In view of the defects in traditional technologies, this paper proposes a video generation method, which can reasonably allocate strong-correlation materials and weak-correlation materials when materials are insufficient, thereby improving the generation effect of the target video. Figure 1 A process 100 according to an embodiment of a video generation method of the present disclosure is shown. The video generation method includes the following steps:

[0020] Step 101 : semantically segment and position-mark the material to be processed to obtain a semantic text block set including at least one semantic text block.

[0021] In this embodiment, the material to be processed is the material for the video to be generated. The material to be processed may include one or more of text, voice, and image. By processing the material to be processed, the text to be processed is obtained, and the video is generated by the text to be processed, so as to obtain the target video corresponding to the material to be processed.

[0022] In this embodiment, semantic segmentation refers to dividing materials with the same semantics in the material to be processed together, or extracting text materials from the material to be processed and dividing text materials with the same semantics in the text materials together; position marking refers to dividing materials according to position relationship marks corresponding to materials in the material to be processed, or dividing text materials according to position relationship marks corresponding to materials in the material to be processed.

[0023] In this embodiment, a semantic text block is a text segment obtained by semantically segmenting and position-marking the material to be processed. A semantic text block can be a text segment obtained by first performing text extraction on the material to be processed, then performing semantic segmentation and position-marking. A semantic text block can also be a text segment obtained by first performing semantic segmentation and position-marking on the material to be processed, then performing text extraction. Each semantic text block in the semantic text block set has a location identifier that identifies its location in the material to be processed.

[0024] In this embodiment, position marking of the material to be processed can provide corresponding position references for the materials (such as text and images) used to generate the target video subsequently, thereby providing a reliable position basis for obtaining the target video in the same order as the material to be processed.

[0025] The collection, storage, use, processing, transmission, provision, and disclosure of the processed material in this disclosure are performed with authorization and in compliance with relevant laws and regulations. User-related information in the processed material is obtained with the user's permission, and the generation of user-related videos is performed under confidentiality conditions.

[0026] In this embodiment, when the material to be processed is text, it contains multiple aspects. Therefore, text segments with the same main body can be aggregated to form semantic text blocks. For example, if the material to be processed is: "F is an ancient military fortification, a tall, solid, and continuous wall used to restrict the movement of enemy cavalry; E not only brings together several famous gardens and scenic spots in Jiangnan, but also transplants Western garden architecture, integrating the essence of ancient and modern Chinese and foreign garden art at the time," this material will be segmented into two semantic text blocks, one for identifying F and the other for identifying E.

[0027] Step 102 : Based on the key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set.

[0028] In this embodiment, the key event information is the main information representing the key event, and the key event information includes: the subject of the key event, the activity scene of the key event, the event description of the key event, etc.

[0029] In this embodiment, the key text block set includes at least one key text block, and the non-key text block set includes at least one non-key text block. The semantic text blocks in the semantic text block set are distinguished by key event information. When the text in the semantic text block has key event information, the semantic text block is determined to be a key text block in the key text block set; when the text in the semantic text block does not have key event information, the semantic text block is determined to be a non-key text block.

[0030] In this embodiment, the semantic text blocks in the semantic text block set have only two categories. Based on key event information, part of the semantic text blocks in the semantic text block set is divided into a key text block set, and the other part is divided into a non-key text block set.

[0031] Step 103: Generate a key image sequence based on the key text block set, the material to be processed, and the video material library.

[0032] In this embodiment, the video material library is a database storing video materials, and the video materials may be pre-stored video content stored in an existing network or locally.

[0033] In this embodiment, after obtaining the key text block set, each key text block in the key text block set is generated into a first generated image sequence including at least one first generated image. At this time, each first generated image carries a position identifier of the corresponding key text block in the key text block set; each first generated image in the first image sequence is matched with the material to be processed and the image in the video material library, and the first generated image with successful matching is used as the key image.

[0034] In this embodiment, the key image sequence is an image sequence generated by a key text block set, the material to be processed, and a video material library. The key image sequence includes at least one key image, and each key image in the key image sequence has a position identifier representing the position in the material to be processed.

[0035] Step 104 : Generate a non-key image sequence based on the non-key text block set and the video material library.

[0036] In this embodiment, after obtaining the non-key text block set, each non-key text block in the non-key text block set is used to generate a second generated image sequence including at least one second generated image. At this time, each second generated image carries a position identifier of the corresponding non-key text block in the non-key text block set; each second generated image in the second image sequence is matched with an image in the video material library, and the second generated image with a successful match is used as a non-key image.

[0037] In this embodiment, the non-key image sequence is an image sequence generated by a non-key text block set and a video material library. The non-key image sequence includes at least one non-key image, and each non-key image in the non-key image sequence has a position identifier representing the position in the material to be processed.

[0038] Step 105 : Generate a target video based on the key image sequence and the non-key image sequence.

[0039] In this embodiment, the target video is a video corresponding to the material to be processed, and is also a video generated by the material to be processed.

[0040] In this embodiment, the key images in the key image sequence and the non-key images in the non-key image sequence both have position identifiers that represent the order of the materials to be processed. The key images in the key image sequence and the non-key images in the non-key image sequence are sorted and arranged according to the positions identified by the position identifiers to obtain a target image sequence. The target image sequence is rendered and the video is merged to obtain a target video.

[0041] In this embodiment, during the process of generating the target video, it is respectively detected whether the difference between the broadcast duration of the key image and the non-key image and the preset duration is within the time threshold range. If it is within the time threshold range, the playback duration is extended, wherein the preset duration is the playback duration set for the key image and the non-key image respectively. The specific value of the preset duration can be set based on development requirements, for example, the preset duration is 5s; the time threshold range is a value within the range of 0 to 3s, and the broadcast duration is the duration of the text-to-speech conversion in the matching image, that is, the TTS (Text-to-Speech) duration.

[0042] The video generation method provided by the embodiment of the present disclosure first performs semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; secondly, based on key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set; thirdly, based on the key text block set, the material to be processed and the video material library, a key image sequence is generated; thirdly, based on the non-key text block set and the video material library, a non-key image sequence is generated; finally, based on the key image sequence and the non-key image sequence, a target video is generated. Thus, by performing semantic segmentation on the material to be processed to obtain a semantic text block set, the material belonging to the same subject is effectively distinguished; based on the classification of the semantic text block set, a non-key image sequence and a key image sequence are generated, which can strongly correlate key images and weakly correlate non-key images, thereby improving the rationality of target video generation; generating a target video through the material to be processed saves video generation costs and improves the generation effect of the target video.

[0043] In some optional implementations of the present disclosure, the above-mentioned semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block includes: processing the material to be processed to obtain the text to be processed; identifying and clustering text segments belonging to the same semantics in the text to be processed; segmenting the clustered text segments to obtain multiple segmentation units; marking the position of each segmentation unit according to the sequence of the text to be processed to obtain a semantic text block set including at least one semantic text block.

[0044] In this optional implementation, the text to be processed may be text directly extracted from the material to be processed, or may be text obtained by performing text conversion on the material to be processed.

[0045] In this optional implementation, the texts of different subjects in the text to be processed can be divided into multiple text segments, and the text segments belonging to the same semantics can be clustered. The clustered text segments are segmented to obtain multiple segmentation units, each of which is a text segment of a subject, and each segmentation unit is an independent text segment.

[0046] In this optional implementation, the order of the text to be processed refers to the order of each text segment in the text to be processed, and the order of each text segment is obtained by the distribution position relationship of the material in the material to be processed.

[0047] In this optional implementation, when marking the position of each segmentation unit according to the order of the text to be processed, if the segmentation unit includes multiple text segments at different positions, the position of the text segment at the front is used to mark the position of the segmentation unit.

[0048] In this optional implementation, the above-mentioned processing of the material to be processed to obtain the text to be processed includes: detecting whether the material to be processed includes voice information; in response to the material to be processed including voice information, converting the voice information into a first text; extracting the text in the material to be processed to obtain a second text; and combining the first text and the second text according to the position of the voice information and text in the material to be processed to obtain the text to be processed.

[0049] In this optional implementation, the same semantics means that the various parts of the text to be processed describe the information content of the same subject. For example, when multiple text segments represent different aspects of the same subject, the multiple text segments can be aggregated together to obtain clustered text segments.

[0050] In this optional implementation, a pre-trained sequence labeling model can be used to identify and cluster text segments with the same semantics in the text to be processed, wherein the sequence labeling model is used to perform semantic analysis on the text sequence and to perform symbolic labeling on the text segments with the same semantics.

[0051] The method for obtaining a semantic text block set provided by this optional implementation method processes the material to be processed to obtain the text to be processed; identifies and clusters text segments belonging to the same semantics in the text to be processed; segments the clustered text segments to obtain multiple segmentation units; marks the position of each segmentation unit according to the sequence of the text to be processed to obtain a semantic text block set including at least one semantic text block, and divides the text to be processed by the main body, thereby providing a reliable implementation method for obtaining the semantic text block set; marks the position of the segmentation unit according to the sequence of the text to be processed, thereby improving the accuracy of obtaining the semantic text block set.

[0052] Optionally, the above-mentioned semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block includes: identifying and clustering material segments belonging to the same semantics in the material to be processed; segmenting the clustered material segments to obtain a plurality of segmented materials; marking the position of each segmented material according to the order of each material segment in the material to be processed, and converting each segmented material into a semantic text block to obtain a semantic text block set including at least one semantic text block. It should be noted that when marking the segmented material according to the order of each material segment in the material to be processed, if the segmented material includes multiple material segments in different positions, the position of the segmented material is marked with the position of the material segment that is at the front.

[0053] In some optional implementations of the present disclosure, the above-mentioned classification of the semantic text block set based on key event information to obtain the key text block set and the non-key text block set includes: inputting each semantic text block, key event information and event judgment prompt words in the semantic text block set into the classification large language model to obtain the classification results of each semantic text block output by the classification large language model; based on the classification results of the semantic text block set, dividing each semantic text block into a corresponding set to obtain the key text block set and the non-key text block set.

[0054] In this optional implementation, the classification large language model is a pre-trained large language model. Large Language Models (LLMs) are artificial intelligence models trained with large amounts of data, designed to understand and generate natural language text.

[0055] In this optional implementation, the event judgment prompt word is a prompt word that prompts the classification large language model to classify whether a semantic text block in the semantic text block set belongs to the event corresponding to the key event information. The classification large language model classifies the semantic text blocks in the semantic text block set and determines whether the semantic text block contains content corresponding to the key event information. If so, the semantic text block is classified as belonging to the key event corresponding to the key event information; if not, the semantic text block is classified as not belonging to the key event corresponding to the key event information.

[0056] In this optional implementation, the classification results include: belonging to a key event and not belonging to a key event. For a semantic text block that belongs to a key event, the semantic text block can be directly divided into a key text block set; for a semantic text block that does not belong to a key event, the semantic text block can be directly divided into a non-key text block set.

[0057] This optional implementation provides a method for obtaining a key text block set and a non-key text block set, which inputs each semantic text block, key event information, and event judgment prompt words in the semantic text block set into a large language model to obtain classification results of each semantic text block output by the large language model; based on the classification results of the semantic text block set, each semantic text block is divided into a corresponding set to obtain a key text block set and a non-key text block set. Through the classification of the large language model, the reliability of obtaining the key text block set and the non-key text block set is improved.

[0058] Optionally, the semantic text block set is classified based on the key event information to obtain the key text block set and the non-key text block set including:

[0059] Based on key event information, the core elements of the event are determined, where the core elements include: event characters and detailed descriptions of the event; whether each semantic text block in the semantic text block set has core elements is identified; the semantic text blocks with core elements in the semantic text block set are divided into the key text block set, and the semantic text blocks without core elements in the semantic text block set are divided into the non-key text block set.

[0060] In some optional implementations of the present disclosure, the above-mentioned generation of a key image sequence based on a key text block set, materials to be processed, and a video material library includes: determining a set of images to be processed based on the materials to be processed; extracting images to be processed from the set of images to be processed that match the key text block set to obtain a matching image set; obtaining a matching text set based on the matching image set and a multimodal large model; matching the matching text set with the key text block set to obtain a set of relevant text blocks in the key text block set that match the matching text set and a set of non-relevant text blocks that do not match the matching text set; obtaining a first image sequence based on the relevant text block set and the matching image set; obtaining a second image sequence based on the non-relevant text block set and the video material library; and merging the first image sequence and the second image sequence to obtain a key image sequence.

[0061] In this optional implementation, the step of determining the image set to be processed based on the material to be processed includes: extracting images from the material to be processed to obtain the image set to be processed.

[0062] Optionally, the above-mentioned determining the image set to be processed based on the material to be processed may further include: performing image conversion on text in the material to be processed to obtain a converted image, and using the converted image and the image in the material to be processed as the image set to be processed.

[0063] In this optional implementation, the matching image set includes at least one matching image, which is an image to be processed that is related to a key text block. The above-mentioned extraction of the image to be processed that matches the key text block set in the image set to be processed, and the resulting matching image set includes: calculating the correlation between the image to be processed in the image set to be processed and each key text block in the key text block set; if the image to be processed is related to the key text block, the image to be processed is used as the matching image.

[0064] In this optional implementation, the matching text set includes at least one matching text, each matching text corresponding to a matching image. Obtaining the matching text set based on the matching image set and the multimodal macromodel includes: inputting the matching image set and image-to-text prompt words into the multimodal macromodel, thereby obtaining the matching text set output by the multimodal macromodel. The image-to-text prompt words are used to prompt the multimodal macromodel to perform image-to-text conversion on the matching images in the matching image set.

[0065] In this optional implementation, the relevant text block set includes at least one relevant text block, the non-relevant text block set includes at least one non-relevant text block, both the relevant text blocks and the non-relevant text blocks are key text blocks in the key text block set, and both the relevant text blocks and the non-relevant text blocks have position identifiers in the key text blocks that represent the positions of the materials to be processed. The above-mentioned matching of the matching text set with the key text block set to obtain the relevant text block set that matches the matching text set in the key text block set and the non-relevant text block set that does not match the matching text set includes: calculating the similarity between each matching text set in the matching text set and the key text blocks in the key text block set, and taking the key text blocks with a similarity greater than a similarity threshold (which can be set, for example, the similarity threshold is 80%) as the relevant text blocks in the relevant text block set. Taking the key text blocks with a similarity less than or equal to the similarity threshold as the non-relevant text blocks in the non-relevant text block set.

[0066] In this optional implementation, the above-mentioned first image sequence includes at least one first image, each first image has a position identifier representing the position of the material to be processed, and based on the relevant text block set and the matching image set, obtaining the first image sequence includes: converting the relevant text block set into a relevant image sequence through a multimodal large model, wherein the relevant image sequence includes at least one relevant image, and each relevant image is obtained by converting text into an image by performing text-to-image conversion on the relevant text block; calculating the similarity between the relevant image sequence and the matching images in the matching image set; taking the matching image with a similarity greater than a similarity threshold as the first image, and migrating the position identifier of the relevant text block to the first image.

[0067] In this optional implementation, the above-mentioned second image sequence includes at least one second image, each second image has a position identifier representing the position of the material to be processed, and based on the non-correlated text block set and the video material library, obtaining the second image sequence includes: converting the non-correlated text block set into a non-correlated image sequence through a multimodal large model, wherein the non-correlated image sequence includes at least one non-correlated image, and each non-correlated image is obtained by converting text into an image by performing text-to-image conversion on a non-correlated text block; calculating the similarity between the non-correlated image sequence and the video material in the video material library; using the matching image with a similarity greater than a similarity threshold as the second image, and migrating the position identifier of the non-correlated text block to the second image.

[0068] The method for generating a key image sequence provided by this optional implementation method determines a set of images to be processed based on the material to be processed; extracts the images to be processed that match the key text block set from the set of images to be processed to obtain a matching image set; obtains a matching text set based on the matching image set and a multimodal large model; matches the matching text set with the key text block set to obtain a set of related text blocks that match the matching text set and a set of non-related text blocks that do not match the matching text set in the key text block set; obtains a first image sequence based on the related text block set and the matching image set; obtains a second image sequence based on the non-related text block set and a video material library; merges the first image sequence and the second image sequence to obtain a key image sequence, and effectively generates a key image sequence through mutual matching and transformation between text and image, thereby improving the accuracy of key image sequence generation.

[0069] Optionally, the above-mentioned generation of a key image sequence based on the key text block set, the material to be processed and the video material library includes: determining the image set to be processed based on the material to be processed; generating multiple image sets to be matched based on the key text block set; matching the multiple image sets to be matched with the image set to be processed to obtain an initial image sequence in the image set to be processed that matches the image set to be matched; matching the initial image sequence with the video material in the video material library to obtain a matching image set; and marking the positions of the matching image set and the initial image sequence according to the order of the images to be processed in the image set to be processed to obtain a key image sequence.

[0070] In some embodiments of the present disclosure, obtaining the first image sequence based on the relevant text block set and the matching image set includes: selecting matching images related to the relevant text block set in the matching image set; marking the positions of the matching images according to the positions of the relevant text blocks in the relevant text block set, to obtain the first image sequence.

[0071] In this optional implementation, the correlation between text and image can be obtained through vector comparison. Specifically, the matching image set and the relevant text block are converted into vectors, and the similarity of the two vectors is compared. When the similarity is greater than a similarity threshold, it is determined to be related; otherwise, it is determined to be unrelated.

[0072] In this optional implementation, the first image in the first image sequence is an image obtained by adding a position marker to the matching image, that is, the first image is a matching image with the added position marker. Marking the positions of the matching images according to the positions of the relevant text blocks in the relevant text block set, and obtaining the first image sequence includes: assigning the positions of the relevant text blocks to the matching images to obtain the positions of the matching images, and obtaining the first image after marking the corresponding position markers for the unmatched images.

[0073] The method for obtaining the first image sequence provided by this optional implementation method selects matching images related to the relevant text block set in the matching image set; marks the positions of the matching images according to the positions of the relevant text blocks in the relevant text block set to obtain the first image sequence, thereby improving the reliability of obtaining the first image sequence.

[0074] In some embodiments of the present disclosure, obtaining the first image sequence based on the relevant text block set and the matching image set includes: selecting matching images related to the relevant text block set in the matching image set; and migrating the position identifiers of the relevant text blocks to the matching images to obtain the first image sequence.

[0075] Optionally, the above-mentioned obtaining of the first image sequence based on the relevant text block set and the matching image set includes: selecting a matching image related to the relevant text block in the matching image set, detecting whether the difference between the broadcasting time of the matching image and the preset time is within a time threshold range, and if it is within the time threshold range, extending the playback time to obtain the first image.

[0076] In this embodiment, a preset playback duration is set for each matching image. If the difference between the TTS duration and the preset duration is within a certain range, such as 0 to 3 seconds, the playback duration can be extended. If the difference is too large, the image will be played for more than ten seconds, resulting in a poor experience for the generated target video.

[0077] In some optional implementations of the present disclosure, obtaining the second image sequence based on the non-relevant text block set and the video material library includes: searching the video material library for search images related to the non-relevant text block set; marking the position of the search image according to the position of the non-relevant text blocks in the non-relevant text block set, to obtain the second image sequence.

[0078] In this optional implementation, the search image is an image obtained by searching the video material library and associated with the non-relevant text block. The specific calculation of the correlation is described in the above embodiment. The second image in the second image sequence is an image obtained by adding a position marker to the search image, that is, the second image is the search image with the added position marker. The positions of the matching images are marked according to the positions of the non-relevant text blocks in the non-relevant text block set. Obtaining the second image sequence includes: assigning the positions of the non-relevant text blocks to the matching images to obtain the positions of the matching images; and obtaining the second image after marking the corresponding position markers for the non-matching images.

[0079] The method for obtaining the second image sequence provided by this optional implementation method selects matching images from the matching image set that are not related to the non-related text block set; marks the positions of the matching images according to the positions of the non-related text blocks in the non-related text block set to obtain the second image sequence, thereby improving the reliability of obtaining the second image sequence.

[0080] The method for obtaining the second image sequence provided by this optional implementation method searches the video material library for search images related to the non-related text block set; marks the position of the search image according to the position of the relevant text block in the relevant text block set, and obtains the second image sequence, thereby improving the accuracy of obtaining the second image sequence.

[0081] In some optional implementations of the present disclosure, the above-mentioned merging of the first image sequence and the second image sequence to obtain the key image sequence includes: merging the first image sequence and the second image sequence to obtain an initial image sequence; and sorting all the initial images according to the position of each initial image in the initial image sequence to obtain the key image sequence.

[0082] In this optional implementation, merging the first image sequence and the second image sequence to obtain the initial image sequence means directly placing the first image sequence and the second image sequence together.

[0083] In this optional implementation, the initial image sequence includes at least one initial image, each initial image can be a first image or a second image. Since the first image in the first image sequence and the second image in the second image sequence both have position identifiers representing the positions of each element in the material to be processed, the initial images are sorted in the order of positions identified by the position identifiers to obtain a key image sequence, wherein the key image sequence is an image sequence obtained after sorting the initial images in the initial image sequence.

[0084] The method for obtaining a key image sequence provided by this optional implementation combines a first image sequence and a second image sequence to obtain an initial image sequence; sorts all initial images according to the position of each initial image in the initial image sequence to obtain a key image sequence; and combines the first image sequence and the second image sequence in sequence, thereby improving the accuracy of obtaining the key image sequence.

[0085] In some optional implementations of the present disclosure, the above-mentioned generation of a non-key image sequence based on a non-key text block set and a video material library includes: inputting the non-key text block set and a pointing prompt word into a large language model to obtain a pointing result of the non-key text block set output by the large language model; and obtaining a non-key image sequence based on the non-key text block set, the pointing result and the video material library.

[0086] In this optional implementation, the directional large language model is a pre-trained large language model, and the directional prompt word is used to prompt the directional large language model to perform directional analysis on the non-key text blocks in the non-key text block set, thereby obtaining a directional result for each non-key text block. The directional analysis refers to analyzing whether the content of the non-key text block has a clear subject or activity theme.

[0087] In this optional implementation, the pointing results include: having clear pointing and not having clear pointing, wherein having clear pointing means that the non-key text block has a clear subject or activity theme, for example, the non-key text block is: "Ms. So-and-so, born on September 20, 1982 in District B, City A"; and not having clear pointing means that the non-key text block does not have a clear subject or activity theme, for example, the non-key text block is: "It's a beautiful day" or "Keep a positive attitude"; by pointing to the large language model, each non-key text block in the non-key text block set can have a corresponding pointing result.

[0088] In this optional implementation, the non-key image sequence includes at least one non-key image, each non-key image has a position identifier representing the position of the material to be processed. For a non-key text block with a clear pointing result, the video material related to the non-key text block can be directly selected from the video material library, and the position identifier representing the corresponding position of the material to be processed can be used as the video material identifier to obtain the non-key image; for a non-key text block without a clear pointing result, the network can be searched for an identification image related to the non-key text block, and the position identifier representing the corresponding position of the material to be processed can be used as the identification image identifier to obtain the non-key image.

[0089] The method for generating a non-critical image sequence provided by this optional implementation inputs a non-critical text block set and a pointing prompt word into a large language model to obtain a pointing result of the non-critical text block set output by the large language model; based on the non-critical text block set, the pointing result, and a video material library, a non-critical image sequence is obtained, and the pointing result of the non-critical text block set is analyzed by pointing to the large language model, thereby improving the accuracy of the pointing result; and a non-critical image sequence is generated based on the pointing result and the video material library, thereby improving the accuracy of the non-critical image sequence.

[0090] In some optional implementations of the present disclosure, the above-mentioned non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library, including: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, image materials related to the subject pointing are selected from the video material library; based on the position of the non-key text block, the position of the image material is marked, and the marked image materials are placed in sequence in the non-key image sequence.

[0091] In this optional implementation, subject pointing means that the non-key text block has a clear subject.

[0092] In this optional implementation, selecting image materials related to the subject pointing from the video material library includes: selecting image materials of the category of the subject corresponding to the subject pointing from the video material library.

[0093] In this optional implementation, the above-mentioned marking of the position of the image material based on the position of the non-critical text block and placing the marked image material in sequence in the non-critical image sequence includes: migrating the position identifier of the non-critical text block to the image material, and sorting the marked image material according to the position size corresponding to the position identifier.

[0094] This optional implementation provides a method for obtaining a non-key image sequence. In response to the non-key text block's pointing result indicating that the non-key text block has a subject pointing, the method selects image materials related to the subject pointing from a video material library. Based on the position of the non-key text block, the position of the image materials is marked, and the marked image materials are sequentially placed in the non-key image sequence. Thus, based on the subject pointing of the non-key text block, image materials related to the subject pointing are selected from the video material library, and based on the position of the non-key text block, the position of the image materials is marked. This provides a reliable implementation method for obtaining a non-key image sequence and improves the accuracy of obtaining the non-key image sequence.

[0095] Optionally, the above-mentioned non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library, including: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, image materials related to the subject pointing are selected from the video material library; based on the position of the non-key text block, the position of the image material is marked, and the marked image materials are placed in sequence in the non-key image sequence.

[0096] In some optional implementations of the present disclosure, the above-mentioned obtaining of a non-key image sequence based on a non-key text block set, a pointing result and a video material library further includes: in response to the pointing result of a non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing, generating an identification image related to the non-key text; based on the position of the non-key text block, marking the position of the identification image, and placing the marked identification image in the non-key image sequence.

[0097] In this optional implementation, the identification image can be an image directly generated from a non-key text block, or it can be an image searched from the Internet that is consistent with the style of the non-key text block. For example, for the non-key text block "It's a beautiful day", an image of a smiling face or landscape with a consistent style is directly generated as the identification image.

[0098] In this optional implementation, the above-mentioned marking of the position of the identification image based on the position of the non-critical text block and placing the marked identification image in the non-critical image sequence include: migrating the position identifier of the non-critical text block to the identification image, and sorting the marked identification image according to the position size corresponding to the position identifier to obtain a non-critical image.

[0099] The method for obtaining a non-key image sequence provided by this optional implementation method generates an identification image related to the non-key text in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing; based on the position of the non-key text block, the position of the identification image is marked, and the marked identification image is placed in the non-key image sequence, thereby improving the reliability of obtaining images in the non-key image sequence.

[0100] Optionally, the above-mentioned obtaining of a non-key image sequence based on a non-key text block set, pointing results and a video material library further includes: in response to the pointing result of a non-key text block in the non-key text block set characterizing that the non-key text block does not have a subject pointing and a theme pointing, generating an identification image related to the non-key text; based on the position of the non-key text block, marking the position of the identification image, and placing the marked identification image in the non-key image sequence.

[0101] In some optional implementations of the present disclosure, the above-mentioned generation of the target video based on the key image sequence and the non-key image sequence includes: merging the key image sequence and the non-key image sequence to obtain a mixed image sequence; sorting all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtaining the target video based on the target image sequence.

[0102] In this optional implementation, the key images in the key image sequence and the non-key images in the non-key image sequence both have position identifiers that represent the positional relationship of the materials to be processed. After merging the key image sequence and the non-key image sequence, the mixed images are sorted based on the position sizes corresponding to the position identifiers to obtain the target image sequence.

[0103] In this optional implementation, the target image sequence is rendered and merged to obtain the target video. Processing the image sequence to obtain the video is a mature method in this field and will not be described in detail here.

[0104] The method for generating a target video provided by this optional implementation combines a key image sequence and a non-key image sequence to obtain a mixed image sequence; sorts all mixed images according to the position of the mixed images in the mixed image sequence to obtain a target image sequence; obtains a target video based on the target image sequence, effectively sorts the target images, and improves the reliability and accuracy of obtaining the target video.

[0105] Optionally, the above-mentioned generation of the target video based on the key image sequence and the non-key image sequence includes: sorting the key images in the key image sequence and the non-key images in the non-key image sequence in sequence according to the position size corresponding to the position identifier of the key image in the key image sequence and the position size corresponding to the position identifier of the non-key image in the non-key image sequence to obtain the target image sequence; and obtaining the target video based on the target image sequence.

[0106] In one example of the present disclosure, the following Figure 2 , the video generation method disclosed in the present invention is described in detail:

[0107] 1) Input the material to be processed, for example, the material to be processed is a video containing text and accompanying pictures.

[0108] 2) Semantic Chunking: Using a sequence tagging model, we perform semantic chunking on the text. We cluster text with the same semantic meaning into a chunk in the available material to be processed, generating a set of semantic text chunks.

[0109] 3) Key event classification: Based on the key event information, a large language model is used to classify semantic text blocks to determine whether they are key events. The judgment criteria are: the semantic text blocks in the semantic text block set represent a specific event, with core elements such as characters and detailed descriptions.

[0110] If the semantic text block is a key event, then it is of high importance and highly relevant materials will be prioritized. If it is not a key event, weakly relevant materials will be supplemented. Specifically:

[0111] It is a key event: using semantic relevance, we select n (n≥1) images with relevance greater than a certain threshold from the original image / hot image search, that is, Figure 2 For image recall, a large multimodal model is used to generate descriptions for these n images. The generated matching text set is then compared with multiple sentences in the semantic text block, and the text relevance is calculated. The most relevant sentences for the current image are selected to obtain the relevant text block set. The playback duration of each image is preset to 5 seconds. If the playback duration of the matching image obtained from the relevant text block set is longer than 5 seconds and the difference with the preset duration is within the time threshold, the playback duration of the matching image is extended to obtain the first image sequence.

[0112] For the remaining sentences in the semantic block that are not filled with picture materials or the sentences whose picture materials are insufficient in length, weakly relevant video materials are retrieved from the video material library to fill them, thereby obtaining a second image sequence.

[0113] Non-critical events: For non-critical events, the directional large language model is used to determine whether the semantic text block has a clear direction. For semantic text blocks with a clear direction, weakly related materials are used to fill them and obtain the identification image.

[0114] For the copywriting, based on the above-mentioned supplementary materials from beginning to end, it is rendered and merged into a complete video.

[0115] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video generation device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically implemented in various electronic devices.

[0116] like Figure 3As shown, the video generation device 300 provided in this embodiment includes: a segmentation unit 301, a classification unit 302, a key generation unit 303, a non-key generation unit 304, and a video generation unit 305. The segmentation unit 301 can be configured to perform semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block. The classification unit 302 can be configured to classify the semantic text block set based on key event information to obtain a key text block set and a non-key text block set. The key generation unit 303 can be configured to generate a key image sequence based on the key text block set, the material to be processed, and the video material library. The non-key generation unit 304 can be configured to generate a non-key image sequence based on the non-key text block set and the video material library. The video generation unit 305 can be configured to generate a target video based on the key image sequence and the non-key image sequence.

[0117] In this embodiment, the specific processing of the video generation device 300: the block unit 301, the classification unit 302, the key generation unit 303, the non-key generation unit 304, and the video generation unit 305 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104 and step 105 in the corresponding embodiment are not repeated here.

[0118] In some optional implementations of this embodiment, the above-mentioned block segmentation unit 301 is configured to: process the material to be processed to obtain the text to be processed; identify and cluster text segments belonging to the same semantics in the text to be processed; segment the clustered text segments to obtain multiple segmentation units; mark the position of each segmentation unit according to the sequence of the text to be processed, and obtain a semantic text block set including at least one semantic text block.

[0119] In some optional implementations of this embodiment, the above-mentioned classification unit 302 is configured to: input each semantic text block, key event information and event judgment prompt words in the semantic text block set into the classification large language model to obtain the classification results of each semantic text block output by the classification large language model; based on the classification results of the semantic text block set, divide each semantic text block into corresponding sets to obtain a key text block set and a non-key text block set.

[0120] In some optional implementations of this embodiment, the key generation unit 303 is configured to: determine the image set to be processed based on the material to be processed; extract the images to be processed that match the key text block set in the image set to be processed to obtain a matching image set; obtain a matching text set based on the matching image set and the multimodal large model; match the matching text set with the key text block set to obtain a related text block set in the key text block set that matches the matching text set and a non-related text block set that does not match the matching text set; obtain a first image sequence based on the related text block set and the matching image set; obtain a second image sequence based on the non-related text block set and the video material library; and merge the first image sequence and the second image sequence to obtain a key image sequence.

[0121] In some optional implementations of this embodiment, the key generation unit 303 is configured to: select matching images related to the relevant text block set in the matching image set; mark the matching image positions according to the positions of the relevant text blocks in the relevant text block set to obtain a first image sequence.

[0122] In some optional implementations of this embodiment, the key generation unit 303 is configured to: search the video material library for a search image related to the non-relevant text block set; mark the position of the search image according to the position of the non-relevant text block in the non-relevant text block set, and obtain a second image sequence.

[0123] In some optional implementations of this embodiment, the key generation unit 303 is configured to: merge the first image sequence and the second image sequence to obtain an initial image sequence; and sort all initial images according to the position of each initial image in the initial image sequence to obtain a key image sequence.

[0124] In some optional implementations of this embodiment, the above-mentioned non-key generation unit 304 is configured to: input the non-key text block set and the pointing prompt word into the large language model to obtain a pointing result of the non-key text block set output by the large language model; and obtain a non-key image sequence based on the non-key text block set, the pointing result and the video material library.

[0125] In some optional implementations of this embodiment, the above-mentioned non-key generation unit 304 is configured to: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, select image materials related to the subject pointing from the video material library; based on the position of the non-key text block, mark the position of the image material, and place the marked image materials in sequence in the non-key image sequence.

[0126] In some optional implementations of the present disclosure, the above-mentioned non-key generation unit 304 is configured to: generate an identification image related to the non-key text in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing; mark the position of the identification image based on the position of the non-key text block, and place the marked identification image in the non-key image sequence.

[0127] In some optional implementations of the present disclosure, the above-mentioned video generation unit 305 is configured to: merge the key image sequence and the non-key image sequence to obtain a mixed image sequence; sort all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtain a target video based on the target image sequence.

[0128] The video generation device provided by the embodiment of the present disclosure is as follows: first, the segmentation unit 301 performs semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; second, the classification unit 302 classifies the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; third, the key generation unit 303 generates a key image sequence based on the key text block set, the material to be processed and the video material library; fourth, the non-key generation unit 304 generates a non-key image sequence based on the non-key text block set and the video material library; finally, the video generation unit 305 generates a target video based on the key image sequence and the non-key image sequence. Thus, by semantically segmenting the material to be processed to obtain a semantic text block set, the material belonging to the same theme is effectively distinguished; based on the classification of the semantic text block set, the non-key image sequence and the key image sequence are generated, which can strongly correlate the key image and weakly correlate the non-key image, thereby improving the rationality of the target video generation; generating the target video by the material to be processed saves the video generation cost and improves the generation effect of the target video.

[0129] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0131] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0133] Various components in device 400 are connected to I / O interface 405, including an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0134] The computing unit 401 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the video generation method. For example, in some embodiments, the video generation method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to perform the video generation method by any other appropriate means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable video generation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0141] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0142] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A video generation method, the method comprising: Performing semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; Based on the key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the material to be processed, and a video material library; generating a non-key image sequence based on the non-key text block set and the video material library; generating a target video based on the key image sequence and the non-key image sequence; The generating of a key image sequence based on the key text block set, the to-be-processed material, and the video material library includes: Determining a set of images to be processed based on the material to be processed; Extracting the images to be processed from the set of images to be processed that match the set of key text blocks to obtain a set of matching images; Based on the matching image set and the multimodal large model, a matching text set is obtained; Matching the matching text set with the key text block set to obtain a related text block set in the key text block set that matches the matching text set and a non-related text block set that does not match the matching text set; Obtaining a first image sequence based on the relevant text block set and the matching image set; obtaining a second image sequence based on the non-correlated text block set and the video material library; The first image sequence and the second image sequence are merged to obtain the key image sequence.

2. The method according to claim 1, wherein The semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block comprises: Processing the material to be processed to obtain a text to be processed; Identifying and clustering text segments with the same semantics in the text to be processed; Segment the clustered text segments to obtain multiple segmentation units; According to the sequence of the text to be processed, the position of each segmentation unit is marked to obtain a semantic text block set including at least one semantic text block.

3. The method according to claim 1, wherein The semantic text block set is classified based on the key event information to obtain a key text block set and a non-key text block set, including: Inputting each semantic text block, key event information, and event judgment prompt words in the semantic text block set into a classification large language model to obtain a classification result of each semantic text block output by the classification large language model; Based on the classification result of the semantic text block set, each semantic text block is divided into a corresponding set to obtain a key text block set and a non-key text block set.

4. The method according to claim 1, wherein The obtaining of a first image sequence based on the relevant text block set and the matching image set includes: Selecting a matching image from the matching image set that is related to the related text block set; According to the positions of the relevant text blocks in the relevant text block set, the matching image positions are marked to obtain a first image sequence.

5. The method according to claim 1, wherein The obtaining of a second image sequence based on the non-correlated text block set and the video material library comprises: Searching the video material library for search images related to the non-related text block set; According to the positions of the non-relevant text blocks in the non-relevant text block set, the positions of the search images are marked to obtain a second image sequence.

6. The method according to claim 1, wherein The merging the first image sequence and the second image sequence to obtain the key image sequence includes: merging the first image sequence and the second image sequence to obtain an initial image sequence; All initial images are sorted according to the position of each initial image in the initial image sequence to obtain a key image sequence.

7. The method according to claim 1, wherein The generating of a non-key image sequence based on the non-key text block set and the video material library includes: Inputting the non-key text block set and the pointing prompt word into a pointing large language model, and obtaining a pointing result of the non-key text block set output by the pointing large language model; the pointing large language model is a pre-trained large language model; A non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library.

8. The method according to claim 7, wherein: The obtaining of a non-key image sequence based on the non-key text block set, the pointing result, and the video material library includes: In response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, selecting image material related to the subject pointing from the video material library; Based on the position of the non-key text block, the position of the image material is marked, and the marked image material is placed in sequence in the non-key image sequence.

9. The method according to claim 8, wherein The obtaining of a non-key image sequence based on the non-key text block set, the pointing result, and the video material library further comprises: In response to a pointing result of a non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing, generating a logo image related to the non-key text; Based on the position of the non-key text block, the position of the logo image is marked, and the marked logo image is placed in the non-key image sequence.

10. The method according to claim 1, wherein The generating a target video based on the key image sequence and the non-key image sequence includes: merging the key image sequence and the non-key image sequence to obtain a mixed image sequence; Sort all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; A target video is obtained based on the target image sequence.

11. A video generation device, comprising: A block division unit is configured to perform semantic block division and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; A classification unit is configured to classify the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; A key generation unit is configured to generate a key image sequence based on the key text block set, the material to be processed and a video material library; a non-key generating unit configured to generate a non-key image sequence based on the non-key text block set and the video material library; a video generating unit configured to generate a target video based on the key image sequence and the non-key image sequence; The key generation unit is configured to: determine a set of images to be processed based on the material to be processed; extract the images to be processed that match the key text block set from the set of images to be processed to obtain a matching image set; obtain a matching text set based on the matching image set and a multimodal large model; match the matching text set with the key text block set to obtain a set of relevant text blocks in the key text block set that match the matching text set and a set of non-relevant text blocks that do not match the matching text set; obtain a first image sequence based on the relevant text block set and the matching image set; obtain a second image sequence based on the non-relevant text block set and the video material library; and merge the first image sequence and the second image sequence to obtain the key image sequence.

12. According to the device according to claim 11, the segmentation unit is configured to: process the material to be processed to obtain the text to be processed; identify and cluster text segments belonging to the same semantics in the text to be processed; segment the clustered text segments to obtain multiple segmentation units; mark the position of each segmentation unit according to the sequence of the text to be processed, and obtain a semantic text block set including at least one semantic text block.

13. The apparatus according to claim 11, wherein the classification unit is configured to: Each semantic text block, key event information, and event judgment prompt word in the semantic text block set is input into a classification large language model to obtain a classification result of each semantic text block output by the classification large language model; based on the classification result of the semantic text block set, each semantic text block is divided into a corresponding set to obtain a key text block set and a non-key text block set.

14. The device according to claim 11, wherein The key generation unit is configured to: select matching images related to the relevant text block set from the matching image set; mark the positions of the matching images according to the positions of the relevant text blocks in the relevant text block set, and obtain a first image sequence.

15. The device according to claim 11, wherein The key generation unit is configured to: search the video material library for a search image related to the non-related text block set; mark the position of the search image according to the position of the non-related text blocks in the non-related text block set to obtain a second image sequence.

16. The device according to claim 11, wherein The key generation unit is configured to: merge the first image sequence and the second image sequence to obtain an initial image sequence; and sort all initial images according to positions of the initial images in the initial image sequence to obtain a key image sequence.

17. The device according to claim 11, wherein The non-key generating unit is configured to: input the non-key text block set and the pointing prompt word into the pointing large language model, and obtain the pointing result of the non-key text block set output by the pointing large language model; Based on the non-key text block set, the pointing result and the video material library, a non-key image sequence is obtained. The pointing large language model is a pre-trained large language model.

18. The device according to claim 17, wherein The non-key generating unit is configured to: in response to a pointing result of a non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, select an image material related to the subject pointing from the video material library; Based on the position of the non-key text block, the position of the image material is marked, and the marked image material is placed in sequence in the non-key image sequence.

19. The device according to claim 18, wherein The non-key generation unit is configured to: generate an identification image related to the non-key text in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing; mark the position of the identification image based on the position of the non-key text block, and place the marked identification image in the non-key image sequence.

20. The device according to claim 11, wherein The video generation unit is configured to: merge the key image sequence and the non-key image sequence to obtain a mixed image sequence; sort all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtain a target video based on the target image sequence.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images