Video generation method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a video generation method and device, and relates to the technical field of computers, in particular to the technical fields of large models, deep learning, image processing, voice recognition and the like. According to the specific implementation scheme, semantic partitioning and position marking are conducted on a to-be-processed material, and a semantic text block set comprising at least one semantic text block is obtained; based on the key event information, classifying semantic text blocks in the semantic text block set to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the to-be-processed material and a video material library; generating a non-key image sequence based on the non-key text block set and a video material library; the target video is generated based on the key image sequence and the non-key image sequence, and the video generation effect is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, specifically to technical fields such as large models, deep learning, image processing, and speech recognition, and in particular to a video generation method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] As a kind of information medium, video can provide an immersive experience due to its rich media form. For this reason, video is becoming more and more popular.

[0003] The current text-generated videos mainly adopt the video generation method of editing. The main process of the video generation method of editing is: based on the copywriting, the lens language is understood and the lens text is generated, and then a large amount of material is retrieved from the material library, and then the editing algorithm is used to match the lens text with the image material of the video to form the timeline of the video. Although this method can obtain some image materials, the algorithm ignores the situation of insufficient effective materials when the video is generated, resulting in the image material relevance gradually becoming worse as the video progresses. Summary of the invention

[0004] The present disclosure provides a video generation method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a video generation method is provided, the method comprising: semantically segmenting and position-marking a material to be processed to obtain a semantic text block set including at least one semantic text block; classifying the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the material to be processed and a video material library; generating a non-key image sequence based on the non-key text block set and the video material library; generating a target video based on the key image sequence and the non-key image sequence.

[0006] According to a second aspect, a video generating device is provided, which includes: a segmentation unit, configured to perform semantic segmentation and position marking on a material to be processed, and obtain a semantic text block set including at least one semantic text block; a classification unit, configured to classify the semantic text block set based on key event information, and obtain a key text block set and a non-key text block set; a key generation unit, configured to generate a key image sequence based on the key text block set, the material to be processed, and a video material library; a non-key generation unit, configured to generate a non-key image sequence based on the non-key text block set and the video material library; and a video generation unit, configured to generate a target video based on the key image sequence and the non-key image sequence.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation manner of the first aspect.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0009] According to a fifth aspect, a computer program product is provided, comprising a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0010] The video generation method and device provided by the embodiments of the present disclosure firstly perform semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; secondly, classify the semantic text block set based on key event information to obtain a key text block set and a non-key text block set; thirdly, generate a key image sequence based on the key text block set, the material to be processed and the video material library; thirdly, generate a non-key image sequence based on the non-key text block set and the video material library; finally, generate a target video based on the key image sequence and the non-key image sequence. Thus, by performing semantic segmentation on the material to be processed to obtain a semantic text block set, the material belonging to the same subject is effectively distinguished; based on the classification of the semantic text block set, a non-key image sequence and a key image sequence are generated, which can strongly correlate key images and weakly correlate non-key images, thereby improving the rationality of target video generation; generating a target video through the material to be processed saves the video generation cost and improves the generation effect of the target video.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a flow chart of an embodiment of a video generation method according to the present disclosure;

[0014] Figure 2 is a structural schematic diagram of generating a target video in the present disclosure;

[0015] Figure 3is a structural schematic diagram of an embodiment of a video generating device according to the present disclosure;

[0016] Figure 4 It is a block diagram of an electronic device used to implement the video generating method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0018] Some existing methods for generating videos from text are based on greedy algorithms, which select the image material with the highest relevance score for each sentence in the text to generate a video. This algorithm ignores the situation of insufficient effective material when generating a video, resulting in the relevance of the video material gradually deteriorating.

[0019] In view of the defects in the traditional technology, the present disclosure proposes a video generation method, which can reasonably allocate strong-correlation materials and weak-correlation materials when the materials are insufficient, thereby improving the generation effect of the target video. Figure 1 A process 100 according to an embodiment of a video generation method of the present disclosure is shown. The video generation method comprises the following steps:

[0020] Step 101 , semantically segment and position-mark the material to be processed to obtain a semantic text block set including at least one semantic text block.

[0021] In this embodiment, the material to be processed is the material for the video to be generated, and the material to be processed may include: one or more of text, voice, and image. By processing the material to be processed, the text to be processed is obtained, and the video is generated by the text to be processed, so as to obtain the target video corresponding to the material to be processed.

[0022] In this embodiment, semantic segmentation refers to dividing materials with the same semantics in the material to be processed together, or extracting text materials of the material to be processed and dividing text materials with the same semantics in the text materials together; position marking refers to dividing materials according to position relationship marks corresponding to materials in the material to be processed, or dividing text materials according to position relationship marks corresponding to materials in the material to be processed.

[0023] In this embodiment, the semantic text block is a text segment obtained by semantically segmenting and position marking the material to be processed, wherein the semantic text block can be a text segment obtained by first performing text extraction on the material to be processed, then performing semantic segmentation and position marking; the semantic text block can also be a text segment obtained by first performing semantic segmentation and position marking on the material to be processed, then performing text extraction. Each semantic text block in the semantic text block set has a position identifier that identifies the position in the material to be processed.

[0024] In this embodiment, position marking of the material to be processed can provide corresponding position references for the materials (such as texts and images) used to generate the target video, thereby providing a reliable position basis for obtaining the target video in the same sequence as the material to be processed.

[0025] In the technical solution disclosed in this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of the materials to be processed are carried out after authorization and in compliance with relevant laws and regulations. The information related to the user in the materials to be processed is the information obtained after the user's permission, and the video generation related to the user is processed under confidentiality conditions.

[0026] In this embodiment, when the material to be processed is a text, the text includes multiple aspects of content, so the text segments with the same subject in the text can be aggregated together to obtain semantic text blocks. For example, the material to be processed is: "F is an ancient military fortification, a tall, solid and continuous long wall used to limit the movement of enemy cavalry; E not only brings together several famous gardens and scenic spots in Jiangnan, but also transplants Western garden architecture, and is a culmination of the garden art of ancient and modern China and foreign countries at that time", the material will be divided into two semantic text blocks, one of which is a text block used to identify F, and the other is a text block used to identify E.

[0027] Step 102: Classify the semantic text block set based on the key event information to obtain a key text block set and a non-key text block set.

[0028] In this embodiment, the key event information is the main information that characterizes the key event, and the key event information includes: the subject of the key event, the activity scene of the key event, the event description of the key event, and the like.

[0029] In this embodiment, the key text block set includes at least one key text block, the non-key text block set includes at least one non-key text block, and the semantic text blocks in the semantic text block set are distinguished by key event information. When the text in the semantic text block has key event information, the semantic text block is determined to be a key text block in the key text block set; when the text in the semantic text block does not have key event information, the semantic text block is determined to be a non-key text block.

[0030] In this embodiment, there are only two categories of semantic text blocks in the semantic text block set. Based on key event information, part of the semantic text blocks in the semantic text block set is divided into a key text block set, and the other part is divided into a non-key text block set.

[0031] Step 103: Generate a key image sequence based on the key text block set, the material to be processed and the video material library.

[0032] In this embodiment, the video material library is a database storing video materials, and the video materials may be pre-stored video content stored in an existing network or locally.

[0033] In this embodiment, after obtaining the key text block set, each key text block in the key text block set is generated into a first generated image sequence including at least one first generated image. At this time, each first generated image carries a position identifier of the corresponding key text block in the key text block set; each first generated image in the first image sequence is matched with the material to be processed and the image in the video material library, and the first generated image with successful matching is used as the key image.

[0034] In this embodiment, the key image sequence is an image sequence generated by a key text block set, material to be processed and a video material library. The key image sequence includes at least one key image. Each key image in the key image sequence has a position identifier representing the position in the material to be processed.

[0035] Step 104: Generate a non-key image sequence based on the non-key text block set and the video material library.

[0036] In this embodiment, after obtaining the non-critical text block set, each non-critical text block in the non-critical text block set is used to generate a second generated image sequence including at least one second generated image, and each second generated image carries a position identifier of the corresponding non-critical text block in the non-critical text block set; each second generated image in the second image sequence is matched with an image in the video material library, and the second generated image with a successful match is used as a non-critical image.

[0037] In this embodiment, the non-key image sequence is an image sequence generated by a non-key text block set and a video material library. The non-key image sequence includes at least one non-key image. Each non-key image in the non-key image sequence has a position identifier representing the position in the material to be processed.

[0038] Step 105: Generate a target video based on the key image sequence and the non-key image sequence.

[0039] In this embodiment, the target video is a video corresponding to the material to be processed, and is also a video generated by the material to be processed.

[0040] In this embodiment, the key images in the key image sequence and the non-key images in the non-key image sequence both have position identifiers that represent the order of materials in the material to be processed. The key images in the key image sequence and the non-key images in the non-key image sequence are sorted and arranged according to the positions identified by the position identifiers to obtain a target image sequence, and the target image sequence is rendered and the video is merged to obtain a target video.

[0041] In this embodiment, in the process of generating the target video, it is respectively detected whether the difference between the broadcast time of the key image and the non-key image and the preset time is within the time threshold range. If it is within the time threshold range, the playback time is extended, wherein the preset time is the playback time set for the key image and the non-key image respectively, and the specific value of the preset time can be set based on development requirements, for example, the preset time is 5s; the time threshold range is a value within the range of 0 to 3s, and the broadcast time is the time for matching the text-to-speech in the image, that is, the TTS (Text-to-Speech) time.

[0042] The video generation method provided by the embodiment of the present disclosure first performs semantic segmentation and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; secondly, based on the key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set; thirdly, based on the key text block set, the material to be processed and the video material library, a key image sequence is generated; thirdly, based on the non-key text block set and the video material library, a non-key image sequence is generated; finally, based on the key image sequence and the non-key image sequence, a target video is generated. Thus, by performing semantic segmentation on the material to be processed to obtain a semantic text block set, the material belonging to the same subject is effectively distinguished; based on the classification of the semantic text block set, a non-key image sequence and a key image sequence are generated, which can strongly correlate key images and weakly correlate non-key images, thereby improving the rationality of target video generation; generating a target video through the material to be processed saves the video generation cost and improves the generation effect of the target video.

[0043] In some optional implementations of the present disclosure, the above-mentioned semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block includes: processing the material to be processed to obtain the text to be processed; identifying and clustering text segments belonging to the same semantics in the text to be processed; segmenting the clustered text segments to obtain a plurality of segmentation units; marking the position of each segmentation unit according to the sequence of the text to be processed to obtain a semantic text block set including at least one semantic text block.

[0044] In this optional implementation, the text to be processed may be text directly extracted from the material to be processed, or may be text obtained by performing text conversion on the material to be processed.

[0045] In this optional implementation, the texts of different subjects in the text to be processed can be divided into multiple text segments, and the text segments belonging to the same semantics can be clustered. The clustered text segments can be segmented to obtain multiple segmentation units, each of which is a text segment of a subject, and each segmentation unit is an independent text segment.

[0046] In this optional implementation, the sequence of the text to be processed refers to the sequence of the text segments in the text to be processed, and the sequence of the text segments is obtained through the distribution position relationship of the materials in the material to be processed.

[0047] In this optional implementation, when marking the position of each segmentation unit according to the sequence of the text to be processed, if the segmentation unit includes multiple text segments at different positions, the position of the text segment at the front is used to mark the position of the segmentation unit.

[0048] In this optional implementation, the above-mentioned processing of the material to be processed to obtain the text to be processed includes: detecting whether the material to be processed includes voice information; in response to the material to be processed including voice information, converting the voice information into a first text; extracting the text in the material to be processed to obtain a second text; and combining the first text and the second text according to the position of the voice information and text in the material to be processed to obtain the text to be processed.

[0049] In this optional implementation, the same semantics means that the various parts of the text to be processed describe the information content of the same subject. For example, when multiple text segments represent different aspects of the same subject, the multiple text segments can be aggregated together to obtain clustered text segments.

[0050] In this optional implementation, a pre-trained sequence labeling model can be used to identify and cluster text segments with the same semantics in the text to be processed, wherein the sequence labeling model is used to perform semantic analysis on the text sequence and to perform symbol labeling on the text segments with the same semantics.

[0051] The method for obtaining a semantic text block set provided by this optional implementation method processes the material to be processed to obtain the text to be processed; identifies and clusters text segments belonging to the same semantics in the text to be processed; segments the clustered text segments to obtain a plurality of segmentation units; marks the position of each segmentation unit according to the sequence of the text to be processed to obtain a semantic text block set including at least one semantic text block, and provides a reliable implementation method for obtaining the semantic text block set by dividing the text to be processed into main parts; marks the position of the segmentation unit according to the sequence of the text to be processed, thereby improving the accuracy of obtaining the semantic text block set.

[0052] Optionally, the above-mentioned semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block includes: identifying and clustering material segments belonging to the same semantics in the material to be processed; segmenting the clustered material segments to obtain multiple segmented materials; marking the position of each segmented material according to the sequence of each material segment in the material to be processed, and converting each segmented material into a semantic text block to obtain a semantic text block set including at least one semantic text block. It should be noted that when marking the segmented material according to the sequence of each material segment in the material to be processed, if the segmented material includes multiple material segments in different positions, the position of the segmented material is marked with the position of the material segment that is closest to the front.

[0053] In some optional implementations of the present disclosure, the above-mentioned classification of the semantic text block set based on key event information to obtain the key text block set and the non-key text block set includes: inputting each semantic text block in the semantic text block set, key event information and event judgment prompt words into the classification large language model to obtain the classification results of each semantic text block output by the classification large language model; based on the classification result of the semantic text block set, dividing each semantic text block into a corresponding set to obtain the key text block set and the non-key text block set.

[0054] In this optional implementation, the classification large language model is a pre-trained large language model. A large language model (LLMs) is an artificial intelligence model trained with a large amount of data, and is designed to understand and generate natural language text.

[0055] In this optional implementation, the event judgment prompt word is a prompt word that prompts the classification large language model to classify whether the semantic text block in the semantic text block set belongs to the event corresponding to the key event information. The classification large language model classifies the semantic text blocks in the semantic text block set, and determines whether the semantic text block has the content of the corresponding key event information. If it has the content of the corresponding key event information, a classification result is given that the semantic text block belongs to the key event corresponding to the key event information; if it does not have the content of the key event information, a classification result is given that the semantic text block does not belong to the key event corresponding to the key event information.

[0056] In this optional implementation, the classification results include: belonging to a key event and not belonging to a key event. For a semantic text block that belongs to a key event, the semantic text block can be directly divided into a key text block set; for a semantic text block that does not belong to a key event, the semantic text block can be directly divided into a non-key text block set.

[0057] The method for obtaining the key text block set and the non-key text block set provided by this optional implementation mode inputs each semantic text block, key event information and event judgment prompt words in the semantic text block set into the large language model to obtain the classification results of each semantic text block output by the large language model; based on the classification results of the semantic text block set, each semantic text block is divided into a corresponding set to obtain the key text block set and the non-key text block set, and the reliability of obtaining the key text block set and the non-key text block set is improved through the classification of the large language model.

[0058] Optionally, the semantic text block set is classified based on the key event information to obtain the key text block set and the non-key text block set including:

[0059] Based on key event information, the core elements of the event are determined, wherein the core elements include: event characters and event detailed descriptions; whether each semantic text block in the semantic text block set has core elements is identified; the semantic text blocks with core elements in the semantic text block set are divided into the key text block set, and the semantic text blocks without core elements in the semantic text block set are divided into the non-key text block set.

[0060] In some optional implementations of the present disclosure, the above-mentioned generation of a key image sequence based on a key text block set, materials to be processed and a video material library includes: determining a set of images to be processed based on the materials to be processed; extracting images to be processed from the set of images to be processed that match the key text block set, to obtain a matching image set; obtaining a matching text set based on the matching image set and a multimodal large model; matching the matching text set with the key text block set to obtain a set of related text blocks in the key text block set that match the matching text set and a set of non-related text blocks that do not match the matching text set; obtaining a first image sequence based on the related text block set and the matching image set; obtaining a second image sequence based on the non-related text block set and the video material library; and merging the first image sequence and the second image sequence to obtain a key image sequence.

[0061] In this optional implementation, the above-mentioned determining the image set to be processed based on the material to be processed includes: extracting images from the material to be processed to obtain the image set to be processed.

[0062] Optionally, the determining of the image set to be processed based on the material to be processed may further include: performing image conversion on text in the material to be processed to obtain a converted image, and taking the converted image and the image in the material to be processed as the image set to be processed.

[0063] In this optional implementation, the matching image set includes at least one matching image, and the matching image is an image to be processed that is related to a key text block. The above-mentioned extraction of the image to be processed that matches the key text block set in the image set to be processed includes: calculating the correlation between the image to be processed in the image set to be processed and each key text block in the key text block set; if the image to be processed is related to the key text block, the image to be processed is used as the matching image.

[0064] In this optional implementation, the above-mentioned matching text set includes at least one matching text, each matching text corresponds to a matching image, and the above-mentioned matching text set based on the matching image set and the multimodal large model includes: inputting the matching image set and the image-to-text prompt word into the multimodal large model to obtain the matching text set output by the multimodal large model. Among them, the image-to-text prompt word is used to prompt the multimodal large model to convert the matching image in the matching image set from image to text.

[0065] In this optional implementation, the related text block set includes at least one related text block, the non-related text block set includes at least one non-related text block, both the related text blocks and the non-related text blocks are key text blocks in the key text block set, and both the related text blocks and the non-related text blocks have position identifiers in the key text blocks that represent the positions of the materials to be processed. The above-mentioned matching of the matching text set with the key text block set to obtain the related text block set in the key text block set that matches the matching text set and the non-related text block set that does not match the matching text set includes: calculating the similarity between each matching text set in the matching text set and the key text blocks in the key text block set, and taking the key text blocks with a similarity greater than a similarity threshold (which can be set, for example, the similarity threshold is 80%) as the related text blocks in the related text block set. Taking the key text blocks with a similarity less than or equal to the similarity threshold as the non-related text blocks in the non-related text block set.

[0066] In this optional implementation, the above-mentioned first image sequence includes at least one first image, each first image has a position identifier representing the position of the material to be processed, and based on the relevant text block set and the matching image set, obtaining the first image sequence includes: converting the relevant text block set into a relevant image sequence through a multimodal large model, wherein the relevant image sequence includes at least one relevant image, and each relevant image is obtained by converting text into an image by performing text-to-image conversion on the relevant text block; calculating the similarity between the relevant image sequence and the matching images in the matching image set; taking the matching image with a similarity greater than a similarity threshold as the first image, and migrating the position identifier of the relevant text block to the first image.

[0067] In this optional implementation, the above-mentioned second image sequence includes at least one second image, each second image has a position identifier representing the position of the material to be processed, and based on the non-correlated text block set and the video material library, obtaining the second image sequence includes: converting the non-correlated text block set into a non-correlated image sequence through a multimodal large model, wherein the non-correlated image sequence includes at least one non-correlated image, and each non-correlated image is obtained by converting text into image by performing text conversion on a non-correlated text block; calculating the similarity between the non-correlated image sequence and the video material in the video material library; taking the matching image with a similarity greater than a similarity threshold as the second image, and migrating the position identifier of the non-correlated text block to the second image.

[0068] The method for generating a key image sequence provided by this optional implementation mode determines a set of images to be processed based on the material to be processed; extracts the images to be processed that match the key text block set from the set of images to be processed to obtain a matching image set; obtains a matching text set based on the matching image set and a multimodal large model; matches the matching text set with the key text block set to obtain a related text block set that matches the matching text set and a non-related text block set that does not match the matching text set in the key text block set; obtains a first image sequence based on the related text block set and the matching image set; obtains a second image sequence based on the non-related text block set and a video material library; and merges the first image sequence and the second image sequence to obtain a key image sequence, thereby effectively generating a key image sequence through mutual matching and conversion between text and image, thereby improving the accuracy of key image sequence generation.

[0069] Optionally, the above-mentioned generation of a key image sequence based on a key text block set, materials to be processed and a video material library includes: determining an image set to be processed based on the materials to be processed; generating multiple image sets to be matched based on the key text block set; matching the multiple image sets to be matched with the image set to be processed to obtain an initial image sequence in the image set to be processed that matches the image set to be matched; matching the initial image sequence with video materials in the video material library to obtain a matching image set; and marking the positions of the matching image set and the initial image sequence according to the sequence of images to be processed in the image set to be processed to obtain a key image sequence.

[0070] In some embodiments of the present disclosure, obtaining the first image sequence based on the relevant text block set and the matching image set includes: selecting matching images related to the relevant text block set in the matching image set; marking the matching image positions according to the positions of the relevant text blocks in the relevant text block set, to obtain the first image sequence.

[0071] In this optional implementation, the correlation between text and image can be obtained through vector comparison. Specifically, the matching image set and the relevant text block are converted into vectors, and the similarity of the two vectors is compared. When the similarity is greater than a similarity threshold, it is determined to be related; otherwise, it is determined to be irrelevant.

[0072] In this optional implementation, the first image in the first image sequence is an image obtained after adding a position identifier to the matching image, that is, the first image is a matching image with a position identifier added. According to the position of the relevant text block in the relevant text block set, the matching image position is marked, and obtaining the first image sequence includes: assigning the position of the relevant text block to the matching image, obtaining the position of the matching image, and obtaining the first image after marking the corresponding position identifier for the unmatched image.

[0073] The method for obtaining the first image sequence provided by this optional implementation method selects matching images related to the relevant text block set in the matching image set; marks the matching image positions according to the positions of the relevant text blocks in the relevant text block set to obtain the first image sequence, thereby improving the reliability of obtaining the first image sequence.

[0074] In some embodiments of the present disclosure, obtaining the first image sequence based on the relevant text block set and the matching image set includes: selecting matching images related to the relevant text block set in the matching image set; and migrating the position identifiers of the relevant text blocks to the matching images to obtain the first image sequence.

[0075] Optionally, the above-mentioned obtaining of the first image sequence based on the relevant text block set and the matching image set includes: selecting a matching image related to the relevant text block in the matching image set, detecting whether the difference between the broadcasting time of the matching image and the preset time is within a time threshold range, and if within the time threshold range, extending the broadcasting time to obtain the first image.

[0076] In this embodiment, the playback time is set to a preset time for each matching image. When the difference between the tts time and the preset time is within a certain range, such as 0 to 3 seconds, the playback time can be extended. If the difference is too large, the image will be played for more than ten seconds, which will make the generated target video experience poor.

[0077] In some optional implementations of the present disclosure, the above-mentioned obtaining of the second image sequence based on the non-relevant text block set and the video material library includes: searching the video material library for a search image related to the non-relevant text block set; marking the position of the search image according to the position of the non-relevant text block in the non-relevant text block set, to obtain the second image sequence.

[0078] In this optional implementation, the search image is an image obtained by searching the video material library and related to the non-related text block. The specific calculation of the correlation is shown in the above embodiment. The second image in the second image sequence is an image obtained after adding a position mark to the search image, that is, the second image is a search image with a position mark added. According to the position of the non-related text block in the non-related text block set, the matching image position is marked, and obtaining the second image sequence includes: assigning the position of the non-related text block to the matching image, obtaining the position of the matching image, and obtaining the second image after marking the corresponding position mark for the unmatched image.

[0079] The method for obtaining the second image sequence provided by this optional implementation method selects matching images that are not related to the non-related text block set in the matching image set; marks the matching image positions according to the positions of the non-related text blocks in the non-related text block set, and obtains the second image sequence, thereby improving the reliability of obtaining the second image sequence.

[0080] The method for obtaining the second image sequence provided by this optional implementation method searches the video material library for search images related to the non-related text block set; marks the position of the search image according to the position of the related text block in the related text block set, and obtains the second image sequence, thereby improving the accuracy of obtaining the second image sequence.

[0081] In some optional implementations of the present disclosure, the above-mentioned merging the first image sequence and the second image sequence to obtain the key image sequence includes: merging the first image sequence and the second image sequence to obtain an initial image sequence; and sorting all the initial images according to the position of each initial image in the initial image sequence to obtain the key image sequence.

[0082] In this optional implementation, merging the first image sequence and the second image sequence to obtain the initial image sequence means directly placing the first image sequence and the second image sequence together.

[0083] In this optional implementation, the initial image sequence includes at least one initial image, each initial image can be a first image or a second image. Since the first image in the first image sequence and the second image in the second image sequence both have position identifiers representing the positions of each element in the material to be processed, the initial images are sorted in the order of positions identified by the position identifiers to obtain a key image sequence, wherein the key image sequence is an image sequence obtained after sorting the initial images in the initial image sequence.

[0084] The method for obtaining a key image sequence provided by this optional implementation mode combines a first image sequence and a second image sequence to obtain an initial image sequence; sorts all initial images according to the position of each initial image in the initial image sequence to obtain a key image sequence; and combines the first image sequence and the second image sequence in order to improve the accuracy of obtaining the key image sequence.

[0085] In some optional implementations of the present disclosure, the above-mentioned generation of a non-key image sequence based on a non-key text block set and a video material library includes: inputting the non-key text block set and a pointing prompt word into a large language model to obtain a pointing result of the non-key text block set output by the large language model; and obtaining a non-key image sequence based on the non-key text block set, the pointing result and the video material library.

[0086] In this optional implementation, the pointing large language model is a pre-trained large language model, and the pointing prompt words are used to prompt the pointing large language model to perform pointing analysis on the non-key text blocks in the non-key text block set to obtain the prompt words of the pointing results of each non-key text block. The pointing analysis refers to analyzing whether the content of the non-key text block has a clear subject or activity theme.

[0087] In this optional implementation, the pointing results include: having clear pointing and not having clear pointing, wherein having clear pointing means that the non-key text block has a clear subject or activity theme, for example, the non-key text block is: "Mr. So-and-so, born on September 20, 1982 in District B, City A"; and not having clear pointing means that the non-key text block does not have a clear subject or activity theme, for example, the non-key text block is: "It's a beautiful day" or "Keep a positive attitude"; by pointing to the large language model, each non-key text block in the non-key text block set can have a corresponding pointing result.

[0088] In this optional implementation, the non-key image sequence includes at least one non-key image, each non-key image has a position identifier representing the position of the material to be processed. For a non-key text block with a pointing result having a clear direction, the video material related to the non-key text block can be directly selected from the video material library, and the position identifier representing the corresponding position of the material to be processed is used as the video material identifier to obtain the non-key image; for a non-key text block without a pointing result having a clear direction, an identification image related to the non-key text block can be searched on the network, and the position identifier representing the corresponding position of the material to be processed is used as the identification image identifier to obtain the non-key image.

[0089] The method for generating a non-critical image sequence provided by this optional implementation mode inputs a non-critical text block set and a pointing prompt word into a large language model to obtain a pointing result of the non-critical text block set output by the large language model; based on the non-critical text block set, the pointing result and the video material library, a non-critical image sequence is obtained, and the pointing result of the non-critical text block set is analyzed by pointing to the large language model, thereby improving the accuracy of the pointing result; a non-critical image sequence is generated based on the pointing result and the video material library, thereby improving the accuracy of the non-critical image sequence.

[0090] In some optional implementations of the present disclosure, the above-mentioned non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library, including: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, image materials related to the subject pointing are selected from the video material library; based on the position of the non-key text block, the position of the image material is marked, and the marked image materials are placed in sequence in the non-key image sequence.

[0091] In this optional implementation, subject pointing means that the non-key text block has a clear subject.

[0092] In this optional implementation, selecting image material related to subject pointing from the video material library includes: selecting image material of the category in which the subject corresponding to the subject pointing belongs from the video material library.

[0093] In this optional implementation, the above-mentioned marking of the position of the image material based on the position of the non-critical text block and placing the marked image material in sequence in the non-critical image sequence includes: migrating the position identifier of the non-critical text block to the image material, and sorting the marked image material according to the position size corresponding to the position identifier.

[0094] The method for obtaining a non-key image sequence provided by this optional implementation is to select image materials related to the subject pointing from the video material library in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing; based on the position of the non-key text block, mark the position of the image material, and place the marked image materials in sequence in the non-key image sequence. Thus, based on the subject pointing of the non-key text block, select image materials related to the subject pointing from the video material library, and based on the position of the non-key text block, mark the position of the image material, which provides a reliable implementation method for obtaining a non-key image sequence and improves the accuracy of obtaining a non-key image sequence.

[0095] Optionally, the above-mentioned non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library, including: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, image materials related to the subject pointing are selected from the video material library; based on the position of the non-key text block, the position of the image material is marked, and the marked image materials are placed in sequence in the non-key image sequence.

[0096] In some optional implementations of the present disclosure, the above-mentioned obtaining of the non-key image sequence based on the non-key text block set, the pointing results and the video material library also includes: in response to the pointing result of the non-key text block in the non-key text block set characterizing that the non-key text block does not have a subject pointing, generating an identification image related to the non-key text; based on the position of the non-key text block, marking the position of the identification image, and placing the marked identification image in the non-key image sequence.

[0097] In this optional implementation, the logo image can be an image directly generated from a non-key text block, or it can be an image searched from the Internet that is consistent with the style of the non-key text block. For example, for the non-key text block "It's a beautiful day", an image of a smiling face or landscape with a consistent style is directly generated as the logo image.

[0098] In this optional implementation, the above-mentioned marking of the position of the identification image based on the position of the non-critical text block and placing the marked identification image in the non-critical image sequence include: migrating the position identifier of the non-critical text block to the identification image, and sorting the marked identification image according to the position size corresponding to the position identifier to obtain a non-critical image.

[0099] The method for obtaining a non-key image sequence provided by this optional implementation manner generates an identification image related to the non-key text in response to the pointing result of the non-key text block in the non-key text block set characterizing that the non-key text block does not have a subject pointing; based on the position of the non-key text block, the position of the identification image is marked, and the marked identification image is placed in the non-key image sequence, thereby improving the reliability of obtaining images in the non-key image sequence.

[0100] Optionally, the above-mentioned obtaining of the non-key image sequence based on the non-key text block set, the pointing results and the video material library also includes: in response to the pointing results of the non-key text blocks in the non-key text block set characterizing that the non-key text blocks do not have subject pointing and theme pointing, generating an identification image related to the non-key text; based on the position of the non-key text block, marking the position of the identification image, and placing the marked identification image in the non-key image sequence.

[0101] In some optional implementations of the present disclosure, the above-mentioned generation of the target video based on the key image sequence and the non-key image sequence includes: merging the key image sequence and the non-key image sequence to obtain a mixed image sequence; sorting all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtaining a target video based on the target image sequence.

[0102] In this optional implementation, the key images in the key image sequence and the non-key images in the non-key image sequence both have position identifiers that represent the positional relationship of the materials to be processed. After the key image sequence and the non-key image sequence are merged, the mixed images are sorted based on the position sizes corresponding to the position identifiers to obtain the target image sequence.

[0103] In this optional implementation, the target image sequence is rendered and merged to obtain the target video, wherein processing the image sequence to obtain the video is a mature method in the art and will not be described in detail here.

[0104] The method for generating a target video provided by this optional implementation combines a key image sequence and a non-key image sequence to obtain a mixed image sequence; sorts all mixed images according to the positions of the mixed images in the mixed image sequence to obtain a target image sequence; obtains a target video based on the target image sequence, effectively sorts the target images, and improves the reliability and accuracy of obtaining the target video.

[0105] Optionally, the above-mentioned generation of the target video based on the key image sequence and the non-key image sequence includes: sorting the key images in the key image sequence and the non-key images in the non-key image sequence in sequence according to the position sizes corresponding to the position identifiers of the key images in the key image sequence and the position sizes corresponding to the position identifiers of the non-key images in the non-key image sequence to obtain the target image sequence; and obtaining the target video based on the target image sequence.

[0106] In one example of the present disclosure, the following is combined Figure 2 , the video generation method disclosed in the present invention is described in detail:

[0107] 1) Input the material to be processed, for example, the material to be processed is a video containing text and illustrations for the article.

[0108] 2) Semantic Chunking: Sequence labeling model is used to perform semantic chunking on the text. The text with the same semantics in the available materials is clustered into one chunk to obtain a semantic text chunk set.

[0109] 3) Critical event classification: for semantic text blocks, based on key event information, a classification large language model is used to determine whether it is a critical event. The judgment criteria are: the semantic text block in the semantic text block set indicates a specific event, with core elements such as characters and detailed descriptions.

[0110] If the semantic text block is a key event, it is more important and highly relevant materials will be arranged first. If it is not a key event, weakly relevant materials will be added. Specifically:

[0111] is a key event: using semantic relevance, select n (n ≥ 1) images with relevance greater than a certain threshold from the original image / hot image search, that is, Figure 2 In the image recall, for these n images, a multimodal large model is used to generate descriptions. The generated matching text set and multiple sentences in the semantic text block are used to calculate the text relevance, select the sentence most relevant to the current image, and obtain the relevant text block set. The preset playback time of each image material is 5s. If the playback time of the matching image obtained by the relevant text block set is greater than 5s, and the difference with the preset time is within the time threshold, the playback time of the matching image is extended to obtain the first image sequence.

[0112] For the remaining sentences in the semantic block that are not filled with picture materials or the sentences whose picture materials are insufficient in length, the video material library is retrieved for weakly relevant video materials to fill them, thereby obtaining a second image sequence.

[0113] Non-critical events: For non-critical events, the directional large language model is used to determine whether the semantic text block has a clear direction. For semantic text blocks with clear directions, weakly related materials are used to fill them and obtain identification images.

[0114] For the copy, based on the above-mentioned supplementary materials from beginning to end, it is rendered and merged into a complete video.

[0115] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a video generation device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically implemented in various electronic devices.

[0116] like Figure 3As shown, the video generation device 300 provided in this embodiment includes: a block division unit 301, a classification unit 302, a key generation unit 303, a non-key generation unit 304, and a video generation unit 305. Among them, the above-mentioned block division unit 301 can be configured to perform semantic block division and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block. The above-mentioned classification unit 302 can be configured to classify the semantic text block set based on key event information to obtain a key text block set and a non-key text block set. The above-mentioned key generation unit 303 can be configured to generate a key image sequence based on the key text block set, the material to be processed and the video material library. The above-mentioned non-key generation unit 304 can be configured to generate a non-key image sequence based on the non-key text block set and the video material library. The above-mentioned video generation unit 305 can be configured to generate a target video based on the key image sequence and the non-key image sequence.

[0117] In this embodiment, the specific processing of the block unit 301, the classification unit 302, the key generation unit 303, the non-key generation unit 304, and the video generation unit 305 and the technical effects thereof can be referred to in detail. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104, and step 105 in the corresponding embodiments are not repeated here.

[0118] In some optional implementations of the present embodiment, the above-mentioned block segmentation unit 301 is configured to: process the material to be processed to obtain the text to be processed; identify and cluster text segments belonging to the same semantics in the text to be processed; segment the clustered text segments to obtain multiple segmentation units; mark the position of each segmentation unit according to the sequence of the text to be processed, and obtain a semantic text block set including at least one semantic text block.

[0119] In some optional implementations of the present embodiment, the classification unit 302 is configured to: input each semantic text block, key event information, and event judgment prompt words in the semantic text block set into the classification large language model to obtain the classification results of each semantic text block output by the classification large language model; based on the classification results of the semantic text block set, divide each semantic text block into corresponding sets to obtain a key text block set and a non-key text block set.

[0120] In some optional implementations of the present embodiment, the key generation unit 303 is configured to: determine a set of images to be processed based on the material to be processed; extract the images to be processed that match the key text block set in the set of images to be processed to obtain a matching image set; obtain a matching text set based on the matching image set and the multimodal large model; match the matching text set with the key text block set to obtain a related text block set in the key text block set that matches the matching text set and a non-related text block set that does not match the matching text set; obtain a first image sequence based on the related text block set and the matching image set; obtain a second image sequence based on the non-related text block set and the video material library; and merge the first image sequence and the second image sequence to obtain a key image sequence.

[0121] In some optional implementations of this embodiment, the key generation unit 303 is configured to: select matching images related to the relevant text block set in the matching image set; mark the matching image positions according to the positions of the relevant text blocks in the relevant text block set to obtain a first image sequence.

[0122] In some optional implementations of the present embodiment, the key generation unit 303 is configured to: search the video material library for a search image related to the non-relevant text block set; mark the position of the search image according to the position of the non-relevant text block in the non-relevant text block set, and obtain a second image sequence.

[0123] In some optional implementations of this embodiment, the key generation unit 303 is configured to: merge the first image sequence and the second image sequence to obtain an initial image sequence; sort all initial images according to the position of each initial image in the initial image sequence to obtain a key image sequence.

[0124] In some optional implementations of the present embodiment, the non-key generation unit 304 is configured to: input the non-key text block set and the pointing prompt word into a large language model to obtain a pointing result of the non-key text block set output by the large language model; and obtain a non-key image sequence based on the non-key text block set, the pointing result and the video material library.

[0125] In some optional implementations of the present embodiment, the above-mentioned non-key generation unit 304 is configured to: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, select image materials related to the subject pointing from the video material library; based on the position of the non-key text block, mark the position of the image material, and place the marked image materials in sequence in the non-key image sequence.

[0126] In some optional implementations of the present disclosure, the above-mentioned non-key generation unit 304 is configured to: generate an identification image related to the non-key text in response to the pointing result of the non-key text block in the non-key text block set characterizing that the non-key text block does not have a subject pointing; mark the position of the identification image based on the position of the non-key text block, and place the marked identification image in the non-key image sequence.

[0127] In some optional implementations of the present disclosure, the video generation unit 305 is configured to: merge the key image sequence and the non-key image sequence to obtain a mixed image sequence; sort all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtain a target video based on the target image sequence.

[0128] The video generation device provided by the embodiment of the present disclosure, first, the block unit 301 performs semantic block and position marking on the material to be processed, and obtains a semantic text block set including at least one semantic text block; secondly, the classification unit 302 classifies the semantic text block set based on key event information, and obtains a key text block set and a non-key text block set; thirdly, the key generation unit 303 generates a key image sequence based on the key text block set, the material to be processed and the video material library; from then on, the non-key generation unit 304 generates a non-key image sequence based on the non-key text block set and the video material library; finally, the video generation unit 305 generates a target video based on the key image sequence and the non-key image sequence. Thus, by semantically segmenting the material to be processed to obtain a semantic text block set, the material belonging to the same theme is effectively distinguished; based on the classification of the semantic text block set, the non-key image sequence and the key image sequence are generated, and the key image can be strongly correlated and the non-key image can be weakly correlated, thereby improving the rationality of the target video generation; the target video is generated by the material to be processed, saving the video generation cost and improving the generation effect of the target video.

[0129] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0131] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0132] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0133] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0134] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable video generating device, so that the program code, when executed by the processor or controller, causes the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0137] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an information server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0142] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, the method comprising: Semantically segment and position-mark the material to be processed to obtain a semantic text block set including at least one semantic text block; Based on the key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set; Generate a key image sequence based on the key text block set, the material to be processed and a video material library; Generate a non-key image sequence based on the non-key text block set and the video material library; A target video is generated based on the key image sequence and the non-key image sequence.

2. The method according to claim 1, wherein: The semantic segmentation and position marking of the material to be processed to obtain a semantic text block set including at least one semantic text block comprises: Processing the material to be processed to obtain a text to be processed; Identifying and clustering text segments belonging to the same semantics in the text to be processed; Segment the clustered text segments to obtain multiple segmentation units; According to the sequence of the text to be processed, the position of each segmentation unit is marked to obtain a semantic text block set including at least one semantic text block.

3. The method according to claim 1, wherein: Based on the key event information, the semantic text block set is classified to obtain a key text block set and a non-key text block set, including: Inputting each semantic text block, key event information, and event judgment prompt words in the semantic text block set into a classification large language model to obtain classification results of each semantic text block output by the classification large language model; Based on the classification result of the semantic text block set, each semantic text block is divided into a corresponding set to obtain a key text block set and a non-key text block set.

4. The method according to claim 1, wherein: The generating of a key image sequence based on the key text block set, the material to be processed and the video material library comprises: Based on the material to be processed, determining a set of images to be processed; Extracting the images to be processed from the image set to be processed that match the key text block set to obtain a matching image set; Based on the matching image set and the multimodal large model, a matching text set is obtained; Matching the matching text set with the key text block set to obtain a related text block set in the key text block set that matches the matching text set and a non-related text block set that does not match the matching text set; Based on the relevant text block set and the matching image set, obtaining a first image sequence; Based on the non-correlated text block set and the video material library, obtaining a second image sequence; The first image sequence and the second image sequence are combined to obtain the key image sequence.

5. The method according to claim 4, wherein: The obtaining of a first image sequence based on the relevant text block set and the matching image set comprises: Selecting a matching image from the matching image set that is related to the related text block set; According to the positions of the relevant text blocks in the relevant text block set, the matching image positions are marked to obtain a first image sequence.

6. The method according to claim 4, wherein: The obtaining of a second image sequence based on the non-correlated text block set and the video material library comprises: Searching the video material library for search images related to the non-related text block set; According to the positions of the non-relevant text blocks in the non-relevant text block set, the positions of the search images are marked to obtain a second image sequence.

7. The method according to claim 4, wherein: The merging the first image sequence and the second image sequence to obtain the key image sequence comprises: Merging the first image sequence and the second image sequence to obtain an initial image sequence; All initial images are sorted according to the positions of the initial images in the initial image sequence to obtain a key image sequence.

8. The method according to claim 1, wherein: The generating of a non-key image sequence based on the non-key text block set and the video material library comprises: Inputting the non-key text block set and the pointing prompt word into a pointing large language model, and obtaining a pointing result of the non-key text block set output by the pointing large language model; A non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library.

9. The method according to claim 8, wherein: The obtaining of a non-key image sequence based on the non-key text block set, the pointing result and the video material library comprises: In response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, selecting image material related to the subject pointing from the video material library; Based on the position of the non-critical text block, the position of the image material is marked, and the marked image material is placed in sequence in the non-critical image sequence.

10. The method according to claim 9, wherein: The step of obtaining a non-key image sequence based on the non-key text block set, the pointing result and the video material library further comprises: In response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing, generating a logo image related to the non-key text; Based on the position of the non-key text block, the position of the logo image is marked, and the marked logo image is placed in the non-key image sequence.

11. The method according to claim 1, wherein: The generating a target video based on the key image sequence and the non-key image sequence comprises: Merging the key image sequence and the non-key image sequence to obtain a mixed image sequence; According to the position of each mixed image in the mixed image sequence, all mixed images are sorted to obtain a target image sequence; Based on the target image sequence, a target video is obtained.

12. A video generating device, the device comprising: A block division unit is configured to perform semantic block division and position marking on the material to be processed to obtain a semantic text block set including at least one semantic text block; A classification unit is configured to classify the semantic text block set based on the key event information to obtain a key text block set and a non-key text block set; A key generation unit, configured to generate a key image sequence based on the key text block set, the material to be processed and a video material library; A non-key generation unit, configured to generate a non-key image sequence based on the non-key text block set and the video material library; The video generating unit is configured to generate a target video based on the key image sequence and the non-key image sequence.

13. According to the device according to claim 12, the segmentation unit is configured to: process the material to be processed to obtain the text to be processed; identify and cluster text segments belonging to the same semantics in the text to be processed; segment the clustered text segments to obtain multiple segmentation units; mark the position of each segmentation unit according to the sequence of the text to be processed to obtain a semantic text block set including at least one semantic text block.

14. The apparatus according to claim 12, wherein the classification unit is configured to: Each semantic text block, key event information and event judgment prompt word in the semantic text block set is input into the classification large language model to obtain the classification results of each semantic text block output by the classification large language model; based on the classification results of the semantic text block set, each semantic text block is divided into a corresponding set to obtain a key text block set and a non-key text block set.

15. The device according to claim 12, wherein: The key generation unit is configured to: determine a set of images to be processed based on the material to be processed; extract the images to be processed that match the key text block set from the set of images to be processed to obtain a matching image set; obtain a matching text set based on the matching image set and the multimodal large model; match the matching text set with the key text block set to obtain a set of related text blocks in the key text block set that match the matching text set and a set of non-related text blocks that do not match the matching text set; obtain a first image sequence based on the related text block set and the matching image set; obtain a second image sequence based on the non-related text block set and the video material library; and merge the first image sequence and the second image sequence to obtain the key image sequence.

16. The device according to claim 15, wherein: The key generation unit is configured to: select matching images related to the related text block set in the matching image set; mark the matching image positions according to the positions of the related text blocks in the related text block set, and obtain a first image sequence.

17. The device according to claim 15, wherein: The key generation unit is configured to: search the video material library for a search image related to the non-related text block set; mark the position of the search image according to the position of the non-related text block in the non-related text block set, and obtain a second image sequence.

18. The device according to claim 15, wherein: The key generation unit is configured to: merge the first image sequence and the second image sequence to obtain an initial image sequence; and sort all initial images according to the position of each initial image in the initial image sequence to obtain a key image sequence.

19. The device according to claim 12, wherein: The non-key generation unit is configured to: input the non-key text block set and the pointing prompt word into the pointing large language model, and obtain the pointing result of the non-key text block set output by the pointing large language model; A non-key image sequence is obtained based on the non-key text block set, the pointing result and the video material library.

20. The device according to claim 19, wherein The non-key generation unit is configured to: in response to the pointing result of the non-key text block in the non-key text block set indicating that the non-key text block has a subject pointing, select an image material related to the subject pointing from the video material library; Based on the position of the non-critical text block, the position of the image material is marked, and the marked image material is placed in sequence in the non-critical image sequence.

21. The device according to claim 20, wherein: The non-key generation unit is configured to: generate an identification image related to the non-key text in response to a pointing result of a non-key text block in the non-key text block set indicating that the non-key text block does not have a subject pointing; mark the position of the identification image based on the position of the non-key text block, and place the marked identification image in the non-key image sequence.

22. The device according to claim 12, wherein: The video generation unit is configured to: merge the key image sequence and the non-key image sequence to obtain a mixed image sequence; sort all mixed images according to the position of each mixed image in the mixed image sequence to obtain a target image sequence; and obtain a target video based on the target image sequence.

23. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 11.

Images