Video generation method, electronic device, storage medium and product

Abstract

The present application discloses a video generation method, an electronic device, a storage medium and a product, and relates to the field of artificial intelligence. In the present application, different large language models are used to generate video texts in the process of performing video generation tasks, so different video texts can be generated based on the same text material, thereby ensuring the diversity of video texts generated based on the same text material. The video text will also be split into different text segments, and a target image segment will be selected for each text segment based on similarity, and each target image segment can be combined with the video text into a target video. It can be understood that since each target image segment is selected based on the video text, the video obtained by splicing the target image segments selected based on the video text is also diverse when the generated video text is diverse. Therefore, even when facing similar or identical materials, different video works can be generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, electronic device, storage medium and product. Background Art

[0002] With the rise of short video platforms, people's demand for short videos is also increasing. In order to improve the efficiency of short video editing and generation, short video generation tools have emerged to automatically generate short videos. However, short video generation tools currently have serious homogeneity problems. Especially when faced with similar or identical materials, video generation tools often generate the same or similar video works, which is difficult to meet users' diverse generation needs.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a video generation method, electronic device, storage medium and product, aiming to solve the technical problem that short videos currently generated by short video generation tools are seriously homogenized, especially when faced with similar or identical materials, video generation tools often obtain identical or similar video works.

[0005] To achieve the above object, the present application proposes a video generation method, which includes:

[0006] Receiving image material and text material, segmenting the image material to obtain image segments, and marking labels for the image segments;

[0007] For any video generation task for the image material and the text material, a large language model is selected from different preset large language models as a target model, and a video copy is generated by using the target model and the text material;

[0008] The video text is split into text segments according to sentences, and based on the similarity between the label of the image segment and the text segment, a target image segment is selected from the image segments for each text segment, wherein the image segment with higher similarity between the label and the text segment has a higher probability of being selected as the target image segment;

[0009] A target video is generated based on each target image segment and the video text combination.

[0010] Optionally, the step of selecting a large language model from different preset large language models as the target model includes:

[0011] Determine a candidate large language model from among the preset different large language models, wherein a large language model from among the preset different large language models that has not been used by a historical video generation task is the candidate large language model, and the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0012] A candidate large language model is selected and a sampling strategy parameter of the selected candidate large language model is adjusted to expand a sampling range of the selected candidate large language model to obtain the target model.

[0013] Optionally, the step of generating a video copy using the target model and the text material includes:

[0014] Selecting a target prompt word template from preset different prompt word templates, wherein, among the preset different prompt word templates, a prompt word template that has not been used by a historical video generation task is the target prompt word template, and the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0015] The target prompt word template and the text material are input into the target model to generate the video copy.

[0016] Optionally, the step of selecting a target image segment for each text segment from each image segment based on the similarity between the label of the image segment and the text segment includes:

[0017] For any one of the text segments, determine the similarity between the text segment and the label of each image segment, and use the candidate image segment whose label has the greatest similarity with the text segment and has not been used in the historical video generation task as the target image segment of the text segment, wherein the historical video generation task is a video generation task for the image material and the text material and has been completed currently;

[0018] After traversing the text segments in the text segments, the target image segments are obtained.

[0019] Optionally, after the step of determining the similarity between the text segment and the labels of each image segment, the method includes:

[0020] In the case that there is no candidate image segment that has not been used in the historical video generation task among the image segments, the image segment whose label is most similar to the text segment among the image segments is used as the image segment to be processed;

[0021] The to-be-processed image segment is deduplicated to obtain a target image segment of the text segment, wherein the deduplication process includes at least one of image flipping, random adjustment of image ratio, and random adjustment of image parameters.

[0022] Optionally, the step of generating a target video based on the target image segments and the video text combination includes:

[0023] Based on the order of the text segments corresponding to the target image segments in the video copy, the target image segments are spliced ​​together to obtain a spliced ​​video;

[0024] The video text is imported into the spliced ​​video to form the target video.

[0025] In addition, to achieve the above objectives, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video generation method described above.

[0026] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video generation method described above are implemented.

[0027] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the video generation method described above are implemented.

[0028] One or more technical solutions proposed in this application have at least the following technical effects:

[0029] In the embodiment of the present application, image material and text material will be received, the image material will be segmented to obtain each image segment, and each image segment will be labeled; for any video generation task for the image material and the text material, a large language model is selected from different preset large language models as the target model, and a video copy is generated through the target model and the text material; the video copy is split into each text segment according to the sentence, and based on the similarity between the label of the image segment and the text segment, a target image segment is selected from each image segment for each text segment, wherein the image segment with a higher similarity between the label and the text segment has a higher probability of being selected as the target image segment; a target video is generated based on each target image segment and the video copy combination. That is, after receiving the image material and text material uploaded by the user, the image material will be split first, and each image segment obtained by the split will be labeled as a basis for matching the copy. For any video generation task for the received image material and text material, when executing the video generation task to generate a video, one of the multiple large language models can be selected as the target model, and the video copy is generated through the selected target model and the received text material. Since different large language models have different model structures, model parameters or training samples, different video copy can be generated even based on the same text material by using different large language models, thereby ensuring the diversity of video copy generated based on the same text material. The video copy will also be split into different text segments, and a target image segment will be selected for each text segment based on similarity, and each target image segment can be combined with the video copy to form a target video. It is understandable that since each target image segment is selected based on the video copy, in the case of the diversity of the generated video copy, the video spliced ​​by the target image segment selected according to the video copy also has diversity. Therefore, in an embodiment of the present application, even when facing similar or identical materials, different video works can be generated, thereby meeting the user's video diversity generation needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0032] Figure 1This is a flow chart of the first embodiment of the video generation method of the present application;

[0033] Figure 2 This is a flow chart of the second embodiment of the video generation method of the present application;

[0034] Figure 3 This is a flow chart of the third embodiment of the video generation method of the present application;

[0035] Figure 4 This is a schematic diagram of the structure of a video generating device in the video generating method of the present application;

[0036] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the video generation method in the embodiment of the present application.

[0037] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0038] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0039] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0040] The main solution of the embodiment of the present application is: receiving image materials and text materials, segmenting the image materials to obtain image segments, and marking labels for the image segments; for any video generation task for the image materials and the text materials, selecting a large language model from preset different large language models as a target model, and generating video text through the target model and the text materials; segmenting the video text into text segments according to sentences, and based on the similarity between the labels of the image segments and the text segments, selecting target image segments for the text segments from the image segments, wherein the higher the similarity between the labels and the text segments, the higher the probability that the image segments are selected as target image segments; generating a target video based on the combination of each target image segment and the video copy.

[0041] With the rise of short video platforms, people's demand for short videos is also increasing. In order to improve the efficiency of short video editing and generation, short video generation tools have emerged to automatically generate short videos. However, short video generation tools currently have serious homogeneity problems. Especially when faced with similar or identical materials, video generation tools often generate the same or similar video works, which is difficult to meet users' diverse generation needs.

[0042] The present application provides a solution. After receiving the image material and text material uploaded by the user, the image material will be split first, and each image segment obtained by the splitting will be marked with a label as a basis for matching the copy. For any video generation task for the received image material and text material, when executing the video generation task to generate a video, one of the multiple large language models can be selected as the target model, and the video copy can be generated by the selected target model and the received text material. Since the model structure, model parameters or training samples of different large language models are different, even if based on the same text material, different video copy can be generated by using different large language models, thereby ensuring the diversity of video copy generated based on the same text material. The video copy will also be split into different text segments, and a target image segment will be selected for each text segment based on similarity, and each target image segment can be combined with the video copy to form a target video. It can be understood that since each target image segment is selected based on the video copy, in the case of the diversity of the generated video copy, the video obtained by splicing the target image segments selected according to the video copy is also diverse. Therefore, in the embodiment of the present application, even when faced with similar or identical materials, different video works can be generated, thereby meeting the user's demand for video diversity generation.

[0043] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a car computer, a computer, etc., or an electronic device that can realize the above functions.

[0044] Based on this, the embodiment of the present application provides a video generation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the video generation method of the present application.

[0045] The video generation method comprises steps S10 to S30:

[0046] Step S10, receiving image material and text material, segmenting the image material to obtain image segments, and marking labels for the image segments;

[0047] It should be noted that the implementation subject of the above-mentioned video generation method can be a computer, a cloud server or a mobile terminal, etc., and the above-mentioned image material and text material are image data and text data uploaded or input by the user, respectively. Among them, the image data can be a video or a certain picture, and the text material can be multiple keywords, multiple subject words or a paragraph of text, etc. It should also be noted that the text material can be a text describing the content of the image material, or it can be a text independent of the content of the image material.

[0048] Exemplarily, after receiving the image material and the text material, the image material is first split into multiple image segments (i.e., each image segment), and each image segment includes at least one image frame. The image material splitting process may include using a multimodal AI model to perform scene recognition on the image material, and then splitting the image material based on the scene. For example, the multimodal features of each image frame in the image material are extracted through a multimodal AI model, such as visual features (including objects, characters, actions in the picture) and audio features (including background audio embedded in the image material, character speech audio, etc.), and then scene recognition is performed based on the multimodal features. When it is determined that a scene switch occurs, the image frame of the scene switch is used as the splitting boundary between image segments. Each image segment is then labeled, wherein the label of the image segment can be a description of the image segment scene, such as the behavior of the characters in the picture, or the audio content corresponding to the picture, etc.

[0049] Step S20, for any video generation task for the image material and the text material, a large language model is selected from different preset large language models as a target model, and a video copy is generated by using the target model and the text material;

[0050] It should be noted that after the user uploads the image material and the text material, the number of videos to be generated based on the image material and the text material can be set. Accordingly, generating a video based on the image material and the text material corresponds to a video generation task for the image material and the text material. Accordingly, since the process of generating each video is roughly the same, in this embodiment, the generation process of one video will be taken as an example for description.

[0051] Exemplarily, for any video generation task for the above-mentioned image material and the above-mentioned text material (i.e., a task of generating a video based on the above-mentioned image material and the above-mentioned text material). During the execution of the video generation task, a large language model can be selected from different preset large language models as the target model, wherein the preset multiple large language models can be selected by technical personnel according to actual needs. Different large language models that are open for use, and specifically, the types of large language models are not limited here. A large language model can be randomly selected from multiple large language models as the target model, or a large language model that has not been used in the process of generating a video based on the above-mentioned image material and the above-mentioned text material can be selected as the target model. Generate video copy based on the selected target model and the above-mentioned text material, such as using the target model to expand the above-mentioned text material to generate video copy.

[0052] It can be understood that, since in the embodiment of the present application, each time a video copy is generated, a large language model will be selected from multiple large language models for the generation of the video copy, and different large language models have different model structures, model parameters or training samples, even based on the same text material, different video copy can be generated by using different large language models, thereby ensuring the diversity of video copy generated based on the same text material.

[0053] Step S30, splitting the video text into text segments according to sentences, and selecting target image segments for each text segment from the image segments based on the similarity between the labels of the image segments and the text segments, wherein the image segment with higher similarity between the labels and the text segments has higher probability of being selected as the target image segment;

[0054] Exemplarily, after obtaining the video text, the video text can be split into sentences to obtain various text segments. For example, specific punctuation marks (e.g., periods) in the video text can be used as splitting positions. The video text can also be directly split into various text segments using a public text splitting tool. After completing the splitting of the text segments, the similarity between each text segment and the label of each image segment is determined, such as the label of the image segment and the text segment can be vectorized using a vectorization tool (e.g., embedding technology). The similarity between each vectorized label and the vectorized text segment is then calculated. Then, a target image segment is selected for each text segment based on the similarity. For example, for a text segment, the higher the vector similarity between the label of an image segment and the text segment, the higher the probability that the image segment is selected as the target image segment of the text segment. For example, an image segment with the label with the highest similarity to the text segment can be selected as the target image segment of the text segment.

[0055] Step S40, generating a target video based on the target image segments and the video text combination.

[0056] For example, after obtaining the target image segment of each text segment, the target image segments are integrated and spliced ​​to obtain a spliced ​​video, and then the video copy is imported into the spliced ​​video to obtain the target video. The video copy can be imported into the spliced ​​video in the form of subtitles or voice.

[0057] In this embodiment, image materials and text materials are received, the image materials are segmented to obtain image segments, and labels are marked for the image segments; for any video generation task for the image materials and the text materials, a large language model is selected from different preset large language models as the target model, and a video copy is generated through the target model and the text materials; the video copy is segmented into text segments according to sentences, and based on the similarity between the labels of the image segments and the text segments, a target image segment is selected from the image segments for each text segment, wherein the image segment with a higher similarity between the label and the text segment has a higher probability of being selected as the target image segment; and a target video is generated based on the combination of each target image segment and the video copy. That is, after receiving the image materials and text materials uploaded by the user, the image materials will be segmented first, and labels will be marked for each image segment obtained by the segmentation as a basis for matching the copy. For any video generation task for the received image materials and text materials, when executing the video generation task to generate a video, one of the multiple large language models can be selected as the target model, and the video copy is generated through the selected target model and the received text materials. Since different large language models have different model structures, model parameters or training samples, different video copy can be generated even based on the same text material by using different large language models, thereby ensuring the diversity of video copy generated based on the same text material. The video copy will also be split into different text segments, and a target image segment will be selected for each text segment based on similarity, and each target image segment can be combined with the video copy to form a target video. It is understandable that since each target image segment is selected based on the video copy, in the case of the diversity of the generated video copy, the video spliced ​​by the target image segment selected according to the video copy also has diversity. Therefore, in an embodiment of the present application, even when facing similar or identical materials, different video works can be generated, thereby meeting the user's video diversity generation needs.

[0058] In a feasible implementation manner, the step of selecting a large language model from different preset large language models as the target model includes steps S211 to S212:

[0059] Step S211, determining a candidate large language model among the preset different large language models, wherein a large language model among the preset different large language models that has not been used by a historical video generation task is the candidate large language model, and the historical video generation task is a video generation task for the image material and the text material and that has been completed currently;

[0060] Step S212, selecting a candidate large language model and adjusting the sampling strategy parameters of the selected candidate large language model to expand the sampling range of the selected candidate large language model to obtain the target model.

[0061] Exemplarily, a candidate large language model is determined among different preset large language models, and the candidate large language model refers to a large language model that has not been used by the historical video generation task among the preset multiple large language models. The historical video generation task refers to a video generation task that is also for the above-mentioned image material and text material and has been completed at the current moment. If there are multiple candidate large language models. Then a candidate large language model can be randomly selected, and the sampling strategy parameters of the selected candidate large language model can be adjusted, wherein the sampling strategy parameters refer to parameters such as Temperature or Top-P of the large language model. For example, Temperature is a parameter used to control the smoothness of the probability distribution of the model output. It affects the randomness of the model when generating text: High Temperature (value close to 1 or greater than 1): It will make the probability distribution of the model output smoother, increase the randomness of the output, and the generated text may be more diverse. Top-P is a sampling strategy, also known as nucleus sampling. Instead of adjusting the probability distribution, it directly selects the words with the highest probability until the cumulative probability of these words reaches a threshold P (P in Top-P). High P value: The model will consider more words and the generated text will be more diverse. Therefore, in the embodiment of the present application, the sampling range of the candidate large language model can be expanded by increasing Temperature and Top-P to obtain the target model. It can be understood that in this embodiment, not only the diversity of the video copy is ensured by setting multiple large language models, but also the model parameters of the large language model used to generate the video copy are adjusted, thereby further improving the diversity of the generated video copy.

[0062] In addition, it should be noted that, when it is determined that there is no candidate large language model among the preset different large language models, the large language model that has been used the least number of times in the historical video generation task can be selected as the target large language model, so as to avoid repeated use of the large language model as much as possible.

[0063] In a feasible implementation manner, the step of generating a video copy using the target model and the text material includes steps S221 and S222:

[0064] Step S221, selecting a target prompt word template from preset different prompt word templates, wherein, among the preset different prompt word templates, a prompt word template that has not been used by a historical video generation task is the target prompt word template, and the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0065] Step S222: input the target prompt word template and the text material into the target model to generate the video copy.

[0066] It should be noted that, in this embodiment, a plurality of different prompt word templates are provided, and the prompt word templates are used to prompt or guide the large language model to generate corresponding video texts.

[0067] For example, a target prompt word template is selected from different preset prompt word templates, and among the preset different prompt word templates, the prompt word template that has not been used by the historical video generation task is the target prompt word template. Similarly, the historical video generation task refers to the video generation task that is also for the above-mentioned image material and text material and has been completed at the current moment. The target prompt word template and the text material are input into the target model together, and the target model expands or integrates the text material under the guidance of the target prompt word template to obtain the video copy.

[0068] In addition, if there is no prompt word template that has not been used in the historical video generation task among the preset different prompt word templates, the prompt word template that forms a new pairing relationship with the target model is selected as the target prompt word template, wherein the new pairing relationship refers to the pairing relationship between the large language model and the prompt word template that has not been used in the historical video generation task.

[0069] It is understandable that after determining the large language model used to generate video copy, a target prompt word template will be selected from multiple different prompt word templates, and different prompt word templates will produce different prompts or guidance for the large language model, so that the large language model can generate different video copy based on the same text material, ensuring the diversity of the video copy.

[0070] In a feasible implementation manner, the step of generating a target video based on each target image segment and the video text combination includes step S41 to step S42:

[0071] Step S41, based on the order of the text segments corresponding to the target image segments in the video copy, stitching the target image segments to obtain a stitched video;

[0072] Step S42, importing the video text into the spliced ​​video to form the target video.

[0073] It should be noted that the text segment itself is obtained by splitting the video copy, so each text segment originally has a sequence in the video copy. When each target image segment is spliced, each target image segment can be spliced ​​in the order of its corresponding text segment in the video copy, so as to obtain a spliced ​​video. In addition, before splicing, each target image segment can be adapted to the text segment, such as for any target image segment, the target duration of the target image segment is calculated according to the length of the text segment corresponding to the target image segment, such as after the text segment is converted into speech, the duration of the speech is taken as the target duration, or the result of adding the duration of the speech to the duration of the image segment transition is taken as the target duration, and the target image segment is adapted to make the actual duration of the adapted target image segment the same as the target duration, wherein the adaptation process includes at least one of repeated playback, double-speed playback and screen repetition. After the splicing is completed, the video copy can be imported into the spliced ​​video obtained by splicing in the form of subtitles and / or audio to form the target video.

[0074] It is worth noting that in the process of adapting the target image segment by adding the duration of the voice and the duration of the image segment transition as the target duration, the subtitle of the previous sentence will disappear only after the audio duration is executed, and the transition duration will be separated in the middle, while the subtitle of the next sentence will appear from the starting point of the image segment after the transition, which can avoid the overlap of the two subtitles due to the transition, and can also quickly connect two adjacent image segments. Among them, the transition effect corresponding to the image segment transition duration can be erase, shutters, ripples, etc., and the transition effect is used to connect two adjacent image segments.

[0075] Reference Figure 2 , is a flow chart of a second embodiment proposed based on the first embodiment of the present application. In this embodiment, the same or similar contents as those in the above embodiment can be referred to the above description and will not be described in detail later. The step of selecting a target image segment for each text segment from each image segment based on the similarity between the label of the image segment and the text segment includes steps S31 to S32:

[0076] Step S31, for any one of the text segments, determine the similarity between the text segment and the label of each image segment, and use the candidate image segment whose label has the greatest similarity with the text segment and has not been used by the historical video generation task as the target image segment of the text segment, wherein the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0077] Step S32, after traversing the text segments in the text segments, obtaining the target image segments.

[0078] It should be noted that, since the process of selecting a target image segment for each text segment is the same, in this embodiment, one text segment will be taken as an example for description.

[0079] Exemplarily, for any one of the text segments, the similarity between the text segment and the label of each image segment is determined, that is, the text segment and the label are both vectorized, and then the similarity (such as cosine similarity) between the vectorized text segment and the vectorized label is calculated. Then, the candidate image segment in each image segment whose label has the greatest similarity with the text segment and has not been used by the historical video generation task is used as the target image segment of the text segment. Correspondingly, the historical video generation task is a video generation task for the image material and the text material and has been completed, and the image segment that has not been used by the historical video generation task is the candidate image segment. After traversing the text segments, the target image segments can be obtained.

[0080] It can be understood that in this embodiment, the candidate image segment that is most similar to the text segment and has not been used in the historical video generation task will be selected as the target image segment of the text segment. On the one hand, the purpose of selecting the most similar image segment is to ensure that the target image segment is related to the text segment. On the other hand, selecting the candidate image segment that has not been used can avoid duplication in the spliced ​​video as much as possible, thereby ensuring the diversity of the spliced ​​video.

[0081] In a feasible implementation manner, after the step of determining the similarity between the text segment and the labels of each image segment, the method includes steps S311 and S312:

[0082] Step S311, when there is no candidate image segment that has not been used in the historical video generation task among the image segments, the image segment whose label has the greatest similarity with the text segment among the image segments is used as the image segment to be processed;

[0083] Step S312, performing deduplication processing on the image segment to be processed to obtain a target image segment of the text segment, wherein the deduplication processing includes at least one of image flipping, random adjustment of image ratio, and random adjustment of image parameters.

[0084] It should be noted that, in the case where there are no candidate image segments that have not been used in the historical video generation task in each image video segment, the image segment with the greatest similarity between the label and the text segment in each image segment is used as the image segment to be processed, and the image segment to be processed is deduplicated to obtain the target image segment of the text segment. The deduplication process may include at least one of picture flipping, random adjustment of the picture ratio, and random adjustment of the picture parameters. For picture flipping, the picture frame in the image segment to be processed may be randomly mirrored. The random adjustment of the picture ratio is to randomly take an offset value on the X-axis or Y-axis of the picture frame in the image segment to be processed to stretch the picture of the image segment to be processed. The random adjustment of the picture parameters is to randomly adjust the picture parameters such as brightness, contrast, saturation and color temperature of the image segment to be processed. It can be understood that when it is inevitable to use repeated image segments, this embodiment will perform deduplication on the repeated image segments, so as to use non-repeated image segments as much as possible for splicing to ensure the diversity of the spliced ​​video.

[0085] Reference Figure 3, which is a schematic diagram of the actual scene flow of the video generation method in this application. The user can first set the text material (i.e., the preset text material) and upload the video material (i.e., the above-mentioned image material). After splitting the video material into segments, label each segment. The user can also review and adjust the automatically labeled labels. The user can set the number of videos to be generated, and each video to be generated corresponds to a video generation task. When each video generation task is executed, a large language model will be selected from multiple different large language models for copy generation. After obtaining the video copy, the material matching algorithm is executed for each sentence in the video copy, that is, the target image segment is selected for each sentence (text segment) by calculating the similarity. Then use the transition algorithm to splice the video segments (i.e., image segments) of each sentence. The video segments can be deduplicated before and after splicing. For the specific deduplication process, please refer to the content of the above embodiment. After splicing, elements such as cover, subtitles, music, and stickers can be automatically added to the spliced ​​video. Among them, the part of adding sticker elements includes pre-setting sticker materials, each sticker material will be labeled, and then the keywords in the text segment are understood by NLP (Natural Language Processing) to match the target sticker material with the keywords in the text segment. Among them, the sticker material with the highest similarity between the label and the keyword is the target sticker material, and then the sticker appears in the designated area of ​​the image segment corresponding to the text segment in a random animation, so as to make the overall video screen more vivid. For example, if the sticker material about "greening rate" is pre-set, if the keyword "greening rate" appears in the actual text segment, a "greening rate" sticker animation can pop up in the corresponding video screen. In addition, the part of adding cover elements includes inputting the video copy and task prompt words (such as prompt words for generating text titles) into the big language model, so that the big language model generates a title for the video copy under the guidance of the task prompt words. Then, the candidate image frames are selected from the spliced ​​video, wherein the step of selecting the candidate image frames from the spliced ​​video includes clustering based on the picture features of each video frame in the spliced ​​video to obtain multiple video frame clusters, wherein the picture features include sharpness, contrast, signal-to-noise ratio and multimodal features of the video frame; performing secondary clustering on the preset cover images based on the multiple video frame clusters, and taking the video frame clusters that are assigned the most preset cover images in the secondary clustering results as the target video frame clusters; taking the video frames in the target video frame clusters other than the preset cover images and having the highest similarity with the title features as the cover images, and combining the cover images and titles to obtain the cover of the spliced ​​video. In addition, the prompt words input into the large language model can also include the selection of background music, and the prompt words for the selection of background music are input into the large language model so that the large language model selects background music according to the video text.

[0086] The present application also provides a video generation device, referring to Figure 4 , the video generating device comprises:

[0087] The splitting module 10 is used to receive image materials and text materials, split the image materials into segments to obtain image segments, and mark labels for the image segments;

[0088] A first generating module 20 is used for selecting a large language model from preset different large language models as a target model for any video generation task for the image material and the text material, and generating a video copy by using the target model and the text material;

[0089] A selection module 30 is used to split the video text into text segments according to sentences, and select a target image segment for each text segment from the image segments based on the similarity between the label of the image segment and the text segment, wherein the image segment with a higher similarity between the label and the text segment has a higher probability of being selected as the target image segment;

[0090] The second generating module 40 is used to generate a target video based on each target image segment and the video text combination.

[0091] Optionally, the first generating module 20 is further used for:

[0092] Determine a candidate large language model from among the preset different large language models, wherein a large language model from among the preset different large language models that has not been used by a historical video generation task is the candidate large language model, and the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0093] A candidate large language model is selected and a sampling strategy parameter of the selected candidate large language model is adjusted to expand a sampling range of the selected candidate large language model to obtain the target model.

[0094] Optionally, the first generating module 20 is further used for:

[0095] Selecting a target prompt word template from preset different prompt word templates, wherein, among the preset different prompt word templates, a prompt word template that has not been used by a historical video generation task is the target prompt word template, and the historical video generation task is a video generation task that is currently completed for the image material and the text material;

[0096] The target prompt word template and the text material are input into the target model to generate the video copy.

[0097] Optionally, the selection module 30 is further used for:

[0098] For any one of the text segments, determine the similarity between the text segment and the label of each image segment, and use the candidate image segment whose label has the greatest similarity with the text segment and has not been used in the historical video generation task as the target image segment of the text segment, wherein the historical video generation task is a video generation task for the image material and the text material and has been completed currently;

[0099] After traversing the text segments in the text segments, the target image segments are obtained.

[0100] Optionally, the selection module 30 is further used for:

[0101] In the case that there is no candidate image segment that has not been used in the historical video generation task among the image segments, the image segment whose label is most similar to the text segment among the image segments is used as the image segment to be processed;

[0102] The to-be-processed image segment is deduplicated to obtain a target image segment of the text segment, wherein the deduplication process includes at least one of image flipping, random adjustment of image ratio, and random adjustment of image parameters.

[0103] Optionally, the second generating module 40 is further used for:

[0104] Based on the order of the text segments corresponding to the target image segments in the video copy, the target image segments are spliced ​​together to obtain a spliced ​​video;

[0105] The video text is imported into the spliced ​​video to form the target video.

[0106] The video generation device provided by the present application adopts the video generation method in the above embodiment, aiming to solve the technical problem that the short videos obtained by the current short video generation tools are seriously homogenized, especially when faced with similar or identical materials, the video generation tools often obtain the same or similar video works. Compared with the prior art, the beneficial effects of the video generation device provided by the embodiment of the present application are the same as the beneficial effects of the video generation method provided by the above embodiment, and the other technical features in the video generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0107] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video generation method in the above-mentioned embodiment 1.

[0108] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0109] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0110] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0111] The electronic device provided by the present application adopts the video generation method in the above embodiment, which can solve the technical problem that the short videos obtained by the current short video generation tools are seriously homogenized, especially when faced with similar or identical materials, the video generation tools often obtain the same or similar video works. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the video generation method provided by the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0113] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0114] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video generation method in the above-mentioned embodiment.

[0115] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0116] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0117] The computer-readable storage medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device generates a video.

[0118] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0121] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned video generation method, and can solve the technical problem that the short videos obtained by the current short video generation tools are seriously homogenized, especially when faced with similar or identical materials, the video generation tools often obtain the same or similar video works. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the video generation method provided by the above-mentioned embodiment, and will not be repeated here.

[0122] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video generation method when executed by a processor.

[0123] The computer program product provided by the present application can solve the technical problem of video generation. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the video generation method provided by the above embodiment, which will not be repeated here.

[0124] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video generation method, characterized in that: The video generation method comprises: Receiving image material and text material, segmenting the image material to obtain image segments, and marking labels for the image segments; For any video generation task for the image material and the text material, a large language model is selected from different preset large language models as a target model, and a video copy is generated by using the target model and the text material; The video text is split into text segments according to sentences, and based on the similarity between the label of the image segment and the text segment, a target image segment is selected from the image segments for each text segment, wherein the image segment with higher similarity between the label and the text segment has a higher probability of being selected as the target image segment; Generate a target video based on each target image segment and the video copy combination; The step of selecting a large language model from different preset large language models as a target model includes: Determine a candidate large language model from among the preset different large language models, wherein a large language model from among the preset different large language models that has not been used by a historical video generation task is the candidate large language model, and the historical video generation task is a video generation task that is currently completed for the image material and the text material; A candidate large language model is selected and a sampling strategy parameter of the selected candidate large language model is adjusted to expand a sampling range of the selected candidate large language model to obtain the target model, wherein the sampling strategy parameter refers to a Temperature or Top-P parameter of the large language model.

2. The video generation method according to claim 1, characterized in that: The step of generating a video copy using the target model and the text material comprises: Selecting a target prompt word template from preset different prompt word templates, wherein, among the preset different prompt word templates, a prompt word template that has not been used by a historical video generation task is the target prompt word template, and the historical video generation task is a video generation task that is currently completed for the image material and the text material; The target prompt word template and the text material are input into the target model to generate the video copy.

3. The video generation method according to claim 1, characterized in that: The step of selecting a target image segment for each text segment from each image segment based on the similarity between the label of the image segment and the text segment comprises: For any one of the text segments, determine the similarity between the text segment and the label of each image segment, and use the candidate image segment whose label has the greatest similarity with the text segment and has not been used in the historical video generation task as the target image segment of the text segment, wherein the historical video generation task is a video generation task for the image material and the text material and has been completed currently; After traversing the text segments in the text segments, the target image segments are obtained.

4. The video generation method according to claim 3, characterized in that: After the step of determining the similarity between the text segment and the labels of each image segment, the method comprises: In the case that there is no candidate image segment that has not been used in the historical video generation task among the image segments, the image segment whose label is most similar to the text segment among the image segments is used as the image segment to be processed; The to-be-processed image segment is deduplicated to obtain a target image segment of the text segment, wherein the deduplication process includes at least one of image flipping and random adjustment of the image ratio.

5. The video generation method according to claim 1, characterized in that: The step of generating a target video based on the target image segments and the video text combination comprises: Based on the order of the text segments corresponding to the target image segments in the video copy, the target image segments are spliced ​​together to obtain a spliced ​​video; The video text is imported into the spliced ​​video to form the target video.

6. A video generating device, characterized in that: The video generating device comprises: A splitting module, used for receiving image material and text material, splitting the image material into segments to obtain image segments, and marking labels for the image segments; A first generation module is used to select a large language model from preset different large language models as a target model for any video generation task for the image material and the text material, and generate a video copy by using the target model and the text material; A selection module is used to split the video text into text segments according to sentences, and select a target image segment for each text segment from the image segments based on the similarity between the label of the image segment and the text segment, wherein the image segment with a higher similarity between the label and the text segment has a higher probability of being selected as the target image segment; A second generating module, configured to generate a target video based on each target image segment and the video text combination; The selecting a large language model from different preset large language models as a target model includes: Determine a candidate large language model from among the preset different large language models, wherein a large language model from among the preset different large language models that has not been used by a historical video generation task is the candidate large language model, and the historical video generation task is a video generation task that is currently completed for the image material and the text material; A candidate large language model is selected and a sampling strategy parameter of the selected candidate large language model is adjusted to expand a sampling range of the selected candidate large language model to obtain the target model, wherein the sampling strategy parameter refers to a parameter such as Temperature or Top-P of the large language model.

7. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video generation method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. The computer program is configured to implement the steps of the video generation method according to any one of claims 1 to 5 when executed by a processor.

9. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is configured to implement the steps of the video generation method according to any one of claims 1 to 5 when executed by a processor.

Images