A method and device for generating ancient poetry to video based on large language model

Abstract

The present invention discloses a method and device for generating videos from ancient poems based on a large language model, and relates to the technical field of video generation. The method comprises: constructing an ancient poetry knowledge base; using the Soft Prompt training method according to the large language model to obtain a trained story learning vector; obtaining the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; obtaining the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model; generating the story picture corresponding to the ancient poetry data to be generated according to the script, the picture knowledge and the picture generation model; generating the video corresponding to the ancient poetry data to be generated according to the story picture and the video generation model. The present invention can solve the problem of the scarcity of ancient poetry video data sets and improve the generation quality of ancient poetry videos.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a method and device for generating ancient poems into videos based on a large language model. Background Art

[0002] Chinese traditional culture is extensive and profound, and ancient poetry, as its essence, carries rich historical information and profound cultural heritage. However, current research on ancient poetry mainly focuses on its translation and image generation, which cannot fully display the artistic conception of ancient poetry, while videos have rich modal representations.

[0003] Wensheng video refers to inputting a natural language text and then converting it into video content. Existing video generation cannot generate videos of ancient poems because of the complex semantics of ancient poems and the lack of necessary data sets.

[0004] The translation task of ancient poetry refers to translating the input ancient poetry into other languages, and the translated content is required to be consistent with the artistic conception of the ancient poetry. The translation content of the ancient poem "Two golden orioles sing amid the willows green; A flock of white egrets fly into the blue sky. My window frames the snow-crowned western mountain scene; My door off says to eastward going ships "Goodbye!"" is required to be consistent with the Chinese meaning and artistic conception. Cao et al. proposed a TongGu large language model specifically for the task of Classical Chinese Understanding (CCU). The researchers constructed a two-stage instruction adjustment dataset ACCN-INS. In order to reduce the hallucinations generated by LLMs in knowledge-intensive tasks, the researchers proposed a knowledge-enhanced CCU-RAG Method, this method significantly improves the performance of the model in knowledge-intensive tasks through retrieval enhancement generation technology. Chen et al. proposed a method of using retrieval enhancement to enhance the translation of ancient poems by adding historical background information, author introduction, poetry type and other content, and achieved a lot of improvement. The generation of ancient poetry images mainly inputs the content of the ancient poem and then generates a picture that conforms to the ancient poem. Dan et al. proposed a new task, how to artistically visualize Chinese classical poetry to generate paintings. They manually collected 3,648 pairs of Feng Zikai's painting titles-painting pairs and collected 89,204 pairs of poetry-painting pairs in traditional Chinese painting styles from the Internet. They trained two representative text-to-image generation models-AttnGAN and MirrorGAN, and realized the conversion of Chinese classical poetry into paintings with a specific artistic style. Jiang proposed an iterative correction framework called "Poetry2Image", which aims to solve the common key element loss or semantic confusion problems of text-to-image generation models when generating images from Chinese classical poetry. By introducing an external dataset, the large prediction model detects the difference between the elements on the generated image and the semantics of the ancient poem, and iteratively completes the generation of the image.

[0005] The existing tasks of ancient poetry: translation and image generation of ancient poetry cannot fully reflect the artistic conception of ancient poetry; and for the existing literary video models, the existing video generation models are mostly end-to-end models obtained through training, while the ancient poetry videos lack the necessary training data sets and cannot be trained end-to-end. The language of ancient poetry is often concise and layered, containing a large number of images, metaphors and emotions. These abstract concepts and deep cultural connotations are difficult for existing models to fully understand and process. There is a rich historical and cultural background behind the ancient poetry. Generating videos that match the artistic conception of ancient poetry requires the model to have a deep understanding of this knowledge. Existing models usually rely on large-scale training data, which may not fully cover the cultural knowledge related to ancient poetry, resulting in a lack of cultural authenticity in the generated content, making the generated ancient poetry videos of low quality. Summary of the invention

[0006] In order to solve the technical problems of the lack of ancient poetry video datasets and the low quality of the generated ancient poetry videos in the prior art, the embodiment of the present invention provides a method and device for generating ancient poetry videos based on a large language model. The technical solution is as follows:

[0007] On the one hand, a method for generating ancient poems to videos based on a large language model is provided, the method being implemented by a device for generating ancient poems to videos based on a large language model, the method comprising:

[0008] S1. Constructing an ancient poetry knowledge base, wherein the ancient poetry knowledge base includes text knowledge and image knowledge;

[0009] S2. Use the Soft Prompt training method based on the large language model to obtain the trained story learning vector;

[0010] S3, obtaining the ancient poetry data to be generated, searching the ancient poetry knowledge base for the text knowledge corresponding to the ancient poetry data to be generated, and obtaining the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model;

[0011] S4, obtaining the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model;

[0012] S5, generating a story picture corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated;

[0013] S6. Generate a video corresponding to the ancient poetry data to be generated based on the story pictures corresponding to the ancient poetry data to be generated and the video generation model.

[0014] On the other hand, a device for generating ancient poems to videos based on a large language model is provided, and the device is applied to a method for generating ancient poems to videos based on a large language model, and the device includes:

[0015] A construction module is used to construct an ancient poetry knowledge base, wherein the ancient poetry knowledge base includes text knowledge and image knowledge;

[0016] The training module is used to obtain the trained story learning vector using the Soft Prompt training method based on the large language model;

[0017] An acquisition module, used for acquiring the ancient poetry data to be generated, searching the ancient poetry knowledge base for the text knowledge corresponding to the ancient poetry data to be generated, and obtaining the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model;

[0018] A first generation module, for obtaining a script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, a preset prompt word and the large language model;

[0019] The second generation module is used to generate a story picture corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated;

[0020] The third generation module is used to generate a video corresponding to the ancient poetry data to be generated based on the story pictures corresponding to the ancient poetry data to be generated and the video generation model.

[0021] On the other hand, a device for generating ancient poems to videos based on a large language model is provided, and the device for generating ancient poems to videos based on a large language model comprises: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the methods for generating ancient poems to videos based on a large language model is implemented.

[0022] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for generating ancient poems to videos based on a large language model.

[0023] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0024] Construct an ancient poetry knowledge base, which includes text knowledge and picture knowledge; adopt the Soft Prompt training method according to the large language model to obtain a trained story learning vector; obtain the ancient poetry data to be generated, find out the text knowledge corresponding to the ancient poetry data to be generated in the ancient poetry knowledge base, and obtain the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; obtain the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model; generate the story picture corresponding to the ancient poetry data to be generated according to the script corresponding to the ancient poetry data to be generated, the picture knowledge and the picture generation model; generate the video corresponding to the ancient poetry data to be generated according to the story picture corresponding to the ancient poetry data to be generated and the video generation model. The present invention does not require a large number of samples for training, solves the problem of lack of ancient poetry video data sets and inability to perform end-to-end video generation, realizes the understanding of the complex semantics of ancient poetry and improves the generation quality of ancient poetry videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a flow chart of a method for generating ancient poetry to video based on a large language model provided by an embodiment of the present invention;

[0027] Figure 2 It is a schematic diagram of the structure of three kinds of knowledge included in an ancient poetry knowledge base provided by an embodiment of the present invention;

[0028] Figure 3 It is a structural schematic diagram of a soft prompt training provided by an embodiment of the present invention;

[0029] Figure 4 This is a case diagram of story generation provided by an embodiment of the present invention;

[0030] Figure 5 It is a structural schematic diagram of a story segment picture generation provided by an embodiment of the present invention;

[0031] Figure 6 It is a block diagram of a device for generating ancient poetry to video based on a large language model provided by an embodiment of the present invention;

[0032] Figure 7It is a structural schematic diagram of a device for generating ancient poetry to video based on a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0034] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0035] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0036] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0037] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0038] The embodiment of the present invention provides a method for generating ancient poetry to video based on a large language model. The method can be implemented by a device for generating ancient poetry to video based on a large language model. The device for generating ancient poetry to video based on a large language model can be a terminal or a server. Figure 1 The flowchart of the method for generating ancient poems to videos based on a large language model is shown. The processing flow of the method may include the following steps:

[0039] S1. Build an ancient poetry knowledge base, which includes text knowledge and image knowledge.

[0040] Optionally, the specific operation of S1 may be as follows:

[0041] Obtain the original text, translation and annotations of ancient poems, the creation background of ancient poems and the appreciation of ancient poems from text websites related to ancient poems and determine them as text knowledge;

[0042] We obtain pictures related to ancient poetry from picture websites related to ancient poetry, including pictures of people, scenery, and ancient poetry. We use the CLIP model to filter out pictures that do not meet the standards and identify the filtered pictures as picture knowledge.

[0043] In a feasible implementation, Figure 2 As shown, text knowledge can include: translation, annotation, appreciation, author and dynasty knowledge of ancient poems; picture knowledge can include: author characters, scenery pictures and pictures related to ancient poems.

[0044] S2. Use the Soft Prompt training method based on the large language model to obtain the trained story learning vector.

[0045] Optionally, the specific operation of S2 may include the following steps S21-S24:

[0046] S21, obtaining training samples, where the training samples include sample ancient poems and true story sequences corresponding to the sample ancient poems;

[0047] S22, freezing the model parameters in the initial large language model, inputting the sample ancient poetry into the frozen large language model, and obtaining the predicted story sequence corresponding to the sample ancient poetry according to the story learning vector Prompt to be trained;

[0048] S23. Compare the predicted story sequence corresponding to the sample ancient poetry with the true story sequence, and train the Prompt to be trained according to the cross entropy loss function. The formula of the cross entropy loss function is as follows (1):

[0049] (1)

[0050] Where T is the length of the true story sequence, It means that under the condition of sample ancient poetry X and softprompt P, the large language model predicts the i-th word in the true story sequence probability;

[0051] S24. Iterate the training until the loss function converges, stop the training, and obtain the trained story learning vector Prompt.

[0052] In a feasible implementation, in the ancient poetry video generation task, LLM (Large Language Model) is used as the writer and director task. In order to let LLM find an optimal task prompt, softprompt training is adopted. Figure 3 As shown, when training soft prompt, freeze the pre-trained model parameters (Right now Figure 3 In the text encoder, Prompt is represented by a series of consecutive word segments P, P = {p1, p2, . . . , pn}, where n is the length of P, which is 20. When inputting ancient poems X and related stories Y, freeze the pre-trained model parameters In the case of , by training P, the similarity between the story y generated by the model and the related story Y can be maximized, and the cross entropy loss function can be used as the training loss function, as shown in the above formula (1). Through this training, Prompt can be trained well while retaining the generation ability of the large language model.

[0053] S3. Obtain the ancient poetry data to be generated, search for the text knowledge corresponding to the ancient poetry data to be generated in the ancient poetry knowledge base, and obtain the story corresponding to the ancient poetry data to be generated based on the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model.

[0054] Optionally, the specific operation of S3 may be as follows:

[0055] According to the trained story learning vector, the text knowledge corresponding to the ancient poetry data to be generated is segmented, and the segmentation result is input into the large language model to obtain the story corresponding to the ancient poetry data to be generated.

[0056] In a feasible implementation, Figure 4 As shown, the ancient poetry data to be generated is Yang Wanli's "Little Pond", then the title, content and author of the ancient poetry are input according to the preset format. For example, the input format is as follows: "title": "Little Pond", "author": "Yang Wanli", "content": "The spring eye silently cherishes the trickle, the tree shadow shines on the water and loves the soft sunshine. The little lotus has just revealed its pointed corners, and a dragonfly has already landed on it."

[0057] Use the title and author of the ancient poetry data to be generated, or use the content of the ancient poetry data to be generated, and search the ancient poetry knowledge base for the corresponding text knowledge, such as background knowledge, author knowledge, dynasty knowledge, translation and annotations, etc. Perform word segmentation on the text knowledge corresponding to the ancient poetry data to be generated, and input the word segmentation result and the trained story learning vector into the large language model to obtain the story corresponding to the ancient poetry data to be generated.

[0058] S4. Obtain the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model.

[0059] Optionally, the specific operation of S4 may include the following steps S41-S43:

[0060] S41. Decompose the story corresponding to the ancient poetry data to be generated using preset rules to obtain a text description sequence.

[0061] In one possible implementation, some rules R are used to decompose the story into a series of text descriptions P to bridge the gap.

[0062] S42. Use a large language model to extract character information and location information from the story corresponding to the ancient poetry data to be generated.

[0063] In a feasible implementation, for extracting the main characters and places in the story, given a story S, a large language model is used to extract the main characters and place descriptions, and the character description format is {id:1, description: character description, id:2, description: character description...id:n, description: character description};

[0064] The location description is {pid:1, location description: description, pid:2, location description: description....pid:n, location description: description}.

[0065] S43, generating four scripts according to the text description sequence, the extracted character information and location information, and the preset prompt words.

[0066] Among them, the first script of the four scripts describes the main tasks and main scenes of the story, the second script of the four scripts describes the video clips of the story, the third script of the four scripts describes the correspondence between the characters and scenes, and the fourth script of the four scripts describes the duration.

[0067] In a feasible implementation, using the information of S41 and S42, the large language model can design four scripts through preset prompt words. The preset prompt words can be designed in various forms. A feasible prompt word example is given below, where prompt words 1, 2, 3 and 4 correspond to generated scripts 1, 2, 3 and 4 respectively:

[0068] Prompt word 1:

[0069] ask = f"""I would like to make a video. Here are this video scriptin the following three single quotes {new_video_list} ”""\

[0070] f“““Here are some characters and places in the followingthree single quotes {key_list}””” \

[0071] f“““Please help me identify the characters or places in thelist where each segment of the video script appears””” \

[0072] f“““You can only choose characters and places that matchexactly, and you can't choose even the slightest doubt. ””” \

[0073] f“““Just answer me the serial number(1 selection ispossible, but no more, pick out what you think is most likely. If you selectless than 1, you can fill it with 0.) ””” \

[0074] f“““Provide me the answer in JSON format(do not answeranything else) with the following keys: video segment id, character / place id.””” \

[0075] f“““You must answer like the content in following threesingle quotes: ””” \

[0076] f“““[

[0077] {{

[0078] “video segment id”: 1,

[0079] “character / place id”: [1],

[0080] }},

[0081] {{

[0082] “video segment id”: 2,

[0083] “character / place id”: [0],

[0084] }},

[0085] {{

[0086] “video segment id”: 3,

[0087] “character / place id”: [2],

[0088] }}

[0089] ”””

[0090] Prompt two:

[0091] ask = f“““The following is a story enclosed in three single quotes{story}. I want to make a video according to this story, this is my videoproduction script in the following three single quotes {video_list}, aparagraph in the script corresponds to a clip ””” \

[0092] f“““of the video. However, there may be some plots in thestory missing, such as important plot missing, or transitions betweenpictures, please check and complete it for me. ””” \

[0093] f“““Provide me the answer in JSON format(do not answeranything else) with the following keys: video fragment id, video fragmentdescription. ””” \

[0094] f“““You must must must answer like the content in followingthree single quotes:\n””” \

[0095] f”””[

[0096] {{

[0097] “video fragment id”: 1,

[0098] “video fragment description”: “(the description) ”,

[0099] }},

[0100] {{

[0101] “video fragment id”: 2,

[0102] “video fragment description”: “(the description) ”,

[0103] }},

[0104] {{

[0105] “video fragment id”: 3,

[0106] “video fragment description”: “(the description)”,

[0107] }}

[0108] ] ””” \

[0109] f“““Remember to make sure that the description of each video clip is not long, no more than fifteen words, but there can be so many video clips.\n””” \

[0110] f“““Each description should include the subject, place, and action as much as possible. ””” \

[0111] f“““As more fragment as possible, as detail as possible!\n””” \

[0112] f“““Read this script carefully and don't pull down any details.\n”””

[0113] # f“““Very important!!!: avoid character-to-character interactions and character-to-object interactions in descriptions. ”””

[0114] Prompt 3:

[0115] ask = f“““I would like to make a video. Here are this video script in the following three single quotes {new_video_list} ””” \

[0116] f“““Here are some characters and places in the following three single quotes {key_list}””” \

[0117] f“““Please help me identify the characters or places in thelist where each segment of the video script appears””” \

[0118] f“““You can only choose characters and places that matchexactly, and you can't choose even the slightest doubt. ””” \

[0119] f“““Just answer me the serial number(1 selection ispossible, but no more, pick out what you think is most likely. If you selectless than 1, you can fill it with 0.) ””” \

[0120] f“““Provide me the answer in JSON format(do not answeranything else) with the following keys: video segment id, character / place id.””” \

[0121] f“““You must answer like the content in following threesingle quotes: ””” \

[0122] f“““[

[0123] {{

[0124] “video segment id”: 1,

[0125] “character / place id”: [1],

[0126] }},

[0127] {{

[0128] “video segment id”: 2,

[0129] “character / place id”: [0],

[0130] }},

[0131] {{

[0132] “video segment id”: 3,

[0133] “character / place id”: [2],

[0134] }}

[0135] ”””

[0136] Prompt Four:

[0137] ask = f“““I want to make a video, this is my videoproduction script in the following three single quotes {new_video_list} aparagraph in the script corresponds to a clip ””” \

[0138] f“““of the video, Now that you know that 16-framevideos have a length of 2 seconds, please help me plan how much time it willtake for each video clip to fully interpret the meaning of the script. ””” \

[0139] f“““Each clip can only be 10 seconds maximum. ””” \

[0140] f“““Provide me the answer in JSON format(do not answeranything else) with the following keys: video fragment id, time. ””” \

[0141] f"""You must answer like the content in followingthree single quotes:\n""\

[0142] f"""[

[0143] {{

[0144] “video fragment id”: 1,

[0145] "time": 3,

[0146] }},

[0147] {{

[0148] “video fragment id”: 2,

[0149] "time": 9,

[0150] }},

[0151] {{

[0152] “video fragment id”: 3,

[0153] "time": 7,

[0154] }},

[0155] {{

[0156] “video fragment id”: 4,

[0157] "time": 2,

[0158] }},

[0159] ] ””” \

[0160] f"""Remember that time must be less than 10. """

[0161] The following is an introduction to Script 1, Script 2, Script 3, and Script 4:

[0162] Script 1: Describe the main characters and main scenes in the story, such as the following format:

[0163] {

[0164] "id": 1,

[0165] “name”: “Zhao Xintong”,

[0166] “description”: “A young boy with messy hair and bright, curious eyes. His skin is slightly tanned from the sun and he wears baggy shorts and a shirt.”

[0167] },

[0168] {

[0169] "id": 2,

[0170] "name": "Jiangnan Water Village",

[0171] “description”: “Early morning scene, characterized by diffuse mist and the first rays of sunlight. There is a pond in the scene.”

[0172] },

[0173] Script 2: describes how many scenes the story generates, such as the following format:

[0174] {

[0175] "Video clip id": 1,

[0176] “Video clip description”: “The sunrise atmosphere of Jiangnan water village.”

[0177] },

[0178] {

[0179] "Video clip id": 2,

[0180] “Video clip description”: “A scene of a small town covered in mist.”

[0181] },

[0182] {

[0183] "Video clip id": 3,

[0184] “Video clip description”: “Young Zhao Xintong approaches the pond.”

[0185] }

[0186] Script 3: Describes the characters and places that appear in the scene, such as the following:

[0187] {

[0188] "Video clip id": 1,

[0189] "role / location id": [2]

[0190] },

[0191] {

[0192] "Video clip id": 2,

[0193] "role / location id": [2]

[0194] },

[0195] {

[0196] "Video clip id": 3,

[0197] "role / location id": [1]

[0198] }

[0199] Script and 4: describes how long it takes to generate each scene, such as the following format:

[0200] {

[0201] "Video clip id": 1,

[0202] Time: 3

[0203] },

[0204] {

[0205] "Video clip id": 2,

[0206] Time: 4

[0207] },

[0208] {

[0209] "Video clip id": 3,

[0210] Time: 3

[0211] }

[0212] S5. Generate story pictures corresponding to the ancient poetry data to be generated based on the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated.

[0213] Optionally, the image generation model may include a stable diffusion model and a Multi-subject consistency Generator generation technology;

[0214] The specific operation of S5 may include the following steps S51-S53:

[0215] S51 , inputting text descriptions corresponding to characters and places in the first script into a stable diffusion model to generate pictures corresponding to the characters and places.

[0216] In a feasible implementation, the information of the first script generated in step S43 is used to input the text descriptions of the characters and places into a stable diffusion model to generate corresponding images. The stable diffusion model is a generative model that learns the data distribution of complex data sets. They consist of a forward diffusion process and a backward diffusion process. In the forward process, the input data points are Perform T steps of noise addition until the resulting vector Almost according to the standard Gaussian distribution. The forward process can be shown as follows:

[0217]

[0218] The reverse process can be as follows:

[0219]

[0220] The variables obtained in the process It can be expressed as:

[0221]

[0222] S52, generating pictures of a plurality of scene segments according to the generated pictures corresponding to the characters and places, the second script, the third script, the fourth script and the Multi-subject consistence Generator generation technology.

[0223] In a feasible implementation, Figure 5 As shown, first, the idlayout (id layout) of the task is generated by using the GPT-4O model, and the id layout is input into the trainable Conditional Encoder to obtain the output vector. The output vector, the text encoding vector of the frozen clip, and the picture vector after the picture encoder are input into the UNET network structure as a condition, and the output is controlled by training a Cross attention layer. A generation method of a Multi-subject consistence Generator is used to utilize the pictures generated by S51 and the information in the second script, the third script, and the fourth script to generate a fragment of each video scene. The layout of the characters in the scene is designed using the capabilities of LLM. The layout information is a box (x, y), where x and y represent the coordinates of the object in the picture. The character's id, picture, and box are used as a condition and input into the Condition-Text-to-layout model. The training loss function is:

[0224]

[0225] in It is standard Gaussian noise. The goal is to predict the minimum value of the real noise and predicted noise added by the model after a given person ID, person picture, and person box.

[0226] S53. Use the large language model to optimize the generated images of multiple scene fragments to obtain optimized story images.

[0227] In a feasible implementation, the reflection ability of LLM is used to allow LLM to optimize the generation of the character box for better results. The output image, the image of the original object, and the Character Prompt are input to LLM, so that LLM can optimize the generation of the box through selfreflection ability and generate better images.

[0228] S6. Generate a video corresponding to the ancient poetry data to be generated based on the story pictures corresponding to the ancient poetry data to be generated and the video generation model.

[0229] Optionally, the specific operation of S6 may include S61-S62:

[0230] S61, inputting the story picture corresponding to the ancient poetry data to be generated and the corresponding text prompt into the Pika model to obtain an initial video;

[0231] S62, adding subtitles and sound to the initial video to obtain a video corresponding to the ancient poetry data to be generated.

[0232] In one feasible implementation, the Pika model is a video generation model launched by Pika, an American AI video generation startup. Through the story pictures corresponding to the ancient poetry data to be generated and the corresponding text prompts, the Pika model can generate the corresponding initial video, and then add subtitles and sound to the generated video clips to complete the generation of the ancient poetry video.

[0233] In the embodiment of the present invention, an ancient poetry knowledge base is constructed, and the ancient poetry knowledge base includes text knowledge and picture knowledge; the Soft Prompt training method is adopted according to the large language model to obtain a trained story learning vector; the ancient poetry data to be generated is obtained, and the text knowledge corresponding to the ancient poetry data to be generated is found in the ancient poetry knowledge base, and the story corresponding to the ancient poetry data to be generated is obtained according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; the script corresponding to the ancient poetry data to be generated is obtained according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model; the story picture corresponding to the ancient poetry data to be generated is generated according to the script corresponding to the ancient poetry data to be generated, the picture knowledge and the picture generation model; the video corresponding to the ancient poetry data to be generated is generated according to the story picture corresponding to the ancient poetry data to be generated and the video generation model. The present invention does not require a large number of samples for training, solves the problem of the lack of ancient poetry video data sets and the inability to perform end-to-end video generation, realizes the understanding of the complex semantics of ancient poetry and improves the generation quality of ancient poetry videos.

[0234] Figure 6 The block diagram of a device for generating ancient poems from videos based on a large language model according to an exemplary embodiment is shown. The device is used in a method for generating ancient poems from videos based on a large language model. Figure 6 The device includes a construction module 610, a training module 620, an acquisition module 630, a first generation module 640, a second generation module 650 and a third generation module 660. Among them:

[0235] A construction module 610 is used to construct an ancient poetry knowledge base, wherein the ancient poetry knowledge base includes text knowledge and image knowledge;

[0236] A training module 620 is used to obtain a trained story learning vector using a Soft Prompt training method according to a large language model;

[0237] The acquisition module 630 is used to acquire the ancient poetry data to be generated, find the text knowledge corresponding to the ancient poetry data to be generated in the ancient poetry knowledge base, and obtain the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model;

[0238] The first generation module 640 is used to obtain the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model;

[0239] The second generation module 650 is used to generate a story picture corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated;

[0240] The third generation module 660 is used to generate a video corresponding to the ancient poetry data to be generated based on the story pictures corresponding to the ancient poetry data to be generated and the video generation model.

[0241] Optionally, the construction module 610 is used to:

[0242] Obtain the original text, translation and annotations of ancient poems, the creation background of ancient poems and the appreciation of ancient poems from text websites related to ancient poems and determine them as text knowledge;

[0243] We obtain pictures related to ancient poetry from picture websites related to ancient poetry, including pictures of people, scenery, and ancient poetry. We use the CLIP model to filter out pictures that do not meet the standards and identify the filtered pictures as picture knowledge.

[0244] Optionally, the training module 620 is used to:

[0245] Acquire training samples, wherein the training samples include sample ancient poems and true story sequences corresponding to the sample ancient poems;

[0246] Freeze the model parameters in the initial large language model, input the sample ancient poetry into the frozen large language model, and obtain the predicted story sequence corresponding to the sample ancient poetry according to the story learning vector Prompt to be trained;

[0247] The predicted story sequence corresponding to the sample ancient poetry is compared with the true story sequence, and the Prompt to be trained is trained according to the cross entropy loss function. The formula of the cross entropy loss function is as follows (1):

[0248] (1)

[0249] Where T is the length of the true story sequence, It means that under the condition of sample ancient poetry X and softprompt P, the large language model predicts the i-th word in the true story sequence probability;

[0250] Iterate the training until the loss function reaches convergence, stop the training, and obtain the trained story learning vector Prompt.

[0251] Optionally, the acquisition module 630 is used to:

[0252] According to the trained story learning vector, the text knowledge corresponding to the ancient poetry data to be generated is segmented, and the segmentation result is input into the large language model to obtain the story corresponding to the ancient poetry data to be generated.

[0253] Optionally, the first generating module 640 is used to:

[0254] S41, using preset rules to decompose the story corresponding to the ancient poetry data to be generated to obtain a text description sequence;

[0255] S42, using a large language model to extract character information and location information in the story corresponding to the ancient poetry data to be generated;

[0256] S43. Generate four scripts according to the text description sequence, the extracted character information and location information, and the preset prompt words, wherein the first script of the four scripts describes the main tasks and main scenes of the story, the second script of the four scripts describes the video clips of the story, the third script of the four scripts describes the correspondence between the characters and scenes, and the fourth script of the four scripts describes the duration.

[0257] Optionally, the image generation model includes a stable diffusion model and a Multi-subjectconsistence Generator generation technology;

[0258] The second generating module 650 is used to:

[0259] S51, inputting text descriptions corresponding to characters and places in the first script into a stable diffusion model to generate pictures corresponding to the characters and places;

[0260] S52, generating pictures of a plurality of scene segments according to the generated pictures corresponding to the characters and places, the second script, the third script, the fourth script, and the Multi-subject consistence Generator generation technology;

[0261] S53. Use the large language model to optimize the generated images of multiple scene fragments to obtain optimized story images.

[0262] Optionally, the third generating module 660 is used to:

[0263] Input the story picture corresponding to the ancient poetry data to be generated and the corresponding text prompt into the Pika model to obtain the initial video;

[0264] Subtitles and sound are added to the initial video to obtain a video corresponding to the ancient poetry data to be generated.

[0265] In the embodiment of the present invention, an ancient poetry knowledge base is constructed, and the ancient poetry knowledge base includes text knowledge and picture knowledge; the Soft Prompt training method is adopted according to the large language model to obtain a trained story learning vector; the ancient poetry data to be generated is obtained, and the text knowledge corresponding to the ancient poetry data to be generated is found in the ancient poetry knowledge base, and the story corresponding to the ancient poetry data to be generated is obtained according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; the script corresponding to the ancient poetry data to be generated is obtained according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model; the story picture corresponding to the ancient poetry data to be generated is generated according to the script corresponding to the ancient poetry data to be generated, the picture knowledge and the picture generation model; the video corresponding to the ancient poetry data to be generated is generated according to the story picture corresponding to the ancient poetry data to be generated and the video generation model. The present invention does not require a large number of samples for training, solves the problem of the lack of ancient poetry video data sets and the inability to perform end-to-end video generation, realizes the understanding of the complex semantics of ancient poetry and improves the generation quality of ancient poetry videos.

[0266] Figure 7 is a structural diagram of a device for generating ancient poetry to video based on a large language model provided by an embodiment of the present invention, such as Figure 7 As shown, the device for generating ancient poetry to video based on a large language model may include the above Figure 6 The device for generating ancient poems and lyrics into videos based on a large language model is shown. Optionally, the device for generating ancient poems and lyrics into videos based on a large language model 710 may include a first processor 2001 .

[0267] Optionally, the device 710 for generating ancient poetry to video based on a large language model may further include a memory 2002 and a transceiver 2003 .

[0268] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0269] Combine the following Figure 7 The components of the device 710 for generating ancient poetry to video based on a large language model are specifically introduced as follows:

[0270] The first processor 2001 is the control center of the device 710 for generating ancient poems to videos based on a large language model, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0271] Optionally, the first processor 2001 can perform various functions of the device 710 for generating ancient poems to videos based on a large language model by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0272] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0273] In a specific implementation, as an embodiment, the ancient poetry to video generation device 710 based on a large language model may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0274] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0275] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0276] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0277] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0278] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be connected to the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0279] It should be noted that Figure 7 The structure of the device 710 for generating ancient poetry to video based on a large language model shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0280] In addition, the technical effects of the device 710 for generating ancient poems to videos based on a large language model can refer to the technical effects of the method for generating ancient poems to videos based on a large language model described in the above method embodiment, and will not be repeated here.

[0281] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0282] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0283] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0284] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0285] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0286] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0287] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0288] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0289] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0290] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0291] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0292] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0293] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating ancient poetry to video based on a large language model, characterized in that: The method comprises: S1. Constructing an ancient poetry knowledge base, wherein the ancient poetry knowledge base includes text knowledge and image knowledge; S2. Use the Soft Prompt training method based on the large language model to obtain the trained story learning vector; S3, obtaining the ancient poetry data to be generated, searching the ancient poetry knowledge base for the text knowledge corresponding to the ancient poetry data to be generated, and obtaining the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; S4, obtaining the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model; S5, generating a story picture corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated; S6, generating a video corresponding to the ancient poetry data to be generated according to the story picture corresponding to the ancient poetry data to be generated and the video generation model; Among them, S2 adopts the Soft Prompt training method according to the large language model to obtain the trained story learning vector, including: Acquire training samples, wherein the training samples include sample ancient poems and true story sequences corresponding to the sample ancient poems; Freeze the model parameters in the initial large language model, input the sample ancient poetry into the frozen large language model, and obtain the predicted story sequence corresponding to the sample ancient poetry according to the story learning vector Prompt to be trained; The predicted story sequence corresponding to the sample ancient poetry is compared with the true story sequence, and the Prompt to be trained is trained according to the cross entropy loss function. The formula of the cross entropy loss function is as follows (1): (1) Where T is the length of the true story sequence, It means that under the conditions of sample ancient poetry X and soft prompt P, the large language model predicts the i-th word in the true value story sequence probability; Iterate the training until the loss function reaches convergence, then stop the training and get the trained story learning vector Prompt; The step S4 obtains the script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, the preset prompt words and the large language model, including: S41, using preset rules to decompose the story corresponding to the ancient poetry data to be generated to obtain a text description sequence; S42, using a large language model to extract character information and location information in the story corresponding to the ancient poetry data to be generated; S43. Generate four scripts according to the text description sequence, the extracted character information and location information, and the preset prompt words, wherein the first script of the four scripts describes the main tasks and main scenes of the story, the second script of the four scripts describes the video clips of the story, the third script of the four scripts describes the correspondence between the characters and scenes, and the fourth script of the four scripts describes the duration.

2. The method for generating ancient poetry to video based on a large language model according to claim 1, characterized in that: The S1 constructs an ancient poetry knowledge base, which includes text knowledge and picture knowledge, including: Obtain the original text, translation and annotations of ancient poems, the creation background of ancient poems and the appreciation of ancient poems from text websites related to ancient poems and determine them as text knowledge; We obtain pictures related to ancient poetry from picture websites related to ancient poetry, including pictures of people, scenery, and ancient poetry. We use the CLIP model to filter out pictures that do not meet the standards and identify the filtered pictures as picture knowledge.

3. The method for generating ancient poetry to video based on a large language model according to claim 1, characterized in that: The step S3 obtains the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model, including: According to the trained story learning vector, the text knowledge corresponding to the ancient poetry data to be generated is segmented, and the segmentation result is input into the large language model to obtain the story corresponding to the ancient poetry data to be generated.

4. The method for generating ancient poetry to video based on a large language model according to claim 1, characterized in that: The image generation model includes a stable diffusion model and a Multi-subject consistenceGenerator generation technology; The step S5 generates story pictures corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated, including: S51, inputting text descriptions corresponding to characters and places in the first script into a stable diffusion model to generate pictures corresponding to the characters and places; S52, generating pictures of a plurality of scene segments according to the generated pictures corresponding to the characters and places, the second script, the third script, the fourth script, and the Multi-subject consistence Generator generation technology; S53. Use the large language model to optimize the generated images of multiple scene fragments to obtain optimized story images.

5. The method for generating ancient poetry to video based on a large language model according to claim 3 is characterized in that: The step S6 generates a video corresponding to the ancient poetry data to be generated according to the story picture corresponding to the ancient poetry data to be generated and the video generation model, including: Input the story picture corresponding to the ancient poetry data to be generated and the corresponding text prompt into the Pika model to obtain the initial video; Subtitles and sound are added to the initial video to obtain a video corresponding to the ancient poetry data to be generated.

6. A device for generating ancient poems from videos based on a large language model, the device for generating ancient poems from videos based on a large language model is used to implement the method for generating ancient poems from videos based on a large language model as claimed in any one of claims 1 to 5, characterized in that: The device comprises: A construction module is used to construct an ancient poetry knowledge base, wherein the ancient poetry knowledge base includes text knowledge and image knowledge; The training module is used to obtain the trained story learning vector using the Soft Prompt training method based on the large language model; An acquisition module, used for acquiring the ancient poetry data to be generated, searching the ancient poetry knowledge base for the text knowledge corresponding to the ancient poetry data to be generated, and obtaining the story corresponding to the ancient poetry data to be generated according to the ancient poetry data to be generated, the corresponding text knowledge, the trained story learning vector and the large language model; A first generation module, for obtaining a script corresponding to the ancient poetry data to be generated according to the story corresponding to the ancient poetry data to be generated, a preset prompt word and the large language model; The second generation module is used to generate a story picture corresponding to the ancient poetry data to be generated according to the script, picture knowledge and picture generation model corresponding to the ancient poetry data to be generated; The third generation module is used to generate a video corresponding to the ancient poetry data to be generated based on the story pictures corresponding to the ancient poetry data to be generated and the video generation model.

7. A device for generating ancient poetry to video based on a large language model, characterized in that: The device for generating ancient poetry to video based on a large language model includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 5.

Images