Video generation method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a video generation method and device, computer equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: extracting respective keywords of at least two text segments; the at least two text segments are different text segments in the target text; querying an image database through respective keywords of the at least two text segments to obtain reference images corresponding to the at least two text segments; generating text description images respectively corresponding to the at least two text segments based on the at least two text segments and the reference images respectively corresponding to the at least two text segments; the image content of the text description image is matched with the text content of the corresponding text segment; and generating a video of the target text based on the text description images corresponding to the at least two text segments. According to the scheme, the quality of the video generated based on the text can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of AI (Artificial Intelligence) technology, and in particular to a video generation method, apparatus, computer equipment, storage medium, and program product. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of machine learning models is becoming more and more extensive. In the field of multimedia processing technology, combining machine learning models can realize video generation, such as the generation of audio novel videos.

[0003] In related technologies, the process of generating an audio novel video in combination with a machine learning model can be as follows: the designer can pre-set the character information in the novel, such as the character's name and the character's clothing, and then input the preset character information and the novel text into the machine learning model. The machine learning model generates several pictures based on the character information and different fragments in the novel text. The designer manually selects high-quality pictures from several images, inputs the selected pictures into the machine learning model, and generates the corresponding audio novel video.

[0004] However, in the solution shown in the above-mentioned related art, the matching degree between the image generated by the machine learning model and the fragments in the novel text is not high enough, which easily leads to low quality of video generation. Summary of the invention

[0005] The embodiments of the present application provide a video generation method, apparatus, computer equipment, storage medium and program product, which can improve the quality of video generation. The technical solution is as follows:

[0006] In one aspect, a video generation method is provided, the method comprising:

[0007] Extracting keywords from at least two text segments; the at least two text segments are different text segments in the target text;

[0008] Querying an image database using the keywords of the at least two text segments to obtain reference images corresponding to the at least two text segments respectively;

[0009] Based on the at least two text segments and the reference images respectively corresponding to the at least two text segments, generating text description images respectively corresponding to the at least two text segments; the image content of the text description image matches the text content of the corresponding text segment;

[0010] A video of the target text is generated based on the text description images respectively corresponding to the at least two text segments.

[0011] In another aspect, a video generation device is provided, the device comprising:

[0012] A keyword extraction module, used to extract keywords from at least two text segments; the at least two text segments are different text segments in the target text;

[0013] A reference image acquisition module, configured to query an image database using keywords of the at least two text segments to obtain reference images corresponding to the at least two text segments;

[0014] A text description image generation module, configured to generate text description images corresponding to the at least two text segments respectively based on the at least two text segments and the reference images corresponding to the at least two text segments respectively; the image content of the text description image matches the text content of the corresponding text segment;

[0015] The video generation module is used to generate a video of the target text based on the text description images respectively corresponding to the at least two text segments.

[0016] In some embodiments, the text description image generation module is used to obtain a description sentence of a first text segment; the description sentence is a sentence used to describe the main content of the text segment; the first text segment is any one of the at least two text segments; the description sentence of the first text segment and the reference image corresponding to the first text segment are input into a generative language model to obtain a text description image corresponding to the first text segment output by the generative language model.

[0017] In some embodiments, the text description image generation module is used to perform a summary extraction operation on the first text segment to obtain a description sentence of the first text segment; or to concatenate multiple keywords of the first text segment to generate a description sentence of the first text segment.

[0018] In some embodiments, the reference image includes a character reference image and a background reference image; the text description image generation module is used to input the description sentence of the first text segment, the character reference image corresponding to the first text segment, and the background reference image corresponding to the first text segment into the generative language model to obtain the character description image corresponding to the first text segment and the background description image corresponding to the first text segment output by the generative language model; the character description image corresponding to the first text segment and the background description image corresponding to the first text segment are merged to obtain the text description image corresponding to the first text segment.

[0019] In some embodiments, the reference image acquisition module is used to query the image database through character description keywords among the keywords corresponding to the first text segment to obtain the character reference image corresponding to the first text segment; the character description keywords are used to describe the characters in the text segment; and query the image database through scene description keywords among the keywords corresponding to the first text segment to obtain the background reference image corresponding to the first text segment, and the scene description keywords are used to describe the scene in the text segment.

[0020] In some embodiments, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the character reference image includes a face reference image; when the character description keywords include the character's clothing description words, the character reference image includes a clothing reference image; when the character description keywords include the character's posture description words, the character reference image includes a character posture reference image.

[0021] In some embodiments, the reference image acquisition module is used to query the image database through scene description keywords in the keywords corresponding to the first text segment, to obtain the background template image stored in the image database corresponding to the first text segment; to obtain image keywords of the background template image; and to input the image keywords into the generative language model to obtain the background reference image corresponding to the first text segment output by the generative language model.

[0022] In some embodiments, the video generation module is used to obtain particle tags of text description images corresponding to the at least two text segments respectively; the particle tags are used to indicate particle effects matching the text description images; query a particle effect database through the particle tags of the text description images corresponding to the at least two text segments respectively to obtain the particle effects corresponding to the at least two text segments respectively; generate a video of the target text according to the text description images corresponding to the at least two text segments respectively and the particle effects corresponding to the at least two text segments respectively.

[0023] In some embodiments, the video generation module is used to obtain the text speech corresponding to the at least two text segments respectively; the text speech is the speech of reading the text segment; for each of the at least two text segments, a video clip with a playback duration of the text speech corresponding to the text segment and content of which is a text description image corresponding to the text segment superimposed with a particle effect corresponding to the text segment; the video clips corresponding to the at least two text segments are spliced ​​in the order in which the at least two text segments appear in the target text to obtain a speech-free video of the target text; the text speech corresponding to the at least two text segments are spliced ​​in the order in which the at least two text segments appear in the target text, and then combined with the speech-free video to obtain a video of the target text.

[0024] In some embodiments, the video generation module is used to obtain an image tag of a text description image corresponding to a second text segment, and query the particle tag corresponding to the image tag; the second text segment is any one of the at least two text segments; or, query the particle tag corresponding to a keyword of the second text segment.

[0025] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the video generation method as described above.

[0026] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the video generation method as described above.

[0027] In another aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the video generation method provided in the above-mentioned various optional implementations.

[0028] The technical solution provided by this application may have the following beneficial effects:

[0029] The computer device can obtain key information of the text segment in the target text through keywords, and obtain the reference image corresponding to the text segment based on the keywords, thereby ensuring that the queried reference image is highly relevant to the content of the text segment. After obtaining the reference image that has a certain degree of matching with the content of the text segment, the computer device combines the text segment corresponding to the reference image to generate a text description image corresponding to the text segment, so that the obtained text description image has a higher degree of matching with the content of the text segment. After two image acquisition operations, the computer device generates a video corresponding to the target text based on the text description image that has a higher degree of matching with the content of the text segment, which can effectively ensure a high degree of matching between each text segment of the target text and its corresponding text description image, thereby making the generated video more accurate and effectively ensuring the quality of video generation.

[0030] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0032] Figure 1 is a system configuration diagram of a video generation method involved in one embodiment of the present application;

[0033] Figure 2 is a flow chart of a video generation method provided by an embodiment of the present application;

[0034] Figure 3 is a flow chart of a video generation method provided by an embodiment of the present application;

[0035] Figure 4 is a flow chart of a video generation method provided by an embodiment of the present application;

[0036] Figure 5 is a flow chart of a video generation method provided by an embodiment of the present application;

[0037] Figure 6 It is a framework flow chart of a video generation method provided by an embodiment of the present application;

[0038] Figure 7 is a block diagram of a video generating device provided by an exemplary embodiment of the present application;

[0039] Figure 8 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0041] Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as recited in the appended claims.

[0042] The present application proposes a video solution, which can effectively ensure that each text segment of the target text has a high degree of matching with its corresponding text description image, so that the content accuracy of the video generated based on the text description image is higher, and the quality of the video generation is effectively guaranteed. For ease of understanding, some concepts involved in the present application are explained below.

[0043] 1) LLM (Large Language Model) refers to a type of natural language processing model that is trained through deep learning technology and has a large number of parameters (usually hundreds of millions to hundreds of billions). Large language models are trained on large text data sets and can learn the complex structure and patterns of language, and have strong language understanding and generation capabilities. Large language models can be applied to a variety of natural language processing tasks, such as text generation, machine translation, question-answering systems, sentiment analysis, etc.

[0044] 2) Particle effects are a widely used technology in computer graphics to simulate the effects of various dynamic phenomena in nature. These phenomena include flames, smoke, explosions, water flow, snow, dust, halos, etc. Particle effects achieve these effects by generating and controlling a large number of small elements (called "particles"), each of which has its own properties, such as position, speed, color, transparency, life cycle, etc.

[0045] 3) A generative language model is a model that can generate new text similar to human writing based on existing text data. The core task of a generative language model is to sample new text sequences from a probability distribution so that the generated text is as close to the real language data as possible in terms of syntax and semantics.

[0046] 4) The particle sequence frame is a set of pre-prepared image frames, each of which represents the state of the particle at a certain moment. These frames are arranged in chronological order, and the dynamic changes of the particles are simulated by playing these frames continuously, thus achieving the particle animation effect.

[0047] Figure 1 1 is a system configuration diagram of a video generation method according to an embodiment of the present application. Figure 1As shown, the system includes a requesting device 120 for video generation and a serving device 130 .

[0048] The requester device 120 may be a personal computer device such as a desktop computer, a laptop computer, a personal workstation, or the like.

[0049] The service device 130 may be a server, or a plurality of servers, or a virtualization platform, or a cloud computing service center.

[0050] The requesting device 120 and the serving device 130 may be connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0051] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology.

[0052] The user can upload a target text containing at least two text segments through the video generation requesting device 120 to request the generation of a video of the target text. After receiving the target text, the service device 130 extracts the keywords of the two different text segments in the target text, queries the image database according to the keywords of the at least two text segments, obtains reference images corresponding to the at least two text segments from the image database, combines the at least two received text segments and the reference images corresponding to the at least two obtained text segments to generate at least two text description images whose image contents match the text contents of the corresponding text segments, and generates a video of the target text based on the text description images corresponding to the at least two text segments.

[0053] It should be noted that, in the embodiments of the present application, the above solution is implemented by a single-sided device as an example. Optionally, the above solution can also be implemented by the interaction of two-sided devices.

[0054] Figure 2 : is a flowchart of a video generation method provided by an embodiment of the present application. The video generation method can be executed by a computer device, for example, the computer device can be the above-mentioned Figure 1 The service device 130 shown, or the computer device mentioned above may also be other computer devices, such as a server, or a personal computer, etc. The video generation method mentioned above may include steps 210, 220, 230 and 240, which are specifically implemented as follows:

[0055] Step 210: extract keywords from at least two text segments; the at least two text segments are different text segments in the target text.

[0056] The above-mentioned text segment is a semantic unit with independent meaning extracted from the target text, carrying specific information or description, and can independently express a certain theme or concept. The text segment can be a sentence, a paragraph or a group of related sentences in the target text.

[0057] In the embodiment of the present application, the at least two text segments are different text segments in the target text, that is, the at least two text segments carry different information or descriptions, and the text segment contents are different.

[0058] The above keywords are words or phrases extracted from a text segment that can reflect the core content or theme of the text segment. Keywords can quickly and accurately summarize the main information of the text segment.

[0059] Optionally, the target text is the original text content that needs to be converted into a video; the target text can be any form of written material, such as novels, scripts, news reports, advertising copy, popular science articles, etc.

[0060] In an embodiment of the present application, the target text may include at least two text segments, and each of the text segments may include at least one keyword.

[0061] In an embodiment of the present application, a computer device can extract keywords from at least two text segments based on statistical word frequency. The computer device uses the number of times a certain word appears in a text segment as a standard and sets a number threshold. The computer device counts the number of times each word appears in the corresponding text segment. When the number of appearances meets the number threshold, the word is extracted as the keyword corresponding to the text segment.

[0062] Optionally, the computer device can extract keywords from at least two text segments based on text segmentation. For example, the computer device divides at least two text segments into several sentences through natural language processing technology, and extracts keywords from each sentence through a keyword extraction algorithm as keywords for the corresponding text segment.

[0063] Optionally, the computer device may extract keywords from at least two text segments through a generative language model. For example, the computer device may input at least two text segments into the generative language model respectively, and the generative language model obtains at least one keyword corresponding to the input text segment based on the text segment.

[0064] Step 220: querying an image database using the keywords of at least two text segments to obtain reference images corresponding to the at least two text segments.

[0065] The above-mentioned image database stores keywords corresponding to at least two text segments, and a database of at least one reference image corresponding to the keywords. In an embodiment of the present application, after extracting keywords from at least two text segments, the computer device traverses the image database to obtain a number of reference images corresponding to the keywords.

[0066] In some embodiments, the above-mentioned image database stores a database of image-text pairs consisting of keywords corresponding to at least two text segments and reference images corresponding to the keywords. In an embodiment of the present application, after extracting the keywords of at least two text segments, the computer device traverses the image database, searches for image-text pairs containing keywords corresponding to at least two text segments, and obtains reference images in the image-text pairs as reference images of keywords corresponding to at least two text segments.

[0067] The reference image is an image resource whose image content matches the content described by the keyword.

[0068] Step 230: Based on at least two text segments and reference images corresponding to the at least two text segments, text description images corresponding to the at least two text segments are generated; the image content of the text description image matches the text content of the corresponding text segment.

[0069] Among them, the above-mentioned text description image is an image generated by combining the content of the text segment and the reference image corresponding to the text segment. The text description image can accurately reflect the scene or content described by the text segment, and the image content of the text description image matches the content of the text segment.

[0070] In the embodiment of the present application, the at least two text segments respectively have at least two corresponding reference images, and the two corresponding reference images are different.

[0071] In an embodiment of the present application, a computer device can generate text description images corresponding to at least two text segments respectively through a machine learning model based on at least two text segments and reference images corresponding to at least two text segments respectively. The machine learning model is a model that learns through training data and generates images that match text content. The machine learning model is a machine learning model that has the ability to generate text description images whose image content highly matches the content of the text segment based on the input text segment and the reference image.

[0072] Step 240: Generate a video of the target text based on the text description images corresponding to at least two text segments.

[0073] The target text video is generated by a computer device, and the video content is highly matched with the target text content.

[0074] In some embodiments, the computer device may arrange the at least two generated text description images in the order in which the corresponding text segments appear in the target text. Optionally, a transition effect may be added to the at least two arranged text description images to generate a coherent video as the video of the target text.

[0075] In some embodiments, a computer device generates a video of a target text through a deep learning model based on text description images corresponding to at least two text segments. The computer device can input at least two text segments and the text description images corresponding to the at least two text segments into the deep learning model. The deep learning model generates coherent video frames based on the input at least two text segments and the text description images corresponding to the at least two text segments, and synthesizes the video frames to be generated as the video of the target text, wherein the deep learning model is a machine learning model that has the ability to generate a video that highly matches the target text content based on the input text segments and the text description images corresponding to the text segments.

[0076] In an embodiment of the present application, a computer device can obtain key information of a text segment in a target text through keywords, and obtain a reference image corresponding to the text segment based on the keywords, thereby ensuring that the queried reference image is highly relevant to the content of the text segment. After obtaining a reference image that matches the content of the text segment to a certain extent, the computer device generates a text description image corresponding to the text segment in combination with the text segment corresponding to the reference image, so that the obtained text description image matches the content of the text segment to a higher degree. After two image acquisition operations, the computer device generates a video corresponding to the target text based on the text description image that matches the content of the text segment to a higher degree, thereby effectively ensuring a high degree of match between each text segment of the target text and its corresponding text description image, thereby making the generated video more accurate and effectively ensuring the quality of video generation.

[0077] based on Figure 2 , please refer to Figure 3 , Figure 3 This is a flowchart of a video generation method provided by an embodiment of the present application. Figure 2 Step 230 in the embodiment can be implemented as step 230a and step 230b, as follows:

[0078] Step 230a: Obtain a description sentence of the first text segment; the description sentence is a sentence used to describe the main content of the text segment; the first text segment is any one of the at least two text segments.

[0079] The above-mentioned description sentence is an introduction sentence used to describe the main content of the first text segment, and the description sentence is a refinement and summary of the core content of the first text segment.

[0080] In the embodiment of the present application, after acquiring the first text segment, the computer device can automatically extract the description sentence from the first text segment through natural language processing technology.

[0081] Optionally, the computer device may extract summary information from the first text segment by using a text summary generation technology as a description sentence of the first text segment.

[0082] Optionally, the computer device may obtain a description sentence of the first text segment based on a keyword extraction method. For example, the computer device extracts at least one keyword from the first text segment, and combines the at least one keyword into a description sentence, wherein the description sentence includes at least one key.

[0083] Step 230b: Input the description sentence of the first text segment and the reference image corresponding to the first text segment into the generative language model to obtain the text description image corresponding to the first text segment output by the generative language model.

[0084] Among them, the above-mentioned generative language model is a machine learning model with the ability to generate text description images, which is obtained by fine-tuning and training through description sentence samples, reference image samples and annotated text description images.

[0085] In an embodiment of the present application, after obtaining the description sentence of the first text segment and the reference image corresponding to the first text segment, the computer device inputs the description sentence of the first text segment and the reference image corresponding to the first text segment into a generative language model, and the generative language model generates one or at least two text description images corresponding to the first text segment.

[0086] In an embodiment of the present application, the above-mentioned description sentence can summarize the main information of the first text segment, and the computer device extracts the main information from the first text segment. By using the description sentence and the reference image used to indicate the main information as the input of the generative language model, the generative language model can comprehensively consider the text content and image features of the first text segment, generate a more accurate and high-quality text description image, ensure that the generated text description image is consistent with the content of the first text segment in terms of details and overall effect, effectively improve the accuracy of text description image generation, and improve the quality of text description images.

[0087] Based on the schemes shown in any one or more of the above-mentioned embodiments, in some embodiments, the above-mentioned step 230a can be implemented as: performing a summary extraction operation on the first text segment to obtain a description sentence of the first text segment; or, concatenating multiple keywords of the first text segment to generate a description sentence of the first text segment.

[0088] The above-mentioned summary extraction operation is an operation in which a computer device extracts key information from a first text segment and generates a description sentence based on the key information. In an embodiment of the present application, the computer device may input the first text segment into a summary extraction model to obtain a summary of the first text segment as a description sentence of the first text segment, wherein the above-mentioned summary extraction model is a machine learning model obtained by training a sample of the first text segment with summary annotations and having the ability to extract key information from the first text segment and generate a description sentence based on the key information.

[0089] In an embodiment of the present application, the computer device can splice at least two keywords for each sentence in the first text segment to obtain a description statement for each sentence in the first text segment, and splice the description statement of each sentence as the description statement of the first text segment.

[0090] In an embodiment of the present application, the summary extraction operation can extract the core content from the first text segment, so that the description sentence can accurately reflect the core information of the first text segment, remove redundant and unnecessary details, and be more refined and accurate; by splicing multiple keywords of the first text segment to generate a description sentence, it not only ensures that the generated description sentence accurately reflects the core content of the text segment, but also enables the content indicated by the description sentence to cover the content indicated by the first text segment to a large extent, expands the generation method of the description sentence, and effectively ensures the accuracy of the generation of the description sentence.

[0091] Based on the scheme shown in any one or more of the above embodiments, in some embodiments, the reference image includes a character reference image and a background reference image. The above step 230b can be implemented as follows: inputting the description sentence of the first text segment, the character reference image corresponding to the first text segment, and the background reference image corresponding to the first text segment into the generative language model to obtain the character description image corresponding to the first text segment and the background description image corresponding to the first text segment output by the generative language model; fusing the character description image corresponding to the first text segment and the background description image corresponding to the first text segment to obtain the text description image corresponding to the first text segment.

[0092] In some embodiments, a computer device generates a single prompt sentence including a description sentence of a first text segment, a character reference image corresponding to the first text segment, and a background reference image corresponding to the first text segment, and inputs the single prompt sentence into a generative language model to obtain a character description image and a background description image output by the generative language model.

[0093] In an embodiment of the present application, after obtaining the description sentence of the first text segment, the character reference image corresponding to the first text segment, and the background reference image corresponding to the first text segment, the computer device can use Python or other programming languages ​​to combine the description sentence, the storage path of the character reference image, and the storage path of the background reference image into a structured prompt sentence as a single prompt sentence, and input the single prompt sentence into the generative language model. The generative language model generates a character description image and a background description image corresponding to the content of the description sentence based on the content contained in the single prompt sentence.

[0094] In some embodiments, a computer device generates a first prompt sentence including a description sentence of a first text segment and a character reference image corresponding to the first text segment, inputs the first prompt sentence into a generative language model, and obtains a character description image output by the generative language model; generates a second prompt sentence including a description sentence of the first text segment and a background reference image corresponding to the first text segment, inputs the second prompt sentence into the generative language model, and obtains a background description image output by the generative language model.

[0095] In an embodiment of the present application, after obtaining a description statement of a first text segment and a character reference image corresponding to the text segment, the computer device can use Python or other programming languages ​​to combine the description statement of the first text segment and the storage path of the character reference image of the first text segment into a structured prompt statement as a first prompt statement, and input the first prompt statement into a generative language model, and the generative language model generates a character description image corresponding to the content of the description statement of the first text segment according to the content contained in the first prompt statement; after obtaining the description statement of the first text segment and a background reference image corresponding to the text segment, the computer device can use Python or other programming languages ​​to combine the description statement of the first text segment and the storage path of the background reference image of the first text segment into a structured prompt statement as a second prompt statement, and input the second prompt statement into the generative language model, and the generative language model generates a background description image corresponding to the content of the description statement of the first text segment according to the content contained in the second prompt statement.

[0096] In an embodiment of the present application, after obtaining the character description image corresponding to the first text segment and the background description image corresponding to the first text segment, the computer device can segment the character part from the character description image, superimpose the segmented character part on the background description image, and obtain the text description image corresponding to the first text segment.

[0097] Among them, the computer device can segment the character part from the character description image through a segmentation model, and the segmentation model is an image with the ability to extract the character part in the image from the image; for example, the above-mentioned segmentation model can be a U-Net model, and the computer device inputs the character description image into the U-Net model, and the U-Net model performs semantic segmentation on the character description image, identifies and extracts the character image corresponding to the character area in the character description image, and separates the character part and the background part in the character description image.

[0098] In some embodiments, the PNG image corresponding to the character part segmented out by the computer device is transparent, and the segmented character part can be superimposed on the background description image based on the transparency. If the character image shows the whole body of the character, the character image is pasted in the center of the background image to ensure that the character is displayed in the center of the background description image; if the character image shows half of the character, the character image is pasted below the background image to align the lower half of the character with the bottom of the background.

[0099] In an embodiment of the present application, the reference image includes a character reference image and a background reference image. The computer device provides a specific visual reference for the generative language model by inputting the character reference image and the background reference image into the generative language model, so that the generative language model can generate more accurate and high-quality images. The generated character description image can reflect the character features and actions mentioned in the description sentence, and the content of the background description image can show the background environment mentioned in the description sentence. By synthesizing the character description image and the background description image, a complete text description image is generated. The text description image can not only effectively retain the features and actions of the character in the first text segment, but also ensure the fit between the background environment and the text segment, which is more natural and realistic, effectively improving the fit between the text description image and the first text segment, and effectively improving the accuracy of text description image generation.

[0100] Based on the schemes shown in any one or more of the above-mentioned embodiments, in some embodiments, a computer device queries an image database through character description keywords among the keywords corresponding to a first text segment to obtain a character reference image corresponding to the first text segment; the character description keywords are used to describe the characters in the text segment; and a computer device queries an image database through scene description keywords among the keywords corresponding to the first text segment to obtain a background reference image corresponding to the first text segment, and the scene description keywords are used to describe the scene in the text segment.

[0101] In an embodiment of the present application, the keywords corresponding to the first text segment include character description keywords and scene description keywords. The character description keywords are words that can reflect the characteristics of the characters in the first text segment, and the scene description keywords are words that describe the environment and location in the first text segment.

[0102] In an embodiment of the present application, the above-mentioned image database stores character description keywords corresponding to the first text segment, and at least one character reference image corresponding to the character description keywords. In an embodiment of the present application, after extracting the character description keywords corresponding to the first text segment, the computer device traverses the image database and obtains the character reference image corresponding to the character description keyword in the image database as the character reference image corresponding to the first text segment.

[0103] In some embodiments, the above-mentioned image database stores various character description keywords and a picture-text pair or association relationship consisting of at least one character reference image corresponding to the character description keyword. In an embodiment of the present application, after extracting the character description keyword corresponding to the first text segment, the computer device traverses the image database, queries the picture-text pair or association relationship containing the character description keyword corresponding to the first text segment, and obtains the character reference image in the picture-text pair or association relationship as the character reference image corresponding to the first text segment.

[0104] The above-mentioned character reference image is an image in which the character indicated by the image matches the character described by the character description keyword.

[0105] In an embodiment of the present application, the above-mentioned image database stores scene description keywords corresponding to the first text segment and at least one background reference image corresponding to the scene description keywords. In an embodiment of the present application, after extracting the scene description keywords corresponding to the first text segment, the computer device traverses the image database to obtain the background reference image corresponding to the scene description keyword in the image database as the background reference image corresponding to the first text segment.

[0106] In some embodiments, the above-mentioned image database stores various scene description keywords and at least one background reference image corresponding to the scene description keyword, which constitutes an image-text pair or association relationship. In an embodiment of the present application, after extracting the scene description keyword corresponding to the first text segment, the computer device traverses the image database, searches for the image-text pair or association relationship containing the scene description keyword corresponding to the first text segment, and obtains the background reference image in the searched image-text pair or association relationship as the background reference image corresponding to the first text segment.

[0107] The background reference image is an image in which the scene indicated by the image matches the scene described by the scene description keyword.

[0108] In an embodiment of the present application, character description keywords are used to describe the features and appearance of the characters in the first text segment, ensuring that the obtained character reference image highly matches the character features in the first text segment; scene description keywords are used for the scenes and environments in the first text segment, ensuring that the obtained background reference image highly matches the scenes in the first text segment, effectively ensuring the degree of matching between the character reference image and the background reference image and the first text segment, and improving the accuracy of obtaining the reference image.

[0109] Based on the schemes shown in any one or more of the above-mentioned embodiments, in some embodiments, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the character reference image includes a face reference image; when the character description keywords include the character's clothing description words, the character reference image includes a clothing reference image; when the character description keywords include the character's posture description words, the character reference image includes a character posture reference image.

[0110] In an embodiment of the present application, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the computer device can query the image database through the above-mentioned character description keywords to obtain the facial reference image corresponding to the first text segment.

[0111] In some embodiments, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the character description keywords corresponding to the first text segment and at least one face reference image corresponding to the character description keywords are stored in the image database. In the embodiment of the present application, after extracting the character description keywords corresponding to the first text segment, the computer device traverses the image database to obtain the face reference images corresponding to the character description keywords in the image database as the face reference images corresponding to the first text segment.

[0112] In some embodiments, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the above-mentioned image database stores the character description keywords corresponding to the first text segment and the image-text pair or association relationship formed by at least one human face reference image corresponding to the character description keywords. In the embodiment of the present application, after extracting the character description keywords corresponding to the first text segment, the computer device traverses the image database, queries the image-text pair or association relationship containing the character description keywords corresponding to the first text segment, and obtains the human face reference image in the queried image-text pair or association relationship as the human face reference image corresponding to the first text segment.

[0113] In an embodiment of the present application, when the character description keywords include character clothing description words, the computer device may query the image database through the character clothing description words to obtain a clothing reference image of the character corresponding to the first text segment.

[0114] In some embodiments, when the character description keywords include character clothing description words, the character clothing description words corresponding to the first text segment and at least one clothing reference image corresponding to the character clothing description words are stored in the image database. In the embodiment of the present application, after extracting the character clothing description words corresponding to the first text segment, the computer device traverses the image database to obtain the clothing reference images corresponding to the character clothing description words in the image database as the clothing reference images corresponding to the first text segment.

[0115] In some embodiments, when the character description keywords include character clothing description words, the above-mentioned image database stores the character clothing description words corresponding to the first text segment and the image-text pair or association relationship formed by at least one clothing reference image corresponding to the character clothing description words. In an embodiment of the present application, after extracting the character clothing description words corresponding to the first text segment, the computer device traverses the image database, queries the image-text pair or association relationship containing the character clothing description words corresponding to the first text segment, and obtains the clothing reference image in the queried image-text pair or association relationship as the clothing reference image corresponding to the first text segment.

[0116] In an embodiment of the present application, when the character description keywords include character posture description words, the computer device may query the image database through the character posture description words to obtain the character posture reference image corresponding to the first text segment.

[0117] In some embodiments, when the character description keywords include character posture description words, the image database stores the character posture description words corresponding to the first text segment and at least one character posture reference image corresponding to the character posture description words. In the embodiment of the present application, after extracting the character posture description words corresponding to the first text segment, the computer device traverses the image database to obtain the character posture reference image corresponding to the character posture description words in the image database as the character posture reference image corresponding to the first text segment.

[0118] In some embodiments, when the character description keywords include character posture description words, the above-mentioned image database stores the character posture description words corresponding to the first text segment and the image-text pair or association relationship formed by at least one character posture reference image corresponding to the character posture description words. In an embodiment of the present application, after extracting the character posture description words corresponding to the first text segment, the computer device traverses the image database, queries the image-text pair or association relationship containing the character posture description words corresponding to the first text segment, and obtains the character posture reference image in the queried image-text pair or association relationship as the character posture reference image corresponding to the first text segment.

[0119] In the embodiments of the present application, through specific character description keywords, it is possible to ensure that the face reference image obtained by the computer device is highly matched with the character features in the text segment; the character clothing description words are able to ensure that the clothing reference image obtained by the computer device is highly matched with the character clothing in the text segment; the character posture description words are able to ensure that the clothing reference image obtained by the computer device is highly matched with the character clothing in the text segment; by refining the types of character description keywords, the character reference images are divided into face reference images, clothing reference images and character posture reference images, which can ensure that the reference image obtained by the computer device is highly matched with the character indicated by the text segment, thereby improving the accuracy and quality of reference image generation.

[0120] Based on the scheme shown in any one or more of the above embodiments, in some embodiments, a computer device queries an image database through scene description keywords in the keywords corresponding to the first text segment, obtains a background template image stored in the image database corresponding to the first text segment; obtains image keywords of the background template image; inputs the image keywords into a generative language model, and obtains a background reference image corresponding to the first text segment output by the generative language model.

[0121] The above scene description keywords may correspond to one or more background template images, and each background template image may have one or more corresponding image keywords.

[0122] In the embodiment of the present application, the above-mentioned image database stores the scene description keyword corresponding to the first text segment, at least one background template image corresponding to the scene description keyword, and the image keyword corresponding to the background template image. In the embodiment of the present application, after extracting the scene description keyword corresponding to the first text segment, the computer device traverses the image database to obtain at least one image keyword corresponding to the background template image corresponding to the scene description keyword in the image database.

[0123] In an embodiment of the present application, after obtaining at least one image keyword corresponding to the background template image, the computer device can input the at least one image keyword into the generative language model, and the computer device generates at least one background reference image through the generative language model as the background reference image corresponding to the first text segment.

[0124] In an embodiment of the present application, the computer device can further refine and optimize the generation of the background reference image by obtaining image keywords corresponding to the background template image, so that the generated background reference image not only matches the first text segment in the overall scene, effectively retains the key information of the first text segment, but is also richer and more accurate in details, effectively ensuring the quality of the background reference image, while ensuring the difference between the reference background images corresponding to different text segments.

[0125] Based on the solutions shown in any one or more of the above embodiments, in some embodiments, based on Figure 2 , please refer to Figure 4 , Figure 4 This is a flowchart of a video generation method provided by an embodiment of the present application. Figure 2 Step 240 in the embodiment can be implemented as step 240a, step 240b and step 240c, as follows:

[0126] Step 240a: Obtain particle labels of the text description images corresponding to at least two text segments respectively; the particle labels are used to indicate the particle effects that match the text description images.

[0127] In an embodiment of the present application, the text description image may match one or more particle effects, each particle effect corresponds to a particle label, and different particle effects correspond to different particle labels.

[0128] Optionally, the computer device can obtain particle labels of text description images corresponding to at least two text segments respectively through rule implementation. For example, the computer device creates a rule base, which contains a mapping relationship between text description images and particle labels. After the computer device obtains the text description images of at least two text segments, the rule base is searched to obtain the particle labels corresponding to the text description images.

[0129] Optionally, the computer device can obtain particle labels of text description images corresponding to at least two text segments respectively through a machine learning model, and the machine learning model is an image classification model trained using a labeled data set. The image classification model can classify text segments into different particle effect categories (i.e., different particle labels). For example, the computer device can input at least two text segments into the image classification model in sequence, and the image classification model outputs the particle effect category corresponding to the input text description image, i.e., the particle label corresponding to the text description image.

[0130] Step 240b: querying a particle effect database through the particle labels of the text description images corresponding to at least two text segments, and obtaining the particle effects corresponding to at least two text segments.

[0131] In an embodiment of the present application, a computer device has a particle effect database, which includes particle tags and particle effects corresponding to the particle tags. After obtaining the particle tags corresponding to at least two text segments respectively, the computer device traverses the particle effect database and obtains at least one particle effect corresponding to the particle tags corresponding to the at least two text segments respectively from the particle effect database as the particle effects corresponding to the text description images of the at least two text segments respectively.

[0132] Step 240c: Generate a video of the target text according to the text description images corresponding to at least two text segments, and the particle effects corresponding to at least two text segments.

[0133] In an embodiment of the present application, after obtaining one or more particle effects corresponding to each text description image, the computer device can synthesize the particle effect layer with the text description image to add the particle effect to the corresponding text description image, and obtain at least two text description images with added particle effects.

[0134] The computer device can generate a video of the target text through professional video editing software based on at least two text description images with added particle effects; or it can synthesize the text description images into a final video file through a video encoding library.

[0135] In an embodiment of the present application, a computer device can quickly obtain a particle effect that matches a text description image by querying a particle tag, thereby effectively ensuring the accuracy of the particle effect and its relevance to the text segment, adding the particle effect to the corresponding text description image, generating an image with a particle effect, and generating a video of the target text based on the image with the particle effect, so that the generated video effectively retains key information in the target text, and also enhances the visual effect of the video through the particle effect, thereby effectively ensuring the visual effect of the video and improving the quality of video generation while improving the accuracy of video generation.

[0136] Based on the scheme shown in any one or more of the above-mentioned embodiments, in some embodiments, the above-mentioned step 240c can be implemented as follows: obtaining text speech corresponding to at least two text segments respectively; the text speech is the speech of reading the text segment; for each of the at least two text segments, generating a video clip with a duration equal to the playback duration of the text speech corresponding to the text segment, and the content being a text description image corresponding to the text segment superimposed with a particle effect corresponding to the text segment; splicing the video clips corresponding to the at least two text segments respectively in the order in which the at least two text segments appear in the target text to obtain a speech-free video of the target text; splicing the text speech corresponding to the at least two text segments respectively in the order in which the at least two text segments appear in the target text, and combining them with the speech-free video to obtain a video of the target text.

[0137] In the embodiment of the present application, after obtaining at least two text segments, the computer device can use natural language processing and speech synthesis algorithms to convert the contents of the at least two text segments into speech respectively, and use them as the text speech corresponding to the at least two text segments respectively. For each text segment, the computer device generates a video clip, the content of which is a video of the particle effect corresponding to the text segment superimposed on the text description image corresponding to the text segment, and the playback time of the video clip is the playback time of the text speech corresponding to the text segment.

[0138] In an embodiment of the present application, the speech-free video of the target text obtained by the computer device contains video clips corresponding to at least two text segments respectively. The computer device combines the text speech corresponding to the at least two spliced ​​text segments with the video clips in the speech-free video in a one-to-one correspondence, that is, the text speech of a text segment is combined with the video clip corresponding to the text segment.

[0139] In the embodiment of the present application, text-to-speech ensures that the text content can be presented in the form of speech, providing synchronized audio support for subsequent video generation. For each text segment, the generation duration is the playback duration of the text-to-speech corresponding to the text segment, and the content is a video clip with particle effects superimposed on the text description image corresponding to the text segment, ensuring that the playback duration of the video clip is consistent with the playback duration of the text-to-speech. The text-to-speech with consistent sequence is combined with the video without speech to generate a video of the target text, which can effectively ensure the synchronization of audio and picture of the generated video and improve the accuracy and quality of video generation.

[0140] Based on the scheme shown in any one or more of the above embodiments, in some embodiments, the above step 240a can be implemented as: obtaining the image label of the text description image corresponding to the second text segment, and querying the particle label corresponding to the image label; the second text segment is any one of the at least two text segments; or, querying the particle label corresponding to the keyword of the second text segment.

[0141] In an embodiment of the present application, a computer device includes a particle label database, which stores an image label of a text description image corresponding to the second text segment and at least one particle label corresponding to the image label. In an embodiment of the present application, after extracting the image label of the text description image corresponding to the second text segment, the computer device traverses the particle label database to obtain the particle label in the particle label database that corresponds to the image label.

[0142] In some embodiments, the particle label database stores a label pair consisting of an image label of the text description image corresponding to the second text segment and at least one particle label corresponding to the image label. In an embodiment of the present application, after extracting the image label of the text description image corresponding to the second text segment, the computer device traverses the particle label database, queries the label pair containing the image label of the text description image corresponding to the second text segment, and obtains the particle label in the label pair.

[0143] In an embodiment of the present application, optionally, when there are multiple image tags corresponding to the second text segment, the computer device can obtain particle tags corresponding to the multiple image tags respectively, and the computer device selects the particle tag with the largest number of occurrences among the particle tags corresponding to the multiple image tags respectively as the particle tag of the text description image corresponding to the second text segment.

[0144] In an embodiment of the present application, a computer device includes a particle label database, which stores keywords of the second text segment and at least one particle label corresponding to the keyword. In an embodiment of the present application, after extracting the keyword corresponding to the second text segment, the computer device traverses the particle label database to obtain the particle label corresponding to the keyword in the particle label database.

[0145] In some embodiments, the particle tag database stores various keywords and keyword-tag pairs consisting of at least one particle tag corresponding to each keyword. In an embodiment of the present application, after extracting the keywords corresponding to the second text segment, the computer device traverses the particle tag database, queries the keyword-tag pairs containing the keywords corresponding to the second text segment, and obtains the particle tags in the queried keyword-tag pairs.

[0146] In an embodiment of the present application, when there are multiple keywords corresponding to the second text segment, the computer device can obtain particle labels corresponding to the multiple keywords respectively, and the computer device selects the particle label with the largest number of appearances among the particle labels corresponding to the multiple keywords respectively as the particle label of the text description image corresponding to the second text segment.

[0147] In an embodiment of the present application, the computer device obtains the image tag of the text description image corresponding to the second text segment, queries the particle tag corresponding to the image tag, or queries the particle tag corresponding to the keyword of the second text segment, thereby ensuring a high match between the queried particle tag and the text content, thereby improving the accuracy and quality of the generated image.

[0148] Based on the solutions shown in any one or more of the above embodiments, please refer to Figure 5 , Figure 5 is a flowchart of a video generation method provided by an embodiment of the present application. The video generation method shown in this embodiment can be executed by a computer device, which can be Figure 1 The video generation service device 130 in the embodiment. This solution includes three stages: a reference image acquisition stage, a text description image acquisition stage, and a target text video generation stage. The specific process is as follows.

[0149] 1. Reference Image Acquisition Stage

[0150] After acquiring the target text including at least two text segments (a first text segment and a second text segment), the computer device extracts keywords from the at least two text segments respectively.

[0151] Among them, the keywords include character description keywords and scene description keywords. The character description keywords may include at least one of character personal information description words (for example, character facial features description words, character name or character gender), character clothing description words and character posture description words.

[0152] The computer device queries an image database based on the obtained task description keywords to obtain a character reference image corresponding to the description keywords as a character reference image corresponding to the text segment; and queries an image database based on the obtained scene description keywords to obtain a background reference image corresponding to the scene description keywords as a background reference image corresponding to the text segment. The computer device obtains the character reference images corresponding to at least two text segments, respectively.

[0153] At the same time, the computer device can perform summary extraction operations on at least two text segments respectively to obtain description sentences corresponding to the at least two text segments; or, after obtaining multiple keywords corresponding to at least two text segments respectively, the computer device can splice multiple keywords to obtain description sentences corresponding to the at least two text segments respectively.

[0154] 2. Text description image acquisition stage

[0155] The computer device inputs the description sentences, character reference images and background reference images corresponding to at least two text segments into a generative language model, obtains the character description images and background description images corresponding to the at least two text segments as output, and then fuses the character description images with the background description images to obtain the text description images corresponding to the at least two text segments.

[0156] 3. Video Generation Stage of Target Text

[0157] After obtaining text description images corresponding to at least two text segments, the computer device obtains particle tags corresponding to the text description images, queries a particle effect database through the particle tags, and obtains particle effects corresponding to the particle tags as the particle effects corresponding to the text segments corresponding to the text description images.

[0158] A computer generates a video clip whose duration is the playback duration of the text voice corresponding to the text segment, and whose content is a text description image corresponding to the text segment superimposed with a particle effect corresponding to the text segment. The computer device generates a video clip for each text segment. After generating at least two video clips, the at least two video clips are spliced ​​according to the order in which the at least two text segments appear in the target text to obtain a speech-free video of the target text.

[0159] At the same time, the computer device splices the text voices corresponding to the at least two text segments according to the order in which the at least two text segments appear in the target text, and combines them with the non-voice video to obtain the video of the target text.

[0160] In an embodiment of the present application, in the reference image acquisition stage, the computer device obtains the reference image based on extracting keywords (character description keywords and scene description keywords) from each text segment in the target text, ensuring that the obtained reference image can accurately reflect the core elements of the text segment, helping to capture the detailed description of the text, and effectively avoiding generation errors caused by text comprehension deviations; in the text description image acquisition stage, the computer device uses a generative language model to combine the description sentences corresponding to the text segment with the reference image to generate character description images and background description images that are highly matched with the text content, further enhancing the accuracy and richness of the generated image; in the video generation stage of the target text, the text speech and the voice-free video are spliced ​​in the order of the text paragraphs, ensuring the logical consistency of the timeline of the video content with the target text, and avoiding the audio and video asynchrony caused by the confusion of the timeline. In summary, the above scheme greatly improves the accuracy of video generation and effectively ensures the quality of video generation.

[0161] Based on the above Figures 2 to 5 The steps in the embodiment of the present application illustrate a technology for automatically generating videos of novels with high character consistency. The technology for automatically generating videos of novels includes six parts, which are as follows.

[0162] 1. LLM Text Segmentation

[0163] The computer device segments the novel text (equivalent to the target text in the above embodiments) through the fine-tuned novel segmentation model to ensure the integrity of each paragraph (equivalent to at least two text segments in the above embodiments).

[0164] In an embodiment of the present application, a user can upload a novel text to a computer device, and the computer device inputs the received novel text into a fine-tuned novel segmentation model, and segmentates the novel text based on the fine-tuned novel segmentation model to obtain at least two paragraphs, and the at least two paragraphs are paragraphs in the novel text.

[0165] The novel classification model after fine-tuning can be a novel segmentation model that has been fine-tuned and trained with fine-tuning data by an open source LLM large model (such as Llama, which is friendly to Chinese). The training process of the novel segmentation model is as follows: First, prepare a high-quality fine-tuning dataset in the format of JSON, CSV, TXT file, etc. The content of the fine-tuning dataset is:

[0166] {Novel 1 text: paragraph 1; paragraph 2; paragraph n},

[0167] {Novel 2 text: paragraph 1; paragraph 2; paragraph n},

[0168] }

[0169] In an embodiment of the present application, a computer device may use full fine-tuning or LoRA fine-tuning to perform fine-tuning operations on the novel segmentation model, wherein full fine-tuning is to update all weights of the novel segmentation model, which is suitable for situations with large amounts of data and sufficient computing resources; LoRA fine-tuning modifies some weights of the novel segmentation model by adding a low-rank matrix to reduce the amount of fine-tuning parameters, and is an efficient fine-tuning method.

[0170] After the novel segmentation model is trained and fine-tuned, the computer device can evaluate the fine-tuned novel segmentation model and use the fine-tuning data set or the test data set to calculate indicators such as BLEU, ROUGE or F1 to check the fine-tuning effect; optionally, the trained novel segmentation model can be scored through manual evaluation, the lowest score and the highest score are removed, and the average score is taken as the evaluation score of the novel segmentation model to check the fine-tuning effect.

[0171] The above completeness represents the semantic completeness of the generated paragraph, and the semantic completeness includes characters, plots, scenes, etc. For example: A girl is at the beach. The next day she goes home. The protagonists of these two sentences are the same and need to be divided into one paragraph.

[0172] 2. LLM Keyword Extraction

[0173] The computer equipment uses a fine-tuned keyword extraction model to extract key information from the text of each paragraph. The key information includes the character's name, gender, clothing, posture and scene.

[0174] In an embodiment of the present application, after obtaining at least two paragraphs of a novel text, the computer device sequentially inputs the at least two paragraphs into a fine-tuned keyword extraction model. The keyword extraction model performs a key information extraction operation on each paragraph to obtain key information corresponding to at least two paragraphs, respectively. The key information is at least one of a character name, character gender, character clothing, character posture, and scene in the paragraph.

[0175] The fine-tuned keyword extraction model can be a keyword extraction model that has been fine-tuned and trained using fine-tuning data using an open source LLM large model (such as Chinese-friendly Llama). The input of the keyword extraction model is a paragraph of text, and the output is a keyword list, which contains at least one keyword.

[0176] For example, a computer device inputs a fine-tuned keyword extraction model with a paragraph containing the content "a girl at the beach". The output content is: Lili, female, 25 years old, long skirt, beach, etc.

[0177] It should be noted that the above keyword extraction model has global information, and the current output result will refer to the historical input and output results of the keyword extraction model to ensure that the protagonist and scene of the context remain consistent.

[0178] 3. Automated Character and Scene Knowledge Base

[0179] The computer equipment collects high-quality text-image pairs by storing them in the database and indexes the database when necessary.

[0180] In the embodiment of the present application, the computer device includes a character knowledge base and a scene knowledge base, and the character knowledge base and the scene knowledge base store text-picture pairs consisting of key (text) and value (picture), such as "Girl 1": Picture 1, "Seaside": Picture 2.

[0181] In an embodiment of the present application, pictures of people can be captured using a database entry program (the source can be from the Internet), and then quality filtering is performed (filtering out pictures of poor quality and pictures without people), and then labels are extracted from the pictures to make keys, and the keys and pictures are stored in the knowledge base in the form of text-picture pairs.

[0182] In actual application, the computer device can obtain the required pictures from the text-picture pairs in the knowledge base according to the task requirements. For example, if it is necessary to generate pictures of female characters, the computer device retrieves the text-picture pairs and queries all pictures corresponding to the text (female).

[0183] 4. Precise control generation

[0184] When computer equipment is creating pictures, it controls the characters’ faces, clothing and scenes, so that the characters’ consistency is improved.

[0185] In an embodiment of the present application, a computer device can use IP-Adapter (an innovative image condition generation control method) to add image conditions to a text image model (e.g., Stable Diffusion) to better maintain a specific style, consistency, or detail features when generating an image. Through IP-Adapter, the consistency of faces, clothing, and scenes in the generated image can be controlled with the help of a reference image.

[0186] The input of the above-mentioned text-based graph model is the conditional image and the description text.

[0187] 1. Conditional image: Select one or more reference images (such as a person's face or a specified clothing style) to provide a basis for target consistency. The conditional image contains at least one of the target person's specific facial features, clothing style, or scene elements.

[0188] 2. Description text: Write a description of the generated image, such as "a young woman in a blue dress standing in the forest with long brown hair" (that is, the prompt words extracted before: quality words + role words + scene words).

[0189] 5. Convert pictures to videos

[0190] The computer equipment arranges the pictures according to timestamps to generate a video, matching the particle effects that match the tags.

[0191] Exemplarily, the novel text is divided into 10 paragraphs, and 10 pictures corresponding to the paragraphs are generated. The computer device sorts the pictures in the order of generation, and the duration of each picture is (the duration of the novel text generating speech), and a synthesized video is produced.

[0192] 6. Video Synthesis

[0193] The computer device synthesizes the novel subtitles, audio and video to generate an audio novel video.

[0194] In an embodiment of the present application, the computer device uses fully automated technology to improve character consistency and image quality through precise control of character faces, clothing, postures, and scenes.

[0195] For example, please refer to Figure 6 , Figure 6 It is a framework flow chart of a video generation method provided by an embodiment of the present application.

[0196] Figure 6 The feature extraction operation of the computer device is to first divide the image into specific areas, set other areas except the target feature area to white, and encode the target feature area to obtain the feature value corresponding to the target area.

[0197] Among them, Controlnet has various control models, such as posture control, edge control, etc. A posture image is a specific image input to the SD model (for example, the character posture corresponding to a stickman). The posture image can control the generation of the SD model so that the generated character posture is similar to the input posture image.

[0198] Among them, the secondary rendering is performed by a computer device to redraw the entire picture, reverse the scene picture, extract the prompt words, add quality words to the extracted prompt words, and use the SD model to perform redrawing.

[0199] In an embodiment of the present application, after obtaining the generated person picture, the computer device separates the person from the person picture through person segmentation.

[0200] Assume that the particle images are a1, a2, a3, an, and the generated images are b1, b2, b3. The final synthesized video has a frame rate, for example 30 frames, and each image lasts for one second. Then the generated image b1 lasts for 30 frames, and the generated image b2 lasts for 30 frames. Particle image a1 is attached to the first frame, and particle image a2 is attached to the second frame, and the cycle is repeated in sequence.

[0201] In the embodiment of the present application, the computer device can accurately control the text image and improve the consistency of the characters by establishing a character knowledge base and a scene knowledge base; add particle effects to the video synthesis to make the video more vivid; and effectively improve the quality of the video generated by the automated novel.

[0202] The above is only an embodiment of the present application and should not be regarded as limiting the present application. Professionals should understand that various deformations and modifications can be made to the embodiments to adapt to different application requirements. Therefore, the scope of the present application should be limited by the claims attached to the claims.

[0203] Please refer to Figure 7 , which shows a block diagram of a video generation device provided by an exemplary embodiment of the present application. The data processing device can be implemented as all or part of a computer device by hardware or a combination of hardware and software to achieve the above Figures 2 to 4 All or part of the steps in the embodiments shown. Figure 7 As shown, the video generating device includes:

[0204] A keyword extraction module 701 is used to extract keywords from at least two text segments; the at least two text segments are different text segments in the target text;

[0205] A reference image acquisition module 702 is used to query an image database using keywords of at least two text segments to obtain reference images corresponding to the at least two text segments;

[0206] The text description image generation module 703 is used to generate text description images corresponding to at least two text segments respectively based on at least two text segments and reference images corresponding to the at least two text segments respectively; the image content of the text description image matches the text content of the corresponding text segment;

[0207] The video generation module 704 is used to generate a video of the target text based on the text description images corresponding to at least two text segments.

[0208] In some embodiments, the text description image generation module 703 is used to obtain a description sentence of a first text segment; the description sentence is a sentence used to describe the main content of the text segment; the first text segment is any one of at least two text segments; the description sentence of the first text segment and the reference image corresponding to the first text segment are input into a generative language model to obtain a text description image corresponding to the first text segment output by the generative language model.

[0209] In some embodiments, the text description image generation module 703 is used to perform a summary extraction operation on the first text segment to obtain a description sentence of the first text segment; or to concatenate multiple keywords of the first text segment to generate a description sentence of the first text segment.

[0210] In some embodiments, the reference image includes a character reference image and a background reference image; the text description image generation module 703 is used to input the description sentence of the first text segment, the character reference image corresponding to the first text segment, and the background reference image corresponding to the first text segment into the generative language model to obtain the character description image corresponding to the first text segment and the background description image corresponding to the first text segment output by the generative language model; the character description image corresponding to the first text segment and the background description image corresponding to the first text segment are merged to obtain the text description image corresponding to the first text segment.

[0211] In some embodiments, the reference image acquisition module 702 is used to query the image database through the character description keywords among the keywords corresponding to the first text segment, and obtain the character reference image corresponding to the first text segment; the character description keywords are used to describe the characters in the text segment; and to query the image database through the scene description keywords among the keywords corresponding to the first text segment, and obtain the background reference image corresponding to the first text segment, and the scene description keywords are used to describe the scene in the text segment.

[0212] In some embodiments, when the character description keywords include at least one of the character's facial features description words, the character's name, and the character's gender, the character reference image includes a face reference image; when the character description keywords include the character's clothing description words, the character reference image includes a clothing reference image; when the character description keywords include the character's posture description words, the character reference image includes a character posture reference image.

[0213] In some embodiments, the reference image acquisition module 702 is used to query the image database through the scene description keywords in the keywords corresponding to the first text segment, obtain the background template image stored in the image database corresponding to the first text segment; obtain the image keywords of the background template image; input the image keywords into the generative language model, and obtain the background reference image corresponding to the first text segment output by the generative language model.

[0214] In some embodiments, the video generation module 704 is used to obtain particle tags of text description images corresponding to at least two text segments respectively; the particle tags are used to indicate particle effects that match the text description images; query a particle effect database through the particle tags of the text description images corresponding to at least two text segments respectively, and obtain the particle effects corresponding to at least two text segments respectively; generate a video of the target text according to the text description images corresponding to at least two text segments respectively, and the particle effects corresponding to at least two text segments respectively.

[0215] In some embodiments, the video generation module 704 is used to obtain text speech corresponding to at least two text segments; the text speech is the speech of reading the text segment; for each of the at least two text segments, a video clip is generated with a duration equal to the playback duration of the text speech corresponding to the text segment, and the content is a text description image corresponding to the text segment superimposed with a particle effect corresponding to the text segment; the video clips corresponding to the at least two text segments are spliced ​​in the order in which the at least two text segments appear in the target text to obtain a speech-free video of the target text; the text speech corresponding to the at least two text segments are spliced ​​in the order in which the at least two text segments appear in the target text, and then combined with the speech-free video to obtain a video of the target text.

[0216] In some embodiments, the video generation module 704 is used to obtain the image tag of the text description image corresponding to the second text segment, and query the particle tag corresponding to the image tag; the second text segment is any one of the at least two text segments; or, query the particle tag corresponding to the keyword of the second text segment.

[0217] Please refer to Figure 8 , Figure 8 800 is a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. The computer device 800 also includes a basic input / output system (I / O system) 806 that helps transmit information between various devices in the computer, and a large-capacity storage device 807 for storing an operating system 813, application programs 814 and other program modules 815.

[0218] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 such as a mouse and a keyboard for user inputting information. The display 808 and the input device 809 are connected to the central processing unit 801 through an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, a printer, or other types of output devices.

[0219] The mass storage device 807 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0220] Without loss of generality, computer readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Computer storage media include RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 804 and mass storage device 807 can be collectively referred to as memory.

[0221] The computer device 800 can be connected to the Internet or other network devices through a network interface unit 811 connected to the system bus 805 .

[0222] The memory also includes one or more programs, one or more programs are stored in the memory, and the central processing unit 801 implements the one or more programs by executing the one or more programs. Figures 2 to 4 All or part of the steps in the method shown.

[0223] In an exemplary embodiment, a chip is also provided. The chip includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement all or part of the steps of the method shown in the above embodiments of the present application.

[0224] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement all or part of the steps of the method shown in the above-mentioned various embodiments of the present application.

[0225] In an exemplary embodiment, a computer-readable storage medium is also provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement all or part of the steps of the method shown in the above-mentioned embodiments of the present application.

[0226] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program. The above program may be stored in a computer-readable storage medium. The above storage medium may be a read-only memory, a disk or an optical disk, etc.

[0227] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0228] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: The method comprises: Extracting keywords from at least two text segments; the at least two text segments are different text segments in the target text; Querying an image database using the keywords of the at least two text segments to obtain reference images corresponding to the at least two text segments respectively; Based on the at least two text segments and the reference images respectively corresponding to the at least two text segments, generating text description images respectively corresponding to the at least two text segments; the image content of the text description image matches the text content of the corresponding text segment; A video of the target text is generated based on the text description images respectively corresponding to the at least two text segments.

2. The method according to claim 1, characterized in that The step of generating text description images corresponding to the at least two text segments respectively based on the at least two text segments and the reference images corresponding to the at least two text segments respectively comprises: Acquire a description sentence of a first text segment; the description sentence is a sentence used to describe the main content of the text segment; the first text segment is any one of the at least two text segments; The description sentence of the first text segment and the reference image corresponding to the first text segment are input into a generative language model to obtain a text description image corresponding to the first text segment output by the generative language model.

3. The method according to claim 2, characterized in that The step of obtaining a description statement of the first text segment includes: Performing a summary extraction operation on the first text segment to obtain a description sentence of the first text segment; or, Multiple keywords of the first text segment are concatenated to generate a description sentence of the first text segment.

4. The method according to claim 2, characterized in that: The reference images include a character reference image and a background reference image; The step of inputting the description sentence of the first text segment and the reference image corresponding to the first text segment into a generative language model to obtain a text description image corresponding to the first text segment output by the generative language model includes: Inputting the description sentence of the first text segment, the character reference image corresponding to the first text segment, and the background reference image corresponding to the first text segment into the generative language model, and obtaining the character description image corresponding to the first text segment and the background description image corresponding to the first text segment output by the generative language model; The character description image corresponding to the first text segment and the background description image corresponding to the first text segment are fused to obtain a text description image corresponding to the first text segment.

5. The method according to claim 4, characterized in that The step of querying the image database using the keywords of the at least two text segments comprises: The image database is searched by using a character description keyword among the keywords corresponding to the first text segment to obtain the character reference image corresponding to the first text segment; the character description keyword is used to describe the character in the text segment; The image database is queried through scene description keywords among the keywords corresponding to the first text segment to obtain the background reference image corresponding to the first text segment, wherein the scene description keywords are used to describe the scene in the text segment.

6. The method according to claim 5, characterized in that In the case where the character description keywords include at least one of a character facial features description word, a character name, and a character gender, the character reference image includes a face reference image; In the case where the character description keywords include character clothing description words, the character reference images include clothing reference images; In the case where the character description keyword includes a character posture description word, the character reference image includes a character posture reference image.

7. The method according to claim 5, characterized in that The step of querying the image database using the scene description keywords in the keywords corresponding to the first text segment to obtain the background reference image corresponding to the first text segment includes: Querying the image database using the scene description keywords in the keywords corresponding to the first text segment to obtain a background template image stored in the image database corresponding to the first text segment; Obtain image keywords of the background template image; The image keyword is input into the generative language model to obtain the background reference image corresponding to the first text segment output by the generative language model.

8. The method according to any one of claims 1 to 7, characterized in that: The step of generating a video of the target text based on the text description images respectively corresponding to the at least two text segments includes: Obtaining particle labels of the text description images respectively corresponding to the at least two text segments; the particle labels are used to indicate particle effects matching the text description images; Querying a particle effect database through particle labels of text description images corresponding to the at least two text segments respectively, to obtain particle effects corresponding to the at least two text segments respectively; A video of the target text is generated according to the text description images respectively corresponding to the at least two text segments and the particle effects respectively corresponding to the at least two text segments.

9. The method according to claim 8, characterized in that The step of generating a video of the target text according to the text description images respectively corresponding to the at least two text segments and the particle effects respectively corresponding to the at least two text segments comprises: Acquire text speech corresponding to the at least two text segments respectively; the text speech is the speech of reading the text segment; For each of the at least two text segments, generating a video clip having a duration equal to the playback duration of the text speech corresponding to the text segment and having content of a text description image corresponding to the text segment superimposed with a particle effect corresponding to the text segment; splicing the video segments corresponding to the at least two text segments respectively according to the order in which the at least two text segments appear in the target text, to obtain a voice-free video of the target text; The text speech corresponding to the at least two text segments are spliced ​​according to the order in which the at least two text segments appear in the target text, and then combined with the speech-free video to obtain the video of the target text.

10. The method according to claim 8, characterized in that The step of obtaining particle labels of the text description images respectively corresponding to the at least two text segments includes: Obtaining an image label of a text description image corresponding to a second text segment, and querying the particle label corresponding to the image label; the second text segment is any one of the at least two text segments; or, The particle tags corresponding to the keywords of the second text segment are searched.

11. A video generating device, characterized in that: The device comprises: A keyword extraction module, used to extract keywords from at least two text segments; the at least two text segments are different text segments in the target text; A reference image acquisition module, configured to query an image database using keywords of the at least two text segments to obtain reference images corresponding to the at least two text segments; A text description image generation module, configured to generate text description images corresponding to the at least two text segments respectively based on the at least two text segments and the reference images corresponding to the at least two text segments respectively; the image content of the text description image matches the text content of the corresponding text segment; The video generation module is used to generate a video of the target text based on the text description images respectively corresponding to the at least two text segments.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein instructions are stored in the memory, and the instructions are executed by the processor to implement the video generation method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and the instructions are executed by a processor of a computer device to implement the video generation method according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the video generation method according to any one of claims 1 to 10.

Images