Image generation method and device, electronic equipment and storage medium

Through text splitting and LoRA model update technology, the problem of inconsistent character characteristics in the literary and biographical model is solved, and the continuity and unity of image generation is achieved, which is suitable for a variety of creative scenarios.

CN118982604APending Publication Date: 2024-11-19BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411064369.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When generating images, the existing literary and biographical graphics models cannot guarantee the continuity and unity of the same character in the images corresponding to different text fragments, resulting in inconsistent character characteristics.

Method used

By obtaining the text containing the characters, extracting the description text of the characters to be used, and splitting the text according to the lens, generating the description text of the lens, using the pre-trained literary and genre model and LoRA model to update the model parameters to ensure the characteristics of the same character consistency.

Benefits of technology

It realizes the continuity and unity of character characteristics in different lens images, improves the accuracy and consistency of image generation, and is suitable for a variety of creative scenes such as script storyboards and novel-to-comic strips.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982604A_ABST
    Figure CN118982604A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image generation method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps of obtaining a first text, wherein the first text comprises a to-be-utilized description text of at least one character; extracting a to-be-utilized description text of each character contained in the first text; splitting the first text according to the shot to obtain a shot description text of each shot; one shot description text represents the figure in the shot represented by the shot description text and the state of the figure in the shot; based on each shot description text and a to-be-utilized description text of a character in the shot description text, obtaining a first cue word corresponding to the shot description text; and processing the first cue word corresponding to the shot description text based on a pre-trained first text graph model to obtain an image corresponding to the shot description text. And the continuity and uniformity of the same character in the generated different images are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the use of Wensheng graph models to generate images corresponding to specified texts has been widely used in the creation of film, television and literary works. For example, in the process of filming a movie, the Wensheng graph model can be used to generate the storyboard images corresponding to the text of a certain paragraph in the script; in the process of novel creation, the Wensheng graph model can be used to generate illustrations corresponding to the text of a certain plot in the novel.

[0003] In the related art, for the segments in the text whose corresponding images are to be generated, each text segment is input into a text-generated graph model respectively, and the text-generated graph model obtains the image corresponding to each text segment according to the content of each text segment.

[0004] However, when the text contains characters, the above-mentioned method of generating images cannot guarantee the continuity and uniformity of the same character in the images corresponding to different text segments. That is, the character features of the same character in different images are inconsistent. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an image generation method, device, electronic device and storage medium to ensure the continuity and uniformity of the same person in different generated images. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a method for generating an image is provided, the method comprising:

[0007] Acquire a first text, wherein the first text includes a description text of at least one character to be used;

[0008] extracting the to-be-used description text of each character contained in the first text;

[0009] The first text is split according to the shots to obtain shot description texts of each shot; wherein a shot description text represents: a character in the shot represented by the shot description text, and a state of the character in the shot;

[0010] Based on each shot description text and the to-be-used description text of the character in the shot description text, a first prompt word corresponding to the shot description text is obtained;

[0011] Based on the pre-trained first text-to-image model, the first prompt word corresponding to the shot description text is processed to obtain an image corresponding to the shot description text.

[0012] In some embodiments, after extracting the to-be-used description text of each character contained in the first text, the method further includes:

[0013] Determine a LoRA model that matches the description text to be used of each person from a plurality of preset LoRA (Low-Rank Adaptation) models; wherein the plurality of preset LoRA models are trained based on sample images of a plurality of sample persons, sample labels of each sample image, and a pre-trained second text image model, and a sample label of a sample image represents: features of the sample person contained in the sample image; a LoRA model corresponds to a sample person;

[0014] The first prompt word corresponding to the shot description text is processed based on the pre-trained first text-image model to obtain an image corresponding to the shot description text, including:

[0015] Using the model parameters of the LoRA model corresponding to the person included in the shot description text, the model parameters of the pre-trained first-text image model are updated to obtain a new first-text image model;

[0016] The first prompt word corresponding to the shot description text is input into the new first text-generated image model to obtain an image generated by the new first text-generated image model as the image corresponding to the shot description text.

[0017] In some embodiments, the splitting of the first text according to the shots to obtain the shot description text of each shot includes:

[0018] The first text and the second prompt word are input into a first large language model to obtain a shot description text of each shot output by the first large language model; wherein the second prompt word indicates: splitting the first text according to the shot.

[0019] In some embodiments, obtaining the first text includes:

[0020] Acquire a second text, wherein the second text includes a description text of at least one character to be used;

[0021] Each text containing at least one character obtained by splitting the second text according to a preset splitting rule is used as a first text; wherein the preset splitting rule includes: splitting the second text according to the chapters of the text, and / or splitting the second text according to a preset number of words in the text.

[0022] In some embodiments, extracting the to-be-used description text of each character contained in the first text includes:

[0023] The first text and the third prompt word are input into the second largest language model to obtain the description text to be used for each character contained in the first text; wherein the third prompt word indicates: feature extraction is performed on each character contained in the first text.

[0024] In some embodiments, the number of the first texts is greater than 1;

[0025] The first text and the third prompt word are input into the second largest language model to obtain the description text to be used of each character contained in the first text, including:

[0026] For each first text, the first text and the third prompt word are input into the second largest language model to obtain an initial description text of each character contained in the first text;

[0027] The initial description texts for the same character in the obtained initial description texts are merged to obtain the description text to be used for the character.

[0028] In some embodiments, the use of the model parameters of the LoRA model corresponding to the character included in the shot description text to update the model parameters of the pre-trained first text image model to obtain a new first text image model includes:

[0029] The model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated, and the fusion results of the model parameters of the pre-trained first-text image model are used as the model parameters of the new first-text image model to obtain the new first-text image model.

[0030] In some embodiments, there is only one character in a shot of a shot description text representation;

[0031] The step of calculating the model parameters of the LoRA model corresponding to the characters included in the shot description text and fusing the model parameters of the pre-trained first text image model as the model parameters of the new first text image model to obtain the new first text image model includes:

[0032] The model parameters of the pre-trained first-text image model and the weighted sum of the model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated as the model parameters of the new first-text image model to obtain the new first-text image model.

[0033] In some embodiments, there are multiple characters in a shot represented by a shot description text;

[0034] The step of calculating the model parameters of the LoRA model corresponding to the characters included in the shot description text and fusing the model parameters of the pre-trained first text image model as the model parameters of the new first text image model to obtain the new first text image model includes:

[0035] For multiple characters in a shot represented by a shot description text, determine an image area occupied by each character in an image to be generated;

[0036] For each image area corresponding to a person, the weighted sum of the part of the model parameters of the pre-trained first-text image model corresponding to the image area and the model parameters of the LoRA model corresponding to the person is calculated to obtain the part of the model parameters of the new first-text image model corresponding to the image area.

[0037] In a second aspect of the present invention, there is also provided an image generating device, the device comprising:

[0038] A first text acquisition module, used to acquire a first text, wherein the first text includes a description text of at least one character to be used;

[0039] A feature acquisition module, used for extracting the description text to be used of each character contained in the first text;

[0040] A shot splitting module is used to split the first text according to the shots to obtain shot description texts of each shot; wherein a shot description text represents: a character in the shot represented by the shot description text, and a state of the character in the shot;

[0041] A prompt word generation module, used for obtaining a first prompt word corresponding to each shot description text based on each shot description text and the to-be-used description text of the character in the shot description text;

[0042] The image generation module is used to process the first prompt word corresponding to the shot description text based on the pre-trained first text-to-image model to obtain the image corresponding to the shot description text.

[0043] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement any of the above-mentioned image generation methods when executing the program stored in the memory.

[0046] In another aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned image generation methods is implemented.

[0047] In yet another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image generation methods.

[0048] The embodiment of the present invention provides an image generation method. After obtaining the first text of the image to be generated, the description text to be used of each character contained in the first text can be extracted. That is, the text to be used of each character can reflect the characteristics of the character in the first text. Accordingly, since a shot description text is used to describe the character appearing in a shot and the state of the character appearing, for each shot description text obtained by splitting the first text, the first prompt word corresponding to the shot description text can be obtained based on the shot description text and the description text to be used of the character in the shot description text, and the first prompt word corresponding to the shot description text is processed based on the pre-trained first text-born image model to obtain the image corresponding to the shot description text. It can be understood that since the description text to be used of the same character in different shot description texts is fixed, that is, when generating images for the same character in different shots, the characteristics of the character obtained by the first text-born image model are fixed, it can also ensure that the characteristics of the character in the images corresponding to the generated different shots are consistent, and the continuity and unity of the same character in the generated different images are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0050] Figure 1 A first flow chart of an image generating method provided by an embodiment of the present invention;

[0051] Figure 2 A second flow chart of the image generation method provided by an embodiment of the present invention;

[0052] Figure 3 A third flow chart of the image generating method provided by an embodiment of the present invention;

[0053] Figure 4 A fourth flow chart of the image generating method provided by an embodiment of the present invention;

[0054] FIG5( a ) is an image corresponding to the shot description text of shot 1;

[0055] FIG5( b ) is the image corresponding to the shot description text of shot 2;

[0056] FIG5( c ) is the image corresponding to the shot description text of shot 3;

[0057] FIG5( d ) is an image corresponding to the shot description text of shot 4;

[0058] Figure 6 A fifth flow chart of the image generating method provided by the embodiment of the present invention;

[0059] Figure 7 A flowchart of generating an image provided by an embodiment of the present invention;

[0060] Figure 8 A structural diagram of an image generating device provided by an embodiment of the present invention;

[0061] Fig. 9 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0063] With the development of artificial intelligence technology, the use of text-to-image models to generate images corresponding to specified texts has been widely used in the creation of film, television, and literary works. For example, in the process of filming a movie, the text-to-image model can be used to generate the storyboard images corresponding to the text of a certain paragraph in the script; in the process of novel creation, the text-to-image model can be used to generate illustrations corresponding to the text of a certain plot in the novel.

[0064] In the related art, for the segments in the text whose corresponding images are to be generated, each text segment is input into a text-generated graph model respectively, and the text-generated graph model obtains the image corresponding to each text segment according to the content of each text segment.

[0065] However, when the text contains characters, the above image generation method cannot guarantee the continuity and unity of the same character in the images corresponding to different text segments. That is, the character features of the same character in different images are inconsistent.

[0066] In order to solve the above problems, an embodiment of the present invention provides an image generation method, which can be applied to electronic devices. Figure 1 , Figure 1 A first flow chart of an image generation method provided by an embodiment of the present invention, the method may include the following steps:

[0067] S101: Acquire a first text, wherein the first text includes a description text of at least one character to be used.

[0068] S102: Extract the description text to be used of each character contained in the first text.

[0069] S103: Split the first text according to the shots to obtain shot description texts of the shots.

[0070] Among them, a shot description text represents: the characters in the shot represented by the shot description text, and the state of the characters in the shot.

[0071] S104: Based on each shot description text and the to-be-used description text of the character in the shot description text, obtain a first prompt word corresponding to the shot description text.

[0072] S105: Based on the pre-trained first text-to-image model, the first prompt word corresponding to the shot description text is processed to obtain an image corresponding to the shot description text.

[0073] The embodiment of the present invention provides an image generation method. After obtaining the first text of the image to be generated, the description text to be used of each character contained in the first text can be extracted. That is, the text to be used of each character can reflect the characteristics of the character in the first text. Accordingly, since a shot description text is used to describe the character appearing in a shot and the state of the character appearing, for each shot description text obtained by splitting the first text, the first prompt word corresponding to the shot description text can be obtained based on the shot description text and the description text to be used of the character in the shot description text, and the first prompt word corresponding to the shot description text is processed based on the pre-trained first text-born image model to obtain the image corresponding to the shot description text. It can be understood that since the description text to be used of the same character in different shot description texts is fixed, that is, when generating images for the same character in different shots, the characteristics of the character obtained by the first text-born image model are fixed, it can also ensure that the characteristics of the character in the images corresponding to the generated different shots are consistent, and the continuity and unity of the same character in the generated different images are ensured.

[0074] In addition, the image generation method provided by the present invention improves the usability and scope of application of generating images through the text graph model, and can be applied to various business scenarios such as script storyboard creation, novel conversion to comic strip, and story conversion to animation.

[0075] For steps S101-S102, the first text is the text of the image to be generated acquired by the electronic device. The text of the image to be generated can be the original text of a work in text form such as a novel or a script.

[0076] After obtaining the original text, the electronic device can directly use the original text as the first text. Alternatively, after obtaining the original text, the electronic device can also perform a preprocessing operation on the original text to obtain the first text. The specific process of performing the preprocessing operation on the original text will be described in the subsequent embodiments.

[0077] For example, the text content of a first text (which can be called first text A) obtained is: "Zhang looked inside the supermarket door. He noticed that there were only a few people in the supermarket, including a cashier in work clothes, who was a middle-aged woman with a ponytail. The cashier was busy collecting money. Zhang walked into the supermarket. He had short and messy hair, broad shoulders and narrow waist. He wore a white vest and worn-out jeans, and a shoulder bag was slung diagonally on his shoulder. As Zhang walked, he quietly observed the positions of the surveillance cameras around him. Zhang lowered his head to read the instructions of the food in his hand, glanced at the cashier counter, and quickly stuffed the food into his bag. In the surveillance, Zhang was still holding the food and checking the packaging. The female cashier glanced and continued to collect money. Zhang walked around and observed the movement of the truck outside the door."

[0078] For a character, the characteristics of the character can be divided into multiple types. For example, the characteristics of a character can include: age group, gender, hairstyle, appearance, clothing, etc.

[0079] It is understandable that, for a character, in the first text, some types of characteristics of the character are fixed, and some types of characteristics of the character are variable. For example, the first text describes the growth experience of a man from youth to old age, and the text content of the first text includes "he studied in Beijing in his youth, then went to Shanghai to work, and returned to his hometown in his old age." It can be seen that in the first text, the gender of the man is fixed, and the age range of the man is variable. For another example, the first text describes the life scene of a woman, and the text content of the first text includes "at 7 o'clock, she was still wearing pajamas with her hair loose, washing, and at 8 o'clock, she had changed into formal clothes and had her hair done up and was ready to go." It can be seen that in the first text, the gender of the woman is fixed, and the woman's clothes and hairstyle are variable.

[0080] Correspondingly, for a certain type of feature, if the feature of this character is unchanged in the first text, it means that the feature can reflect a fixed state of the character; for a certain type of feature, if the feature of this character is variable in the first text, it means that the feature can reflect a temporary state of the character.

[0081] The electronic device can process the acquired first text, and for each character included in the first text, among the texts representing various characteristics of the character, determine the text representing the characteristics of the fixed state of the character as the description text to be used for the character. That is, the description text to be used for a character obtained can reflect the fixed state of the character, and is a general description of the character in the first text.

[0082] In some embodiments, the description text to be used for each character contained in the first text can be obtained through a large language model. Figure 2 , Figure 2 A second flow chart of the image generation method provided by the embodiment of the present invention, in Figure 1 On the basis of, step S102 comprises:

[0083] S1021: Input the first text and the third prompt word into the second largest language model to obtain the description text to be used of each character contained in the first text.

[0084] The third prompt word (Prompt) indicates: extracting features of each character included in the first text.

[0085] For example, the third prompt word may be: "Extract the unchanged features of each character contained in the above text." In addition, the third prompt word may further limit the types of the character's features to: age group, gender, hairstyle, appearance, clothing, etc.

[0086] In an embodiment of the present invention, a preset large language model (LLM) (which may be referred to as a second large language model) may be used to obtain the description text to be used for each character contained in the first text. For example, the large language model may be: GPT-3 (Generative Pre-trained Transformer 3), BERT (Bidirectional Encoder Representations from Transformers), etc.

[0087] For example, for the above-mentioned first text A, the electronic device can input the first text and the third prompt word into the second largest language model, and the second largest language model can determine that the first text contains two characters: Zhang (ie, character 1) and the cashier (ie, character 2). Correspondingly, the second largest language model can output the initial description text of each character contained in the first text. That is, for character 1 in the first text A, the initial description text of character 1 is: male, short hair, white vest, bleached jeans. For character 2 in the first text A, the initial description text of character 2 is: female, ponytail, middle-aged, work clothes.

[0088] In one implementation, the number of first texts is greater than 1. Step S1021 includes:

[0089] Step 1: For each first text, the first text and the third prompt word are input into the second largest language model to obtain an initial description text of each character contained in the first text.

[0090] Step 2: The initial description texts for the same character in the obtained initial description texts are merged to obtain the description text to be used for the character.

[0091] In an embodiment of the present invention, if multiple first texts are obtained, for each first text, the first text and the third prompt word can be input into the second largest language model, and feature extraction is performed on each character contained in the first text to obtain the features of the character contained in the first text (i.e., the initial description text).

[0092] It is understandable that, since different first texts may contain text content related to the same character, multiple initial description texts for the same character may be obtained for multiple first texts. For a certain type of feature of the same character, the features of the same type extracted from the initial description texts of the character in different first texts may be the same or different.

[0093] For example, as in the above example, for character 1 in the first text A, the initial description text of character 1 is: male, short hair, white vest, bleached jeans. In addition, in addition to the first text A, the electronic device also obtains the first text B, and the text content of the first text B is: "... Zhang is 20 years old this year. This is his first time in City B. He is wearing a denim jacket and overalls." Correspondingly, for the above-mentioned first text B, the initial description text of Zhang (character 1) in the first text can be obtained. That is, for character 1 in the first text B, the initial description text of character 1 is: male, young, denim jacket, overalls. It can be seen that for the age group of character 1, the age group of character 1 extracted in the first text A and B is the same, and for the clothing of character 1, the clothing of character 1 extracted in the first text A and B is different.

[0094] Furthermore, the initial description texts for the same character in the obtained initial description texts can be merged to obtain the description text to be used for the character. Since the description text to be used for a character is used to reflect the fixed state of the character in multiple first texts, the features other than the features of the same type that have changed in the initial description texts of the character can be determined as the description text to be used for the character.

[0095] For example, for character 1 in the first text A, the initial description text of character 1 is: male, short hair, white vest, bleached jeans. For character 1 in the first text B, the initial description text of character 1 is: male, young, denim jacket, overalls. The features of the same type that have changed are: white vest, bleached jeans, and denim jacket, overalls. Accordingly, the description text to be used for character 1 is: male, young, short hair.

[0096] In one implementation, when the second largest language model outputs an initial description text of each character contained in a first text, for each character, the name of the character and the initial description text of the character to which the name belongs can be determined. Accordingly, the initial description texts of the characters to which the same name belongs (i.e., the initial description texts of the same character) can be fused, and the features of the fixed state of the character in the initial description texts of the character can be obtained and determined as the description text to be used for the character. For example, the initial description texts of the same character and the fourth prompt word can be input into the preset large language model to obtain the description text to be used for the character. Among them, the fourth prompt word can represent: the initial description texts are fused. For example, the fourth prompt word can be: output the same type of invariant features in the initial description texts of the character.

[0097] In this way, when multiple first texts are obtained, the image generation method provided by the embodiment of the present invention can integrate the descriptive texts of the characteristics of the same person extracted from each first text, that is, for each person, the same type of unchanged characteristics of the person in the world (that is, the descriptive text to be used) are obtained, which can improve the accuracy of the obtained descriptive text to be used. Furthermore, when the descriptive text to be used is used to generate an image subsequently, the finally generated image can more accurately reflect the characteristics of the person, thereby improving the accuracy of image generation.

[0098] In another implementation, when only one first text is obtained, the first text and the third prompt word can be directly input into the second language model. Then, the second language model can process the input first text, and output the result of feature extraction of each character contained in the first text according to the text content related to each character in the first text. Accordingly, the electronic device can directly use the features of each character in the first text output in the second language as the description text to be used for each character contained in the first text.

[0099] For example, there is only one first text, namely the first text A mentioned above. The electronic device can input the first text and the third prompt word into the second largest language model, and the second largest language model can determine that the first text contains two characters: Zhang (character 1) and the cashier (character 2). Correspondingly, the second largest language model can output the features of each character contained in the first text as the description text to be used for each character contained in the first text. That is, for character 1, the description text to be used for character 1 is: male, short hair, white vest, bleached jeans. For character 2, the description text to be used for character 2 is: female.

[0100] Based on the above processing, the descriptive text to be used for each character contained in the first text is obtained through a large language model. When the text content of the acquired first text is large, compared with determining the descriptive text to be used for each character contained in the first text by other methods (such as manual method), the efficiency and accuracy of obtaining the descriptive text to be used for each character contained in the first text can be improved.

[0101] In some embodiments, the description text to be used for each character contained in the first text can also be obtained by machine learning. That is, the first text can be input into a pre-trained character feature extraction model to obtain the description text to be used for each character contained in the first text. Among them, the feature extraction model can be trained based on a recurrent neural network (RNN) or a long short-term memory neural network (LSTM). The sample training set of the character feature extraction model: contains text of at least one character, and the label of each text in the sample training set is: the feature of at least one character contained in the text.

[0102] With respect to step S103, in the text content of a first text, the state of a character contained in the first text may be variable. For example, the state of the character may represent the action of the character. Therefore, it is necessary to generate multiple images containing the character, respectively representing different states of the character.

[0103] Accordingly, the first text can be split according to the shots to obtain the shot description texts of each shot. For a shot, the state of at least one character contained in the shot is unchanged. A shot description text represents: the character in the shot represented by the shot description text, and the state of the character in the shot. The state of the character in the shot can be: the action of the character.

[0104] In some embodiments, the first text may be split according to the shots using a large language model, see Figure 3 , Figure 3 A third flow chart of the image generation method provided by the embodiment of the present invention, in Figure 1 On the basis of, step S103 comprises:

[0105] S1031: Input the first text and the second prompt word into the first large language model to obtain the shot description text of each shot output by the first large language model.

[0106] The second prompt word indicates: split the first text according to the shots.

[0107] For example, the second prompt word may be: splitting the first text according to the actions of the characters, or splitting the first text according to the scenes.

[0108] In the embodiment of the present invention, the first text and the second prompt word may be input into a preset large language model (which may be referred to as a first large language model) to obtain the shot description text of each shot output by the first large language model.

[0109] In one implementation, the technician can weigh the number of generated images and the amount of computation consumed, and pre-train the large language model so that the trained large language model (i.e., the first large language model) can split the first text according to the specified granularity. The sample training set of the large language model: contains text of at least one character, and the label of each text in the sample training set is: the sentence containing at least one character and the state of the at least one character obtained by splitting the text.

[0110] It can be understood that, for a first text, the higher the granularity of the split, the more shots are obtained by the split, and accordingly, the more shot description texts are generated. Subsequently, the more images are generated for the first text, and the obtained images can reflect the detailed change process of the shot images in the first text. The lower the granularity of the split, the fewer shots are obtained by the split, and accordingly, the fewer shot description texts are generated. Subsequently, the fewer images are generated for the first text, which can avoid generating a large number of similar shot images, reduce the amount of calculation, and improve the efficiency of generating images.

[0111] In this way, the first text is split according to the lens through the large language model, and the splitting method can be set according to actual needs to control the number of generated images and the amount of calculation.

[0112] In some embodiments, based on a grammar-based splitting method in natural language processing, a grammar analyzer (such as a dependency syntax analyzer) may be used to split the first text according to the grammatical structure.

[0113] When the above grammatical structure is used to split the first text, the main sentence corresponding to a shot and the subordinate clause of the main sentence may be split into two independent sentences. That is, compared with splitting according to grammar, splitting by a large language model can combine the semantics, word order, grammar and other aspects of the text to split the text, thereby improving the accuracy of the splitting. In addition, compared with the manual splitting of the first text, splitting by a large language model can improve the efficiency of the splitting.

[0114] For example, the first text A can be split to obtain the following shot description texts of multiple shots:

[0115] Shot 1: Zhang (character 1) walks into the supermarket.

[0116] Shot 2: Zhang (Person 1) stuffs food into his bag.

[0117] Shot 3: Zhang (Person 1) is checking the packaging of the food while the cashier (Person 2) is collecting the money.

[0118] Shot 4: Zhang (Person 1) looks at the truck outside the door.

[0119] With respect to step S104, for each obtained shot description text, a first prompt word corresponding to the shot description text may be obtained based on the shot description text and the to-be-used description text of the character in the shot description text.

[0120] In one implementation, the shot description text and the to-be-used description text of the character in the shot description text may be spliced ​​according to a preset prompt word template to obtain a first prompt word corresponding to the shot description text.

[0121] For example, the shot description text of shot 1 is: Zhang (character 1) walks into the supermarket. Among them, the description text to be used of character 1 is: male, young, short hair. It can be spliced ​​to get "Character 1 [male, young, short hair] walks into the supermarket".

[0122] The description text of shot 2 is: Zhang (character 1) puts food into the bag. Among them, the description text to be used of character 1 is: male, young, short hair. It can be spliced ​​to get "Character 1 [male, young, short hair] puts food into the bag".

[0123] The description text of shot 3 is: Zhang (character 1) is holding food and checking the packaging, and the cashier (character 2) is collecting money. Among them, the description text to be used for character 1 is: male, young, short hair, and the description text to be used for character 2 is: female. It can be spliced ​​to get "Character 1 [male, young, short hair] is holding food and checking the packaging, and character 2 [female] is collecting money".

[0124] The description text of shot 4 is: Zhang (character 1) looks at the truck outside the door. Among them, the description text to be used of character 1 is: male, young, short hair. It can be spliced ​​to get "character 1 [male, young, short hair] looks at the truck outside the door".

[0125] With respect to step S105 , for each shot description text, the first prompt word corresponding to the shot description text may be processed based on the pre-trained first text-to-image model to obtain an image corresponding to the shot description text.

[0126] In some embodiments, the first prompt word corresponding to the shot description text may be input into a pre-trained first text-to-image model to obtain an image generated by the first text-to-image model as the image corresponding to the shot description text.

[0127] In some embodiments, the model parameters of the pre-trained first text image model may also be updated, and the first prompt word corresponding to the shot description text may be input into the new first text image model to obtain an image generated by the new first text image model as the image corresponding to the shot description text. The specific process of updating the model parameters of the first text image model will be described in subsequent embodiments.

[0128] In one implementation, after obtaining the first prompt word corresponding to each shot description text, the initial description text of the character in the first text can be added to the first prompt word. In this way, the character in the generated image can be more consistent with the characteristics of the character in the first text.

[0129] For example, for the shot 1 in the first text A, the shot description text of shot 1 is: Zhang (character 1) walks into the supermarket. Among them, the description text to be used for character 1 is: male, young, short hair. Accordingly, the first prompt word corresponding to the shot description text of shot 1 can be spliced ​​as: "Character 1 [male, young, short hair] walks into the supermarket".

[0130] In addition, for the first text A, the initial description text of character 1 is: male, short hair, white vest, bleached jeans. For character 2, the initial description text of character 2 is: female, ponytail, middle-aged, work clothes.

[0131] The initial description text of the character in the first text is added to the first prompt word, and the first prompt word corresponding to each shot description text is obtained as follows:

[0132] The first prompt word corresponding to the shot description text of shot 1 is: Person 1 [male, young, short hair, white vest, bleached jeans] walks into the supermarket.

[0133] The first prompt word corresponding to the shot description text of shot 2 is: Person 1 [male, young, short hair, white vest, bleached jeans] puts food into the bag.

[0134] The first prompt word corresponding to the shot description text of shot 3 is: Person 1 [male, young, short hair, white vest, bleached jeans] is holding food and checking the packaging, and Person 2 [female, ponytail, middle-aged, work clothes] is collecting money.

[0135] The first prompt word corresponding to the shot description text of shot 4 is: Person 1 [male, young, short hair, white vest, bleached jeans] looks at the truck outside the door.

[0136] Correspondingly, the first prompt words corresponding to the above-mentioned shot description texts may be input into the first text-generated graph to obtain an image generated by the first text-generated graph model as the image corresponding to the shot description text.

[0137] In this way, since the character's description text to be used can reflect the fixed state of the character, the initial description text can also reflect the temporary state of the character in the first text. Therefore, the consistency of the character in the image of each shot description text contained in the generated first text can be further improved, and while ensuring the consistency of the character, the generated character is made to conform to the temporary state of the character in the first text.

[0138] In some embodiments, the model parameters of the pre-trained first text image model can also be updated. Then, the image corresponding to each shot description text is obtained based on the new first text image model. Figure 4 , Figure 4 A fourth flow chart of the image generation method provided by an embodiment of the present invention. Figure 1 On the basis of, after step S102, the method further includes:

[0139] S106: Determine, from a plurality of preset LoRA (low rank adaptive) models, a LoRA model that matches the description text to be used of each character.

[0140] Among them, the preset multiple LoRA models are trained based on the sample images of multiple sample persons, the sample labels of each sample image and the pre-trained second text image model. The sample label of a sample image represents: the characteristics of the sample person contained in the sample image; one LoRA model corresponds to one sample person.

[0141] Step S105 includes:

[0142] S1051: Using the model parameters of the LoRA model corresponding to the character included in the shot description text, the model parameters of the pre-trained first-text-image model are updated to obtain a new first-text-image model.

[0143] S1052: Input the first prompt word corresponding to the shot description text into the new first text-generated image model to obtain an image generated by the new first text-generated image model as the image corresponding to the shot description text.

[0144] In this embodiment, the LoRA model is used to adjust the text graph model (ie, the first text graph model). The first text graph model and the second text graph model are both SD (Stable Diffusion) models.

[0145] The sample label of a sample image indicates the characteristics of the sample person contained in the sample image. For example, if a sample image represents a young man with short hair, the sample label of the sample image may be: male, teenager, short hair; if a sample image represents an old man with a big nose, the sample label of the sample image may be: male, old, big nose; if a sample image represents a middle-aged woman with long hair and big eyes, the sample label of the sample image may be: female, middle-aged, big eyes, long hair.

[0146] Therefore, since a LoRA model is trained based on a sample image of a sample person, a sample label of each sample image, and a second text image model, the trained LoRA model can learn the characteristics of the sample person. Subsequently, after updating the model parameters of the first text image model based on the LoRA model, the characteristics of the person in the image generated by the new first text image model can be consistent with the characteristics of the sample person learned by the LoRA model.

[0147] In addition, the electronic device may also annotate the features of the sample person corresponding to the LoRA model based on the sample label of the sample image of the sample person. For example, the sample labels of each sample image of the sample person may be combined to annotate the LoRA model corresponding to the sample person.

[0148] For example, for the above sample image representing a young man with short hair and glasses, the sample label of the sample image is: male, teenager, short hair. The LoRA model trained based on the sample image can learn the characteristics of the sample character (which can be called character A), so the label of the LoRA model can be determined as: male, teenager, short hair. For the above sample image representing an old man with a big nose, the sample label of the sample image is: male, old, big nose. The LoRA model trained based on the sample image can learn the characteristics of the sample character (which can be called character B), so the label of the LoRA model can be determined as: male, old, big nose. For the above sample image representing a middle-aged woman with long hair and big eyes, the sample label of the sample image is: female, middle-aged, big eyes, long hair. The LoRA model trained based on the sample image can learn the characteristics of the sample character (which can be called character C), so the label of the LoRA model can be determined as: female, middle-aged, big eyes, long hair.

[0149] It is understandable that in order to ensure that the features of each sample person learned by the preset multiple LoRA models can satisfy the features of any person contained in the first text as much as possible, the number of trained LoRA models can be increased by enriching the sample persons. For example, a large number of sample images of sample persons with different features can be obtained to train the LoRA model corresponding to each sample person.

[0150] Correspondingly, the electronic device can determine a LoRA model that matches the description text to be used of each character from a plurality of preset low-rank adaptive LoRA models.

[0151] In one implementation, for each character, the similarity between the description text to be used of the character and the annotations of each LoRA model can be determined. Then, the LoRA model corresponding to the greatest similarity can be used as the LoRA model corresponding to the character object.

[0152] For each LoRA model, the cosine similarity between the to-be-used description text of the character and the annotation of the LoRA model may be calculated.

[0153] Alternatively, the description text to be used of the character, the annotations of each LoRA model, and the fifth prompt word are input into the first large language model, and the similarity between the description text to be used of the character and the annotations of the LoRA model is obtained through the large language model; the fifth prompt word indicates: calculate the similarity between the two input texts.

[0154] Furthermore, after obtaining the first prompt word corresponding to each shot description text, the model parameters of the LoRA model corresponding to the character contained in the shot description text can be used to update the model parameters of the pre-trained first text-born image model to obtain a new first text-born image model.

[0155] Based on the above processing, a low-rank adaptive LoRA model corresponds to a sample character, and multiple LoRA models learn the characteristics of different sample characters respectively. Accordingly, among the preset multiple LoRA models, a LoRA model that matches the description text to be used for each character is determined, that is, for each character in the first text, a sample character that is closest to the characteristics of the character can be determined, and the LoRA model of the sample character is determined as the LoRA model corresponding to the character in the first text. And because a shot description text is used to describe the characters appearing in a shot and the state of the characters appearing, therefore, when an image corresponding to a shot description text is generated by the first text image model, the model parameters of the first text image model are updated based on the LoRA model corresponding to the characters appearing in the shot, and the first prompt word corresponding to the shot description text is input into the new first text image model to obtain an image representing the characters appearing in the shot and the state of the characters appearing, that is, the image corresponding to the shot description text. It can be understood that since the LoRA model corresponding to the same character in different shot description texts is fixed, that is, the LoRA model used when generating images for the same character in different shots is consistent, and since the features of the character learned by a LoRA model are fixed, it can ensure that the features of the character in the images corresponding to different shots generated are consistent, and ensure the continuity and unity of the same character in the different generated images.

[0156] In some embodiments, step 1051 includes:

[0157] Step 1: Calculate the model parameters of the LoRA model corresponding to the characters contained in the shot description text, and fuse the model parameters of the pre-trained first-text image model as the model parameters of the new first-text image model to obtain a new first-text image model.

[0158] In this embodiment, the pre-trained first text-based graph model can be obtained by training a text-based graph model of an initial structure based on sample prompt words and sample images, wherein the sample prompt words are used to describe the image screen to be generated.

[0159] After obtaining the model parameters of the LoRA model corresponding to the character contained in a shot description text, the model parameters of the LoRA model corresponding to the character contained in the shot description text can be calculated, and the fusion result of the model parameters of the pre-trained first text image model can be calculated, and the fusion result is used as the model parameters of the new first text image model. The process of calculating the fusion result will be described in subsequent embodiments.

[0160] Furthermore, the first prompt word corresponding to the shot description text can be input into a new first text-generated image model to obtain an image generated by the new first text-generated image model as the image corresponding to the shot description text. The first text-generated image model includes: a text encoder, an image generator, and a variational autoencoder. The text encoder is used to convert the input text content into a feature vector, the image generator generates image information according to the feature vector output by the text encoder, and the variational autoencoder decodes the image information generated by the image generator to obtain the final image output.

[0161] In one implementation, there is only one character in a shot represented by a shot description text. The above step 1 includes: calculating the model parameters of a pre-trained first text image model and the weighted sum of the model parameters of the LoRA model corresponding to the character contained in the shot description text as the model parameters of a new first text image model, and obtaining a new first text image model.

[0162] In this embodiment, the model parameters of the pre-trained first text image model can be obtained. The model parameters of the pre-trained first text image model and the weighted sum of the model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated. The weight values ​​of the model parameters of the pre-trained first text image model and the weight values ​​of the model parameters of the LoRA model are both pre-set and are between [0,1].

[0163] If there is only one character in a shot represented by a shot description text, the model parameters of the pre-trained first-text image model and the weighted sum of the model parameters of the LoRA model corresponding to the character can be calculated as the model parameters of the new first-text image model.

[0164] Furthermore, the new text-based graph model can generate an image representing the state of the character in the shot according to the first prompt word, as the image corresponding to the shot description text.

[0165] Based on the above processing, when there is only one character in a shot represented by a shot description text, the model parameters of the first text image model can be updated based on the LoRA model corresponding to the character, and the first prompt word corresponding to the shot description text is input into the new first text image model, so that an image representing the character appearing in the shot and the state of the character appearing can be obtained, that is, the image corresponding to the shot description text. It can be understood that since the LoRA model corresponding to the character is fixed, that is, the LoRA model used when generating images for the character in different shots is consistent, and since the features of the character learned by a LoRA model are fixed, it is possible to ensure that the features of the character in the images corresponding to different shots are consistent, and to ensure the continuity and unity of the same character in the different images generated.

[0166] In another implementation, there are multiple characters in a shot represented by a shot description text. The above step 1 includes:

[0167] Step a: for a plurality of characters in a shot represented by a shot description text, determine the image area occupied by each character in the image to be generated.

[0168] Step b: For each image area corresponding to a character, calculate the weighted sum of the part of the model parameters of the pre-trained first text image model corresponding to the image area and the model parameters of the LoRA model corresponding to the character, and obtain the part of the model parameters of the new first text image model corresponding to the image area.

[0169] In this embodiment, when there are multiple characters in a shot represented by a shot description text, the image area occupied by each character in the image to be generated may be preset.

[0170] For example, for the above-mentioned shot 3, there are person 1 and person 2 in the shot. The image to be generated can be divided into two image regions, the left image region and the right image region, and the area of ​​the left image region is consistent with the area of ​​the right image region. The image region occupied by person 1 in the image to be generated is determined to be the left image region, and the image region occupied by person 2 in the image to be generated is determined to be the right image region.

[0171] For the image area occupied by each character, the weighted sum of the model parameters of the pre-trained first text image model corresponding to the image area and the model parameters of the LoRA model corresponding to the character can be calculated to obtain the model parameters of the new first text image model corresponding to the image area. The calculation process can be referred to the above embodiment, which will not be repeated here.

[0172] Then, the first prompt word corresponding to the shot description text is input into the new first text-generated image model. That is, the new first text-generated image model can generate an image representing the state of each character in the shot according to the first prompt word as the image corresponding to the shot description text.

[0173] Based on the above processing, when there are multiple characters in a shot represented by a shot description text, for each character, the part corresponding to the image area occupied by the character in the model parameters of the first text image model can be updated based on the LoRA model corresponding to the character, so that the character features in the image area occupied by the generated image are consistent with the character features learned by the LoRA model corresponding to the character, thereby ensuring that the features of the character in the images corresponding to different shots are consistent, and ensuring the continuity and unity of the same character in the different generated images.

[0174] For example, the first prompt word corresponding to the shot description text of shot 1 is: Person 1 [male, young, short hair, white vest, bleached jeans] walks into the supermarket.

[0175] The first prompt word corresponding to the shot description text of shot 2 is: Person 1 [male, young, short hair, white vest, bleached jeans] puts food into the bag.

[0176] The first prompt word corresponding to the shot description text of shot 3 is: Person 1 [male, young, short hair, white vest, bleached jeans] is holding food and checking the packaging, and Person 2 [female, ponytail, middle-aged, work clothes] is collecting money.

[0177] The first prompt word corresponding to the shot description text of shot 4 is: Person 1 [male, young, short hair, white vest, bleached jeans] looks at the truck outside the door.

[0178] Accordingly, the first prompt words corresponding to the above-mentioned shot description texts can be input into the new first text-generated image to obtain an image generated by the first text-generated image model, see Figure 5(a)-Figure 5(d) , Figure 5(a) is the image corresponding to the lens description text of lens 1. Figure 5(b) is the image corresponding to the lens description text of lens 2. Figure 5(c) is the image corresponding to the lens description text of lens 3. Figure 5(d) is the image corresponding to the lens description text of lens 4.

[0179] In some embodiments, multiple LoRA models are obtained by the following steps:

[0180] Step (1): According to the current model parameters of the second text image model, the model parameters of the initialized LoRA model are obtained.

[0181] Among them, the model parameters of the initialized LoRA model are: the product of the initialized first low-rank matrix and the second low-rank matrix; the number of rows of the first low-rank matrix is ​​the number of rows of the current parameter matrix of the second text-generated image model, and the number of columns is the rank of the current parameter matrix of the second text-generated image model; the number of rows of the second low-rank matrix is ​​the rank of the current parameter matrix of the second text-generated image model, and the number of columns is the number of columns of the current parameter matrix of the second text-generated image model.

[0182] Step (2): According to the model parameters of the initialized LoRA model and the current model parameters of the second text-generated graph model, the adjusted model parameters of the second text-generated graph model are calculated.

[0183] Step (3): for each sample image of each sample person, the sample label of the sample image is input into the second text image model after the model parameters are adjusted, and the difference between the predicted image output by the second text image model after the model parameters are adjusted and the sample image is obtained.

[0184] Step (4): Based on the obtained difference, gradient adjustment is performed on the first low-rank matrix and the second low-rank matrix of the initialized LoRA model to obtain an adjusted LoRA model as the LoRA model of the sample person.

[0185] In an embodiment of the present invention, a first low-rank matrix can be obtained based on a random initialization algorithm, and the initialized second low-rank matrix is ​​set to a zero matrix, so that the original output of the second text graph model is maintained at the beginning of training (the initial offset is 0). For example, the random initialization algorithm can be: a Gaussian initialization algorithm or a uniform initialization algorithm.

[0186] For each sample image of each sample person, the sample label of the sample image is input into the second Wenshengtu model after the model parameters are adjusted, and the difference between the predicted image output by the second Wenshengtu model after the model parameters are adjusted and the sample image is obtained. Then, according to the obtained difference, only the gradient adjustment of the first low-rank matrix and the second low-rank matrix can be performed, and the model parameters of the second Wenshengtu model are guaranteed to remain unchanged. Accordingly, the product of the adjusted first low-rank matrix and the second low-rank matrix can be calculated as the model parameters of the adjusted LoRA model to obtain the LoRA model of the sample person.

[0187] Among them, the model parameters of the second text graph model can be expressed as W m×n Among them, the first low-rank matrix can be expressed as A m×r , the second low-rank matrix can be expressed as B r×n , m represents the number of rows of the parameter matrix corresponding to the model parameters of the second text-generated graph model, n represents the number of columns of the parameter matrix corresponding to the model parameters of the second text-generated graph model, and r represents the rank of the current parameter matrix of the second text-generated graph model. The initialized A×B is a zero matrix.

[0188] Then, the adjusted model parameters of the second text-generated graph model are calculated based on the model parameters of the initialized LoRA model and the current model parameters of the second text-generated graph model. That is, the adjusted model parameters of the second text-generated graph model W′=W+AB.

[0189] Based on the above processing, when training a LoRA model, using sample images of the same sample person for training can enable the obtained LoRA model to learn the characteristics of the sample person. Subsequently, when generating an image through the first text-generated image model, the model parameters of the first text-generated image model can be updated based on the LoRA model corresponding to the person appearing in the shot, and the shot description text can be input as a prompt word into the new first text-generated image model, so that an image representing the person appearing in the shot and the state of the person appearing can be obtained, that is, the image corresponding to the shot description text.

[0190] In some embodiments, the checkpoints of the first and second text image models may be hand-drawn. Thus, the training and image generation process of the LoRA model may be performed by hand-drawn checkpoints, which may reduce image details and improve error tolerance.

[0191] In some embodiments, after obtaining the original text of a work in text form such as a novel or a script, the electronic device may also split the original text to reduce the text content of the obtained first text. Figure 6 , Figure 6 A fifth flow chart of the image generation method provided by the embodiment of the present invention, in Figure 1 On the basis of, step S101 comprises:

[0192] S1011: Acquire a second text containing at least one character.

[0193] S1012: Splitting the second text according to a preset splitting rule to obtain each text containing at least one character as the first text.

[0194] The preset splitting rule includes: splitting the second text according to the chapters of the text, and / or splitting the second text according to the preset number of words in the text.

[0195] In the embodiment of the present invention, the second text may be the original text of a work in text form such as a novel or a script. Alternatively, the second text may be obtained by performing a preprocessing operation on the original text. For example, the preprocessing operation may be a text processing operation such as spelling correction or grammar correction.

[0196] The preset splitting rule is set by the technician according to actual needs. Among them, regular expressions can be used to split the second text according to the preset splitting rule.

[0197] Specifically, the electronic device can split the second text according to the chapters of the text. For example, for the second text of the novel type, splitting the second text according to the chapters of the text means: splitting the second text according to the units of the novel typesetting, such as volume, chapter, section, etc., to obtain the text corresponding to each smallest unit. For the second text of the script type, splitting the second text according to the chapters of the text means: splitting the second text according to the units of the script typesetting, such as act, scene, etc., to obtain the text corresponding to each smallest unit.

[0198] Alternatively, the electronic device may split the second text according to a preset number of words. For example, the preset number of words may be 300 or 500 words, which is not limited here. It is understood that when the second text is not a text such as a novel or a script that has been set according to a preset typesetting unit, that is, when the second text cannot be split according to the unit set by the text content itself, the second text may be split according to the preset number of words.

[0199] Alternatively, the electronic device may split the second text according to the chapters of the text and the preset number of words in the text. For example, for a second text of a novel type, after splitting the second text according to the chapters of the text, if the number of words in a section of the text is too large, the section of the text may be split again according to the preset number of words.

[0200] The electronic device can use each text obtained by splitting the second text as the first text, and if the text contains at least one character, the electronic device can generate an image according to the above steps S102-S105. That is, the electronic device can use each text obtained by splitting the second text as the first text.

[0201] It can be understood that, for the split text that does not contain characters (such as text that explains a concept in a novel, text that describes the environment in a script), the process of generating a corresponding image for the text that does not contain characters through the text-generated graph model is not limited in the embodiments of the present invention.

[0202] In this way, the acquired second text can be split and processed. When the second text is split and processed according to the chapters of the text, since the split text is obtained according to the preset typesetting unit, the logical integrity of the first text can be guaranteed. Subsequently, the quality of the generated image can be improved in the process of generating an image based on the first text. In addition, when the second text is split and processed according to the preset number of words, the number of words in the first text can be controlled according to actual needs, which can reduce the amount of calculation required in the process of generating an image based on the first text.

[0203] Moreover, splitting based on any of the above preset splitting rules can reduce the number of words in the obtained first text. Furthermore, in the subsequent process of processing the first text to generate an image, the amount of calculation can be reduced, and the first text to be generated can be flexibly determined according to actual needs to obtain an image corresponding to the first text.

[0204] See also Figure 7 , Figure 7 A flowchart for generating an image is provided in an embodiment of the present invention. Figure 7 Including:

[0205] S701: Get text.

[0206] That is, the original text is obtained, and the original text may be a work in text form such as a novel or a script.

[0207] S702: Split the text.

[0208] That is, each text containing at least one character obtained by splitting the original text according to the preset splitting rule is used as the first text. The preset splitting rule includes: splitting the second text according to the chapters of the text, and / or splitting the second text according to the preset number of words of the text.

[0209] S703: Split feature summary.

[0210] That is, for each first text, the first text and the third prompt word are input into the second largest language model to obtain an initial description text of each character contained in the first text.

[0211] S704: Biography.

[0212] That is, the initial description texts for the same character in the obtained initial description texts are merged to obtain the description text to be used for the character.

[0213] S705: LoRA (low rank adaptation) model matching.

[0214] That is, for each character, the similarity between the description text to be used of the character and the annotations of each LoRA model can be determined. Then, the LoRA model corresponding to the greatest similarity can be used as the LoRA model corresponding to the character object.

[0215] S706: Split lens.

[0216] That is, the first text and the second prompt word are input into the first large language model to obtain the shot description text of each shot output by the first large language model, wherein the second prompt word indicates: the first text is split according to the shot.

[0217] S707: Description of the lens image.

[0218] That is, the electronic device may combine the shot description text of each shot obtained by splitting with the initial description text of each character contained in the first text as the text describing the shot picture.

[0219] S708: Generate a screen.

[0220] That is, for each shot description text, the model parameters of the first text-based image model are updated based on the LoRA model corresponding to the character contained in the shot description text, and the first prompt word corresponding to the shot description text is input into the new first text-based image model to obtain an image generated by the new first text-based image model as the image corresponding to the shot description text.

[0221] Based on the same inventive concept, an embodiment of the present invention provides an image generating device, see Figure 8 , Figure 8 This is a structural diagram of an image generation device provided by an embodiment of the present invention. The device includes:

[0222] A first text acquisition module 801 is used to acquire a first text, wherein the first text includes a description text of at least one character to be used;

[0223] A feature acquisition module 802 is used to extract the description text to be used of each character contained in the first text;

[0224] The shot splitting module 803 is used to split the first text according to the shots to obtain shot description texts of each shot; wherein a shot description text represents: a character in the shot represented by the shot description text, and a state of the character in the shot;

[0225] A prompt word generating module 804 is used to obtain a first prompt word corresponding to each shot description text based on each shot description text and the to-be-used description text of the character in the shot description text;

[0226] The image generation module 805 is used to process the first prompt word corresponding to the shot description text based on the pre-trained first text-image model to obtain the image corresponding to the shot description text.

[0227] In some embodiments, the apparatus further comprises:

[0228] A matching module is used to determine, after extracting the to-be-used description text of each character contained in the first text, a LoRA model that matches the to-be-used description text of each character from a plurality of preset low-rank adaptive LoRA models; wherein the plurality of preset LoRA models are trained based on sample images of a plurality of sample characters, sample labels of each sample image, and a pre-trained second text image model, and a sample label of a sample image represents: a feature of a sample character contained in the sample image; a LoRA model corresponds to a sample character;

[0229] The image generation module 805 includes:

[0230] An updating submodule is used to update the model parameters of the pre-trained first-text image model using the model parameters of the LoRA model corresponding to the characters contained in the shot description text to obtain a new first-text image model;

[0231] The image generation submodule is used to input the first prompt word corresponding to the lens description text into the new first text-generated image model to obtain an image generated by the new first text-generated image model as the image corresponding to the lens description text.

[0232] In some embodiments, the lens splitting module 803 is specifically used to:

[0233] The first text and the second prompt word are input into a first large language model to obtain a shot description text of each shot output by the first large language model; wherein the second prompt word indicates: splitting the first text according to the shot.

[0234] In some embodiments, the first text acquisition module 801 is specifically used to:

[0235] Acquire a second text, wherein the second text includes a description text of at least one character to be used;

[0236] Each text containing at least one character obtained by splitting the second text according to a preset splitting rule is used as a first text; wherein the preset splitting rule includes: splitting the second text according to the chapters of the text, and / or splitting the second text according to a preset number of words in the text.

[0237] In some embodiments, the feature acquisition module 802 includes:

[0238] The feature acquisition submodule is used to input the first text and the third prompt word into the second largest language model to obtain the description text to be used for each character contained in the first text; wherein the third prompt word indicates: feature extraction is performed on each character contained in the first text.

[0239] In some embodiments, the number of the first texts is greater than 1;

[0240] The feature acquisition submodule is specifically used for:

[0241] For each first text, the first text and the third prompt word are input into the second largest language model to obtain an initial description text of each character contained in the first text;

[0242] The initial description texts for the same character in the obtained initial description texts are merged to obtain the description text to be used for the character.

[0243] In some embodiments, the updating submodule includes:

[0244] The fusion unit is used to calculate the model parameters of the LoRA model corresponding to the characters contained in the shot description text, and the fusion result of the model parameters of the pre-trained first-text image model as the model parameters of the new first-text image model, so as to obtain the new first-text image model.

[0245] In some embodiments, there is only one character in a shot of a shot description text representation;

[0246] The fusion unit is specifically used for:

[0247] The model parameters of the pre-trained first-text image model and the weighted sum of the model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated as the model parameters of the new first-text image model to obtain the new first-text image model.

[0248] In some embodiments, there are multiple characters in a shot represented by a shot description text;

[0249] The fusion unit is specifically used for:

[0250] For multiple characters in a shot represented by a shot description text, determine an image area occupied by each character in an image to be generated;

[0251] For each image area corresponding to a person, the weighted sum of the part of the model parameters of the pre-trained first-text image model corresponding to the image area and the model parameters of the LoRA model corresponding to the person is calculated to obtain the part of the model parameters of the new first-text image model corresponding to the image area.

[0252] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0253] Memory 903, used for storing computer programs;

[0254] The processor 901 is used to implement the steps of any image generation method in the above embodiments when executing the program stored in the memory 903.

[0255] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0256] The communication interface is used for communication between the above terminal and other devices.

[0257] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0258] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0259] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the image generating method described in any one of the above embodiments is implemented.

[0260] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the image generating method described in any one of the above embodiments.

[0261] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0262] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0263] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0264] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating an image, It is characterized in that The method comprises: Acquire a first text, wherein the first text includes a description text of at least one character to be used; extracting the to-be-used description text of each character contained in the first text; The first text is split according to the shots to obtain shot description texts of the shots; wherein a shot description text represents: a character in the shot represented by the shot description text, and a state of the character in the shot; Based on each shot description text and the to-be-used description text of the character in the shot description text, a first prompt word corresponding to the shot description text is obtained; Based on the pre-trained first text-to-image model, the first prompt word corresponding to the shot description text is processed to obtain an image corresponding to the shot description text.

2. The method according to claim 1, It is characterized in that After extracting the to-be-used description text of each character contained in the first text, the method further includes: Determine a LoRA model that matches the description text to be used of each person from a plurality of preset low-rank adaptive LoRA models; wherein the plurality of preset LoRA models are trained based on sample images of a plurality of sample persons, sample labels of each sample image, and a pre-trained second text image model, and a sample label of a sample image represents: a feature of a sample person contained in the sample image; a LoRA model corresponds to a sample person; The first prompt word corresponding to the shot description text is processed based on the pre-trained first text-image model to obtain an image corresponding to the shot description text, including: Using the model parameters of the LoRA model corresponding to the person included in the shot description text, the model parameters of the pre-trained first-text image model are updated to obtain a new first-text image model; The first prompt word corresponding to the shot description text is input into the new first text-generated image model to obtain an image generated by the new first text-generated image model as the image corresponding to the shot description text.

3. The method according to claim 1, It is characterized in that The first text is split according to the shots to obtain the shot description text of each shot, including: The first text and the second prompt word are input into a first large language model to obtain a shot description text of each shot output by the first large language model; wherein the second prompt word indicates: splitting the first text according to the shot.

4. The method according to claim 1, It is characterized in that The obtaining of the first text comprises: Acquire a second text, wherein the second text includes a description text of at least one character to be used; Each text containing at least one character obtained by splitting the second text according to a preset splitting rule is used as a first text; wherein the preset splitting rule includes: splitting the second text according to the chapters of the text, and / or splitting the second text according to a preset number of words in the text.

5. The method according to claim 1, It is characterized in that The extracting the to-be-used description text of each character contained in the first text includes: The first text and the third prompt word are input into the second largest language model to obtain the description text to be used for each character contained in the first text; wherein the third prompt word indicates: feature extraction is performed on each character contained in the first text.

6. The method according to claim 5, It is characterized in that The number of the first texts is greater than 1; The first text and the third prompt word are input into the second largest language model to obtain the description text to be used of each character contained in the first text, including: For each first text, the first text and the third prompt word are input into the second largest language model to obtain an initial description text of each character contained in the first text; The initial description texts for the same character in the obtained initial description texts are merged to obtain the description text to be used for the character.

7. The method according to claim 2, It is characterized in that The method uses the model parameters of the LoRA model corresponding to the character contained in the shot description text to update the model parameters of the pre-trained first text image model to obtain a new first text image model, including: The model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated, and the fusion results of the model parameters of the pre-trained first-text image model are used as the model parameters of the new first-text image model to obtain the new first-text image model.

8. The method according to claim 7, It is characterized in that There is only one character in a shot describing a textual representation; The step of calculating the model parameters of the LoRA model corresponding to the characters included in the shot description text and fusing the model parameters of the pre-trained first text image model as the model parameters of the new first text image model to obtain the new first text image model includes: The model parameters of the pre-trained first-text image model and the weighted sum of the model parameters of the LoRA model corresponding to the characters included in the shot description text are calculated as the model parameters of the new first-text image model to obtain the new first-text image model.

9. The method according to claim 7, It is characterized in that There are multiple characters in a shot that describes a textual representation; The step of calculating the model parameters of the LoRA model corresponding to the characters included in the shot description text and fusing the model parameters of the pre-trained first text image model as the model parameters of the new first text image model to obtain the new first text image model includes: For multiple characters in a shot represented by a shot description text, determine an image area occupied by each character in an image to be generated; For each image area corresponding to a person, the weighted sum of the part of the model parameters of the pre-trained first-text image model corresponding to the image area and the model parameters of the LoRA model corresponding to the person is calculated to obtain the part of the model parameters of the new first-text image model corresponding to the image area.

10. An image generating device, It is characterized in that The device comprises: A first text acquisition module, used to acquire a first text, wherein the first text includes a description text of at least one character to be used; A feature acquisition module, used for extracting the description text to be used of each character contained in the first text; A shot splitting module is used to split the first text according to the shots to obtain shot description texts of each shot; wherein a shot description text represents: a character in the shot represented by the shot description text, and a state of the character in the shot; A prompt word generation module, used for obtaining a first prompt word corresponding to each shot description text based on each shot description text and the to-be-used description text of the character in the shot description text; The image generation module is used to process the first prompt word corresponding to the shot description text based on the pre-trained first text-to-image model to obtain the image corresponding to the shot description text.

11. An electronic device, It is characterized in that It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.