Cartoon video generation method and device and storage medium

Abstract

The invention relates to the field of video generation, and discloses a cartoon video generation method, cartoon video generation equipment and a storage medium. The method comprises the steps of receiving a text making instruction; judging whether the text making instruction is an instruction for making a specified neural network or not; if yes, inputting the theme content setting, the story background setting and the role information setting into a preset AI text generation model to generate a story text; judging whether the story text limits the image to maintain the original text; when the story text is not limited, performing scene division and retouching processing on the story text to generate scene description; when the story text is limited, performing division processing on the original story text to generate scene description; performing drawing processing on the scene description to generate image data; dubbing the story text to generate dubbing data; and combining the dubbing data and the image data to generate an audio cartoon video. In the embodiment of the invention, the neural network is utilized to draw images and make dubbles for the story text, and the picture book cartoon video is efficiently generated at low cost.

Description

Technical Field

[0001] The present invention relates to the field of video generation, and in particular to a method, device and storage medium for generating a comic video. Background Art

[0002] With the development of economy and society and the advancement of technology, images are increasingly widely used in advertising, film and television publicity, education and training, social sharing and other fields. Compared with text content, image content has higher visual appeal and memory effect, can quickly attract people's attention, has a wider cross-cultural communication ability, and has stronger sharing and interactivity in social media.

[0003] The traditional image production process is costly and cannot generate a large number of images in a short period of time. It takes a longer time to produce them manually frame by frame. It is also impossible to adjust the details of the combined picture, change the color, etc. in batches, and it is necessary to start drawing and modifying independently. There are also limitations in style imitation. It is impossible to imitate and create a variety of different artistic styles, and it is difficult to integrate different abstract arts. In addition to the high cost of drawing time, the story rewriting, storyboard design, and dialogue dubbing processes in the traditional picture book and comic production process also require a lot of manpower and material resources, and each link requires a lot of time to make adjustments. The process of adjusting according to the finished product effect is also very costly, and resources need to be invested again to go through the workflow again. It is difficult to quickly compare the finished product effects of different production ideas, and it is difficult to quickly modify and iterate based on the finished product effect.

[0004] Therefore, in view of the high production cost of current comic picture books, a technology is needed to solve the current problem. Summary of the invention

[0005] The main purpose of the present invention is to solve the technical problem of high production cost of comic picture books.

[0006] A first aspect of the present invention provides a method for generating a comic video, the method comprising:

[0007] Receive text production instructions;

[0008] Determining whether the text creation instruction is an instruction created by a specified neural network;

[0009] When the instruction is made for a designated neural network, the theme content setting, the story background setting, the character information setting, and the image prompt words are received, and the theme content setting, the story background setting, and the character information setting are input into a preset AI text generation model to generate a story text;

[0010] Determining whether the story text restricts the image to maintain the original text;

[0011] When the image is not restricted and the original text is maintained, the story text is divided into scenes and polished to generate scene descriptions;

[0012] When the image is restricted to maintain the original text, the original story text is divided and processed to generate a scene description;

[0013] According to the preset large language model and the image prompt words, the scene description is rendered to generate image data;

[0014] According to a preset AI dubbing generation model, dubbing is performed on the story text to generate dubbing data;

[0015] The dubbing data and the image data are combined to generate a comic video with sound.

[0016] Optionally, in a first implementation of the first aspect of the present invention, the step of rendering the scene description according to the preset large language model and the image prompt word to generate image data includes:

[0017] Vectorizing the scene description to obtain a target vector, and vectorizing the image prompt words to obtain an embedding vector;

[0018] The target vector and the embedding vector are inputted into a preset image generation model to generate image data.

[0019] Optionally, in a second implementation of the first aspect of the present invention, the embedding vector includes: ordered t embedding sub-vectors, where t is a positive integer, and the inputting the target vector and the embedding vector into a preset image generation model together to generate image data includes:

[0020] Generate a preliminary image vector according to preset noise parameters;

[0021] The first embedding sub-vector and the target vector are used to adjust the preliminary image vector to generate a first modified vector;

[0022] Generate a first reference vector based on the first modification vector and the target vector;

[0023] According to the i-th reference vector and the i+1-th embedded sub-vector, the i-th modified vector is adjusted to generate the i+1-th modified vector, where i=1, 2, ..., t-1;

[0024] Generate an i+1th reference vector based on the i+1th modification vector and the target vector;

[0025] Image data is generated based on the tth reference vector.

[0026] Optionally, in a third implementation manner of the first aspect of the present invention, adjusting the i-th modification vector according to the i-th reference vector and the i+1-th embedded sub-vector to generate the i+1-th modification vector includes:

[0027]

[0028] Among them, L(x i+1 , e i，opt , θ) is the loss function used to determine the generation of the i+1th modified vector, ∈ is the noise parameter, X i+1 is the input i-th reference vector and i+1-th embedded sub-vector, θ is the built-in parameter of the image generation model, and f θ (x i+1 , i+1, e i，opt ) is the generation function of the image generation model,

[0029] E i+1，ε is the expected value, e i，opt is the i-th modified vector, It is the L2 norm operation.

[0030] Optionally, in a fourth implementation manner of the first aspect of the present invention, generating an i+1th reference vector based on the i+1th modification vector and the target vector includes:

[0031] e i+1，ex =η*e tgt +(1-η)*e i+1，opt ,

[0032] Among them, e i+1，opt Modify the vector for the i+1th time, e tgt is the target vector, e i+1，ex is the i+1th reference vector, and η is the weight coefficient.

[0033] Optionally, in a fifth implementation of the first aspect of the present invention, the performing scene division and polishing processing on the story text to generate scene descriptions includes:

[0034] Performing scene semantic segmentation on the story text to obtain N semantic scene texts, where N is a positive integer;

[0035] The N semantic scene texts are polished to obtain N scene polished texts, and the N scene polished texts are determined as scene descriptions.

[0036] Optionally, in a sixth implementation of the first aspect of the present invention, performing dubbing processing on the story text according to a preset AI dubbing generation model to generate dubbing data includes:

[0037] Determine whether the story text restricts dubbing to maintain the original text;

[0038] When dubbing is not restricted to maintain the original text, dubbing and polishing are performed on the story text to generate a polished text, and dubbing data is generated based on the polished text;

[0039] When dubbing is restricted to maintain the original text, dubbing data is generated based on the story text.

[0040] Optionally, in a seventh implementation manner of the first aspect of the present invention, the image prompt words include: image style prompt words, additional content prompt words, image quality requirement prompt words, and negative prompt words, and the drawing process of the scene description according to the preset large language model and the image prompt words to generate image data includes:

[0041] The image style prompt word, the additional content prompt word, the image quality requirement prompt word, and the negative prompt word are input into a preset large language model for guidance, and the scene description is rendered to generate image data.

[0042] A second aspect of the present invention provides a comic video generation device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected via lines; the at least one processor calls the instructions in the memory so that the comic video generation device executes the above-mentioned comic video generation method.

[0043] A third aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned method for generating a comic video.

[0044] In an embodiment of the present invention, story text is generated by a neural network, which reduces the manual burden as much as possible, can be quickly and repeatedly produced in batches, and different schemes can be quickly tested to reduce the cost of trial and error. Using a large language model to control the text-to-image model, there is no need to manually produce prompt words one by one. The prompt words of the image model can be automatically generated according to the story content, which is more efficient. Restricting the original text or not restricting the original text to control image and audio generation increases the flexibility of comic picture book video production. Applying neural networks to the scenarios of story text generation, image generation, and audio generation realizes flexible and low-cost generation of comic videos, which can be quickly modified and iterated according to the finished product effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Schematic diagram of an embodiment of a method for generating a comic video in an embodiment of the present invention;

[0046] Figure 2107 is a schematic diagram of a specific embodiment of the method for generating a comic video in an embodiment of the present invention;

[0047] Figure 3 1072 is a schematic diagram of a specific embodiment of the method for generating a comic video in an embodiment of the present invention;

[0048] Figure 4 1 is a schematic diagram of a specific embodiment of step 108 of the method for generating a comic video in an embodiment of the present invention;

[0049] Figure 5 The figure is a schematic diagram of an embodiment of a cartoon video generating device in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention provide a method, device and storage medium for generating a comic video.

[0051] The embodiments disclosed in the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments disclosed in the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments disclosed in the present invention are only for exemplary purposes and are not intended to limit the scope of protection disclosed in the present invention.

[0052] In the description of the embodiments disclosed in the present invention, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0053] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the method for generating a comic video in an embodiment of the present invention includes:

[0054] 101. Receive text production instructions;

[0055] In this embodiment, a text production instruction is received, which is an instruction for producing a text story. The story text can be generated by one-click AI after setting the theme, and it also supports adjusting the story background, character information, story outline and other details or manually writing and importing.

[0056] 102. Determine whether the text creation instruction is an instruction created by a designated neural network;

[0057] In this embodiment, the source of the story text can be imported from outside or generated by a neural network. When processing a text production instruction, it is first determined whether it is a specified neural network production. When it is determined that it is an instruction that requires neural network production, the neural network model needs to receive the specified text generation restrictions so that the user's requirements for story text generation can be met.

[0058] 103. When the instruction is made for a designated neural network, the theme content setting, the story background setting, the character information setting, and the image prompt word are received, and the theme content setting, the story background setting, and the character information setting are input into a preset AI text generation model to generate a story text;

[0059] In this embodiment, when it is determined that the text production instruction is an instruction that needs to be produced by a neural network, it is necessary to receive user information about theme content settings, story background settings, character information settings, etc., so as to generate story text for the story file based on the theme content settings, story background settings, and character information settings. In addition, it is also necessary to input related image prompt words at the same time, so that after the story text is completed, the image prompt words corresponding to the story text can be generated based on the image prompt words.

[0060] 104. Determine whether the story text restricts the image to maintain the original text;

[0061] In this embodiment, it is necessary to analyze the settings of the story text to see whether there are settings that restrict the image to maintain the original text. When setting the story text, you can set the following settings: 1. Set tasks to divide the storyline: "Please divide the story by scene, if necessary..., when dividing, it is necessary / not necessary to maintain the original text unchanged, each scene paragraph cannot exceed xxx words / can be divided into xxx paragraphs at most."

[0062] 2. Perform a secondary analysis of each scene and generate a description of the scene content: "Please help me analyze each story scene. Now I need to add an illustration to them and generate a description of the image content. It should include a description of the main content, a description of the scenery and objects... The description should use short words, such as... The description length of each scene should not exceed xxx length..."

[0063] 3. Specify the output data structure to facilitate further processing: "Use json format to output, its structure is shown in xxx, where the xxx segment stores xxx information..."

[0064] 105. When the image is not restricted and the original text is maintained, the story text is divided into scenes and polished to generate scene descriptions;

[0065] In this embodiment, when it is set that the image generation process does not need to maintain the original text, the story text can be first divided and polished based on the scenes to generate the polished overall scene descriptions of different story scenes.

[0066] Specifically, step 105 includes the following specific implementation methods:

[0067] 1051. Perform scene semantic segmentation on the story text to obtain N semantic scene texts, where N is a positive integer;

[0068] 1052. Polish the N semantic scene texts to obtain N scene polished texts, and determine the N scene polished texts as scene descriptions.

[0069] In steps 1051-1052, the story text is first split into multiple semantic scene texts based on semantics, and each semantic scene text is polished in turn, and restrictions are placed on the main description, background description, description method, and description length of each semantic scene text to obtain multiple scene polished texts, and the multiple scene polished texts are determined as scene descriptions.

[0070] 106. When the image is restricted to maintain the original text, the original story text is divided and processed to generate a scene description;

[0071] In this embodiment, when the story text is restricted to the original text, there is no need to polish the original text, and the story text is directly divided based on the scene to obtain the scene description.

[0072] 107. According to the preset large language model and the image prompt words, the scene description is rendered to generate image data;

[0073] In this embodiment, the image prompt word will limit the style of the image to use the corresponding style prompt word. The image prompt word can also be a prompt word for the image content, and the description of the image quality accuracy can be added to limit the quality of the generated image. The image prompt word can also set a negative prompt word to prevent the text from generating specific content during the image generation process.

[0074] See also Figure 2 , Figure 2 This is a specific embodiment of step 107 of the method for generating a comic video in an embodiment of the present invention, and step 107 includes the following specific implementation methods:

[0075] 1071. Vectorize the scene description to obtain a target vector, and vectorize the image prompt word to obtain an embedding vector;

[0076] 1072. Input the target vector and the embedding vector into a preset image generation model to generate image data.

[0077] In steps 1071-1072, the text describing the scene is first vectorized to obtain a target vector, and then the image prompt words are vectorized to obtain an embedded vector.

[0078] The target vector and the embedding vector are used as the basis for guiding the image generation model to generate images. The target vector and the embedding vector are input into the image generation model together to obtain image data.

[0079] See also Figure 3 , Figure 3 This is a specific embodiment of step 1072 of the method for generating a comic video in an embodiment of the present invention. The embedding vector includes: ordered t embedding sub-vectors, t is a positive integer, and step 1072 includes the following specific implementation methods:

[0080] 10721. Generate a preliminary image vector according to preset noise parameters;

[0081] 10722. The first embedded sub-vector and the target vector are used to adjust the preliminary image vector to generate a first modified vector;

[0082] 10723. Generate a first reference vector based on the first modification vector and the target vector;

[0083] 10724. Adjust the i-th modification vector according to the i-th reference vector and the i+1-th embedded sub-vector to generate the i+1-th modification vector, where i=1, 2, ..., t-1;

[0084] 10725. Generate an i+1th reference vector based on the i+1th modification vector and the target vector;

[0085] 10726. Generate image data according to the tth reference vector.

[0086] In steps 10721-1726, in the process of generating an image, image data can be generated by a joint combination of a diffusion model, a target vector, and an embedding vector.

[0087] First, a preliminary image vector is generated by the extended model based on the noise parameter ∈.

[0088] Since the embedding vector is converted from the image prompt words, the embedding vector is composed of multiple embedding sub-vectors converted from restrictive words.

[0089] According to the order of the embedded sub-vectors, the preliminary image vector is adjusted by the first sorted embedded sub-vector and the target vector to generate the first modified vector.

[0090] The first modified vector and the target vector are combined and adjusted to obtain the first reference vector.

[0091] Then, the first reference vector and the second sorted embedded sub-vector are used as the basis to adjust the first modified vector to obtain the second modified vector. Similarly, the second modified vector and the target vector are combined and adjusted to obtain the second reference vector.

[0092] Based on the above logic, the second reference vector and the third sorted embedded sub-vector are used as the basis to adjust the second modified vector to obtain the third modified vector. Similarly, the third modified vector and the target vector are combined and adjusted to obtain the third reference vector.

[0093] All embedded sub-vectors are adjusted to the modified vector in sequence to obtain the final image data.

[0094] Specifically, step 10724 includes the following specific implementation methods:

[0095]

[0096] Among them, L(x i+1 , e i，opt , θ) is the loss function used to determine the generation of the i+1th modified vector, ∈ is the noise parameter, X i+1 is the input i-th reference vector and i+1-th embedded sub-vector, θ is the built-in parameter of the image generation model, and f θ (x i+1 , i+1, e i，opt ) is the generation function of the image generation model, E i+1，ε is the expected value, e i，opt is the i-th modified vector, It is the L2 norm operation.

[0097] In this embodiment, X i+1 Take the input i-th reference vector and the i+1-th embedded sub-vector as the input vector and adjust e i，opt Modify the vector for the i-th time until L(x i+1 , e i，opt , θ) is an adjustment to achieve the convergence of the loss function to a reasonable range, and the adjustment is repeated. In addition, the image generation model itself has θ built-in parameters and noise parameters ∈, and the output of the adjusted e after each adjustment is performed i+1，opt Modify the vector for the i+1th element.

[0098] Specifically, step 10725 includes the following specific implementation methods:

[0099] e i+1，ex =η*e tgt +(1-η)*e i+1，opt,

[0100] Among them, e i+1，opt Modify the vector for the i+1th time, e tgt is the target vector, e i+1，ex is the i+1th reference vector, and η is the weight coefficient.

[0101] In this embodiment, the weight coefficient η is used to perform weighted adjustment on each generated modification vector to obtain the i+1th reference vector e i+1，ex , cyclically adjust until the last reference vector, and generate image data based on the last reference vector.

[0102] In one embodiment, the image prompt words include: image style prompt words, additional content prompt words, image quality requirement prompt words, and negative prompt words. Step 107 includes the following specific implementation methods:

[0103] 107X, input the image style prompt word, the additional content prompt word, the image quality requirement prompt word, and the negative prompt word into a preset large language model for guidance, perform drawing processing on the scene description, and generate image data.

[0104] In step 107X, image prompt words are words that restrict the drawing process. Through the restrictions of image style prompt words, additional content prompt words, image quality requirement prompt words, and negative prompt words, the generated scene image meets the needs of the user.

[0105] 108. Perform dubbing processing on the story text according to a preset AI dubbing generation model to generate dubbing data;

[0106] In this embodiment, an AI dubbing generation model is used to perform dubbing processing on each sentence of the story text to obtain the required dubbing data.

[0107] See also Figure 4 , Figure 4 This is a specific embodiment of step 108 of the method for generating a comic video in an embodiment of the present invention, and step 108 includes the following specific implementation methods:

[0108] 1081. Determine whether the story text restricts dubbing to maintain the original text;

[0109] 1082. When dubbing is not restricted to maintain the original text, dubbing and polishing are performed on the story text to generate a polished text, and dubbing data is generated based on the polished text;

[0110] 1083. When the dubbing is restricted to maintain the original text, dubbing data is generated based on the story text.

[0111] In steps 1081-1083, it is first determined whether the story text has a setting that limits dubbing to maintain the original text. If there is a setting that limits dubbing to maintain the original text, the boundaries of scene division are made based on the story text, and dubbing data is generated so that it can be combined with image data through scenes.

[0112] If there is no setting to maintain the original text with dubbing, the story text is first divided into scenes, and then the divided text data is polished, and the dubbing data is generated based on the polished text.

[0113] The text corresponding to each scene is dubbed by AI separately, and you can choose specific dubbing features such as gender, age, different timbres, etc.

[0114] 109. Combine the dubbing data and the image data to generate a comic video with sound.

[0115] In this embodiment, based on the matching relationship of each scene, the dubbing data and image data are combined, and the corresponding dubbing data is played on the image corresponding to each scene, and the dubbing data corresponding to the next scene is played on the image of the next scene until all scenes of the entire story text are assembled to generate an audio comic video.

[0116] In an embodiment of the present invention, story text is generated by a neural network, which reduces the manual burden as much as possible, can be quickly and repeatedly produced in batches, and different schemes can be quickly tested to reduce the cost of trial and error. Using a large language model to control the text-to-image model, there is no need to manually produce prompt words one by one. The prompt words of the image model can be automatically generated according to the story content, which is more efficient. Restricting the original text or not restricting the original text to control image and audio generation increases the flexibility of comic picture book video production. Applying neural networks to the scenarios of story text generation, image generation, and audio generation realizes flexible and low-cost generation of comic videos, which can be quickly modified and iterated according to the finished product effect.

[0117] Figure 51 is a schematic diagram of the structure of a comic video generation device provided by an embodiment of the present invention. The comic video generation device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the comic video generation device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the comic video generation device 500.

[0118] The comic video-based generating device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 5 The structure of the comic video generation device shown does not constitute a limitation on the comic video generation device, and may include more or less components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0119] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the method for generating a comic video.

[0120] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0122] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating a comic video, characterized in that: Includes steps: Receive text production instructions; Determining whether the text creation instruction is an instruction created by a specified neural network; When the instruction is made for a designated neural network, the theme content setting, the story background setting, the character information setting, and the image prompt words are received, and the theme content setting, the story background setting, and the character information setting are input into a preset AI text generation model to generate a story text; Determining whether the story text restricts the image to maintain the original text; When the image is not restricted and the original text is maintained, the story text is divided into scenes and polished to generate scene descriptions; When the image is restricted to maintain the original text, the original story text is divided and processed to generate a scene description; According to the preset large language model and the image prompt words, the scene description is rendered to generate image data; According to a preset AI dubbing generation model, dubbing is performed on the story text to generate dubbing data; The dubbing data and the image data are combined to generate a comic video with sound.

2. The method for generating a comic video according to claim 1, characterized in that: The step of rendering the scene description according to the preset large language model and the image prompt words to generate image data comprises: Vectorizing the scene description to obtain a target vector, and vectorizing the image prompt words to obtain an embedding vector; The target vector and the embedding vector are inputted together into a preset image generation model to generate image data.

3. The method for generating a comic video according to claim 2, characterized in that: The embedding vector includes: ordered t embedding sub-vectors, t is a positive integer, and the target vector and the embedding vector are inputted into a preset image generation model to generate image data, including: Generate a preliminary image vector according to preset noise parameters; The first embedding sub-vector and the target vector are used to adjust the preliminary image vector to generate a first modified vector; Generate a first reference vector based on the first modification vector and the target vector; According to the i-th reference vector and the i+1-th embedded sub-vector, the i-th modified vector is adjusted to generate the i+1-th modified vector, where i=1, 2, ..., t-1; Generate an i+1th reference vector based on the i+1th modification vector and the target vector; Image data is generated based on the tth reference vector.

4. The method for generating a comic video according to claim 3, characterized in that: The step of adjusting the i-th modified vector according to the i-th reference vector and the i+1-th embedded sub-vector to generate the i+1-th modified vector comprises: Among them, L(x i+1 , e o，ppt , θ) is the loss function used to determine the generation of the i+1th modified vector, ∈ is the noise parameter, X i+1 is the input i-th reference vector and i+1-th embedded sub-vector, θ is the built-in parameter of the image generation model, and f θ (x i+1 , i+1, e i，opt ) is the generation function of the image generation model, E i+1，ε is the expected value, e i，opt is the i-th modified vector, It is the L2 norm operation.

5. The method for generating a comic video according to claim 4, characterized in that: The generating the i+1th reference vector based on the i+1th modification vector and the target vector comprises: And i+1，ex =η*e tgt +(1-η)*e i+1，opt , Among them, e i+1，opt Modify the vector for the i+1th time, e tgt is the target vector, e i+1，ex is the i+1th reference vector, and η is the weight coefficient.

6. The method for generating a comic video according to claim 1, characterized in that: The step of dividing and polishing the story text to generate scene descriptions includes: Performing scene semantic segmentation on the story text to obtain N semantic scene texts, where N is a positive integer; The N semantic scene texts are polished to obtain N scene polished texts, and the N scene polished texts are determined as scene descriptions.

7. The method for generating a comic video according to claim 1, characterized in that: The dubbing process of the story text is performed according to the preset AI dubbing generation model to generate dubbing data, including: Determine whether the story text restricts dubbing to maintain the original text; When dubbing is not restricted to maintain the original text, dubbing and polishing are performed on the story text to generate a polished text, and dubbing data is generated based on the polished text; When dubbing is restricted to maintain the original text, dubbing data is generated based on the story text.

8. The method for generating a comic video according to claim 1, characterized in that: The image prompt words include: image style prompt words, additional content prompt words, image quality requirement prompt words, and negative prompt words. The scene description is drawn based on the preset large language model and the image prompt words to generate image data, including: The image style prompt word, the additional content prompt word, the image quality requirement prompt word, and the negative prompt word are input into a preset large language model for guidance, and the scene description is rendered to generate image data.

9. A device for generating comic videos, characterized in that: The comic video generation device comprises: a memory and at least one processor, the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor calls the instructions in the memory so that the comic video generation device executes the comic video generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for generating a comic video according to any one of claims 1 to 8 is implemented.

Images