Cartoon image generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of image processing, and provides a cartoon image generation method and device, equipment and a storage medium, and the method comprises the steps: obtaining an initial role image corresponding to each role in a novel text; based on a first language sub-model of a preset language model, determining a split text corresponding to at least one split in the novel text; based on a second language sub-model of the language model, according to the split text, determining a split prompt word, a role prompt word and a role distribution position corresponding to the split; based on a first image sub-model of a preset text graph model, according to the initial role image, a role prompt word corresponding to the sub-mirror and a role distribution position, determining a posture of a role in the sub-mirror; and based on a second image sub-model of the text graph model, according to the initial role image corresponding to each role in the sub-mirror, the posture of the role and the sub-mirror cue word, determining a sub-mirror image corresponding to the sub-mirror. The role and the text description in the split image generated through the preset language model and the text graph model are consistent.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a comic image generation method, device, equipment and storage medium. Background Art

[0002] The traditional comic creation process generally includes many and complex steps, such as screenwriting, character design, storyboard scripts, draft drawing, and coloring. Therefore, the production of a comic generally requires a large number of staff to work together. In recent years, with the breakthroughs of artificial intelligence technology in the fields of natural language processing and computer vision, large language models and image generation technology have continued to advance, making the conversion of novel content into comic images practical, greatly improving the efficiency of comic creation and the visual presentation of novel content.

[0003] The large language model performs well in understanding and generating natural language text. It can analyze the complex plot content, character dialogues, and narrative style in novels, and convert this information into storyboard descriptions suitable for comics and prompt words for the text-based image model. It does not require significant human intervention, reducing the reliance on manual drawing in traditional comic creation.

[0004] However, when generating images including multiple characters, the related technology still faces the problem that the characters in the image are inconsistent with the text description. Summary of the invention

[0005] The main purpose of the present application is to provide a comic image generation method, device, equipment and storage medium, in which the characters and text descriptions in the storyboard images generated by a preset language model and a text-based image model are consistent.

[0006] In a first aspect, the present application provides a comic image generation method, comprising:

[0007] Obtaining initial character images corresponding to each character in the novel text;

[0008] Determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model;

[0009] Based on the second language sub-model of the language model, and according to the storyboard text, determining the storyboard prompt words, the role prompt words and the role distribution position of each of the roles corresponding to the storyboard;

[0010] Based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position, determine the posture of the character in the storyboard;

[0011] Based on the second image sub-model of the Wensheng graph model, the storyboard image corresponding to the storyboard is determined according to the initial character image corresponding to each character in the storyboard, the character's posture and the storyboard prompt word.

[0012] In a second aspect, the present application further provides a comic image generating device, comprising:

[0013] An acquisition module, used to acquire initial character images corresponding to each character in the novel text;

[0014] A text determination module, used for determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model;

[0015] A position determination module, for determining, based on the second language sub-model of the language model and according to the storyboard text, the storyboard prompt words and role prompt words corresponding to the storyboard and the role distribution position of each of the roles;

[0016] A posture determination module, configured to determine the posture of the character in the storyboard based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position;

[0017] An image determination module is used to determine the storyboard image corresponding to the storyboard based on the second image sub-model of the Wensheng graph model and according to the initial character image corresponding to each character in the storyboard, the character's posture and the storyboard prompt word.

[0018] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;

[0019] The memory is used to store computer programs;

[0020] The processor is used to execute the computer program and implement the above-mentioned comic image generation method when executing the computer program.

[0021] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the comic image generation method as described above are implemented.

[0022] The present application provides a method, apparatus, device and storage medium for generating comic images, wherein the method comprises: obtaining an initial character image corresponding to each character in a novel text; determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model; determining a storyboard prompt word, a role prompt word and a role distribution position of each character corresponding to the storyboard based on a second language sub-model of the language model according to the storyboard text; determining the posture of the character in the storyboard based on the initial character image, the role prompt word corresponding to the storyboard and the character distribution position based on a first image sub-model of a preset Vincent graph model; determining the storyboard image corresponding to the storyboard based on the initial character image, the character prompt word and the character distribution position based on a second image sub-model of the Vincent graph model. The present application first obtains the initial character images corresponding to each character in the novel text; then, based on a preset language model, determines the storyboard prompt words, character prompt words corresponding to each storyboard, and the character distribution position of each character in the storyboard; then, based on a preset text graph model, determines the posture of each character in the storyboard, and generates a storyboard image including multiple characters and corresponding prompt words based on the initial character images corresponding to each character, the character's posture, and the storyboard prompt words, thereby ensuring that the characters in each storyboard image are consistent with the text description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A flowchart of a comic image generation method provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the connection between the server and the terminal device provided in the embodiment of the present application;

[0026] Figure 3 A schematic diagram of a partial framework of a cultural graph model provided in an embodiment of the present application;

[0027] Figure 4 A schematic block diagram of a comic image generating device provided in an embodiment of the present application;

[0028] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] The embodiments of the present application provide a method, device, equipment and storage medium for generating comic images. The method for generating comic images can be applied to a terminal device, which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer and the like. The method can also be applied to a server, which can be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] See also Figure 1 , Figure 1 This is a flow chart of a comic image generation method provided in an embodiment of the present application. It should be noted that the comic image generation method provided in an embodiment of the present application can be used in a terminal device, and of course can also be used in a server.

[0034] like Figure 2 As shown, the comic image generation method is applied to a server, and the server and the terminal device are connected in communication, and the storyboard image obtained by the comic image generation method can be sent to the terminal device through the server. Of course, it is not limited to this and is not limited here.

[0035] In specific implementation, the terminal device includes but is not limited to: any one of a mobile phone, a tablet computer, a laptop computer, and a desktop computer; the server can be a single server or a server cluster, or a cloud server that provides cloud computing services.

[0036] like Figure 1 As shown, the comic image generation method includes steps S101 to S105.

[0037] Step S101: Obtain the initial character image corresponding to each character in the novel text.

[0038] A novel text generally includes multiple characters, and each character is closely connected to the storyline of the novel text. By combining the descriptions of different characters in the novel text and obtaining the initial character images corresponding to each character, a more three-dimensional character image can be created.

[0039] For example, the initial character image in the embodiment of the present application can be generated according to the character information of each character, such as the character's basic information (name, age, personality), personality traits (positive traits, negative traits), background story (family background, growth experience, social relationships), abilities and skills (magic, superpowers, professional skills), role positioning and relationship (protagonist, supporting role, relationship with other characters), appearance and clothing (facial features, facial contour, clothing) and other character information.

[0040] In some other implementations, the embodiments of the present application may also obtain initial character images corresponding to each character imported by the user.

[0041] Step S102: Determine a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model.

[0042] It should be noted that storyboards are an important part of the creation process of visual media such as movies, animations, advertisements, and comics. They mainly refer to breaking down a continuous story or plot into a series of pictures or shots that can be filmed or drawn. Storyboard text, also known as storyboard text or storyboard script, is a very important part of the production of visual media such as movies, animations, advertisements, and comics. It describes in detail the content, actions, dialogues, sound effects, picture composition, and other information of the storyboards.

[0043] The preset language model in the embodiment of the present application may include a first language sub-model for splitting the novel text. Based on the first language sub-model, the novel text can be split into storyboard texts corresponding to multiple storyboards. Specifically, the semantic structure in the novel text can be analyzed based on the first language sub-model, and semantic breakpoints can be found and split. For example, the first language sub-model may include a BERT pre-trained language model, a GPT model, and the like. It should be noted that the BERT pre-trained language model is based on the Transformer architecture and is trained through a large-scale text corpus to capture the deep-level features of the language. The GPT model is based on the Transformer architecture and learns the statistical features and intrinsic structure of the language through unsupervised learning on a large-scale text corpus. This pre-training method enables the GPT model to generate coherent and grammatically correct texts, which is suitable for a variety of text generation tasks. The Transformer architecture is a model composed of multiple self-attention mechanisms and positional feedforward neural networks, which can capture long-distance dependencies in sequences and has the advantage of parallel computing.

[0044] Step S103: Based on the second language sub-model of the language model, determine the storyboard prompt words, role prompt words and role distribution positions of the characters corresponding to the storyboards according to the storyboard text.

[0045] For example, the storyboard prompts corresponding to the storyboard refer to the keywords used to guide the specific content and expression of each shot. The character prompts can include the content of the character's speech, keywords used to indicate the action, etc. These prompts can help staff members (such as directors, photographers, animators, etc.) quickly understand the visual style, emotional atmosphere, action requirements, etc. of each shot. The storyboard prompts can include the composition of the picture, light and color; the character prompts can include the action expressions, character dialogues, narration, etc. of each character.

[0046] The character distribution position in the embodiment of the present application refers to the position of one or more characters in a storyboard. For example, a storyboard may include a character A, and character A is located in the middle, upper left side, or lower right side of the storyboard; for another example, a storyboard may include three characters, namely character A, character B, and character C, and character A and character B are on the right side of the storyboard, and character C is on the left side of the storyboard.

[0047] Specifically, the embodiments of the present application can extract key information in the storyboard text based on the second language sub-model of the language model to obtain the storyboard prompt words, role prompt words and role distribution positions of each role corresponding to the storyboard. For example, the storyboard text is "On the evening of October 20th at 20:05, in XX convenience store, character A and character B stood face to face, character A held a bottle of beverage and pointed at character B and said, "Why sell expired products?" Character B's palm was facing up and stretched out to character A and said, "Our store will not sell expired products. Please provide me with the receipt for the bottle of beverage you bought." The embodiment of the present application can input the storyboard text into the BERT pre-trained language model to perform semantic analysis on the storyboard text through the BERT pre-trained language model, and determine the storyboard prompt words for the convenience store scene (for example, at 20:05 on the evening of October 20th at XX convenience store), character A's lines and actions, character B's lines and actions, and the distribution positions of character A and character B. The embodiment of the present application can extract the prompt words for the corresponding storyboard scenes and different characters respectively through the second language sub-model of the language model to ensure that the characters and the description text are consistent.

[0048] Exemplarily, the second language sub-model in the embodiments of the present application may also include a BERT pre-trained language model and a GPT model.

[0049] Step S104: Based on the first image sub-model of the preset Wensheng graph model, the posture of the character in the storyboard is determined according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position.

[0050] The character's posture usually refers to the body position, movement and expression of the character in the storyboard. These postures can convey the character's emotions, personality and storyline. For example, a brave character may show a posture with his head held high, while a character with low self-esteem may show a posture with his head lowered and shoulders drooped.

[0051] It can be understood that the text-based graph model in the embodiment of the present application can transform abstract text into concrete visual representations by learning the relationship between a large number of images and text descriptions. The text-based graph model is generally composed of several parts, including a text understanding component, image information creation, and an image decoder. The text understanding component refers to converting the input text information into a digital representation to capture the ideas in the text; the image information creation generates image information based on these digital representations and noise data; finally, the image decoder converts the image information into a visual image.

[0052] The first image sub-model of the Wensheng graph model in the embodiment of the present application may include a diffusion model, an autoregressive model, a generative adversarial network model, etc. It should be noted that the diffusion model generates a new image by gradually adding noise to the image and then learning how to restore the original image from the noise distribution. Among them, the more typical diffusion model includes the Stable Diffusion model.

[0053] The embodiment of the present application can first process the initial character image and the character prompt words corresponding to the storyboard based on the first image sub-model of the preset Wensheng graph model to obtain the image corresponding to the character, and then determine the posture of the character in the storyboard according to the processed image corresponding to the character and the character distribution position.

[0054] Step S105 , based on the second image sub-model of the Wensheng graph model, according to the initial character images corresponding to the characters in the storyboard, the character's postures and the storyboard prompt words, determine the storyboard image corresponding to the storyboard.

[0055] Storyboards refer to the draft of lens design, which is mainly used to show the author's arrangement of the plot and the picture, including the ratio of separation, lens editing, the posture of the character's performance, and the placement of lines. In the process of comic production, the storyboard images of comics are generally divided into grids to show continuous pictures and tell the story through these pictures. It should be noted that grid division is a form unique to comics, which generally refers to plane cutting in a specified size, and the number and size ratio of grids are reasonably arranged according to the content. Common grid division methods include broken format, regular format, oblique division, full page format, etc.

[0056] Exemplarily, the second image submodel of the Wensheng graph model in the embodiment of the present application may include a Stable Diffusion model, which is a model based on denoising diffusion probability and can generate images according to text prompts. Based on the second image submodel of the Wensheng graph model, a storyboard image including at least one character, a posture corresponding to the character, and a prompt word can be generated according to the initial character image corresponding to each character in the storyboard, the character's posture, and the storyboard prompt word.

[0057] The comic image generation method provided by the above embodiment includes: obtaining initial character images corresponding to each character in a novel text; determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model; determining storyboard prompt words, character prompt words and character distribution positions of each character corresponding to the storyboard based on a second language sub-model of the language model according to the storyboard text; determining the posture of the character in the storyboard based on the initial character image, the character prompt words corresponding to the storyboard and the character distribution position based on a first image sub-model of a preset Vincent graph model; determining the storyboard image corresponding to the storyboard based on the initial character image, the character prompt words corresponding to the storyboard and the character distribution position based on a second image sub-model of the Vincent graph model. The embodiment of the present application first obtains the initial character images corresponding to each character in the novel text; then, based on a preset language model, determines the storyboard prompt words, character prompt words and the character distribution position of each character corresponding to each storyboard; then, based on a preset text graph model, determines the posture of each character in the storyboard, and generates a storyboard image including multiple characters and corresponding prompt words based on the initial character images corresponding to each character, the character's posture and the storyboard prompt words, thereby ensuring that the characters in each storyboard image are consistent with the text description.

[0058] In an exemplary implementation, step S101 may include step S1011 and step S1012.

[0059] Step S1011: Obtain role information corresponding to each role in the novel text based on the third language sub-model of the language model.

[0060] Step S1012: Based on the third image sub-model of the Wensheng graph model and according to the role information corresponding to each role, an initial role image corresponding to each role is obtained.

[0061] The preset initial character image in the embodiment of the present application refers to a character setting image, that is, an image designed for a fictional character in the production process of animation, games, movies, etc. The character setting image can show the appearance characteristics of the character, such as hairstyle, clothing, skin color, facial features, etc.

[0062] The third language sub-model of the language model in the embodiment of the present application may also include a BERT pre-trained language model and a GPT model, and the third image sub-model of the text graph model may also include a Stable Diffusion model.

[0063] For example, an embodiment of the present application can first extract character information related to describing different characters in a novel text (i.e., character information such as hairstyle, clothing, skin color, etc.) based on the third language sub-model of the language model, and input the character information related to describing the characters into a Stable Diffusion model. The extracted character information is fully understood through the Stable Diffusion model, thereby generating an initial character image corresponding to each character, i.e., a character setting image.

[0064] In an exemplary implementation, step S104 may include step S401 and step S402.

[0065] Step S401: Based on the first image sub-model of the preset Wensheng graph model, and according to the preset initial character image and the character prompt words corresponding to the storyboard, a target character image corresponding to the character is obtained.

[0066] Step S402: Determine the posture of the character in the storyboard according to the target character image and the character distribution position.

[0067] After obtaining the initial character image, the embodiment of the present application can also generate a target character image (i.e., the character image of each storyboard) that is consistent with the character and text description based on the first image sub-model of the text graph model, according to the initial character image and the extracted character prompt words corresponding to the storyboard, so as to accurately present the image characteristics of the character in the corresponding storyboard.

[0068] After obtaining the corresponding target character image (character image), the embodiment of the present application can determine the posture of the character in the storyboard based on the target character image and the character distribution position obtained above, which can not only more accurately present the postures of different characters in the storyboard, but also make the characters displayed in the storyboard more vivid and lifelike.

[0069] In an exemplary implementation, step S402 may include step S4021 and step S4022.

[0070] Step S4021: Perform biological body detection and key point position detection on the target character image to obtain key point coordinates corresponding to the target character image.

[0071] Step S4022: Map the key point coordinates to the character distribution position to obtain the character's posture in the storyboard.

[0072] For example, the embodiment of the present application can perform biological body detection on the target character image through deep learning models such as deep neural networks, convolutional neural networks, and recurrent neural networks, that is, perform human body detection on the character image to accurately identify the human body area in the image and obtain the human body boundary box; then extract the features in the human body boundary box through the posture estimation method to obtain the key point coordinates of the human body; then, map the key point coordinates and the character distribution position respectively. Specifically, match the key point coordinates with the character distribution position to obtain a matching result, and adjust and optimize the key point coordinates according to the matching result, so that the posture of each character in the storyboard can be accurately obtained.

[0073] In an exemplary implementation, step S105 may include step S501.

[0074] Step S501, inputting the initial character images, character postures and storyboard prompt words corresponding to each character in the storyboard into the second image sub-model of the Wensheng graph model, so as to fuse the initial character images, character postures and storyboard prompt words in the storyboard through the second image sub-model to obtain the storyboard image.

[0075] The embodiment of the present application can fuse the initial character image, the character's posture, and the storyboard prompt words in the storyboard through the second image sub-model of the text graph model to draw a storyboard image that is consistent with the character and the text description.

[0076] In an exemplary implementation, the second image sub-model may include a first adapter network and a second adapter network; step S501 may include steps S5011 to S5013.

[0077] Step S5011: extracting posture information corresponding to the posture of the character through the first adapter network of the second image sub-model.

[0078] Step S5012: extracting appearance features corresponding to the initial character image in the storyboard through the second adapter network of the second image sub-model.

[0079] Step S5013: Fuse the storyboard prompt words, posture information and appearance features to obtain a storyboard image.

[0080] like Figure 3 As shown, the second image sub-model may include a first adapter network and a second adapter network. The first adapter network may be used to control the pose of the generated image to be consistent with the pose of the character, and the second adapter network may be used to control the appearance features of the character in the generated image to be consistent with the appearance of the character in the initial character image.

[0081] Specifically, the embodiment of the present application can extract the posture information corresponding to the posture of the character through the first adapter network of the second image sub-model, so as to control the posture of the generated image to be consistent with the posture of the character. The second adapter network of the second image sub-model can extract the appearance features of the character from the initial character image, so as to control the appearance features of the character in the generated image to be consistent with the appearance of the character in the initial character image. Finally, the posture information, appearance features and storyboard prompt words are fused together to obtain a storyboard image that meets the requirements of the storyboard context and character interaction, which not only allows multiple characters to be presented in the same storyboard image, but also maintains the consistency of the character's appearance, posture and text description.

[0082] In an exemplary implementation, step S106 is further included after step S105.

[0083] Step S106, synthesizing the storyboard images corresponding to the multiple storyboards into a video.

[0084] After obtaining the storyboard images corresponding to the multiple storyboards, the embodiment of the present application can stitch the multiple storyboard images in sequence to form a synthesized video.

[0085] See also Figure 4 , Figure 4 A schematic block diagram of a comic image generation device provided in an embodiment of the present application. The comic image generation device can be configured in a server or a terminal device to execute the aforementioned comic image generation method.

[0086] like Figure 4 As shown, the comic image generating device includes: an acquisition module 110 , a text determination module 120 , a position determination module 130 , a posture determination module 140 and an image determination module 150 .

[0087] The acquisition module 110 is used to acquire the initial character image corresponding to each character in the novel text.

[0088] The text determination module 120 is used to determine the storyboard text corresponding to at least one storyboard in the novel text based on the first language sub-model of the preset language model.

[0089] The position determination module 130 is used to determine the storyboard prompt words, role prompt words and role distribution positions of each role corresponding to the storyboard based on the second language sub-model of the language model and the storyboard text.

[0090] The posture determination module 140 is used to determine the posture of the character in the storyboard based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position.

[0091] The image determination module 150 is used to determine the storyboard image corresponding to the storyboard based on the second image sub-model of the Wensheng graph model and according to the initial character images, character postures and storyboard prompt words corresponding to each character in the storyboard.

[0092] In an exemplary implementation, the acquisition module 110 may include a first acquisition submodule and a second acquisition submodule.

[0093] The first acquisition submodule is used to acquire the role information corresponding to each role in the novel text based on the third language submodel of the language model.

[0094] The second acquisition submodule is used to obtain the initial character image corresponding to each character based on the third image submodel of the Wensheng graph model and according to the character information corresponding to each character.

[0095] In an exemplary embodiment, the posture determination module 140 may include a target determination submodule and a posture determination submodule.

[0096] The target determination submodule is used to obtain a target character image corresponding to the character based on the first image submodel of the preset Wensheng picture model and the character prompt words corresponding to the preset initial character image and the storyboard.

[0097] The posture determination submodule is used to determine the posture of the character in the storyboard according to the target character image and the character distribution position.

[0098] In an exemplary embodiment, the posture determination submodule may include a coordinate acquisition submodule and a mapping submodule.

[0099] The coordinate acquisition submodule is used to perform biological body detection and key point position detection on the target character image to obtain the key point coordinates corresponding to the target character image.

[0100] The mapping submodule is used to map the key point coordinates to the character distribution position to obtain the character's posture in the storyboard.

[0101] In an exemplary embodiment, the image determination module 150 can be specifically used to input the initial character image, the character's posture and the storyboard prompt words corresponding to each character in the storyboard into the second image sub-model of the Wensheng graph model, so as to fuse the initial character image, the character's posture and the storyboard prompt words in the storyboard through the second image sub-model to obtain the storyboard image.

[0102] In an exemplary embodiment, the second image sub-model includes a first adapter network and a second adapter network. The image determination module 150 may include a first extraction sub-module, a second extraction sub-module, and a fusion sub-module.

[0103] The first extraction submodule is used to extract posture information corresponding to the posture of the character through the first adapter network of the second image submodel.

[0104] The second extraction submodule is used to extract the appearance features corresponding to the initial character image in the storyboard through the second adapter network of the second image submodel.

[0105] The fusion submodule is used to fuse the storyboard prompt words, posture information and appearance features to obtain the storyboard image.

[0106] In an exemplary embodiment, the apparatus may further include a synthesis module.

[0107] The synthesis module is used to synthesize the storyboard images corresponding to multiple storyboards into a video.

[0108] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0109] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0110] Exemplarily, the above method and apparatus may be implemented in the form of a computer program, which may be run on a computer device.

[0111] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal device.

[0112] like Figure 5 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0113] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any step of the comic image generation method.

[0114] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0115] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any step of the comic image generation method.

[0116] The network interface is used for network communication, such as sending assigned tasks.

[0117] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0119] In one embodiment, the processor is used to execute a computer program and implement the following steps when executing the computer program:

[0120] Obtaining initial character images corresponding to each character in the novel text;

[0121] Determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model;

[0122] Based on the second language sub-model of the language model, determine the storyboard prompt words, role prompt words and role distribution positions of each role corresponding to the storyboard according to the storyboard text;

[0123] Based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position, determine the posture of the character in the storyboard;

[0124] The second image sub-model based on the Wensheng graph model determines the storyboard image corresponding to the storyboard according to the initial character images corresponding to each character in the storyboard, the character's posture and the storyboard prompt words.

[0125] In an exemplary embodiment, obtaining an initial character image corresponding to each character in a novel text includes:

[0126] The third language sub-model based on the language model obtains the role information corresponding to each role in the novel text;

[0127] Based on the third image sub-model of the Wensheng graph model, an initial character image corresponding to each character is obtained according to the character information corresponding to each character.

[0128] In an exemplary embodiment, based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position, the posture of the character in the storyboard is determined, including:

[0129] Based on the first image sub-model of the preset Wensheng graph model, a target character image corresponding to the character is obtained according to the initial character image and the character prompt words corresponding to the storyboard;

[0130] Determine the posture of the character in the storyboard based on the target character image and character distribution position.

[0131] In an exemplary embodiment, determining the posture of a character in a storyboard according to a target character image and a character distribution position includes:

[0132] Perform biological body detection and key point position detection on the target character image to obtain the key point coordinates corresponding to the target character image;

[0133] Map the key point coordinates to the character distribution position to obtain the character's posture in the storyboard.

[0134] In an exemplary embodiment, based on the second image sub-model of the Wensheng graph model, according to the initial character images corresponding to each character in the storyboard, the character's posture and the storyboard prompt words, determining the storyboard image corresponding to the storyboard includes:

[0135] The initial character images, character postures and storyboard prompts corresponding to each character in the storyboard are input into the second image sub-model of the Wensheng graph model, so that the initial character images, character postures and storyboard prompts in the storyboard are fused through the second image sub-model to obtain the storyboard image.

[0136] In an exemplary embodiment, it is characterized in that the second image sub-model includes a first adapter network and a second adapter network; the initial character image, the character's posture, and the storyboard prompt words in the storyboard are fused by the second image sub-model to obtain the storyboard image, including:

[0137] extracting posture information corresponding to the posture of the character through the first adapter network of the second image sub-model;

[0138] extracting appearance features corresponding to the initial character image in the storyboard through a second adapter network of the second image sub-model;

[0139] The storyboard prompt words, posture information and appearance features are integrated to obtain the storyboard image.

[0140] In an exemplary embodiment, after determining the storyboard image corresponding to the storyboard according to the initial character image, the character's posture and the storyboard prompt words corresponding to each character in the storyboard, the second image sub-model based on the Wensheng graph model further includes:

[0141] The storyboard images corresponding to multiple storyboards are combined into a video.

[0142] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above description of comic image generation can refer to the corresponding process in the embodiment of the aforementioned comic image generation method, and will not be repeated here.

[0143] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the comic image generation method of the present application.

[0144] The computer-readable storage medium may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0145] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0146] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0147] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a comic image, characterized in that: include: Obtaining initial character images corresponding to each character in the novel text; Determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model; Based on the second language sub-model of the language model, and according to the storyboard text, determining the storyboard prompt words, the role prompt words and the role distribution position of each of the roles corresponding to the storyboard; Based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position, determine the posture of the character in the storyboard; Based on the second image sub-model of the Wensheng graph model, the storyboard image corresponding to the storyboard is determined according to the initial character image corresponding to each character in the storyboard, the character's posture and the storyboard prompt word.

2. The method for generating a comic image according to claim 1, wherein: The step of obtaining the initial character images corresponding to the characters in the novel text includes: Acquire the role information corresponding to each of the roles in the novel text based on the third language sub-model of the language model; Based on the third image sub-model of the Wensheng graph model, and according to the role information corresponding to each of the roles, an initial role image corresponding to each of the roles is obtained.

3. The method for generating a comic image according to claim 1, wherein: The first image sub-model based on the preset Wensheng graph model determines the posture of the character in the storyboard according to the initial character image, the character prompt word corresponding to the storyboard, and the character distribution position, including: Based on the first image sub-model of the preset Wensheng graph model, according to the initial character image and the character prompt words corresponding to the storyboard, a target character image corresponding to the character is obtained; The posture of the character in the storyboard is determined according to the target character image and the character distribution position.

4. The method for generating a comic image according to claim 3, wherein: The step of determining the posture of the character in the storyboard according to the target character image and the character distribution position includes: Performing biological body detection and key point position detection on the target character image to obtain key point coordinates corresponding to the target character image; The key point coordinates are mapped to the character distribution positions to obtain the posture of the character in the storyboard.

5. The method for generating a comic image according to claim 1, wherein: The second image sub-model based on the Wensheng graph model determines the storyboard image corresponding to the storyboard according to the initial character image corresponding to each character in the storyboard, the character's posture and the storyboard prompt word, including: The initial character images, the character's postures and the storyboard prompt words corresponding to each of the characters in the storyboard are input into the second image sub-model of the Wensheng graph model, so as to fuse the initial character images, the character's postures and the storyboard prompt words in the storyboard through the second image sub-model to obtain a storyboard image.

6. The method for generating a comic image according to claim 5, wherein: The second image sub-model includes a first adapter network and a second adapter network; the initial character image, the character's posture, and the storyboard prompt word in the storyboard are fused by the second image sub-model to obtain the storyboard image, including: extracting posture information corresponding to the posture of the character through a first adapter network of the second image sub-model; extracting appearance features corresponding to the initial character image in the storyboard through a second adapter network of the second image sub-model; The storyboard prompt words, the posture information and the appearance features are fused to obtain a storyboard image.

7. The method for generating a comic image according to any one of claims 1 to 6, characterized in that: After the second image sub-model based on the Wensheng graph model determines the storyboard image corresponding to the storyboard according to the initial character image, the character's posture and the storyboard prompt word corresponding to each character in the storyboard, the method further includes: The storyboard images corresponding to the multiple storyboards are synthesized into a video.

8. A comic image generating device, characterized in that: include: An acquisition module, used to acquire initial character images corresponding to each character in the novel text; A text determination module, used for determining a storyboard text corresponding to at least one storyboard in the novel text based on a first language sub-model of a preset language model; A position determination module, for determining, based on the second language sub-model of the language model and according to the storyboard text, the storyboard prompt words and role prompt words corresponding to the storyboard and the role distribution position of each of the roles; A posture determination module, configured to determine the posture of the character in the storyboard based on the first image sub-model of the preset Wensheng graph model, according to the initial character image, the character prompt words corresponding to the storyboard, and the character distribution position; An image determination module is used to determine the storyboard image corresponding to the storyboard based on the second image sub-model of the Wensheng graph model and according to the initial character image corresponding to each character in the storyboard, the character's posture and the storyboard prompt word.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the comic image generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the comic image generation method according to any one of claims 1 to 7 are implemented.

Images