Image generation method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides an image generation method and device, electronic equipment and a medium. The method specifically comprises the steps that an original composition corresponding to a text is generated; determining an object image in the original composition and target role information corresponding to the object image; according to the original composition, determining a role mask pattern corresponding to the target role information; determining role guide information according to a role design drawing and a role mask drawing corresponding to the target role information; determining layout guide information according to the depth map of the original composition; and guiding a diffusion model to generate a target image according to the role guiding information and the layout guiding information. According to the embodiment of the invention, the consistency of the character image of the same target role under different sub-mirrors is effectively improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device and medium. Background Art

[0002] With the development of artificial intelligence technology, the technology of Wenshengtu has emerged. Wenshengtu refers to the intelligent generation of images of scenes described by text based on the text input by the user. Wenshengtu technology has been widely used in application scenarios such as film and television production and picture drawing. For example, in the film and television production scene, the text in the script to be filmed is imaged, and the generated images can be provided to the director as a reference before the filming of the film and television. For another example, in the picture drawing scene, the image of the scene to be drawn is generated, and the generated images can be provided to the painter for reference to inspire the painter to create at a deeper level.

[0003] In related technologies, diffusion models are usually used to generate images corresponding to text. Specifically, the diffusion model first determines the semantic vector corresponding to the text and receives a noisy image; then, based on the semantic vector, the noise of the noisy image is gradually removed to obtain image data; finally, the image data is converted into a visual image.

[0004] In practical applications, film and television production and other application scenarios have strict requirements for the consistency of characters, that is, the same person must have highly similar facial features and hairstyles in different shots. However, the current diffusion model gradually constructs images based on probability distribution. Since it does not accurately grasp the detailed features of the character's facial features, hairstyle, etc., the current diffusion model has poor control over the consistency of the character, resulting in poor consistency in the facial features and hairstyle of the same person in images from different shots. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an image generation method, device, electronic device and medium, which can effectively improve the consistency of the character image of the same target character in different storyboards.

[0006] The specific technical solutions are as follows:

[0007] In a first aspect of the present invention, there is provided an image generation method, the method comprising:

[0008] Generate the original composition corresponding to the text;

[0009] Determine an object image in the original composition and target character information corresponding to the object image;

[0010] According to the original composition, a character mask image corresponding to the target character information is determined; wherein the resolution of the character mask image is the same as the resolution of the original composition, a first image area corresponding to the target character information in the character mask image is in a visible state, and a second image area is in a masked state; the second image area is an area in the character mask image other than the first image area;

[0011] Determine the role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information; the role guidance information includes: the target face features contained in the role design drawing, and the redrawing range corresponding to the role mask drawing;

[0012] Determine layout guidance information according to the depth map of the original composition;

[0013] According to the role guidance information and the layout guidance information, a diffusion model is guided to generate a target image.

[0014] In a second aspect of the present invention, there is provided an image generating device, the device comprising:

[0015] An original composition generation module, used to generate the original composition corresponding to the text;

[0016] An object and role determination module, used to determine an object image in an original composition and target role information corresponding to the object image;

[0017] A role mask determination module is used to determine a role mask map corresponding to the target role information according to the original composition; wherein the resolution of the role mask map is the same as the resolution of the original composition, the first image area corresponding to the target role information in the role mask map is in a visible state, and the second image area is in a masked state; the second image area is the area in the role mask map except the first image area;

[0018] The role guide determination module is used to determine the role guide information according to the role design diagram and the role mask diagram corresponding to the target role information; the role guide information includes: the target face features contained in the role design diagram, and the redrawing range corresponding to the role mask diagram;

[0019] A layout guidance determination module, used to determine layout guidance information according to the depth map of the original composition;

[0020] The target image generation module is used to guide the diffusion model to generate a target image according to the role guidance information and the layout guidance information.

[0021] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0022] Memory, used to store computer programs;

[0023] The processor is used to implement the aforementioned method steps when executing the program stored in the memory.

[0024] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above method.

[0025] In a fifth aspect of the implementation of the present invention, a computer program product is also provided, including a computer program / instruction, which implements the above method when executed by a processor.

[0026] The image generation method, device, electronic device and medium provided by the embodiments of the present invention first generate an original composition corresponding to the text; then determine the object image and the corresponding target character information in the original composition; then determine the character mask map corresponding to the target character information based on the original composition; wherein the resolution of the character mask map is the same as the resolution of the original composition, the first image area corresponding to the target character information in the character mask map is in a visible state, and the second image area is in a masked state; the second image area is the area in the character mask map except the first image area; then determine the character guide information based on the character design map and the character mask map corresponding to the target character information; the above-mentioned character guide information specifically includes: the target facial features contained in the character design map, and the redrawing range corresponding to the character mask map; then determine the layout guide information based on the depth map of the original composition; and then guide the diffusion model to generate the target image based on the character guide information and the layout guide information.

[0027] Since the embodiment of the present invention uses the role guidance information determined based on the role design drawing of the target role to guide the diffusion model to draw the character image in the target image. In this way, if the same role design drawing is used for the same target role in different storyboards, the role guidance information can contain the same target facial features. Regardless of which storyboard the target role is in, the diffusion model can draw the character image based on the target facial features in the same role design drawing; therefore, the embodiment of the present invention can effectively improve the consistency of the character image of the same target role in different storyboards. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0029] Figure 1 A flowchart of the steps of an image generation method according to an embodiment of the present invention;

[0030] Figure 2 A flowchart of the steps of an image generation method according to an embodiment of the present invention;

[0031] Figure 3 is a structural schematic diagram of an image generating device according to an embodiment of the present invention;

[0032] Figure 4 The structure block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] The image generation method of the embodiment of the present invention can be used to generate a target image based on text to improve the consistency of the same person under different lenses.

[0035] The embodiment of the present invention does not limit the specific application scenarios corresponding to the image generation method. For example, the application scenarios of the image generation method may include: film and television production, animation production, picture drawing and many other application scenarios.

[0036] In film and television production and animation production, storyboards are a visual narrative tool. It breaks down the content of the text script into continuous shots, and each storyboard is like a separate small picture. Through a series of storyboards, many elements of the whole story, such as the plot development, character actions, scene changes, etc., can be displayed. For example, in a movie, a simple plot of a character walking from one end of the room to the other may be broken down into multiple storyboards, including a close-up shot of the character standing up, a medium shot of the character walking, and a close-up shot of the character reaching the destination.

[0037] In the related art, a diffusion model is usually used to generate an image corresponding to a text. Specifically, the diffusion model first determines the semantic vector corresponding to the text and receives a noisy image; then, based on the semantic vector, the noise of the noisy image is gradually removed to obtain image data; finally, the image data is converted into a visualized image. However, the current diffusion model has poor control over the consistency of characters, resulting in poor consistency in facial features and hairstyles of the same person in images of different lenses.

[0038] In view of the technical problem in the related art that the facial features and hairstyle of the same person in images of different lenses have poor consistency, an embodiment of the present invention provides an image generation method, which specifically includes:

[0039] Generate the original composition corresponding to the text;

[0040] Determine an object image in the original composition and target character information corresponding to the object image;

[0041] According to the original composition, a character mask image corresponding to the target character information is determined; wherein the resolution of the character mask image is the same as the resolution of the original composition, a first image area corresponding to the target character information in the character mask image is in a visible state, and a second image area is in a masked state; the second image area is an area in the character mask image other than the first image area;

[0042] Determine the role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information; the role guidance information specifically includes: the target face features contained in the role design drawing, and the redrawing range corresponding to the role mask drawing;

[0043] Determine layout guidance information according to the depth map of the original composition;

[0044] According to the above-mentioned role guidance information and layout guidance information, the diffusion model is guided to generate a target image.

[0045] In the technical solution of the embodiment of the present invention, firstly, an original composition corresponding to the text is generated; then, the object image and the corresponding target character information in the original composition are determined; then, according to the original composition, a character mask map corresponding to the target character information is determined; wherein, the resolution of the character mask map is the same as the resolution of the original composition, the first image area corresponding to the target character information in the character mask map is in a visible state, and the second image area is in a masked state; the second image area is the area other than the first image area in the character mask map; then, according to the character design map and the character mask map corresponding to the target character information, the character guide information is determined; the above-mentioned character guide information specifically includes: the target facial features contained in the character design map, and the redrawing range corresponding to the character mask map; then, according to the depth map of the original composition, the layout guide information is determined; and then, according to the character guide information and the layout guide information, the diffusion model is guided to generate the target image.

[0046] Among them, the target facial features contained in the character guidance information specify the specific details of the facial features, eyebrow shape, mouth and nose ratio, etc. that the generated face should have; the redrawing range contained in the character guidance information clarifies the specific position of the redrawing area in the original composition.

[0047] The layout guidance information represents the layout of the original composition, so that the layout structure of the target image after redrawing will not deviate from the layout structure of the original composition. Therefore, the redrawing operation of the embodiment of the present invention can reduce the risk of destroying the original composition to a certain extent, and thus can improve the consistency of the layout between the target image and the original composition.

[0048] The process of generating the target image by the diffusion model is essentially: redrawing the character image at the position corresponding to the redrawing range while maintaining the layout of the original composition. In other words, in the process of guiding the diffusion model to generate the target image using the character guidance information and layout guidance information, the diffusion model can accurately find the corresponding position in the original composition according to the redrawing range while maintaining the layout of the original composition, and generate the corresponding pixel information according to the detailed requirements such as the facial features of the target person, thereby generating the target image.

[0049] The embodiment of the present invention uses the role guidance information determined based on the role design drawing of the target role to guide the diffusion model to draw the character image in the target image. In this way, if the same role design drawing is used for the same target role in different storyboards, the role guidance information can contain the same target facial features. Regardless of which storyboard the target role is in, the diffusion model can draw the character image based on the target facial features in the same role design drawing; therefore, the embodiment of the present invention can effectively improve the consistency of the character image of the same target role in different storyboards.

[0050] Moreover, in scenes such as film and television production, the complete content of a film and television work is usually presented by combining target images of many storyboards. Since the embodiment of the present invention improves the consistency of the image of the same target character in different storyboards, the coherence between the target image combinations of many storyboards can be enhanced, thereby accurately meeting the needs of scenes such as film and television production for image generation coherence and consistency.

[0051] In addition, the embodiment of the present invention realizes automatic determination of the redrawing range according to the character mask map, and the entire process does not require manual intervention, thereby improving the efficiency and accuracy of determining the redrawing range.

[0052] The embodiments of the present invention are described below through specific examples.

[0053] Reference Figure 1 , shows a flowchart of the steps of an image generation method according to an embodiment of the present invention, the method may specifically include the following steps:

[0054] Step 101, generating an original composition corresponding to the text;

[0055] Step 102: determining an object image in the original composition and target character information corresponding to the object image;

[0056] Step 103, determining a character mask image corresponding to the target character information according to the original composition; wherein the resolution of the character mask image is the same as the resolution of the original composition, the first image area corresponding to the target character information in the character mask image is in a visible state, and the second image area is in a masked state; the second image area is the area in the character mask image except the first image area;

[0057] Step 104: determining role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information; the role guidance information includes: the target face features contained in the role design drawing, and the redrawing range corresponding to the role mask drawing;

[0058] Step 105: Determine layout guidance information according to the depth map of the original composition;

[0059] Step 106: Guide the diffusion model to generate a target image according to the above-mentioned role guidance information and layout guidance information.

[0060] Figure 1 The illustrated method embodiment may be used to generate a video based on text to improve the consistency of the same person in different shots. Figure 1 The method embodiment shown can be executed by a client or a server. It can be understood that the embodiment of the present invention is for Figure 1 The specific execution subject of the illustrated method embodiment is not limited.

[0061] The embodiment of the present invention is applied to the storyboard generation function of a film and television production scene, which can solve the problem of inconsistent appearance of a character in multiple storyboards, enhance the consistency of a character in the target images of multiple storyboards, save the cost of storyboard production in the early stage of film and television shooting, and speed up the film and television shooting process.

[0062] In step 101, a preset diffusion model may be used to generate an original composition corresponding to the text.

[0063] The above text carries the description information of the target image (image description information), and its role is to provide clear guidance and reference content for the generation of the original composition. Therefore, the above text can be called image description text.

[0064] For example, in a film and television production scenario, the text can be the text in the script that needs to be filmed, which will describe the scene settings, character characteristics, plot atmosphere, etc. in detail. The preset diffusion model will extract key elements from these texts and generate an original composition that matches the key elements.

[0065] The preset diffusion model of the embodiment of the present invention may be any diffusion model. The preset diffusion model may be used to generate an image corresponding to a text. The embodiment of the present invention does not limit the process of using the preset diffusion model to generate an image corresponding to a text. For example, the preset diffusion model may first determine a semantic vector corresponding to the text and receive a noisy image; then, based on the semantic vector, gradually remove noise from the noisy image to obtain image data; and finally, convert the image data into a visual image.

[0066] In one example, the preset diffusion model may be a Flux model, which may include an MMDIT (Multi-Modal Diffusion Transformer) module.

[0067] The MMDIT module uses the CLIP (Contrastive Language-Image Pretraining) module and the T5 (Text-to-Text Transfer Transformer) module to determine the semantic vector corresponding to the text. Since the ability to understand text can be enhanced with the help of the CLIP module and the T5 module, the Flux model can enhance the quality of the original composition. In particular, the Flux model can enhance the text alignment ability of the original composition, that is, it can improve the matching degree between the original composition and the text.

[0068] Among them, the training data of the CLIP module is a massive amount of image-text pairs. After training with the training data, the CLIP module learns the ability to map text and images to the same joint representation space. With this ability, the corresponding vector distances of matching images and texts in the joint representation space can be made closer, and the vector distances of mismatched images and texts can be made farther, thereby accurately judging whether the image and text match. After receiving the input text, the CLIP module first tokenizes the text, and then processes it through a multi-head attention mechanism and a feedforward neural network layer to convert the text into a semantic vector. This semantic vector follows the inherent rules of the joint representation space. Relying on these rules, the semantic vector can be accurately compared and effectively associated with the image vector in the subsequent process, thereby improving the quality of the original composition in the image generation task.

[0069] The T5 module adopts a converter structure, which first tokenizes the text and splits it into word or subword sequences. These tokens then enter the multi-head attention layer, where the attention weights between the tokens are accurately calculated to capture the potential semantic connections between words. They then enter the feedforward neural network layer to further strengthen and regularize the semantic features, ultimately converting the text into a semantic vector rich in syntactic and semantic information.

[0070] In step 102, an object refers to an entity with distinguishable and independent visual features in an image. These entities can be semantic, such as people, animals, vehicles, buildings and other things with actual physical meanings, or abstract shapes or patterns. In the embodiment of the present invention, people are mainly used as an example to illustrate objects, and objects of other categories can be referenced to each other.

[0071] The object image in the original composition may refer to the image portion corresponding to the object in the original composition. The target role information corresponding to the object image may refer to the role identification corresponding to the object image. The role identification may be a role number, or the role identification may be a role name. Relative to the object, the target role information includes the identity information of the person.

[0072] The process of determining the object image in the original composition and the target character information corresponding to the object image specifically includes:

[0073] Step A1: performing target detection on the original composition to obtain position information of the object in the original composition;

[0074] Step A2: performing semantic segmentation on the original composition according to the position information of the object to obtain an object image;

[0075] Step A3: using a large language model to determine the target role information corresponding to the object image.

[0076] In step A1, object detection is used to identify the location information of the object category from the original composition. Object identification can be used to distinguish different object categories, and different object categories correspond to different objects. The location information of the object can be represented by a bounding box. The bounding box is a rectangular box used to frame the object detected in the original composition in object detection. It can be defined by the upper left corner coordinates and width and height. Its main function is to determine the position of the object in the original composition.

[0077] The target detection technology can be implemented using a target detection model. The target detection model usually first uses a convolutional neural network to extract image features of the original composition, and then calculates the coordinates of the bounding box based on the image features. It can be understood that the embodiments of the present invention do not limit the specific target detection technology.

[0078] It should be noted that the object detection model can output bounding boxes of M object categories. In one example, the text can contain a preset number of characters, and M can be greater than or equal to a preset number N. Both M and N can be positive integers.

[0079] It should also be noted that, when the number M of bounding boxes obtained based on target detection is greater than the preset number N, N target bounding boxes can be selected from the M bounding boxes according to the depth value of the bounding box image, and subsequent semantic segmentation can be performed for the N target bounding boxes. The bounding box image can refer to the image portion surrounded by the bounding box in the original composition.

[0080] In a specific implementation, a depth estimation algorithm may be used to perform depth estimation on the bounding box image to obtain a depth value of the bounding box image. The embodiment of the present invention does not limit the specific depth estimation algorithm.

[0081] The process of selecting N target bounding boxes from the M bounding boxes according to the depth value of the bounding box image may include: taking a bounding box having a depth value greater than a first threshold as the target bounding box, or sorting the M bounding boxes in descending order of the depth value, and selecting the front N bounding boxes as the N target bounding boxes.

[0082] In the field of computer vision, an image can be viewed as a concrete presentation of a visual scene mapped to a two-dimensional plane via an imaging device. Among them, the object corresponding to the bounding box with a large depth value is closer to the observation point and located more forward in the actual visual scene, and thus is likely to become the core component element in the image, carrying key semantics related to scene understanding and analysis; in comparison, the object corresponding to the bounding box with a small depth value is basically in a relatively backward position in the visual scene, with a relatively low degree of importance, and is regarded as a relatively minor object. Based on this, the embodiment of the present invention gives priority to selecting a target bounding box with a large depth value, thereby accurately focusing on key objects, effectively avoiding interference factors generated by minor objects in the subsequent semantic segmentation process, helping to improve the accuracy of semantic segmentation, and making the image processing results more in line with the expected goals.

[0083] In step A2, semantic segmentation is used to further divide the original composition to obtain the object image based on the position information obtained in step A1. Its function is to accurately distinguish the pixels in the bounding box that belong to the object identifier and the pixels that do not belong to the object identifier, and remove the pixels that do not belong to the object identifier, so as to accurately extract the complete image part corresponding to each object (that is, the object image). Different object images can be distinguished according to the object identifier.

[0084] The embodiments of the present invention do not limit the specific semantic segmentation process. For example, the semantic segmentation process may include: first, using a fully convolutional network to extract feature representations of pixels within a bounding box; then, based on the feature representations, determining the probability that the pixels within the bounding box belong to a preset object category; and then, classifying the pixels within the bounding box whose probability values ​​are higher than a second threshold into a preset object category, thereby obtaining an object image of the preset object category. If there are multiple preset object categories within the bounding box, the above semantic segmentation process may be repeated for each preset object category.

[0085] In step A3, a large language model is an artificial intelligence model designed to understand and generate human language. The large language model can perform a wide range of tasks by training on a large amount of text data. The large language model is usually based on a deep learning architecture, such as a transformer structure. The embodiment of the present invention does not limit the specific type of the large language model. For example, the type of the large language model may include: GPT (Generative Pre-Trained Transformer) model and GLM (Generative Language Model, General Language Model), etc.

[0086] After being trained with a large amount of text and other data, the large language model has strong semantic understanding and knowledge reasoning capabilities. When the object image and the role description information of the character in the text are input, the large language model can match the visual features in the object image with the role description information based on semantic understanding and knowledge reasoning capabilities, thereby obtaining the target role information corresponding to the object image.

[0087] In one implementation of the present invention, assuming that the original composition contains N object images, the large language model can be called N times, and one call is used to obtain the corresponding target character information for one object image. However, N calls require a relatively high call cost; the above call cost may include: time cost, computing cost, or economic cost, etc.

[0088] In another implementation of the present invention, the target character information corresponding to N object images can be determined based on a single call of the large language model, thereby saving the cost of obtaining the target character information.

[0089] Accordingly, the process of determining the target character information corresponding to the object image using the large language model specifically includes:

[0090] Step B1, selecting a target object image from the object images of the original composition according to the image area and / or the face orientation category corresponding to the object image; the number of the target object images is a preset number;

[0091] Step B2, synthesizing a preset number of target object images according to the position information of the objects contained in the target object images to obtain a synthesized image;

[0092] Step B3: input the synthesized image into a large language model, and the large language model outputs target character information corresponding to the target object images in the synthesized image.

[0093] In step B1, the image area is correlated with the visual significance and importance of the object. According to the law of visual presentation, an object image with a larger image area occupies a more critical position in the original composition and can carry more visual information. For example, in a group photo, the details such as the expression and posture of the person who is in the center and occupies a larger proportion of the picture are easier to notice, and these are important visual elements. The embodiment of the present invention uses the image area to filter out relatively more important object images, which clarifies the focus of the subsequent process of obtaining the target character information, and helps to improve the efficiency of the overall processing.

[0094] In a specific implementation, the pixel counting method can be used to determine the image area of ​​the object image. An image is composed of pixels, and each pixel has its position information in the image. Therefore, the number of pixels occupied by the object image can be counted to obtain the image area of ​​the object image.

[0095] According to the rules of visual presentation, the direction of the face is a key element in the image. Because viewers will intuitively judge the degree of participation and role status of the characters in the scene based on the direction of the face, the frontal direction can fully display information such as facial expressions and eyes, which means that the characters actively participate in the communication and are in a key position, while the side or back direction will cause some information to be missing, suggesting that the characters are in a passive auxiliary state. Therefore, the embodiment of the present invention selects relatively more important object images from the original composition based on the face direction category, clarifies the focus of the subsequent process of obtaining the target character information, and helps to improve the efficiency of the overall processing.

[0096] The process of determining the face orientation category may include: first, using face recognition technology to determine the face area in the object image; then, using posture recognition methods to determine the facial feature key points in the face area; and then, judging the face orientation category based on the facial feature key points. Specifically, the positional relationship, angle characteristics and other information presented by the facial feature key points can be analyzed to accurately determine whether the face is in the frontal face category, the side face / half face category, or the back face category.

[0097] The process of selecting a target object image from the object images of the original composition according to the image area and / or the face orientation category corresponding to the object image specifically includes:

[0098] Step B11, sorting the object images under each face orientation category in order from large to small image areas, to obtain a sorting result for each face orientation category;

[0099] Step B12: Obtain target object images from the sorting results of the frontal face category in order from front to back; the initial value of the number of target object images is 0, and the number of target object images increases by one each time a target object image is obtained.

[0100] Step B13, determine whether the number of target objects is greater than or equal to a preset number, if so, end the process, otherwise, execute step B14;

[0101] Step B14, determining whether the target object image of the front face category has been selected, if not, executing step B12, if yes, executing step B15;

[0102] Step B15: Obtain target object images from the sorting results of the side face / half face categories in order from front to back; each time a target object image is obtained, the number of target object images is increased by one.

[0103] Step B16: determine whether the number of target objects is greater than or equal to a preset number. If so, end the process; otherwise, execute step B17;

[0104] Step B17, determining whether the target object image of the side face / half face category has been selected, if not, executing step B15, if yes, executing step B18;

[0105] Step B18: Acquire the target object image from the sorting results of the back-to-back categories in order from front to back; each time a target object image is acquired, the number of target object images is increased by one.

[0106] Step B19, determine whether the number of target objects is greater than or equal to the preset number, if so, end the process, otherwise, execute step B110;

[0107] Step B110, determine whether the target object image of the back-facing category has been selected, if not, execute step B18, if yes, end the process.

[0108] The process of selecting the target object image corresponding to step B11 to step B110 performs orderly screening of the object images according to the image area and the face orientation category, and uses the method of giving priority to those images with large image area and frontal face orientation category, through layer-by-layer judgment and loop operation, efficiently extracting the target object image with a high probability of representing the important object.

[0109] In step B2, the role of synthesizing the image specifically includes:

[0110] On the one hand, synthetic images can help save the cost of calling large models. By extracting and sorting the important characters in the original composition and synthesizing new images in a specific order, unnecessary data input is reduced, the number of calls to the large language model is reduced, and the call cost is reduced, making resource utilization more efficient and reasonable.

[0111] On the other hand, the synthetic image helps solve the problem of the large language model's fuzzy left-right relationship. The target characters originally scattered in the original composition are pasted on the blank image from left to right to form a synthetic image. The relative position relationship between the characters becomes clear, allowing the large language model to more accurately grasp the spatial relationship such as the left-right position of the characters in the picture during analysis and processing, thereby improving the accuracy of the target character information.

[0112] The process of synthesizing a preset number of target object images includes:

[0113] Step B21, while retaining the resolution size of the original composition, cropping the target object image in the original composition along the edge of the object to obtain a cropped image of the target object;

[0114] Step B22, creating a blank image, the resolution of the blank image being the same as the resolution of the original composition;

[0115] Step B23: Place the cropped images one by one on the blank image in order from left to right to obtain a composite image.

[0116] Taking the object image as a person image as an example, in the synthesized image, all pixels except the person image are white, and there is a fixed interval between each person image.

[0117] In step B3, the composite image and the guide information may be input into the large language model, and the large language model outputs the target character information corresponding to the target object images in the composite image. The above-mentioned guide information is used to guide the large language model to determine and output the target character information corresponding to the target object images in the composite image. The above-mentioned guide information may include: the above-mentioned text, or the above-mentioned guide information may include: character feature information. It can be understood that the embodiment of the present invention does not limit the specific guide information.

[0118] In one example A, the above text specifically includes the following information:

[0119] Scene information: Three people sit together with flashing lights in the background, creating a warm atmosphere.

[0120] Character A (a young man with medium-length hair, wearing a black suit, on the left side of the picture) raises his glass and says, "Zhiyuan, Yuxia, thank you for organizing this party, which gives us the opportunity to be together again."

[0121] Character B (young male, short hair, wearing a white shirt, right of center, looking at the screen) raises his glass and says, "Cheers to our friendship!"

[0122] Character C (young woman, long hair, on the right side of the picture, profile) smiled and agreed: "I hope our friendship can last forever."

[0123] Since the synthesized image retains the resolution of the original composition, the details of the picture are fully preserved, the edges of the characters are accurately cropped, and the target object images are arranged from left to right, with fixed intervals between characters, and all pixels except the characters are white, making the characters particularly prominent and their positions clear at a glance; and the text contains character feature information, such as "young male, medium-length hair, wearing a black suit, on the left side of the screen" such appearance, position information, or speech actions; therefore, the large language model can accurately match key details such as the appearance, position information, speech actions, etc. of the characters in the text with the corresponding visual presentation of the characters in the synthesized image, and then determine the target character information.

[0124] The output result of the large language model may include: target character information arranged in order. For example, the output result of example A specifically includes: character A, character B, and character C from left to right.

[0125] In step 103, the character mask image refers to an image with the same resolution as the original composition. In the character mask image, the corresponding area is divided according to the target character information, wherein the first image area representing the target character itself is in a visible state, that is, the pixel information of the first image area can be normally acquired and processed; and the second image area other than the first image area is in a mask state, that is, the pixel information of the second image area is shielded when a specific operation is performed on the target character, and is not included in the processing flow for the time being. The character mask image is used to accurately locate the position of the target character in the original composition, and to assist in subsequent related image processing work such as background mask image determination, and different target characters can correspond to different character mask images.

[0126] The visible state may mean that the pixels in the first image area are visible to the naked eye. The pixels corresponding to the visible state and the target character related information carried by them can participate in subsequent operations related to the target character.

[0127] The mask state may mean that the pixels in the second image area are invisible to the naked eye. When redrawing, adjusting, etc. the target character, the information contained in the pixels corresponding to the mask state will not be involved in the relevant operations of the target character.

[0128] In practical applications, pixels in the character mask map may correspond to a transparency channel. The visible state or mask state of a pixel may be characterized based on the value of the transparency channel. For example, the value of the transparency channel is a first preset value, indicating the visible state of a pixel; and for another example, the value of the transparency channel is a second preset value, indicating the mask state of a pixel. The first preset value may be 1, and the second preset value may be 0. Of course, the embodiment of the present invention does not limit the specific value of the transparency channel corresponding to the state of the pixel.

[0129] Therefore, based on the original composition, the process of determining the character mask image corresponding to the target character information specifically includes: determining the first image area and the second image area from the original composition; setting the value of the transparency channel of the pixel points in the first image area to a first preset value, and setting the value of the transparency channel of the pixel points in the first image area to a second preset value.

[0130] In step 104, the character design corresponding to the target character information may be specified by the user.

[0131] Specifically, users can upload existing character designs, or they can generate character designs that meet the requirements by using the Wensheng graph model. The character image presented in the character design is the object to be used for subsequent redrawing operations, that is, to replace the original face in the original composition with the appearance of the character in the character design.

[0132] The process of determining character guidance information according to a character design drawing and a character mask drawing corresponding to target character information specifically includes: inputting the character design drawing and the character mask drawing corresponding to the target character information into a facial feature extraction model, extracting target facial features from the character design drawing by the facial feature extraction model, and determining a redrawing range according to the character mask drawing.

[0133] The embodiment of the present invention can maintain the mapping relationship between the role information and the role design drawing, and then search can be performed in the above mapping relationship according to the target role information to obtain the role design drawing corresponding to the target role information.

[0134] The embodiments of the present invention do not limit the specific facial feature extraction model. For example, the instantID node is a facial feature extraction model that has good performance in facial feature extraction and has the ability to efficiently and accurately extract key facial features. The instantID node uses a face encoder to extract semantic facial features, and uses a trainable projection layer to map these facial features to a text feature space, thereby generating a facial embedding vector, which is used as a target facial feature.

[0135] The redraw range specifies the specific location of the redraw area in the original composition. The redraw range can be determined using coordinates. For example, the upper left corner of the original composition can be used as the origin of the coordinate system, and then the coordinate values ​​of the points in the redraw area in the coordinate system can be clearly defined to accurately describe and locate the redraw range.

[0136] In a specific implementation, the target role information may be N types, where N may be a positive integer;

[0137] The process of determining the role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information specifically includes: using N facial feature extraction models respectively to determine the role guidance information corresponding to the N types of target role information. In this case, the above method may also include: fusing the role guidance information corresponding to the N types of target role information respectively, and the obtained fusion result can be used as the role guidance information commonly corresponding to the N types of target role information in the original composition.

[0138] The above fusion specifically includes: splicing the role guidance information corresponding to the N types of target role information in the corresponding dimensions. Assuming that the role guidance information is a 5-dimensional vector, the above fusion can be a splicing process of N 5-dimensional vectors. Assuming that the first vector is [a,b,c,d,e] and the second vector is [f,g,h,i,j], the splicing is performed in the same way on the dimension of the vector element order, and the new vector [a,b,c,d,e,f,g,h,i,j] is obtained after splicing.

[0139] In step 105, the layout guide information represents the layout of the original composition, so that the layout structure of the redrawn target image does not deviate from the layout structure of the original composition.

[0140] In practical applications, a depth estimation method can be first used to generate a depth map of the original composition, and then layout guidance information can be extracted from the depth map. The above layout guidance information can be understood by the diffusion model and can guide the diffusion model to not deviate from the layout structure of the original composition when redrawing.

[0141] The process of determining the layout guidance information can be performed by the ControlNet model. The ControlNet model is a neural network used to control the diffusion model. It guides the image generation process of the diffusion model by introducing additional input conditions, such as edge maps, semantic segmentation maps, key points, etc., thereby achieving fine control of the details of the target image.

[0142] In step 106, the process of generating the target image by the diffusion model specifically includes: guiding the diffusion model to gradually denoise the initial noise image according to the role guidance information and the layout guidance information to obtain the target image.

[0143] The embodiment of the present invention does not limit the diffusion model used to generate the target image. For example, the diffusion model used to generate the target image may be a SDXL (i.e., Stable Diffusion eXtra Large) model. Of course, other suitable diffusion models may also be selected according to actual conditions to complete the corresponding target image generation task.

[0144] It should be noted that the embodiments of the present invention can use image samples of one or more image styles to train the diffusion model. The above image styles may include but are not limited to: realistic style, cartoon style, line draft style, illustration style, etc. The diffusion model can learn the pattern information corresponding to the above image styles and encode the pattern information into its own parameters. For example, the above pattern information may include: the line thickness change law corresponding to the image style, the connection method of the lines under the image style, the composition characteristics and color matching corresponding to the image style, etc.

[0145] Therefore, the diffusion model of the embodiment of the present invention can generate a target image with an image style such as a line drawing style by virtue of its powerful learning ability and generation mechanism, thereby meeting diverse image generation requirements.

[0146] In one implementation, the initial noise image may be random Gaussian noise.

[0147] In another implementation, the initial noise image may be a binary image corresponding to the original composition. The binary image corresponding to the original composition is used to generate the target image, which can provide a white background for the target image, thereby improving the visual effect of the target image.

[0148] Accordingly, the above method may also include: determining an edge detection image corresponding to the original composition; binarizing the edge detection image to obtain a binary image; wherein the edge lines in the binary image are black and other areas are white, so that the target image has a clean background color.

[0149] The process of guiding the diffusion model to generate a target image based on the character guidance information and the layout guidance information specifically includes: using the binary image as the initial noise image of the diffusion model; the diffusion model determines the semantic vector corresponding to the text and receives the initial noise image; performing cross-attention processing on the character guidance information and the layout guidance information to obtain a first cross-attention vector; and performing noise removal and stylization processing on the initial noise image based on the semantic vector and the first cross-attention vector to obtain a target image.

[0150] Among them, stylization processing can be used to integrate image style elements into the generation process of the target image, and with the advantage of the background color of the initial noise image, generate a target image that meets the requirements and has image style. Specifically, the diffusion model can use the pattern information of the image style to integrate image style elements (such as line features, color application, composition form, etc.) into the image being constructed. In addition, with the advantage of the white background color of the initial binary noise image, the image style elements can be better displayed and integrated to avoid background color interference, thereby generating a target image that not only meets the text semantics, role and layout guidance requirements, but also highlights the image style.

[0151] The first cross-attention vector represents the key features and mutual relationships after the fusion of the role guidance information and the layout guidance information. It comprehensively reflects the guiding role of the details of the character image in the target image to be generated (derived from the target face features in the role guidance information, etc.) and the overall layout structure characteristics (element position relationship determined by the layout guidance information, etc.) in the image generation process. Therefore, the cross-attention vector can guide the diffusion model to generate a target image that meets the requirements of the role guidance information and the layout guidance information.

[0152] In a specific implementation, the binary image can be encoded by VAE (Variational Auto-Encoder). VAE encoding can map the image data of the binary image to a low-dimensional latent vector space. The image vector in this latent vector space can represent the features of the binary image in a more abstract and compact way. The image vector can represent the initial noise image.

[0153] In summary, the image generation method of the embodiment of the present invention uses the role guidance information determined based on the role design drawing of the target role to guide the diffusion model to draw the character image in the target image. In this way, if the same role design drawing is used for the same target role in different storyboards, the role guidance information can include the same target facial features. Regardless of which storyboard the target role is in, the diffusion model can draw the character image based on the target facial features in the same role design drawing; therefore, the embodiment of the present invention can effectively improve the consistency of the character image of the same target role in different storyboards.

[0154] Moreover, in scenes such as film and television production, the complete content of a film and television work is usually presented by combining target images of many storyboards. Since the embodiment of the present invention improves the consistency of the image of the same target character in different storyboards, the coherence between the target image combinations of many storyboards can be enhanced, thereby accurately meeting the needs of scenes such as film and television production for image generation coherence and consistency.

[0155] In addition, the embodiment of the present invention realizes automatic determination of the redrawing range according to the character mask map, and the entire process does not require manual intervention, thereby improving the efficiency and accuracy of determining the redrawing range.

[0156] Reference Figure 2 , shows a flowchart of the steps of an image generation method according to an embodiment of the present invention, the method may specifically include the following steps:

[0157] Step 201, generating an original composition corresponding to the text;

[0158] Step 202: determining an object image in the original composition and target character information corresponding to the object image;

[0159] Step 203: determining a character mask image corresponding to the target character information according to the original composition; wherein the resolution of the character mask image is the same as the resolution of the original composition, the first image area corresponding to the target character information in the character mask image is in a visible state, and the second image area is in a masked state; the second image area is the area in the character mask image other than the first image area;

[0160] Step 204: determining role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information; the role guidance information includes: the target face features contained in the role design drawing, and the redrawing range corresponding to the role mask drawing;

[0161] Step 205: Determine layout guidance information according to the depth map of the original composition;

[0162] Step 206: Determine a background mask image corresponding to the target character information according to the original composition; the resolution of the background mask image is the same as the resolution of the original composition; the third image area corresponding to the various target character information in the background mask image is in a masked state, and the fourth image area is in a visible state; the fourth image area is an area in the background mask image except the third image area;

[0163] Step 207: determining background guide information according to the background mask image and the text;

[0164] Step 208: Guide the diffusion model to generate a target image according to the character guidance information, the background guidance information and the layout guidance information.

[0165] exist Figure 1 On the basis of the method embodiment shown, this embodiment adds step 206 and step 207, and modifies the process of generating the target image by guiding the diffusion model.

[0166] Among them, step 206 and step 207 determine the background guidance information, and step 208 uses the background guidance information in the generation process of the target image, which can maintain the background area in the background mask image and can control the area in the target image that matches the background mask image to not be redrawn incorrectly.

[0167] The third image area corresponding to the various target character information in the background mask image is in a masked state, and the fourth image area is in a visible state. Assuming that the number of target character information is N, the third image area may include: image areas corresponding to N types of target character information.

[0168] In practical applications, pixels in the background mask image may correspond to a transparency channel. The visible state or mask state of a pixel may be characterized based on the value of the transparency channel. For example, the value of the transparency channel is a first preset value, indicating the visible state of a pixel; and for another example, the value of the transparency channel is a second preset value, indicating the mask state of a pixel.

[0169] Therefore, based on the original composition, the process of determining the background mask image corresponding to the target character information specifically includes: determining the third image area and the fourth image area from the original composition; setting the value of the transparency channel of the pixel points in the fourth image area to the first preset value, and setting the value of the transparency channel of the pixel points in the third image area to the second preset value.

[0170] In a specific implementation, the character guidance information and the background guidance information can be fused to obtain fused information. The process of the diffusion model generating the target image specifically includes: firstly determining the semantic vector corresponding to the text and receiving the initial noise image; then, performing cross-attention processing on the fused information and the layout guidance information to obtain a second cross-attention vector; then, according to the semantic vector and the second cross-attention vector, performing noise removal processing and stylization processing on the initial noise image to obtain the target image.

[0171] Among them, the second cross-attention vector integrates multiple aspects of information such as role, background, and layout, providing key guiding directions and detail constraints for the subsequent removal of noise from the initial noisy image based on the semantic vector and generating a target image that meets the requirements.

[0172] In summary, since the embodiments of the present invention can enable the characters in the target image to follow the target facial features in the character design drawing, this can avoid the problem of lack of continuity in the storyboards due to differences in the same target character information under multiple storyboards. In addition, the present invention can automatically specify the redrawing area without manual intervention, and the redrawing result does not change the original composition structure.

[0173] In addition, the embodiment of the present invention redraws the face of the target character information on the basis of stabilizing the layout of the original composition, thereby converting the overall style of the image into a line drawing while importing the specified facial features.

[0174] Furthermore, the embodiment of the present invention can guide the diffusion model to redraw multiple target characters based on a single call of the diffusion model, which reduces the occupation time of the GPU (Graphics Processing Unit) and improves the overall utilization efficiency.

[0175] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0176] Reference Figure 3 , shows a schematic diagram of the structure of an image generating device according to an embodiment of the present invention, which may specifically include the following modules: an original composition generating module 301, an object and role determining module 302, a role mask determining module 303, a role guide determining module 304, a layout guide determining module 305 and a target image generating module 306.

[0177] The original composition generation module 301 is used to generate the original composition corresponding to the text;

[0178] An object and role determination module 302 is used to determine an object image in the original composition and target role information corresponding to the object image;

[0179] The role mask determination module 303 is used to determine the role mask map corresponding to the target role information according to the original composition; wherein the resolution of the role mask map is the same as the resolution of the original composition, the first image area corresponding to the target role information in the role mask map is in a visible state, and the second image area is in a masked state; the second image area is the area in the role mask map except the first image area;

[0180] The role guidance determination module 304 is used to determine the role guidance information according to the role design diagram and the role mask diagram corresponding to the target role information; the role guidance information includes: the target face features contained in the role design diagram, and the redrawing range corresponding to the role mask diagram;

[0181] A layout guidance determination module 305, configured to determine layout guidance information according to the depth map of the original composition;

[0182] The target image generation module 306 is used to guide the diffusion model to generate a target image according to the role guidance information and the layout guidance information.

[0183] Optionally, the character guidance determination module is specifically used to input the character design drawing and the character mask drawing corresponding to the target character information into a facial feature extraction model, and the facial feature extraction model extracts the target facial features from the character design drawing, determines the redrawing range according to the character mask map, and the target facial features and the redrawing range are used as character guidance information.

[0184] Optionally, the target role information is N types, where N is a positive integer;

[0185] The role guidance determination module includes:

[0186] A single character processing module is used to respectively use N facial feature extraction models to determine the character guidance information corresponding to N types of target character information;

[0187] The multi-role fusion module is used to fuse the role guidance information corresponding to N types of target role information using conditional fusion nodes.

[0188] Optionally, the object and role determination module includes:

[0189] The target detection module is used to perform target detection on the original composition to obtain the position information of the object in the original composition;

[0190] A semantic segmentation module, used to perform semantic segmentation on the original composition according to the position information of the object to obtain an object image;

[0191] The target role determination module is used to determine the target role information corresponding to the object image by using a large language model.

[0192] Optionally, the target role determination module includes:

[0193] A selection module, configured to select a target object image from the object images of the original composition according to the image area and / or the face orientation category corresponding to the object image; the number of the target object images is a preset number;

[0194] A synthesis module, used for synthesizing a preset number of target object images according to position information of objects contained in the target object images to obtain a synthesized image;

[0195] The large language model processing module is used to input the synthetic image into the large language model, and the large language model outputs the target role information corresponding to the target object images in the synthetic image.

[0196] Optionally, the above-mentioned large language model processing module is specifically used to input the synthetic image and guidance information into the large language model, and the guidance information is used to guide the large language model to determine and output target character information corresponding to the target object images in the synthetic image.

[0197] Optionally, the device further comprises:

[0198] An edge detection module, used to determine an edge detection image corresponding to the original composition before guiding the diffusion model to generate a target image according to the character guidance information and the layout guidance information;

[0199] A binarization module, used for binarizing the edge detection image to obtain a binarized image;

[0200] Then the target image generation module includes:

[0201] An initial noise determination module, used for taking the binarized image as an initial noise image of a diffusion model;

[0202] The diffusion model processing module is used to determine the semantic vector corresponding to the text using the diffusion model and receive the initial noisy image; perform cross-attention processing on the character guidance information and the layout guidance information to obtain a first cross-attention vector; and perform noise removal and stylization processing on the initial noisy image according to the semantic vector and the first cross-attention vector to obtain a target image.

[0203] Optionally, the device further comprises:

[0204] A background mask module is used to determine a background mask image corresponding to the target character information according to the original composition before guiding the diffusion model to generate the target image according to the character guidance information and the layout guidance information; the resolution of the background mask image is the same as the resolution of the original composition, the third image area corresponding to the multiple target character information in the background mask image is in a masked state, and the fourth image area is in a visible state; the fourth image area is the area in the background mask image except the third image area;

[0205] A background guidance determination module, used to determine background guidance information according to the background mask image and the text;

[0206] Then the target image generation module includes:

[0207] The second target image generation module is used to guide the diffusion model to generate a target image according to the role guidance information, the background guidance information and the layout guidance information.

[0208] Optionally, the same target character information in different shots can correspond to the same character design drawing. No matter which shot the target character is in, the diffusion model can draw the character image according to the target face features in the same character design drawing; therefore, the embodiment of the present invention can effectively improve the consistency of the character image of the same target character in different shots.

[0209] An embodiment of the present invention further provides an electronic device that can implement the functions of the aforementioned storage node or scheduling node or ledger node or management node.

[0210] like Figure 4 As shown, the electronic device may include a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other via the communication bus 1004.

[0211] Memory 1003, used for storing computer programs;

[0212] The processor 1001 is used to execute the program stored in the memory 1003, and implements the following steps:

[0213] Storing the storage object;

[0214] Sending the storage object to a coupled second storage node, so that the second storage node stores and / or forwards the storage object;

[0215] After the storage object is successfully stored, the object information of the storage object is sent to the ledger node, so that the ledger node saves the ledger information of the first storage node; the ledger information includes: the object information stored in the corresponding storage node.

[0216] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0217] The communication interface is used for communication between the above electronic device and other devices.

[0218] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one image generating device located away from the aforementioned processor.

[0219] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0220] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image generation method described in any one of the above embodiments.

[0221] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the image generating method described in any one of the above embodiments.

[0222] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc. Examples of optical media may include DVD (Digital Video Disc) and the like.

[0223] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0224] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0225] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An image generation method, characterized in that: The method comprises: Generate the original composition corresponding to the text; Determine an object image in the original composition and target character information corresponding to the object image; According to the original composition, a character mask image corresponding to the target character information is determined; wherein the resolution of the character mask image is the same as the resolution of the original composition, a first image area corresponding to the target character information in the character mask image is in a visible state, and a second image area is in a masked state; the second image area is an area in the character mask image other than the first image area; Determine the role guidance information according to the role design drawing and the role mask drawing corresponding to the target role information; the role guidance information includes: the target face features contained in the role design drawing, and the redrawing range corresponding to the role mask drawing; Determine layout guidance information according to the depth map of the original composition; According to the role guidance information and the layout guidance information, a diffusion model is guided to generate a target image.

2. The method according to claim 1, characterized in that The step of determining the role guiding information according to the role design diagram and the role mask diagram corresponding to the target role information includes: The character design drawing and character mask drawing corresponding to the target character information are input into the facial feature extraction model, and the facial feature extraction model extracts the target facial features from the character design drawing. The redrawing range is determined according to the character mask drawing, and the target facial features and the redrawing range are used as character guidance information.

3. The method according to claim 1, characterized in that The target role information is N types, where N is a positive integer; The step of determining the role guiding information according to the role design diagram and the role mask diagram corresponding to the target role information includes: Using N facial feature extraction models respectively, determine the role guidance information corresponding to N target role information respectively; The method further comprises: The role guidance information corresponding to the N types of target role information is fused to obtain the role guidance information corresponding to the N types of target role information.

4. The method according to claim 1, characterized in that: The determining of the object image in the original composition and the target character information corresponding to the object image includes: Performing target detection on the original composition to obtain position information of the object in the original composition; According to the position information of the object, semantically segment the original composition to obtain an object image; The target role information corresponding to the object image is determined by using a large language model.

5. The method according to claim 4, characterized in that The using of the large language model to determine the target role information corresponding to the object image includes: Selecting a target object image from the object images of the original composition according to the image area and / or the face orientation category corresponding to the object image; the number of the target object images is a preset number; Synthesizing a preset number of target object images according to position information of objects contained in the target object images to obtain a synthesized image; The synthesized image is input into a large language model, and the large language model outputs target character information corresponding to the target object images in the synthesized image.

6. The method according to claim 5, characterized in that The step of inputting the synthesized image into a large language model comprises: The composite image and the guide information are input into a large language model, and the guide information is used to guide the large language model to determine and output target character information corresponding to the target object images in the composite image.

7. The method according to claim 1, characterized in that Before guiding the diffusion model to generate the target image according to the role guidance information and the layout guidance information, the method further includes: Determine an edge detection image corresponding to the original composition; Binarizing the edge detection image to obtain a binary image; Then, guiding the diffusion model to generate a target image according to the role guidance information and the layout guidance information includes: Using the binarized image as an initial noise image of a diffusion model; The diffusion model determines the semantic vector corresponding to the text and receives an initial noisy image; cross-attention processing is performed on the character guidance information and the layout guidance information to obtain a first cross-attention vector; and according to the semantic vector and the first cross-attention vector, the initial noisy image is subjected to noise removal processing and stylization processing to obtain a target image.

8. The method according to any one of claims 1 to 7, characterized in that: Before guiding the diffusion model to generate the target image according to the role guidance information and the layout guidance information, the method further includes: According to the original composition, a background mask image corresponding to the target character information is determined; the resolution of the background mask image is the same as the resolution of the original composition, the third image area corresponding to the various target character information in the background mask image is in a masked state, and the fourth image area is in a visible state; the fourth image area is an area in the background mask image except the third image area; Determining background guide information according to the background mask image and the text; Then, guiding the diffusion model to generate a target image according to the role guidance information and the layout guidance information includes: According to the character guiding information, the background guiding information and the layout guiding information, a diffusion model is guided to generate a target image.

9. The method according to any one of claims 1 to 7, characterized in that: The same target character information corresponds to the same character design drawing in different storyboards.

10. An image generating device, characterized in that: The device comprises: An original composition generation module, used to generate the original composition corresponding to the text; An object and role determination module, used to determine an object image in an original composition and target role information corresponding to the object image; A role mask determination module is used to determine a role mask map corresponding to the target role information according to the original composition; wherein the resolution of the role mask map is the same as the resolution of the original composition, the first image area corresponding to the target role information in the role mask map is in a visible state, and the second image area is in a masked state; the second image area is the area in the role mask map except the first image area; The role guide determination module is used to determine the role guide information according to the role design diagram and the role mask diagram corresponding to the target role information; the role guide information includes: the target face features contained in the role design diagram, and the redrawing range corresponding to the role mask diagram; A layout guidance determination module, used to determine layout guidance information according to the depth map of the original composition; The target image generation module is used to guide the diffusion model to generate a target image according to the role guidance information and the layout guidance information.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method described in claims 1-9 is implemented.

Images