A method, medium and system for directional drawing of AI painting model

Through semantic segmentation, batch map generic and user interaction, combined with similarity filtering and hierarchical relationship optimization, the problem of lack of flexible editing of AI painting models is solved, and the efficient, personalized and high-quality effect of targeted map generation is achieved.

CN118351217BActive Publication Date: 2024-11-01YANTAI XUNING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410448255.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-15
Publication Date
2024-11-01
Estimated Expiration
2044-04-15

AI Technical Summary

Technical Problem

Most of the existing AI painting models stay at the level of 'one-click generation', lack flexible editing functions, and cannot be directional and mapped according to the direction selected by the operator, making it difficult to meet the customized needs of professional designers and artists.

Method used

By obtaining the images input by the user, generate text and initial images, perform semantic segmentation, transparent filling, batch image generation operations, and combine user interaction needs to gradually complete semantic segmentation, supplementary generation, synthetic editing and visual optimization of images, using similarity thresholds Filter the reference image, determine the image hierarchy relationship, and optimize color, lighting, and texture to generate directional generated images.

Benefits of technology

It realizes an efficient editing process from coarse to fine, improves the pertinence and personalization of image generation, meets users' customized needs, and improves editing efficiency and the quality of generated images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118351217B_ABST
    Figure CN118351217B_ABST
Patent Text Reader

Abstract

The present invention provides an AI painting model directional drawing method, medium and system, belonging to the field of AI painting technology, comprising: obtaining a user input text and an image group generated by Stable Diffusion, and selecting a basic image; performing semantic segmentation on the basic image to generate multiple segmented images, and transparently filling each segmented image; using the segmented image as a base image, using the semantic segmentation result corresponding to the segmented image as input, and batch generating multiple segmented supplementary images; selecting a suitable image as a second segmented image from the segmented supplementary image, and obtaining a segmented similar image with a high similarity to the second segmented image; determining the main body of each segmented image and the number of layers thereof; obtaining the semantic part that the user needs to update, and regenerating the corresponding updated partial image; merging the unchanged second segmented image and the updated partial image corresponding to the second segmented image that needs to be updated by layer to obtain a directional generated image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of AI painting technology, and in particular, relates to a method, medium and system for directional drawing of an AI painting model. Background Art

[0002] As an important branch of artificial intelligence, image generation technology has developed rapidly in recent years. Image generation models based on deep learning, such as generative adversarial networks (GANs) and diffusion models, can generate realistic images based on text descriptions or other inputs, and have broad application prospects in creative design, game development, and auxiliary creation. Among them, the StableDiffusion model based on the diffusion model has attracted widespread attention in the industry. It not only has high generation quality, but also high computational efficiency, and can support interactive image editing.

[0003] However, most of the existing image generation technologies are aimed at ordinary users and cannot meet the needs of professional designers, artists, etc. for personalized image editing. These users usually have specific creative demands and aesthetic pursuits, and hope to be able to customize and optimize the details while retaining the core content of the image. For example, in the design industry, designers may need to make detailed edits to the color matching, light and shadow effects, texture, etc. of the generated images according to specific brand styles and aesthetic standards. Or in game development, artists may need to make targeted modifications to the generated character images, scene elements, etc. according to the plot and scene requirements.

[0004] Most existing image generation tools remain at the "one-click generation" level, lacking flexible editing functions and unable to meet the customized needs of professional users. Even though some tools provide basic editing functions such as layer management and filter adjustment, the operation is complex and not very targeted. They cannot well integrate the advantages of image generation, and it is difficult to achieve an efficient editing process from coarse to fine, and it is impossible to produce directional images according to the direction selected by the operator. Summary of the invention

[0005] In view of this, the present invention provides an AI painting model directional drawing method, medium and system, which can solve the technical problems that most existing AI painting models remain at the level of "one-click generation", lack flexible editing functions, and cannot be directional drawn according to the direction selected by the operator.

[0006] The present invention is achieved in that:

[0007] A first aspect of the present invention provides an AI painting model directional drawing method, which includes the following steps:

[0008] S10, obtaining the image-generated text input by the user into the image generation AI model tool StableDiffusion, recorded as the first text; obtaining the image group generated by StableDiffusion, and obtaining the basic image selected by the operator from the image group; obtaining multiple images in the image group whose similarity to the basic image is greater than a preset first similarity threshold, recorded as the basic similar image group, including multiple basic similar images;

[0009] S20, performing semantic segmentation of the first text on the basic image to obtain a plurality of segmented images, and performing transparent filling on each segmented image to generate a first segmented image;

[0010] S30, taking each first segmented image as the base image of the image generation, taking the semantics of the semantic segmentation corresponding to the first segmented image as the input word, inputting it into StableDiffusion for batch image generation operation, and obtaining a plurality of corresponding segmented supplementary images;

[0011] S40, for each first segmented image, obtaining an image selected by an operator from the corresponding multiple segmented supplementary images as a basic segmented supplementary image, recorded as a second segmented image; obtaining multiple images from the multiple segmented supplementary images corresponding to the second segmented image, whose similarity to the second segmented image is greater than a preset second similarity threshold, recorded as a segmented similar image group, including multiple segmented similar images;

[0012] S50, determining the main body of each first segmented image, and determining the number of layers of each first segmented image according to the size ratio of the main body to the basic image and the semantics corresponding to the first image, and assigning the number of layers of the corresponding first segmented image to each second segmented image;

[0013] S60, outputting each second segmented image and the corresponding semantics to the operator, obtaining the semantics of a part that the operator needs to update, and determining the second segmented images that the operator needs to maintain unchanged and the second segmented images that need to be updated;

[0014] S70, using each second segmented image to be updated as a base image of the image generation, using the partial semantics to be updated corresponding to the second segmented image to be updated as input words, and inputting them into StableDiffusion for batch image generation operation to obtain multiple updated partial images;

[0015] S80, obtaining an updated partial image selected by an operator from a plurality of updated partial images as an updated partial image corresponding to the second segmented image to be updated, and merging the unchanged second segmented image and the updated partial images corresponding to each second segmented image to be updated according to the number of layers to obtain a pending generated image;

[0016] S90, based on the basic image, the second segmented image, each basic similar image in the basic similar image group, and each segmented similar image in the segmented similar image group, the color, lighting, and texture of the undetermined generated image are optimized and adjusted to obtain a final directional generated image.

[0017] On the basis of the above technical solution, the AI ​​painting model directional drawing method of the present invention can also be improved as follows:

[0018] The first similarity threshold is 0.85-0.95.

[0019] The second similarity threshold is 0.75-0.85.

[0020] Among them, the step of determining the number of layers of each first segmented image includes: determining the height of the subject in the image according to the area ratio of the subject to the basic image; further adjusting its hierarchical relationship in the basic image according to the semantic information of the subject, and finally obtaining the number of layers of the first segmented image corresponding to the subject.

[0021] In the step of merging the unchanged second segmented image and the updated partial image corresponding to each second segmented image to be updated according to the number of layers, alpha blending is used to perform layer synthesis.

[0022] Wherein, in the step of optimizing and adjusting the color, illumination, and texture of the undetermined generated image based on the basic image, the second segmented image, each basic similar image in the basic similar image group, and each segmented similar image in the segmented similar image group, the following method is adopted:

[0023] Color optimization: According to the color histogram of the basic similar image, the color of the generated image is corrected to make it more saturated and coordinated;

[0024] Lighting optimization: Based on the lighting information of the basic similar images, the shadows and highlights of the image to be generated are simulated to make it more natural and three-dimensional;

[0025] Texture optimization: Based on the texture features of segmented similar images, texture synthesis is performed on the local area of ​​the image to be generated to improve the detail texture.

[0026] The method for determining the subject of each first segmented image is to adopt a target detection algorithm based on Mask R-CNN.

[0027] Among them, the step of merging the unchanged second segmented image and the updated partial image corresponding to each second segmented image that needs to be updated according to a hierarchical relationship also includes an image synthesis model based on deep learning to further optimize the details of the synthesized image.

[0028] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned AI painting model directional drawing method.

[0029] The third aspect of the present invention provides an AI painting model directional drawing system, which includes the above-mentioned computer-readable storage medium.

[0030] Compared with the prior art, the AI ​​painting model directional drawing method, medium and system provided by the present invention have the following beneficial effects:

[0031] First, this method can fully mine the information of various reference images, including the text description entered by the user, the initial image group generated by StableDiffusion, and similar reference images. Through in-depth analysis and utilization of this information, a solid foundation is laid for subsequent image editing. Compared with the existing one-click generation method, this approach of making full use of various information sources greatly improves the pertinence and personalization of the generated images.

[0032] Secondly, the method combines the user's interactive needs to gradually complete the semantic segmentation, supplementary generation, synthetic editing and visual optimization of the image. Through multiple rounds of interaction and fine processing, users can customize the editing and optimization of the image according to their needs while retaining the core content of the image. This editing process from coarse to fine not only greatly improves user participation and satisfaction, but also ensures the quality of the final generated image and meets expectations.

[0033] Thirdly, this method makes full use of advanced computer vision and image generation technologies, including semantic segmentation, object detection, image synthesis, visual optimization and other algorithms and principles. These technical means are highly consistent with the goals and implementation of this method, ensuring the visual effect and quality of the generated images. Compared with traditional manual editing methods, this method has strong automation and intelligent features, greatly improving editing efficiency, and is particularly suitable for scenarios that require a lot of repetitive editing.

[0034] In summary, the solution of the present invention solves the technical problem that most existing AI painting models remain at the level of "one-click generation", lack flexible editing functions, and cannot produce directional drawings according to the direction selected by the operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0036] Figure 1 A flow chart of the method provided by the present invention;

[0037] Figure 2 is a basic image in the embodiment;

[0038] Figure 3 This is the final generated image in the embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] like Figure 1 As shown, it is a flow chart of a method for directional drawing output of an AI painting model provided by the present invention, and the method comprises the following steps:

[0041] S10, obtaining the image-generated text input by the user into the image generation AI model tool StableDiffusion, recorded as the first text; obtaining the image group generated by StableDiffusion, and obtaining the basic image selected by the operator from the image group; obtaining multiple images in the image group whose similarity with the basic image is greater than a preset first similarity threshold, recorded as the basic similar image group, including multiple basic similar images;

[0042] S20, performing semantic segmentation of the base image fused with the first text to obtain a plurality of segmented images, and performing transparent filling on each segmented image to generate a first segmented image;

[0043] S30, taking each first segmented image as the base image of the image generation, taking the semantics of the semantic segmentation corresponding to the first segmented image as the input word, inputting it into StableDiffusion for batch image generation operation, and obtaining a corresponding plurality of segmented supplementary images;

[0044] S40, for each first segmented image, obtaining an image selected by an operator from the corresponding multiple segmented supplementary images as a basic segmented supplementary image, recorded as a second segmented image; obtaining multiple images from the multiple segmented supplementary images corresponding to the second segmented image, whose similarity with the second segmented image is greater than a preset second similarity threshold, recorded as a segmented similar image group, including the multiple segmented similar images;

[0045] S50, determining the subject of each first segmented image, and determining the number of layers of each first segmented image according to the size ratio of the subject to the base image and the semantics corresponding to the first image, and assigning the number of layers of the corresponding first segmented image to each second segmented image;

[0046] S60, outputting each second segmented image and the corresponding semantics to the operator, obtaining the semantics of a part that the operator needs to update, and determining the second segmented images that the operator needs to maintain unchanged and the second segmented images that need to be updated;

[0047] S70, using each second segmented image to be updated as a base image of the image generation, using the partial semantics to be updated corresponding to the second segmented image to be updated as input words, and inputting them into StableDiffusion for batch image generation operation to obtain multiple updated partial images;

[0048] S80, obtaining an updated partial image selected by an operator from a plurality of updated partial images as an updated partial image corresponding to the second segmented image to be updated, and merging the unchanged second segmented image and the updated partial images corresponding to each second segmented image to be updated according to the number of layers to obtain a pending generated image;

[0049] S90, optimizing and adjusting the color, illumination, and texture of the to-be-determined generated image based on the basic image, the second segmented image, and each basic similar image in the basic similar image group and each segmented similar image in the segmented similar image group to obtain a final directional generated image.

[0050] The specific implementation methods of the above steps are described in detail below:

[0051] The specific implementation of step S10 is as follows:

[0052] First, this step aims to obtain the image generation text input by the user and the image group generated by StableDiffusion, and select the base image from them. Specifically, this step includes the following sub-steps:

[0053] 1. Obtain the image generation text input by the user into the image generation AI tool StableDiffusion, which is recorded as the first text. The first text is a semantic description of the target image that the user wants to generate.

[0054] 2. Get the image group generated by StableDiffusion. StableDiffusion is a diffusion model based on large-scale unsupervised pre-training, which can generate corresponding images based on the input text description. In this method, StableDiffusion generates a group of images as candidate outputs based on the first text.

[0055] 3. Obtaining a basic image selected by an operator from the image group. Since the generated image group may contain multiple images, the operator needs to select one of them as a basic image as a basis for subsequent processing.

[0056] 4. Obtain multiple images in the image group whose similarity with the basic image is greater than a preset first similarity threshold, and record them as a basic similar image group, including multiple basic similar images. In addition to the basic image, it is also necessary to obtain some images with higher similarity with the basic image as a reference for subsequent processing. Here, a similarity threshold is used to screen out some images with higher similarity.

[0057] In a specific implementation, cosine similarity or other common image similarity measurement methods can be used to calculate the similarity between each generated image and the base image, and images with similarity greater than a preset threshold (e.g., 0.8) are selected as the base similar image group. This ensures that there are sufficiently similar reference images for subsequent processing.

[0058] Through the above sub-steps, step S10 completes the goals of obtaining the image generated text input by the user, the image group generated by StableDiffusion, selecting the basic image, and obtaining the basic similar image group, laying the foundation for the processing of subsequent steps.

[0059] The specific implementation of step S20 is as follows:

[0060] The purpose of this step is to perform semantic segmentation on the base image and generate a transparently filled segmented image. It specifically includes the following sub-steps:

[0061] 1. Perform semantic segmentation on the base image. Semantic segmentation is an important technology in the field of computer vision, which can segment an image into multiple semantic regions, such as the sky, people, buildings, etc. Semantic segmentation technology is used here to divide the base image into multiple semantic regions according to its content.

[0062] 2. Fill each segmented area with transparent filling to generate the first segmented image. Since these segmented areas need to be supplemented and modified later, these areas need to be set to transparent to facilitate subsequent operations.

[0063] Through these sub-steps, step S20 obtains a set of transparent segmented images, which provides a basis for subsequent graphics generation and editing. Semantic segmentation technology can fully understand the content structure of the image, laying the foundation for subsequent refined processing, while transparent filling facilitates image editing.

[0064] The specific implementation of step S30 is as follows:

[0065] The purpose of this step is to perform image-to-image operations on each first segmented image to generate a corresponding segmented supplementary image. Specifically, it includes the following sub-steps:

[0066] 1. Each first segmented image is used as the base image of the image generation image. In other words, the first segmented image is used as the basic image, and the image generation is performed on this basis.

[0067] 2. The semantic segmentation information corresponding to the first segmented image is used as the input word and input into StableDiffusion for batch image generation. In other words, the semantic segmentation information of the first segmented image is used as the text description and input into the StableDiffusion model for image generation. Batch operation is used here to generate multiple segmented supplementary images at one time.

[0068] Through these sub-steps, step S30 obtains a set of segmented supplementary images corresponding to each first segmented image. These supplementary images can be used for subsequent modification and improvement of the first segmented image. As a powerful image generation model, StableDiffusion can generate corresponding images based on the input semantic description, providing important support for subsequent image editing.

[0069] The specific implementation of step S40 is as follows:

[0070] The purpose of this step is to obtain the segmented supplementary image selected by the operator and filter out the image groups similar to it. It specifically includes the following sub-steps:

[0071] 1. For each first segmented image, an image selected by the operator from the corresponding multiple segmented supplementary images is obtained as a basic segmented supplementary image, recorded as the second segmented image. In other words, the operator needs to select one second segmented image as a basic image from the segmented supplementary images generated in step S30.

[0072] 2. Obtain multiple images corresponding to the second segmented image, and the images whose similarity with the second segmented image is greater than a preset second similarity threshold are recorded as a segmented similar image group, including multiple segmented similar images. Similar to step S10, here it is also necessary to screen out some images with high similarity with the second segmented image according to the similarity threshold (e.g., 0.7) as a reference for subsequent processing.

[0073] Through these sub-steps, step S40 obtains the segmented supplementary image selected by the operator, that is, the second segmented image, and also obtains a group of segmented similar images similar thereto, providing sufficient materials for subsequent image editing.

[0074] The specific implementation of step S50 is as follows:

[0075] The purpose of this step is to determine the subject of each first segmented image, and determine the number of layers of each first segmented image according to the size ratio of the subject to the base image and the semantic information, and assign the corresponding number of layers to the second segmented image. It specifically includes the following sub-steps:

[0076] 1. Determine the subject of each first segmented image. By performing target detection and semantic analysis on the first segmented image, the main target object in the image can be identified and recorded as the subject. The subject here can be a significant semantic entity such as a person, a building, or a vehicle.

[0077] 2. Determine the number of layers of each first segmented image based on the size ratio of the subject to the base image and the semantics corresponding to the first segmented image. The number of layers reflects the importance and positional relationship of the subject in the entire image. For example, a subject that occupies a larger proportion of the image should be located at a lower level, while a subject that occupies a smaller proportion should be located at an upper level. At the same time, subjects with different semantics should also have different hierarchical relationships.

[0078] 3. For each second segmented image, assign the number of layers of the corresponding first segmented image. In other words, the hierarchical relationship of the first segmented image is inherited to the second segmented image to ensure that they maintain the correct hierarchical relationship when they are finally synthesized.

[0079] Through these sub-steps, step S50 establishes the hierarchical relationship between the first segmented image and the second segmented image, providing an important basis for subsequent image synthesis. This hierarchical determination method based on subject, size and semantics can fully reflect the spatial structure and semantic structure of the image, providing a good foundation for image editing operations.

[0080] The specific implementation of step S60 is as follows:

[0081] The purpose of this step is to obtain the operator's update requirements for the second segmented image and determine which parts need to remain unchanged and which parts need to be updated. It specifically includes the following sub-steps:

[0082] 1. Output each second segmented image and its corresponding semantics to the operator. In other words, the second segmented image and its semantic information are displayed to the operator for evaluation and modification.

[0083] 2. Obtain the semantics that the operator needs to update. The operator can mark which semantic areas need to be updated or modified according to their needs.

[0084] 3. Determine the second segmented image that the operator needs to maintain and the second segmented image that needs to be updated. According to the operator's feedback, the second segmented image is divided into two parts, which need to be maintained and which need to be updated.

[0085] Through these sub-steps, step S60 obtains the operator's modification requirements for the second segmented image, providing specific guidance for subsequent image updates. This interactive approach can fully meet the user's personalized image editing requirements and improve the satisfaction of the final generated image.

[0086] The specific implementation of step S70 is as follows:

[0087] The purpose of this step is to perform an image-to-image operation on the second segmented image to be updated to generate an updated partial image. Specifically, it includes the following sub-steps:

[0088] 1. Each second segmented image to be updated is used as the base image of the image generation image. In other words, the second segmented image to be updated is used as the basic image, and the image generation is performed on this basis.

[0089] 2. Take the semantic part of the second segmented image that needs to be updated as the input word and input it into StableDiffusion for batch image generation. Use the semantic area that needs to be updated marked by the operator as the text description and input it into the StableDiffusion model for image generation. Here, batch operation is also used to generate multiple updated partial images at one time.

[0090] Through these sub-steps, step S70 obtains a set of updated partial images corresponding to the second segmented image to be updated. These updated partial images can be used to replace the original second segmented image to achieve local update of the image. The use of the StableDiffusion model ensures the quality of the updated partial images and the coordination with the style of the original image.

[0091] The specific implementation of step S80 is as follows:

[0092] The purpose of this step is to merge the unchanged second segmented image and the updated part of the second segmented image that needs to be updated according to the determined hierarchical relationship to generate the final directional generation map. Specifically, it includes the following sub-steps:

[0093] 1. Obtain an updated partial image selected by the operator from multiple updated partial images as the updated partial image corresponding to the second segmented image to be updated. In other words, the operator needs to select an updated partial image from the updated partial images generated in step S70 to replace the original second segmented image.

[0094] 2. The unchanged second segmented image and the updated partial image corresponding to each second segmented image to be updated are synthesized according to the hierarchical relationship determined in step S50 to obtain a pending generated image. Through reasonable hierarchical arrangement, the unchanged part and the updated part are organically combined to form the final image.

[0095] Through these sub-steps, step S80 completes the final synthesis of the image and generates a pending directional generation map. This step integrates the processing results of the previous steps, fully reflects the user's editing needs, and lays the foundation for subsequent further optimization.

[0096] The specific implementation of step S90 is as follows:

[0097] The purpose of this step is to further optimize and adjust the pending generated graph to obtain the final directional generated graph. It specifically includes the following sub-steps:

[0098] 1. Based on the basic image, the second segmented image, each basic similar image in the basic similar image group, and each segmented similar image in the segmented similar image group, the color, lighting, and texture of the to-be-determined generated image are optimized and adjusted. In other words, the visual effect of the to-be-determined generated image is optimized by using the various reference images obtained previously, so that its color, lighting, texture, etc. are more coordinated and natural.

[0099] 2. Through the above optimization and adjustment, the final directional generation map is obtained. After this step of fine processing, the final generated image will be more in line with the user's expectations and aesthetics, achieving the goal of directional generation.

[0100] Through these sub-steps, step S90 completes the optimization processing of the final image, ensuring the visual effect and quality of the generated image. Here, various reference images obtained previously are comprehensively utilized to fully explore their visual features, making the directional generation result more natural and coordinated.

[0101] In summary, the method for directional drawing of an AI painting model proposed in the present invention, through step-by-step processing, from obtaining user input and initial images to the final directional generated image, makes full use of advanced computer vision and image generation technology, and provides users with a highly customized image editing experience. The core of this method is to fully explore the information of various reference images, combine the user's interactive needs, and gradually complete the semantic segmentation, supplementary generation, synthetic editing and visual optimization of the image, and finally generate a directional image that meets user expectations. The key algorithms and principles involved in the whole process include semantic segmentation, image generation, similarity calculation, hierarchy determination, image synthesis and visual optimization, etc. These technical means are highly consistent with the goals and implementations of this method. Through this method, users can customize the editing and optimization of images according to their needs while retaining the core content of the image, which greatly improves the flexibility and satisfaction of image generation.

[0102] The technical solution of this patent has the following advantages:

[0103] 1. Make full use of advanced computer vision and image generation technology to provide users with a highly customized image editing experience.

[0104] 2. Through step-by-step processing, from obtaining user input and initial image to the final directional generated graph, a coarse-to-fine image editing process is achieved.

[0105] 3. Combined with the user's interactive needs, the image's semantic segmentation, supplementary generation, synthetic editing and visual optimization are gradually completed to meet the user's personalized needs.

[0106] 4. Fully exploit the information of various reference images to improve the visual effect and quality of the final generated image.

[0107] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned AI painting model directional drawing method.

[0108] The third aspect of the present invention provides an AI painting model directional drawing system, which includes the above-mentioned computer-readable storage medium.

[0109] The following is a specific example to illustrate the technical implementation and effect of the AI ​​painting model directional drawing method of the present invention.

[0110] The art team of a game company is designing the protagonist image for a role-playing game that is about to be launched. They need to generate a series of distinctive character modeling pictures based on the game plot and style characteristics as a reference for subsequent modeling and rendering. This is exactly the typical application scenario of the method of the present invention.

[0111] 1. Step S10: Obtain initial input

[0112] First, the art team used the StableDiffusion tool to input a text describing the protagonist: "A weathered but brave old warrior, wearing dark blue armor, a steel helmet, holding a Hopps double-edged battle axe, with a resolute expression." StableDiffusion generated 20 candidate images based on this text. The art team selected one as the base image, such as Figure 2 As shown, 5 images with similarity greater than 0.8 to the basic image are obtained at the same time to form a basic similar image group.

[0113] 2. Step S20: semantic segmentation and transparency

[0114] Next, based on the selected base image, a semantic segmentation algorithm (such as Mask R-CNN) is used to divide it into seven semantic regions: helmet, face, body armor, arms, battle axe, background environment, and robe folds. Each semantic region is assigned a different label. Then, these semantic regions are transparently filled to generate the first segmented image.

[0115] 3. Step S30: Segmentation and supplementation generation

[0116] The first segmented image was used as the base image, and its corresponding semantic labels were input into the StableDiffusion model for batch image generation. After 50 sampling steps, the temperature parameter was set to 0.7, and 5 supplementary images were generated for each semantic region. In this way, a total of 35 segmented supplementary images were obtained.

[0117] 4. Step S40: Segmentation Image Selection

[0118] The art team selected the 7 most expected images from the 35 segmentation supplementary images as the second segmentation images. At the same time, the similarity between these 7 second segmentation images and the other 28 supplementary images was calculated (using the Siamese network), and 4 images with a similarity greater than 0.7 with each second segmentation image were selected to form a segmentation similarity image group.

[0119] 5. Step S50: Level determination

[0120] According to the object detection algorithm (such as Mask R-CNN), the main area of ​​each first segmented image is determined: helmet, body armor and battle axe. Combined with the size ratio and semantic information of the main area, the hierarchical relationship of each semantic area is determined according to the following rules:

[0121] (1) The helmet and face are on the top layer;

[0122] (2) Body armor is located in the middle layer;

[0123] (3) The battle axe and background environment are at the bottom layer;

[0124] (4) The folds of the robe are appropriately distributed in the middle and bottom layers according to their proportions.

[0125] The hierarchical relationship determined in this way can better reflect the spatial structure and semantic relationship of the character modeling. This hierarchical relationship is given to the second segmented image.

[0126] 6. Step S60: User interactive editing

[0127] Each second segmentation image and its semantic information were presented to the art team through a friendly UI interface, allowing them to review and modify. After communication, the art team wanted to keep 5 second segmentation images and needed to update the helmet and battle axe parts of 2 second segmentation images.

[0128] 7. Step S70: Update part generation

[0129] For the two second segmented images that need to be updated, we used them as the base images and input the corresponding "helmet" and "tomahawk" semantic labels into StableDiffusion for batch image generation. After 50 sampling steps and a temperature parameter of 0.7, 10 updated partial images were generated. The art team selected the two most satisfactory images.

[0130] 8. Step S80: Image synthesis

[0131] The five second segmented images that the art team selected to be retained, as well as the updated partial images corresponding to the two second segmented images that needed to be updated, were synthesized according to the previously determined hierarchical relationship. Alpha blending was used to blend the layers to ensure a natural transition. After the synthesis was completed, basic adjustments such as saturation and contrast were made to the overall image to further enhance the visual effect.

[0132] 9. Step S90: Visual Optimization

[0133] Finally, based on the base image, the second segmented image, the base similar image group, and the segmented similar image group, the synthesized image is deeply visually optimized:

[0134] (1) Color optimization: Based on the color histogram of the basic similar image, the color of the main character’s image is corrected to make it more saturated and coordinated;

[0135] (2) Lighting optimization: Refer to the lighting information of similar basic images to add natural shadows and highlights to the main character's shape to enhance the three-dimensional effect;

[0136] (3) Texture optimization: Using the texture features of segmented similar images, detailed texture synthesis is performed on the protagonist’s armor, weapons and other parts to enhance the texture.

[0137] Through these targeted visual optimizations, we finally got a protagonist image that meets the game style and the expectations of the art team, such as Figure 3 shown.

[0138] 2. Technical Effect Verification

[0139] Through the above specific embodiments, it can be seen that the AI ​​painting model directional drawing method of the present invention has achieved remarkable results in solving existing technical problems and improving user experience.

[0140] 1. Make full use of reference information to improve generation targeting

[0141] Compared with the traditional image generation method driven by single text input, the method of the present invention makes full use of the initial image group and its similar images generated by Stable Diffusion, providing rich reference materials for subsequent customized editing.

[0142] Taking the above embodiment as an example, the art team not only inputs text describing the image of the protagonist, but also selects a basic image, such as Figure 2 As shown in the figure, five images with high similarity are obtained as references. These diverse reference information enable subsequent steps such as semantic segmentation, supplementary generation, and image synthesis to better meet the actual needs of users and produce highly targeted results.

[0143] In contrast, if only text input is relied upon, the generated image is likely to fail to fully meet the user's expectations and requires a lot of manual modification. The method of the present invention makes full use of various reference information sources, greatly improving the pertinence and fit of the generated image.

[0144] 2. Interactive editing process to meet personalized needs

[0145] The method of the present invention adopts a step-by-step, interactive image editing process, which fully respects the personalized needs of users. As shown in the above embodiments, user interaction is integrated into key steps such as semantic segmentation, supplementary generation, and image synthesis, allowing the art team to adjust and optimize the generated results in real time according to their own creative demands and aesthetic preferences.

[0146] For example, in step S60, each second segmented image and its semantic information are displayed to the art team through a friendly UI interface, allowing them to select the parts to be retained and updated. This interactive editing method not only improves user participation, but also ensures that the final generated image can meet personalized needs.

[0147] Compared with the traditional "one-size-fits-all" image generation method, the method of the present invention incorporates sufficient user interaction links, so that the generated results can better reflect the user's creative intentions and aesthetic pursuits, greatly improving the user experience.

[0148] 3. Combining automation and intelligence to improve editing efficiency

[0149] The method of the present invention makes full use of advanced image generation models such as StableDiffusion to achieve multiple rounds of automatic supplementation and optimization. At the same time, it also incorporates human interaction, allowing users to fine-tune the generation results according to their preferences. This human-computer collaboration method greatly improves editing efficiency while ensuring the quality of generation.

[0150] Taking the above embodiment as an example, in steps S30 and S70, StableDiffusion is used to perform batch segmentation, supplementation and update of partial generation, which greatly saves the time and effort of manual operation. At the same time, in steps S40 and S80, user selection feedback is incorporated to ensure that the generated results can meet personalized needs.

[0151] Compared with the traditional manual image editing method, the method of the present invention can give full play to the advantages of automation technology, while integrating human interaction, greatly improving the editing efficiency while ensuring the quality of generation. This human-computer collaborative design is particularly suitable for scenarios that require a lot of repetitive editing (such as game art asset generation).

[0152] Specifically, the principle of the present invention is:

[0153] 1. Make full use of various reference information sources to achieve targeted image generation. This method not only obtains the text description of the target image that the user expects to generate, but also uses the initial image group and its similar images generated by StableDiffusion to provide rich materials for subsequent image editing. Compared with relying solely on a single text input for image generation, this method can better understand the user's creative appeal and lay a good foundation for customized editing.

[0154] 2. Adopt a step-by-step, interactive image editing process to meet the personalized needs of users. This method divides the image editing process into multiple steps, such as semantic segmentation, supplementary generation, synthetic editing, and visual optimization, making full use of advanced algorithms in the field of computer vision, such as target detection and image synthesis. At the same time, it also incorporates user interactive feedback, allowing users to adjust and optimize the generated results at each step. This editing process from coarse to fine not only improves user participation, but also ensures that the final generated image can meet personalized needs.

[0155] 3. Organically combine automation technology with human interaction to improve editing efficiency. This method makes full use of powerful image generation models such as StableDiffusion to achieve multiple rounds of automated image supplementation and optimization. At the same time, it also incorporates human interaction, allowing users to fine-tune the generated results according to their own aesthetic preferences. This human-computer collaborative approach greatly improves editing efficiency while ensuring generation quality, and is particularly suitable for scenarios that require a large amount of repetitive editing.

[0156] 4. Give full play to the advantages of computer vision technology to ensure the visual effect of the generated image. The key technologies involved in this method include semantic segmentation, target detection, image synthesis, visual optimization, etc. These algorithms and principles are highly consistent with the goals and implementation of this method. By organically integrating these advanced technologies into the image editing process, not only the quality and detail of the generated image are ensured, but also the intelligent level of editing is greatly improved.

[0157] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for directional drawing of AI painting model, It is characterized in that The following steps are involved: S10, obtaining the image-generated text input by the user into the image generation AI model tool StableDiffusion, recorded as the first text; obtaining the image group generated by StableDiffusion, and obtaining the basic image selected by the operator from the image group; obtaining multiple images in the image group whose similarity to the basic image is greater than a preset first similarity threshold, recorded as the basic similar image group, including multiple basic similar images; S20, performing semantic segmentation of the first text on the basic image to obtain a plurality of segmented images, and performing transparent filling on each segmented image to generate a first segmented image; S30, taking each first segmented image as the base image of the image generation, taking the semantics of the semantic segmentation corresponding to the first segmented image as the input word, inputting it into StableDiffusion for batch image generation operation, and obtaining a plurality of corresponding segmented supplementary images; S40, for each first segmented image, obtaining an image selected by an operator from the corresponding plurality of segmented supplementary images as a basic segmented supplementary image, recorded as a second segmented image; Acquire, from among the plurality of segmented supplementary images corresponding to the second segmented image, a plurality of images whose similarity to the second segmented image is greater than a preset second similarity threshold, and record them as a segmented similar image group, including the plurality of segmented similar images; S50, determining the main body of each first segmented image, and determining the number of layers of each first segmented image according to the size ratio of the main body to the basic image and the semantics corresponding to the first segmented image, and assigning the number of layers of the corresponding first segmented image to each second segmented image; S60, outputting each second segmented image and the corresponding semantics to the operator, obtaining the semantics of a part that the operator needs to update, and determining the second segmented images that the operator needs to maintain unchanged and the second segmented images that need to be updated; S70, using each second segmented image to be updated as a base image of the image generation, using the partial semantics to be updated corresponding to the second segmented image to be updated as input words, and inputting them into StableDiffusion for batch image generation operation to obtain multiple updated partial images; S80, obtaining an updated partial image selected by an operator from a plurality of updated partial images as an updated partial image corresponding to the second segmented image to be updated, and merging the unchanged second segmented image and the updated partial images corresponding to each second segmented image to be updated according to the number of layers to obtain a pending generated image; S90, based on the basic image, the second segmented image, each basic similar image in the basic similar image group, and each segmented similar image in the segmented similar image group, the color, illumination, and texture of the to-be-determined generated image are optimized and adjusted to obtain a final directional generated image, specifically including: Color optimization: According to the color histogram of the basic similar image, the color of the generated image is corrected to make it more saturated and coordinated; Lighting optimization: Based on the lighting information of the basic similar images, the shadows and highlights of the image to be generated are simulated to make it more natural and three-dimensional; Texture optimization: Based on the texture features of segmented similar images, texture synthesis is performed on the local area of ​​the image to be generated to improve the detail texture.

2. According to claim 1, a method for directional drawing of an AI painting model, It is characterized in that The first similarity threshold is 0.85-0.

95.

3. According to claim 1, a method for directional drawing of an AI painting model, It is characterized in that The second similarity threshold is 0.75-0.

85.

4. According to claim 1, a method for directional drawing of an AI painting model, It is characterized in that The step of determining the number of layers of each first segmented image includes: determining the height of the subject in the image according to the area ratio of the subject to the basic image; further adjusting the hierarchical relationship of the subject in the basic image according to the semantic information of the subject, and finally obtaining the number of layers of the first segmented image corresponding to the subject.

5. According to claim 1, a method for directional drawing of an AI painting model, It is characterized in that In the step of merging the unchanged second segmented image and the updated partial image corresponding to each second segmented image to be updated according to the number of layers, alpha blending is used to perform layer synthesis.

6. The method for directional drawing of an AI painting model according to claim 1, It is characterized in that The method for determining the subject of each first segmented image is to adopt an object detection algorithm based on Mask R-CNN.

7. The method for directional drawing of an AI painting model according to claim 1, It is characterized in that The step of merging the unchanged second segmented image and the updated partial image corresponding to each second segmented image that needs to be updated according to a hierarchical relationship also includes an image synthesis model based on deep learning to further optimize the details of the synthesized image.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program instructions, which, when executed, are used to execute the AI ​​painting model directional drawing method according to any one of claims 1 to 7.

9. An AI painting model directional drawing system, It is characterized in that Contains the computer-readable storage medium of claim 8.