Picture customization generation method based on diffusion model and multi-modal large language model

Abstract

The application discloses a picture customization generation method based on a diffusion model and a multimodal large language model, performs feature extraction on an input image, splices output features, and obtains a high-dimensional image feature vector of each object and overall style in the image; performs multimodal large model image-text feature fusion based on a mask attention mechanism on the image features and a scene description text input by a user to obtain target image features; inputs the target image features into an image generator based on the diffusion model, introduces a mask-based attention injection mechanism in the image generation process to accurately control the layout and local details of each object in the picture; and performs post-processing to obtain a high-quality picture that finally meets the customization needs of the user. The design can reduce the dependence on large-scale data sets, has good generalization ability, and is suitable for customized picture generation in different styles and application scenarios.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a method for customized image generation based on a diffusion model and a multimodal large language model. Background Technology

[0002] In recent years, deep learning-driven image generation technology has made significant progress and has been applied in various fields such as artistic creation, advertising design, and industrial manufacturing. While existing technologies based on image style transfer or image-to-image generation can achieve a certain degree of image enhancement, they have significant shortcomings in terms of controllable image generation, including:

[0003] Most existing methods directly convert the input image, making it difficult to flexibly adjust the details, object states, and composition of the generated image based on text prompts.

[0004] Traditional image generation techniques often lack targeted control when dealing with complex scenes and multi-object interactions, resulting in poor performance in terms of content consistency and visual coherence.

[0005] Due to the lack of a mechanism for efficient fusion of cross-modal information (text and images), existing systems have limitations in meeting user customization requirements and are unable to satisfy the practical application's need for accurate and dynamic image generation. Summary of the Invention

[0006] To overcome the shortcomings of the existing technologies, this invention provides a method for customizing and generating images based on a diffusion model and a multimodal large language model. The aim is to customize and dynamically adjust multiple elements (such as objects, backgrounds, and details) in an image based on user-input text descriptions and example images, while ensuring controllability in the overall composition, style, and detail of the generated image. This allows for precise control over the content, style, composition, and details of the generated image, thereby producing high-quality, personalized custom images suitable for various application scenarios requiring controllable image generation.

[0007] The core of this invention is: First, by using a pre-trained image encoder (such as the multimodal model CLIP (Contrastive Language-Image Pre-training) model) and an encoder optimized for a specific style (such as Magi, see the paper The Manga Whisperer: Automatically Generating Transcriptions for Comics) to extract features from the input example image, a high-dimensional representation of each object and the overall style in the image is obtained.

[0008] Secondly, the extracted image features are fused with the scene description text input by the user. A multimodal large language model (such as SEED-X, from the paper SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation) is used as a feature adapter to perform multimodal large model image-text feature fusion based on mask attention mechanism. The image features are adjusted for text compatibility to obtain the adjusted and adapted target image features, so as to achieve precise control of object state, expression, posture or background details based on text.

[0009] Next, the adapted target image features and the scene description text input by the user are input into the diffusion model-based image generator (e.g., Stable Diffusion XL, from the paper SDXL: Improving LatentDiffusion Models for High-Resolution Image Synthesis). A mask-based attention injection mechanism is introduced during the image generation process to precisely control the layout and local details of each object in the image.

[0010] Finally, the generated preliminary images are post-processed to optimize details and enhance visual effects, resulting in high-quality images that ultimately meet the user's customized needs.

[0011] The specific technical solution is as follows:

[0012] A method for customized image generation based on a diffusion model and a multimodal large language model includes the following steps:

[0013] 1) Image feature extraction

[0014] The image is forward-inferenced using a pre-trained CLIP image encoder (such as ViT-B / 16 or ViT-L / 14) to obtain its image feature representation in CLIP space, denoted as f. CLIP = CLIPEncoder(I). The CLIP model is used to capture the semantic content of an image and the high-level semantic features corresponding to the text.

[0015] In specific visual domain scenarios such as comics / illustrations, to better capture the style and local details of images, an image encoder (referred to here as the "Magi model") specifically pre-trained or fine-tuned for the comics / illustrations domain can be used. The image is input into this image encoder model to obtain its style and local feature representation, denoted as f. Magi=MagiEncoder(I). The Magi model was originally used for image object detection. This invention uses the Magi model to save detailed features such as specific line drawings, brushstrokes, and grid information, thereby improving the controllability of example images in customized image generation.

[0016] This invention concatenates the output features of the two models at the hidden layer dimension (or channel dimension) to obtain a more comprehensive high-dimensional image feature vector c. C = [f CLIP ||f Magi The high-dimensional image feature vector c obtained in this way can simultaneously contain multiple aspects of the image, such as semantic content, style, and local details, thereby improving the textual controllability of the generated image.

[0017] 2) Parse the text prompt to obtain the text token;

[0018] Natural language processing is performed on the image scene description text provided by the user to extract key information such as object state, composition layout, and detailed requirements (text segmentation).

[0019] 3) Perform multimodal large-scale model image-text feature fusion based on mask attention mechanism;

[0020] By using a multimodal large language model as a feature adapter, pre-extracted image features are fused with text information. Specifically, an attention mechanism with a mask is used so that the corresponding image features are only subject to attention calculation with the corresponding text description, while other features are covered by a mask.

[0021] This invention proposes a masked attention mechanism. In the self-attention layer of a multimodal large language model, a masked attention mechanism is introduced to match and focus on the features of different example images (such as multiple characters in an image) with their corresponding text segments, and an irrelevant information is masked using a mask. The mask matrix M is defined by the following formula:

[0022]

[0023] When calculating the attention score, a maximum negative value (negative infinity, such as -1e6) is subtracted from the positions in the attention matrix where the mask matrix M is 0, thereby masking the attention connections of image features and irrelevant text segmentation.

[0024] 4) Image generation

[0025] The adjusted target image features and text prompts are input into a diffusion-based image generator. At each generation stage, the generator employs a mask-based attention injection mechanism to precisely position and control details of objects and local regions within the image. A local region control vector is introduced, and this vector is summed with image features using positional weights to achieve precise positioning for each image. This process progressively generates preliminary image results that meet the text description and customization requirements.

[0026] 5) Image post-processing

[0027] The initially generated images undergo detailed optimization and local adjustments (such as image enhancement, edge sharpening, and color correction) to improve the overall visual effect and artistic expression, resulting in the final customized image.

[0028] In specific implementation, the system constructed using the method of this invention mainly includes four modules: CLIP model, Magi model, multimodal large language model, and image generator based on diffusion model. CLIP model and Magi model are responsible for encoding reference image information, multimodal large language model is responsible for fusing the encoded reference image information with text input, and image generator based on diffusion model is responsible for generating the final image.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] This invention provides a method for customizing and generating images based on a diffusion model and a multimodal large language model. It enables dynamic customization of multiple objects in an image according to text prompts, thus solving the problem of fixed object states in existing image conversion methods.

[0031] The method of this invention introduces a multimodal large language model as a feature adapter to fuse pre-extracted image features with text information, effectively alleviating the rigidity caused by directly copying example image features, and making the generated images more diverse and flexible in terms of object representation and detail processing.

[0032] The method of this invention ensures the accurate layout of objects and local details in an image through a mask-based attention injection mechanism, thereby improving the overall coherence and aesthetics of the generated image. Attached Figure Description

[0033] Figure 1 The overall flowchart of the image customization generation method provided by the present invention is shown. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0035] This invention provides a method for customized image generation based on a diffusion model and a multimodal large language model, comprising the following steps:

[0036] ① Image preprocessing and feature extraction

[0037] The user-provided sample images are first preprocessed, including image normalization, resizing, and noise removal. After preprocessing, the CLIP model and an image encoder trained for a specific style are used to extract image features, obtaining high-dimensional feature vectors that reflect the image content, style, and details.

[0038] ②Text prompt parsing and information fusion

[0039] The system performs word segmentation, syntactic analysis, and key information extraction on the user-input image scene description text to clarify the object states (such as expressions, postures, and actions), composition requirements, and local details involved in the description. Subsequently, the text information and image features are fused using a multimodal large language model. During the adaptation process, the image features are dynamically adjusted by calculating the language model loss and mean squared error loss, enabling them to respond to text prompts and reflect changes in object states and details.

[0040] ③ Image generation and local control

[0041] The adapted target image features and text prompts are input into an image generator based on a diffusion model. During each diffusion step, the generator introduces a mask-based attention injection mechanism to perform cross-attention calculations on each object and local region in the image, ensuring that each object is generated within a preset area. Simultaneously, it performs specialized control over key local details, thereby generating a preliminary customized image.

[0042] ④ Image post-processing

[0043] The initially generated images undergo post-processing optimization, including image enhancement, edge sharpening, and color correction, to improve the overall visual effect and artistic expression, ultimately outputting high-quality customized images.

[0044] In the above embodiments, each module and step can be adjusted according to specific application requirements. For example, different image encoders or large language models can be selected, or the diffusion model parameters can be fine-tuned according to the style of the generated image. The above embodiments are only used to illustrate the technical solution of the present invention and do not limit the scope of protection of the present invention.

[0045] This invention utilizes a diffusion model and a multimodal large language model to achieve customized generation of multiple objects in an image. Compared with existing techniques that rely solely on image-to-image conversion, this solution achieves dynamic adjustments to various aspects such as object state, expression, posture, and background details through deep fusion of text and image features, demonstrating significant technological innovation.

[0046] This invention employs a mask-based attention injection mechanism, which can precisely control the layout and presentation of each object and local detail during the generation process. This ensures that the generated image achieves the expected results in terms of overall composition, local details, and visual coherence, significantly improving the controllability and artistic expression of image generation.

[0047] In specific implementation, the system implemented by this invention mainly includes four modules: CLIP model, Magi model, multimodal large language model, and image generator based on diffusion model. CLIP model and Magi model are responsible for encoding reference image information, multimodal large language model is responsible for fusing the encoded reference image information with text input, and image generator based on diffusion model is responsible for generating the final image.

[0048] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for customized image generation based on a diffusion model and a multimodal large language model, characterized in that, include: The input image is processed by pre-trained image encoder and style-optimized encoder. The output features are concatenated in the hidden layer dimension or channel dimension to obtain high-dimensional image feature vectors of each object and the overall style in the image. The high-dimensional image feature vectors contain the semantic content, style and local detail information of the image. Natural language processing is performed on the image scene description text provided by the user to extract key information about object state, composition layout, and detail requirements from the image scene description; The extracted image features are fused with the scene description text input by the user. A multimodal large language model is used as a feature adapter to perform multimodal large model image-text feature fusion based on mask attention mechanism. The image features are then adjusted for text compatibility to obtain the target image features. The target image features and the scene description text input by the user are fed into the diffusion model-based image generator. A mask-based attention injection mechanism is introduced during the image generation process to precisely control the layout and local details of each object in the image. The generated preliminary images are post-processed to optimize details and enhance visual effects, resulting in high-quality images that ultimately meet the user's customized needs.

2. The image customization generation method based on diffusion model and multimodal large language model as described in claim 1, characterized in that, The pre-trained image encoder uses the CLIP multimodal model; by performing forward reasoning on the image, the image feature representation in the CLIP space is obtained, including the semantic content of the image and the high-level semantic features corresponding to the text; the encoder optimized for a specific style uses the Magi model; the image is input into the Magi model to obtain the style and local feature representation of the image, which is used to save the specific line drawing, brushstroke, and grid information details.

3. The image customization generation method based on diffusion model and multimodal large language model as described in claim 2, characterized in that, The encodings obtained using the Magi model and the CLIP model are concatenated at the hidden layer level to obtain high-dimensional image feature vectors for each object and the overall style in the image.

4. The image customization generation method based on diffusion model and multimodal large language model as described in claim 1, characterized in that, Multimodal large-scale model image-text feature fusion based on a masked attention mechanism is performed. Specifically, an attention mechanism incorporating a mask is introduced, ensuring that corresponding image features only undergo attention calculation with their corresponding text descriptions, while irrelevant information is masked out. The mask is defined as follows: When calculating the attention score, a maximum negative value is subtracted from the position where the mask matrix M is 0 in the attention matrix, thereby masking the attention connections of image features and irrelevant text segmentation.

Images