Modal language model image editing technology fusing non-perpetual culture elements

Abstract

The invention discloses a modal language model image editing technology fusing non-perpetual culture elements. The modal language model image editing technology comprises the steps that a model is selected and trained, an LLaMA model is selected as a basis, LoRA is introduced for adaptive fine adjustment, and in this way, the model is subjected to adaptive adjustment under the condition that original parameter freezing is kept; according to the method, the understanding and reasoning capabilities in instruction editing are enhanced by combining MLLM (such as LLaVA), the MLLM can perform collaborative learning across text and image modalities and extract deep semantic information, so that the model can not only process basic instructions, but also understand complex non-perpetual culture elements, and the method has great significance in improving the understanding of the model on the non-perpetual culture elements. An enhanced bidirectional interaction mechanism is designed, and the mechanism realizes deep interaction between image and text features through a cross attention mechanism, so that the image features can be used as query and key value pairs to perform bidirectional communication with the text features, and the expression of a model in a complex non-perpetual scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of instruction image editing, and specifically to a modal language model image editing technology integrating intangible cultural elements. Background Art

[0002] Command image editing is a technology that allows users to modify images through natural language commands. This field has developed rapidly in recent years. In order to help intangible cultural heritage be better inherited and developed in the digital age and empower intangible cultural heritage inheritance with digital intelligence technology, we proposed an image editing method combined with a multimodal large language model (MLLM) to use digital intelligence technology to co-create the value of intangible cultural heritage craftsmen's products, called "immersive". This method aims to enhance the understanding and reasoning capabilities of existing command editing technology, especially in complex command scenarios, such as processing multiple objects and multiple attributes, and ultimately apply it to the digital transformation and innovative development of intangible cultural heritage represented by the "three knives" hairpin technique.

[0003] In the field of instructional image editing, existing technologies such as "InstructPix2Pix" rely on simple text encoders to parse the instructions of the diffusion model. These methods work well when processing simple image editing instructions, but face the following challenges when facing complex scenes, especially when editing intangible cultural heritage (ICH) elements:

[0004] Difficulty in multi-object editing: When an image contains multiple objects and the instruction specifies to modify a specific attribute of an object (such as position, size, color), existing methods often have difficulty in executing accurately.

[0005] Deficiencies in complex reasoning scenarios: Some editing instructions require the model to have world knowledge (such as time, spatial relationships or common sense) for reasoning. Existing methods perform poorly in understanding and identifying intangible cultural elements that need to be edited.

[0006] Limitations of information interaction: In traditional methods, the interaction between images and texts is mostly one-way, lacking effective two-way information flow, which limits the ability to deeply understand and edit intangible cultural heritage elements. Summary of the invention

[0007] The purpose of the present invention is to provide a modal language model image editing technology that integrates intangible cultural elements, which has the advantages of improving the model's understanding and reasoning capabilities of complex instructions and solves the problems raised by the background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a modal language model image editing technology integrating intangible cultural elements, comprising:

[0009] Model selection and training, we selected the LLaMA model as the basis and introduced LoRA for adaptive fine-tuning. In this way, the model can make adaptive adjustments while keeping the original parameters frozen, which greatly improves the training efficiency and command comprehension ability. We use intangible cultural heritage-related training data for model training;

[0010] A two-way interaction module, the two-way interaction module includes: a feature extraction layer, an interaction layer and a fusion layer, the feature extraction layer includes text feature extraction and image feature extraction, the text feature extraction uses a pre-trained text encoder to extract a feature vector of a text instruction; the image feature extraction uses a pre-trained image encoder to extract a feature vector of an image; the interaction layer includes text-to-image interaction and image-to-text interaction, the text-to-image interaction inputs the text feature vector as conditional information into an image generation or editing model to affect the image generation or editing process, the image-to-text interaction feeds back the image feature vector to a text processing module to adjust the understanding of the text instruction or generate a text description related to the image, the fusion layer fuses the text and image feature vectors output by the interaction layer to generate a fused feature vector for subsequent image generation and editing tasks, and realizes multiple rounds of information flow between image and text features, verifies the effectiveness of the two-way interaction module through experiments, and adjusts model parameters to optimize performance.

[0011] Diffusion model, the diffusion model generates images through gradual iteration, starting from a random noise image. Through the step-by-step guidance and generation process of instructions, the model gradually approaches the target image. Therefore, when dealing with complex intangible cultural heritage image editing tasks, the diffusion model can ensure that the generated results have sufficient details and accuracy through its phased generation strategy. The diffusion model can gradually generate these details so that they are highly consistent with the original image and conform to the design style of cultural elements. Combined with the two-way interaction module, the diffusion model can not only generate high-quality images that meet user requirements when executing editing instructions, but also ensure that the uniqueness of cultural symbols is retained during the image generation process through interaction with text features.

[0012] Preferably, the intangible cultural heritage related training data includes image segmentation data, complex instruction editing data and world knowledge reasoning data.

[0013] Preferably, the bidirectional interaction module also includes an attention mechanism, which strengthens the association between image and text features. In text-to-image interaction, the attention mechanism is used to focus on the image area most relevant to the text instructions; in image-to-text interaction, the attention mechanism is used to focus on the text part most relevant to the image features.

[0014] Preferably, the bidirectional interaction module further includes a multimodal fusion mechanism, which effectively fuses text and image feature vectors in a fusion layer. The multimodal fusion mechanism includes but is not limited to splicing, weighted summation, and bilinear pooling.

[0015] A modal language model image editing technology that integrates intangible cultural heritage elements also includes a data enhancement strategy, which introduces multiple data sources related to intangible cultural heritage for model training, including image segmentation data, complex instruction editing data and world knowledge reasoning data.

[0016] Preferably, as described.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention enhances the understanding and reasoning capabilities in instruction editing by combining MLLM (such as LLaVA). MLLM can perform collaborative learning across text and image modalities and extract deep semantic information, so that the model can not only process basic instructions but also understand complex intangible cultural elements.

[0019] 2. To improve the model’s understanding of intangible cultural heritage elements, we designed an enhanced two-way interaction mechanism that achieves deep interaction between image and text features through a cross-attention mechanism, enabling image features to serve as query and key-value pairs for two-way communication with text features.

[0020] 3. In order to improve the performance of the model in complex intangible cultural heritage scenarios, we adopted an effective data augmentation strategy. The training data not only includes traditional image editing data, but also integrates perceptual data related to intangible cultural heritage, such as image segmentation data, as well as synthetic high-quality complex instruction editing data for scenarios that require world knowledge reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The present invention provides a technical solution: a modal language model image editing technology integrating intangible cultural elements, comprising:

[0024] Model selection and training, we selected the LLaMA model as the basis and introduced LoRA for adaptive fine-tuning. In this way, the model can make adaptive adjustments while keeping the original parameters frozen, which greatly improves the training efficiency and command comprehension ability. We use intangible cultural heritage-related training data for model training;

[0025] A two-way interaction module, the two-way interaction module includes: a feature extraction layer, an interaction layer and a fusion layer, the feature extraction layer includes text feature extraction and image feature extraction, the text feature extraction uses a pre-trained text encoder to extract a feature vector of a text instruction; the image feature extraction uses a pre-trained image encoder to extract a feature vector of an image; the interaction layer includes text-to-image interaction and image-to-text interaction, the text-to-image interaction inputs the text feature vector as conditional information into an image generation or editing model to affect the image generation or editing process, the image-to-text interaction feeds back the image feature vector to a text processing module to adjust the understanding of the text instruction or generate a text description related to the image, the fusion layer fuses the text and image feature vectors output by the interaction layer to generate a fused feature vector for subsequent image generation and editing tasks, and realizes multiple rounds of information flow between image and text features, verifies the effectiveness of the two-way interaction module through experiments, and adjusts model parameters to optimize performance.

[0026] Diffusion model. The working principle of the diffusion model is to generate images through step-by-step iteration, starting from a random noise image. Through the step-by-step guidance and generation process of instructions, the model gradually approaches the target image. Therefore, when dealing with complex intangible cultural heritage image editing tasks, the diffusion model can ensure that the generated results have sufficient details and accuracy through its phased generation strategy. When the user requires to modify the subtle texture or decorative pattern of the hairpin in the image, the diffusion model can gradually generate these details to keep them highly consistent with the original image and conform to the design style of the cultural elements. Combined with the two-way interaction module, the diffusion model can not only generate high-quality images that meet user requirements when executing editing instructions, but also ensure that the uniqueness of cultural symbols is retained during the image generation process through interaction with text features.

[0027] In the present invention: the training data related to intangible cultural heritage includes image segmentation data, complex instruction editing data and world knowledge reasoning data.

[0028] In the present invention: the two-way interaction module also includes an attention mechanism, which strengthens the association between image and text features. In text-to-image interaction, the attention mechanism is used to focus on the image area most relevant to the text instruction; in image-to-text interaction, the attention mechanism is used to focus on the text part most relevant to the image feature.

[0029] In the present invention: the bidirectional interaction module also includes a multimodal fusion mechanism, which effectively fuses text and image feature vectors in a fusion layer. The multimodal fusion mechanism includes but is not limited to splicing, weighted summation, and bilinear pooling.

[0030] A modal language model image editing technology that integrates intangible cultural heritage elements also includes a data augmentation strategy, which aims to enrich the model's training data and improve its ability to understand and process multimodal inputs. Traditional image editing training data usually only includes simple image modification or generation tasks, which is insufficient when dealing with complex scenes involving cultural backgrounds. Therefore, we introduced a variety of data sources related to intangible cultural heritage for model training, including image segmentation data, complex instruction editing data, and world knowledge reasoning data.

[0031] Existing technologies often have difficulty accurately distinguishing target objects when dealing with multi-object editing, resulting in errors or deviations in editing results. In the "immersive" technology, the model can accurately identify multiple objects in the image through its two-way interactive module and the multimodal collaborative learning ability of MLLM, and selectively edit the specified object attributes according to the user's natural language instructions. Compared with existing technologies, the accuracy of this model in multi-object editing reaches 92.7%, which is significantly better than other baseline methods. When dealing with multiple objects and requiring the modification of specific attributes of an object (such as color, size, position), traditional methods often confuse the attribute associations between different objects, while "immersive" can ensure that the instructions are effective for the correct target object through its enhanced text-image two-way interaction mechanism. In this way, the model effectively overcomes the limitations of existing technologies in multi-object scenarios and can provide users with more accurate editing effects.

[0032] Since the image editing task of intangible cultural heritage elements is often not just a simple object editing, the model also needs to have certain reasoning capabilities, especially in complex scenarios involving cultural background knowledge. The performance of the model in the present invention in complex instruction reasoning tasks significantly improves the reasoning ability. This task requires the model to have rich cultural background knowledge and infer the implicit intentions in the user's instructions through cross-modal information interaction. Compared with the prior art, the reasoning accuracy of the model has been improved by more than 15%, showing its advantage in complex cultural reasoning tasks. This progress is due to the combination of MLLM and LoRA fine-tuning, which enables the model to quickly adapt to the requirements of new tasks without affecting the original parameters. In addition, the two-way interaction module also ensures that the model can better reason between image features and text semantics to achieve accurate execution of complex instructions.

[0033] In order to verify the application effect of "immersive" technology in intangible cultural heritage scenes, we selected the hairpin technique, a representative intangible cultural element, and conducted multiple editing task verifications. In these experiments, the model was required to make innovative modifications to the specific details of the hairpin according to instructions. The experimental results show that "immersive" technology has a high degree of cultural understanding and detail processing capabilities when dealing with intangible cultural heritage scenes. In scenes with complex and changeable intangible cultural heritage symbols, the model can balance cultural fidelity and innovation needs, and flexibly adjust image elements through instructions, thereby achieving innovative image editing results that are in line with cultural conception. This ability not only enhances the application potential of the model in the field of intangible cultural heritage, but also opens up a new path for the modern digital inheritance of intangible cultural heritage.

[0034] In summary: the modal language model image editing technology that integrates intangible cultural heritage elements enhances the understanding and reasoning capabilities in instruction editing by combining MLLM (such as LLaVA). MLLM can collaboratively learn across text and image modalities and extract deep semantic information, so that the model can not only process basic instructions, but also understand complex intangible cultural heritage elements. In order to improve the model's understanding of intangible cultural heritage elements, we designed an enhanced two-way interaction mechanism, which realizes deep interaction between image and text features through a cross-attention mechanism, so that image features can be used as query and key-value pairs to communicate with text features in a two-way manner. In order to improve the performance of the model in complex intangible cultural heritage scenes, we adopted an effective data enhancement strategy. The training data not only includes traditional image editing data, but also integrates perceptual data related to intangible cultural heritage, such as image segmentation data, as well as synthetic high-quality complex instruction editing data for scenarios that require world knowledge reasoning.

[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0036] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A modal language model image editing technology integrating intangible cultural elements, characterized in that: The steps include: Step 1: Select the basic model. Select the LLaMA model as the basic model and introduce LoRA for adaptive fine-tuning. While keeping the original parameters of the LLaMA model frozen, introduce LoRA (Low-Rank Adaptation) technology for adaptive fine-tuning. Use the training data related to intangible cultural heritage for model training. Collect text and image data related to intangible cultural heritage for model training. Through training, the model can understand and generate text and images related to intangible cultural heritage. Step 2: construct a two-way interaction module, which includes: a feature extraction layer, an interaction layer and a fusion layer. The feature extraction layer extracts text and image features, the interaction layer realizes the mutual influence between text and image, and the fusion layer generates a fusion feature vector. Step 3: Apply the diffusion model. Starting from a random noise image, the diffusion model generates high-quality images through step-by-step iteration and instruction guidance. The diffusion model adopts a phased strategy to ensure details and accuracy. At the same time, it combines a two-way interactive module to interact with user editing instructions and text features to generate images that meet the requirements and retain the uniqueness of cultural symbols.

2. According to claim 1, a modal language model image editing technology integrating intangible cultural elements is characterized by: The two-way interaction module in step 2 includes: a feature extraction layer, an interaction layer and a fusion layer. The feature extraction layer includes text feature extraction and image feature extraction. The text feature extraction uses a pre-trained text encoder to extract a feature vector of a text instruction; the image feature extraction uses a pre-trained image encoder to extract a feature vector of an image; the interaction layer includes text-to-image interaction and image-to-text interaction. The text-to-image interaction inputs the text feature vector as conditional information into an image generation or editing model to affect the image generation or editing process. The image-to-text interaction feeds back the image feature vector to the text processing module to adjust the understanding of the text instruction or generate a text description related to the image. The fusion layer fuses the text and image feature vectors output by the interaction layer to generate a fused feature vector for subsequent image generation and editing tasks, thereby realizing multiple rounds of information flow between image and text features. The effectiveness of the two-way interaction module is verified through experiments, and the model parameters are adjusted to optimize performance.

3. The modal language model image editing technology integrating intangible cultural elements according to claim 1 is characterized in that: The intangible cultural heritage-related training data in step one includes image segmentation data, complex instruction editing data and world knowledge reasoning data.

4. The modal language model image editing technology integrating intangible cultural elements according to claim 1 is characterized in that: The bidirectional interaction module also includes an attention mechanism, which strengthens the association between image and text features. In text-to-image interaction, the attention mechanism is used to focus on the image area most relevant to the text instruction; in image-to-text interaction, the attention mechanism is used to focus on the text part most relevant to the image feature.

5. The modal language model image editing technology integrating intangible cultural heritage elements according to claim 1 is characterized in that: The bidirectional interaction module also includes a multimodal fusion mechanism, which effectively fuses text and image feature vectors in a fusion layer. The multimodal fusion mechanism includes but is not limited to splicing, weighted summation, and bilinear pooling.

6. A modal language model image editing technology integrating intangible cultural elements according to any one of claims 1 to 5, characterized in that: It also includes a data enhancement strategy, which introduces multiple data sources related to intangible cultural heritage for model training, including image segmentation data, complex instruction editing data, and world knowledge reasoning data.

Images