Image processing method based on cross attention and related equipment thereof

Through the cross attention-based image processing method, the cross attention map and modified text prompts are used to solve the problem of difficult output of AI GC technology in specific application scenarios, achieving flexibility and accuracy of image generation and ensuring uniform style Coherent with content.

CN118918222APending Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411172671.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-24
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The output of existing AI GC technology is difficult to control in specific application scenarios, resulting in the generated images that may be stiff or discordant, especially in designs that require unified style and coherent content. Existing methods such as spatial mask local editing ignore the original Structure and content.

Method used

采用基于交叉注意力的图像处理方法,通过接收输入文本和图像生成指令,生成初始图像,并利用交叉注意力图和修改后的文本提示进行图像生成,确保图像编辑过程中保留原始结构和布局。

Benefits of technology

It realizes more flexible and accurate control of image generation results, ensuring that the output image style is unified and the content is coherent, and meeting user needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918222A_ABST
    Figure CN118918222A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method and device based on cross attention, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. Firstly, an initial image is generated by using a pre-trained image generation model according to an input text; and adjusting the image according to a user instruction, modifying a corresponding text prompt, combining with the cross attention map of the first image, and carrying out image generation by using the pre-trained image generation model again so as to adjust the image content, and finally generating a second image meeting the user requirement. The invention further relates to the technical field of block chains, and the input text is stored on a block chain network node. According to the method, the corresponding relation between the text description and the image content is fully utilized, the spatial layout and the geometrical shape in the image editing process are controlled by utilizing the cross attention map, and the work of the image generation model is guided by modifying the text prompt and combining the cross attention map; therefore, more flexible and accurate image processing and generating capabilities are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method, apparatus, computer equipment and storage medium based on cross-attention. Background Art

[0002] The highlight of current AI GC technology lies in its diversified generation capability, that is, it can randomly generate diverse results for the same prompt, inspiring unlimited creativity. However, this feature will become a limitation in specific application scenarios, such as the design of booklets, solar terms and festival series posters, which require a unified style and coherent content. However, the output of AI GC technology is currently difficult to control, and post-processing usually requires tedious manual screening and adjustments, which is not only time-consuming and labor-intensive, but may not fully meet design requirements. Therefore, how to maintain the diversity of AI GC creativity while enhancing the ability to accurately control the output results so that the output results are controllable has become an urgent problem to be solved.

[0003] To address the problem that AI GC output is difficult to control, the currently commonly used solution is to provide a spatial mask and perform localized editing of the image. However, this method ignores the original structure and content within the masked area, causing the modified image to appear stiff or disharmonious. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose an image processing method, apparatus, computer device and storage medium based on cross-attention, so as to solve the technical problem that the modified images in the existing image editing schemes may appear stiff or disharmonious.

[0005] In order to solve the above technical problems, the embodiment of the present application provides an image processing method based on cross attention, which adopts the following technical solution:

[0006] An image processing method based on cross attention, comprising:

[0007] Receiving a first image generation instruction, and obtaining input text corresponding to the image to be generated;

[0008] Based on the input text, a pre-trained image generation model is used to generate a first image to obtain a first image;

[0009] receiving a second image generation instruction, and parsing the second image generation instruction to obtain image modification information;

[0010] generating an initial text prompt of the first image, and modifying the initial text prompt based on the image modification information to obtain a modified text prompt;

[0011] A cross-attention map corresponding to the first image is obtained, and based on the cross-attention map and the modified text prompt, an image generation model is used to generate a second image to obtain the second image.

[0012] Furthermore, the image generation model is pre-trained based on the model architecture of the cross attention mechanism, and the first image is generated based on the input text using the pre-trained image generation model to obtain the first image, specifically including:

[0013] Perform word segmentation processing on the input text to obtain the text segmentation of the input text;

[0014] Perform text embedding processing on the text segmentation of the input text to obtain a first text embedding vector;

[0015] Using a preset text encoder to integrate the first text embedding vector to obtain a text vector sequence;

[0016] Based on the text vector sequence, generate the image features of the first image using the cross-attention mechanism;

[0017] Image features of the first image are synthesized to obtain a first image.

[0018] Furthermore, the step of generating an initial text prompt of the first image and modifying the initial text prompt based on the image modification information to obtain a modified text prompt specifically includes:

[0019] Performing part-of-speech tagging on the text segmentation to obtain a part-of-speech tagging result, and generating an initial text prompt for the first image based on the part-of-speech tagging result;

[0020] Obtain modification keywords in the image modification information, and obtain part-of-speech tagging labels corresponding to the modification keywords;

[0021] Compare the part-of-speech tagging label with the part-of-speech tagging result to obtain the part-of-speech comparison result;

[0022] Determine the matching keyword corresponding to the modified keyword in the text segmentation of the input text according to the part-of-speech comparison result;

[0023] Use the modified keywords to replace the matching keywords in the input text to generate a modified text prompt.

[0024] Further, after the step of generating the image feature of the first image using the cross attention mechanism based on the text vector sequence, the method further includes:

[0025] Calculate the attention weights between the text vector sequence and the image features of the first image based on the cross attention mechanism;

[0026] The attention weights are visualized to obtain the cross-attention map corresponding to the first image.

[0027] Furthermore, the step of calculating the attention weight between the text vector sequence and the image features of the first image based on the cross attention mechanism specifically includes:

[0028] constructing a first vector matrix based on the text vector sequence, and constructing a second vector matrix based on the image features of the first image;

[0029] Calculate the similarity between the first vector matrix and the second vector matrix to obtain the attention score;

[0030] Convert the attention scores into probability distributions of attention weights;

[0031] Based on the probability distribution of the attention weights, a weighted summation method is used to calculate the attention weights between the text vector sequence and the image features of the first image.

[0032] Further, a step of obtaining a cross-attention map corresponding to the first image, and generating a second image using an image generation model based on the cross-attention map and the modified text prompt to obtain the second image specifically includes:

[0033] Obtain a cross-attention map corresponding to the first image, and obtain an attention weight in the cross-attention map;

[0034] Performing word segmentation processing on the modified text prompt to obtain the text segmentation of the text prompt;

[0035] Performing text embedding processing on the text segmentation of the text prompt to obtain a second text embedding vector;

[0036] Use the attention weight to perform weighted summation on the second text embedding vector to obtain a vector weighted summation result;

[0037] The attention constraint condition in the image modification information is obtained, and the weighted summation result and the attention constraint condition are imported as conditional input into the image generation model to generate a second image, thereby obtaining a second image.

[0038] Further, the step of performing weighted summation on the second text embedding vector using the attention weight to obtain a weighted summation result specifically includes:

[0039] Determine the part-of-speech tagging label of the text word corresponding to the second text embedding vector to obtain a target part-of-speech tagging label;

[0040] Determine the matching attention weight according to the target part-of-speech tag label to obtain the target attention weight;

[0041] The second text embedding vector is weighted using the target attention weight, and the weighted results of the second text embedding vector are summed to obtain a weighted sum result.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides an image processing device based on cross attention, which adopts the following technical solution:

[0043] An image processing device based on cross attention, comprising:

[0044] A first instruction triggering module, used to receive a first image generation instruction and obtain input text corresponding to the image to be generated;

[0045] A first image generation module, configured to generate a first image based on the input text using a pre-trained image generation model to obtain a first image;

[0046] A second instruction triggering module, used for receiving a second image generation instruction and parsing the second image generation instruction to obtain image modification information;

[0047] A text prompt modification module, used for generating an initial text prompt of the first image, and modifying the initial text prompt based on the image modification information to obtain a modified text prompt;

[0048] The second image generation module is used to obtain the cross-attention map corresponding to the first image, and based on the cross-attention map and the modified text prompt, use the image generation model to generate the second image to obtain the second image.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0050] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the cross-attention-based image processing method as described in any one of the above items are implemented.

[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0052] A computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by a processor, implement the steps of the cross-attention-based image processing method as described in any one of the above.

[0053] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0054] The present application discloses an image processing method, device, computer equipment and storage medium based on cross-attention, which belongs to the field of artificial intelligence technology. First, an initial image is generated according to the input text using a pre-trained image generation model; then the image is adjusted according to the user's instructions, and the image content is adjusted by modifying the corresponding text prompt and combining the cross-attention map of the first image, and the pre-trained image generation model is used again to generate the image to finally generate a second image that meets the user's needs. The present application makes full use of the correspondence between the text description and the image content, and uses the cross-attention map to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying the text prompt and combining the cross-attention map, thereby achieving more flexible and accurate image processing and generation capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0057] Figure 2 A flowchart of an embodiment of an image processing method based on cross-attention according to the present application is shown;

[0058] Figure 3 A schematic structural diagram of an embodiment of an image processing device based on cross attention according to the present application is shown;

[0059] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0064] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers and desktop computers, etc.

[0066] Server 105 can be a server that provides various services, such as a background server that provides support for pages displayed on terminal devices 101, 102, and 103. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms.

[0067] It should be noted that the cross-attention based image processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the cross-attention based image processing device is generally arranged in the server.

[0068] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0069] Continue to refer Figure 2 , showing a flow chart of an embodiment of the image processing method based on cross attention according to the present application. The embodiment of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] To address the problem that AI GC output is difficult to control, the currently commonly used solution is to provide a spatial mask and perform localized editing of the image. However, this method ignores the original structure and content within the masked area, causing the modified image to appear stiff or disharmonious.

[0072] In response to the above technical problems, the present application proposes an image processing method, device, computer equipment and storage medium based on cross-attention, which belongs to the field of artificial intelligence technology. The core of the present application solution is to use the cross-attention map to control the spatial layout and geometry in the image editing process. Specifically, when a generated image needs to be edited, the image itself is not modified directly, but the text prompt for the image is generated by modifying it, and the cross-attention map generated by the original text and the modified image text prompt are injected into the new image generation process. In this way, the newly generated image will not only be adjusted according to the edited text prompt, but also retain the structure and layout of the original image.

[0073] The Cross-Attention Map is a visualization of the Cross-Attention Mechanism in fields such as image processing or natural language processing. The Cross-Attention Mechanism is an attention mechanism that enables one sequence (or part of a dataset) to "pay attention" to another sequence (or other parts of a dataset). In image processing or image generation tasks, the Cross-Attention Map is used to represent the degree of association between different regions in an image and the input text or another dataset element.

[0074] In text-based image editing tasks, cross-attention maps can help the model understand the correspondence between textual cues and image content, thereby preserving the structure and key information of the original image during the editing process.

[0075] Please refer to Figure 2 , this embodiment discloses an image processing method based on cross attention, which specifically includes the following steps:

[0076] S201, receiving a first image generation instruction, and obtaining input text corresponding to an image to be generated.

[0077] Specifically, first, the system receives a first image generation instruction input by a user, and the first image generation instruction is used to indicate the content of the image to be generated, for example, generating an image related to "a red cat". Subsequently, the system extracts or obtains the corresponding input text from the instruction, and the input text contains descriptive information about the content of the target image.

[0078] S202: Based on the input text, use a pre-trained image generation model to generate a first image to obtain a first image.

[0079] Specifically, the system generates the first image (i.e., the first image) based on the input text using a pre-trained image generation model. The pre-trained image generation model uses a cross-attention mechanism, which enables the model to focus on both the text information and the generated image parts during the image generation process, thereby more accurately generating the corresponding image content based on the text description.

[0080] S203: Receive a second image generation instruction, and parse the second image generation instruction to obtain image modification information.

[0081] Specifically, after the system generates the first image, when it receives the user input instruction again, the second image generation instruction is a control instruction for modifying or adjusting the image. The system obtains specific image modification information by parsing the second image generation instruction, which includes image elements that need to be added, deleted or modified.

[0082] S204: Generate an initial text prompt for the first image, and modify the initial text prompt based on the image modification information to obtain a modified text prompt.

[0083] Specifically, the system first generates an initial text prompt corresponding to the first image. The initial text prompt can be directly derived from the generation process of the first image or an abstract description of the input text content. Then, based on the acquired image modification information, the initial text prompt is modified, such as by text replacement, to reflect the desired image changes. For example, the user can specify the changes they want to make to the image by modifying the text prompt. For example, modify "a red cat" to "a blue dog" to change the subject and color in the image.

[0084] S205, obtaining a cross-attention map corresponding to the first image, and based on the cross-attention map and the modified text prompt, using an image generation model to generate a second image to obtain a second image.

[0085] Cross-attention maps play a key role in the text-to-image generation process, reflecting the degree of association between each pixel in the image and each word in the text prompt. By adjusting the cross-attention map, the content of different regions in the image can be indirectly controlled.

[0086] In the image generation model, the cross-attention map is a key component. The cross-attention map records the attention relationship between the text and different areas of the image when the model generates the first image. This map can help the model understand the correspondence between the text description and the image content.

[0087] By combining the modified textual prompt with the cross-attention map of the first image, the system can more accurately locate the image area that needs to be modified, and guide the image generation model to generate the second image (i.e., the second image) accordingly.

[0088] Using the modified textual cues and the injected cross-attention maps, the image generation model starts a new image generation process (also known as the image adjustment process), in which the image generation model adjusts the image content according to the new textual cues while being constrained by the cross-attention maps of the original image to maintain the structure of the original image.

[0089] In the above embodiments, the present application makes full use of the correspondence between text description and image content, uses cross-attention maps to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying text prompts and combining cross-attention maps, thereby achieving more flexible and accurate image processing and generation capabilities.

[0090] Furthermore, the image generation model is pre-trained based on the model architecture of the cross attention mechanism, and the first image is generated based on the input text using the pre-trained image generation model to obtain the first image, specifically including:

[0091] Perform word segmentation processing on the input text to obtain the text segmentation of the input text;

[0092] Perform text embedding processing on the text segmentation of the input text to obtain a first text embedding vector;

[0093] Using a preset text encoder to integrate the first text embedding vector to obtain a text vector sequence;

[0094] Based on the text vector sequence, generate the image features of the first image using the cross-attention mechanism;

[0095] Image features of the first image are synthesized to obtain a first image.

[0096] In this embodiment, the system first preprocesses the input text, including word segmentation, stop word removal, stem extraction or word form restoration, etc., in order to better extract text features. Then, the preprocessed text word segmentation is converted into a dense vector representation in a high-dimensional vector space using text embedding technology (such as Word2Vec, G loVe, BERT, etc.), capturing the semantic relationship between words. After the text is embedded, a text encoder is usually used to further process these embedded vectors. The text encoder can be a recurrent neural network (RNN), a long short-term memory network (LSTM), a Transformer, etc. model, which is used to encode the embedded vector of the entire text sequence into one or more vectors, which contain the overall semantic information of the text.

[0097] The pre-trained image generation model is trained based on the model architecture of the cross-attention mechanism. The pre-trained image generation model can automatically generate the target image based on the user input text. In the process of image generation by the model, the cross-attention mechanism plays a vital role. The cross-attention mechanism allows the model to pay attention to the text description and the generated image part at each step of the image generation. The image generation model will gradually generate each pixel or feature of the image based on the text vector sequence. This process is iterative until the complete image is generated. In each iteration, the model will update the state of the image based on the current text embedding, the generated image content, and the cross-attention weight map.

[0098] Under the guidance of the cross-attention mechanism, the model begins to gradually generate feature representations of the image. These features include basic information such as color, texture, shape, and higher-level semantic information. This process is usually achieved through a generative network consisting of a series of convolutional layers, upsampling layers, normalization layers, etc. As the generation process proceeds, the feature representation of the image gradually develops from low-level to high-level, and finally forms a complete image content. Finally, the model synthesizes the generated image feature representation into a complete image.

[0099] Through the above steps, the system accurately converts the input text into visual images through the image generation model pre-trained by the cross-attention mechanism. Through word segmentation, text embedding, text encoder integration and cross-attention mechanism, the deep semantics of the text is effectively extracted and mapped to the image feature space, improving the accuracy of image generation.

[0100] Furthermore, the step of generating an initial text prompt of the first image and modifying the initial text prompt based on the image modification information to obtain a modified text prompt specifically includes:

[0101] Performing part-of-speech tagging on the text segmentation to obtain a part-of-speech tagging result, and generating an initial text prompt for the first image based on the part-of-speech tagging result;

[0102] Obtain modification keywords in the image modification information, and obtain part-of-speech tagging labels corresponding to the modification keywords;

[0103] Compare the part-of-speech tagging label with the part-of-speech tagging result to obtain the part-of-speech comparison result;

[0104] Determine the matching keyword corresponding to the modified keyword in the text segmentation of the input text according to the part-of-speech comparison result;

[0105] Use the modified keywords to replace the matching keywords in the input text to generate a modified text prompt.

[0106] In this embodiment, by integrating the word segmentation and part-of-speech tagging technology in natural language processing (NLP), intelligent processing of text content and automatic generation of image prompts are achieved. First, the original input text is segmented and tagged with parts of speech. Then, the modification keywords and their part-of-speech tagging tags are extracted according to the image modification requirements. By comparing the part-of-speech tagging tags of the modified keywords with the word segmentation and part-of-speech tagging results of the original input text, the keywords to be replaced in the original input text are identified. Finally, the matching items are replaced with the modified keywords to generate new text prompts that meet the modification intention.

[0107] Through the above steps, the system realizes the modification of text prompts through part-of-speech tagging and comparison, ensuring that when modifying the text description corresponding to the image, it can accurately identify and replace keywords with the same part of speech, maintain the correctness of the text grammar and the semantic coherence, thereby generating a modified text prompt that conforms to the modification intention and maintains the original context, enhancing the efficiency and accuracy of text editing.

[0108] Further, after the step of generating the image feature of the first image using the cross attention mechanism based on the text vector sequence, the method further includes:

[0109] Calculate the attention weights between the text vector sequence and the image features of the first image based on the cross attention mechanism;

[0110] The attention weights are visualized to obtain the cross-attention map corresponding to the first image.

[0111] In this embodiment, the system uses a cross-attention mechanism to calculate the attention weights between each pixel in the first image (or each position in the feature map) and each word in the text segmentation of the input text (or each element in the input data set). These weights reflect the degree of association between the image pixels and the text segmentation. Finally, the calculated attention weights are displayed in the form of an image, namely a cross-attention map. In the cross-attention map, different colors or brightness may represent different attention weight values, thereby intuitively showing the association between the image area and the text or data element.

[0112] The attention weight matrix can be converted into a visualization form such as a grayscale image or a heat map. In a grayscale image, brighter areas represent higher attention weights; in a heat map, different colors represent different weight ranges. Adjust the visualization parameters (such as color mapping, brightness range, etc.) as needed to better show the distribution of attention weights.

[0113] Through the above steps, the system further generates an intuitive cross-attention map by calculating and visualizing the attention weights between text and image features, which not only enhances the interpretability of image generation, but also enables users to intuitively understand how text affects the generation of image content, providing more transparent and flexible visual feedback for image creation and editing.

[0114] Furthermore, the step of calculating the attention weight between the text vector sequence and the image features of the first image based on the cross attention mechanism specifically includes:

[0115] constructing a first vector matrix based on the text vector sequence, and constructing a second vector matrix based on the image features of the first image;

[0116] Calculate the similarity between the first vector matrix and the second vector matrix to obtain the attention score;

[0117] Convert the attention scores into probability distributions of attention weights;

[0118] Based on the probability distribution of the attention weights, a weighted summation method is used to calculate the attention weights between the text vector sequence and the image features of the first image.

[0119] In this embodiment, the system can use the query matrix (i.e., the first vector matrix), and the key and value matrices (i.e., the second vector matrix) to calculate the attention weights, wherein the query sequence is usually a sequence that needs to pay attention to other sequences, and the key and value sequences are the object sequences being queried. In deep learning, especially in the intersection of natural language processing (NLP) and computer vision (CV), such as multimodal learning or visual-language tasks, attention mechanisms are often used to calculate the correlation between text embeddings and image features. Using query, key, and value matrices to calculate attention weights is a common form of attention mechanism, especially widely used in Transformer models.

[0120] In this embodiment, the first text embedding vector in the text vector sequence is represented as a series of query vectors, and a query matrix is ​​constructed. The image features of the first image are taken as a series of key and value vectors, and a key and value matrix is ​​constructed, wherein the key and value are usually the same. The attention score is calculated based on the query matrix, the key and value vectors, wherein the attention score is a measure of the similarity between the query and the key, which can be calculated by methods such as dot product or cosine similarity. The score is then converted into a probability distribution (i.e., attention weight) using a softmax function, which is applied independently on the row corresponding to each query so that the sum of the elements of each row is 1, indicating the attention allocation of the query to each key. Finally, the value is weighted and summed using the attention weight to obtain the final output attention weight.

[0121] Through the above steps, the system calculates the attention weights between text embeddings (as queries) and image features (as keys and values), and uses these weights to generate an enhanced representation of text embeddings that takes image features into account. This mechanism effectively integrates text and image information, allowing the model to focus on the most relevant parts of the image, enhancing the accuracy and efficiency of cross-modal information interaction, and improving the performance and accuracy of text-image joint analysis tasks.

[0122] Further, a step of obtaining a cross-attention map corresponding to the first image, and generating a second image using an image generation model based on the cross-attention map and the modified text prompt to obtain the second image specifically includes:

[0123] Obtain a cross-attention map corresponding to the first image, and obtain an attention weight in the cross-attention map;

[0124] Performing word segmentation processing on the modified text prompt to obtain the text segmentation of the text prompt;

[0125] Performing text embedding processing on the text segmentation of the text prompt to obtain a second text embedding vector;

[0126] Use the attention weight to perform weighted summation on the second text embedding vector to obtain a vector weighted summation result;

[0127] The attention constraint condition in the image modification information is obtained, and the weighted summation result and the attention constraint condition are imported as conditional input into the image generation model to generate a second image, thereby obtaining a second image.

[0128] In this embodiment, the system first parses the modified text prompt to understand the modified content and constraints in the text prompt, and can use natural language processing (NLP) technology for semantic understanding, such as word segmentation, part-of-speech tagging, syntactic analysis, etc., so that the model can accurately capture the semantic information in the text. The model then fuses the injected cross-attention map with the modified text prompt, and the system uses the information provided by the cross-attention map inside the model to fuse the features in the text prompt to adjust its image generation strategy. As a priori knowledge, the cross-attention map can guide the model on how to allocate attention resources during the generation process, that is, which text elements should have a stronger association with which areas in the image (or text).

[0129] Based on the parsed text prompts and the fused cross-attention map, the model starts to generate new content. During the generation process, the model will follow the guidance of the cross-attention map to ensure that the generated content is consistent with the modified text prompts and retain as much useful information (such as structure, style, etc.) as possible in the original generated results.

[0130] Specifically, the system uses the attention weights of the model's cross-attention map to perform weighted summation on the second text embedding vector to obtain a weighted summation result of the vector, and obtains the attention constraints in the image modification information. The attention constraints refer to the constraints used to guide the model to focus on specific image areas or features during the image generation process. Subsequently, the weighted summation result and the attention constraints are imported into the image generation model as conditional inputs, and the weighted summation result is converted into one or more vectors. Similarly, the attention constraints are also converted into vector form. The weighted summation result vector and the attention constraint vector are directly spliced ​​into a longer spliced ​​vector. Finally, the image generation model generates a second image based on the spliced ​​vector to obtain a second image.

[0131] Attention constraints include the image areas that need to be focused on, the image features that need to remain unchanged, and the specific requirements that should be met after modification. Attention constraints can be formulated based on a variety of factors, including but not limited to:

[0132] Area attention: Specify the area in the image that needs to be focused on, such as faces, vehicles and other key objects.

[0133] Feature preservation: The model is required to keep certain features unchanged when modifying the image, such as texture, color, etc.

[0134] Modification Limits: Limit the scope and degree of modification to avoid generating exaggerated or unrealistic images.

[0135] Semantic consistency: Ensure that the modified image is semantically consistent with the original image.

[0136] User preferences: Constraints set based on the user's personalized needs.

[0137] Through the above steps, the system achieves accurate image modification by combining the cross-attention map of the first image with the modified text prompt, and uses the attention weight to weight the text embedding vector, thereby enhancing the relevance between the text and the key areas of the image. At the same time, the attention constraint condition is introduced as an additional guide to ensure that the image modification process follows specific requirements, which improves the accuracy and controllability of the image generation model when modifying images according to text instructions, and generates images that better meet user expectations.

[0138] Further, the step of performing weighted summation on the second text embedding vector using the attention weight to obtain a weighted summation result specifically includes:

[0139] Determine the part-of-speech tagging label of the text word corresponding to the second text embedding vector to obtain a target part-of-speech tagging label;

[0140] Determine the matching attention weight according to the target part-of-speech tag label to obtain the target attention weight;

[0141] The second text embedding vector is weighted using the target attention weight, and the weighted results of the second text embedding vector are summed to obtain a weighted sum result.

[0142] In this embodiment, in the context of image generation or text-image interaction, the utilization and accuracy of text information are improved through the attention mechanism. First, the system identifies the different importance of different parts of speech (such as nouns, verbs, etc.) in the image modification task by tagging each text word in the second text embedding vector. Then, according to these part-of-speech tagging labels, matching attention weights are selected from the cross-attention map to ensure that the model can pay more attention to key information in the text (such as actions, objects, etc.). Finally, the text embedding vector is weighted and summed using these target attention weights.

[0143] Through the above steps, the system enhances the contribution of key information in the text embedding vector by accurately matching part-of-speech tagging with attention weights. By assigning specific attention weights to each text segment, differentiated processing of text content is achieved, so that the generated weighted summation results are more focused on the core points of the modification instructions, thereby improving the accuracy and effectiveness of text guidance during image generation or modification.

[0144] In the above embodiment, the present application discloses an image processing method based on cross-attention, which belongs to the field of artificial intelligence technology. First, an initial image is generated according to the input text using a pre-trained image generation model; then the image is adjusted according to the user's instructions, and the image content is adjusted by modifying the corresponding text prompt and combining the cross-attention map of the first image, and the pre-trained image generation model is used again to generate the image to finally generate a second image that meets the user's needs. The present application makes full use of the correspondence between the text description and the image content, and uses the cross-attention map to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying the text prompt and combining the cross-attention map, thereby achieving more flexible and accurate image processing and generation capabilities.

[0145] In this embodiment, the electronic device (eg, Figure 1The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0146] It should be emphasized that in order to further ensure the privacy and security of the above input text, the above input text can also be stored in a node of a blockchain.

[0147] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0149] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0150] Further references Figure 3 , as a response to the above Figure 2The present application provides an embodiment of an image processing device based on cross attention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0151] like Figure 3 As shown, the image processing device 300 based on cross attention described in this embodiment includes:

[0152] The first instruction triggering module 301 is used to receive a first image generation instruction and obtain an input text corresponding to the image to be generated;

[0153] A first image generation module 302 is used to generate a first image based on the input text using a pre-trained image generation model to obtain a first image;

[0154] A second instruction triggering module 303 is used to receive a second image generation instruction and parse the second image generation instruction to obtain image modification information;

[0155] A text prompt modification module 304, configured to generate an initial text prompt for the first image, and modify the initial text prompt based on the image modification information to obtain a modified text prompt;

[0156] The second image generation module 305 is used to obtain the cross-attention map corresponding to the first image, and generate the second image using the image generation model based on the cross-attention map and the modified text prompt to obtain the second image.

[0157] Furthermore, based on the model architecture pre-training of the cross-attention mechanism to generate an image generation model, the first image generation module 302 is specifically used for:

[0158] Perform word segmentation processing on the input text to obtain text segmentation of the input text; perform text embedding processing on the text segmentation of the input text to obtain a first text embedding vector; use a preset text encoder to integrate the first text embedding vector to obtain a text vector sequence; based on the text vector sequence, use a cross-attention mechanism to generate image features of a first image; perform image feature synthesis on the image features of the first image to obtain a first image.

[0159] Furthermore, the text prompt modification module 304 is specifically used for:

[0160] Performing part-of-speech tagging on the text segmentation to obtain a part-of-speech tagging result, and generating an initial text prompt for the first image based on the part-of-speech tagging result;

[0161] Obtain modification keywords in the image modification information, and obtain part-of-speech tagging labels corresponding to the modification keywords;

[0162] Compare the part-of-speech tagging label with the part-of-speech tagging result to obtain the part-of-speech comparison result;

[0163] Determine the matching keyword corresponding to the modified keyword in the text segmentation of the input text according to the part-of-speech comparison result;

[0164] Use the modified keywords to replace the matching keywords in the input text to generate a modified text prompt.

[0165] Furthermore, the cross-attention based image processing apparatus 300 further includes an attention map generation module, and the attention map generation module is specifically used for:

[0166] The attention weights between the text vector sequence and the image features of the first image are calculated based on the cross-attention mechanism; the attention weights are visualized to obtain a cross-attention map corresponding to the first image.

[0167] Furthermore, the attention map generation module is further used to:

[0168] constructing a first vector matrix based on the text vector sequence, and constructing a second vector matrix based on the image features of the first image;

[0169] Calculate the similarity between the first vector matrix and the second vector matrix to obtain the attention score;

[0170] Convert the attention scores into probability distributions of attention weights;

[0171] Based on the probability distribution of the attention weights, a weighted summation method is used to calculate the attention weights between the text vector sequence and the image features of the first image.

[0172] Furthermore, the second image generating module 305 is specifically used for:

[0173] Obtain a cross-attention map corresponding to the first image, and obtain the attention weight in the cross-attention map; perform word segmentation processing on the modified text prompt to obtain text segmentation of the text prompt; perform text embedding processing on the text segmentation of the text prompt to obtain a second text embedding vector; use the attention weight to perform weighted summation on the second text embedding vector to obtain a vector weighted summation result; obtain the attention constraint condition in the image modification information, and import the weighted summation result and the attention constraint condition as conditional input into the image generation model to generate a second image, thereby obtaining a second image.

[0174] Furthermore, the second image generating module 305 is further used for:

[0175] Determine the part-of-speech tagging label of the text word corresponding to the second text embedding vector to obtain a target part-of-speech tagging label;

[0176] Determine the matching attention weight according to the target part-of-speech tag label to obtain the target attention weight;

[0177] The second text embedding vector is weighted using the target attention weight, and the weighted results of the second text embedding vector are summed to obtain a weighted sum result.

[0178] In the above embodiment, the present application discloses an image processing device based on cross-attention, which belongs to the field of artificial intelligence technology. First, an initial image is generated according to the input text using a pre-trained image generation model; then the image is adjusted according to the user's instructions, and the image content is adjusted by modifying the corresponding text prompt and combining the cross-attention map of the first image, and the pre-trained image generation model is used again to generate the image to finally generate a second image that meets the user's needs. The present application makes full use of the correspondence between the text description and the image content, and uses the cross-attention map to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying the text prompt and combining the cross-attention map, thereby achieving more flexible and accurate image processing and generation capabilities.

[0179] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0180] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmab l eGate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0181] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0182] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image processing method based on cross attention, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0183] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the cross-attention-based image processing method.

[0184] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0185] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. First, an initial image is generated according to the input text using a pre-trained image generation model; then the image is adjusted according to the user's instructions, and the image content is adjusted by modifying the corresponding text prompt and combining the cross-attention map of the first image, and the pre-trained image generation model is used again to generate the image to finally generate a second image that meets the user's needs. The present application makes full use of the correspondence between the text description and the image content, and uses the cross-attention map to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying the text prompt and combining the cross-attention map, thereby achieving more flexible and accurate image processing and generation capabilities.

[0186] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the cross-attention based image processing method as described above.

[0187] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. First, an initial image is generated according to the input text using a pre-trained image generation model; then the image is adjusted according to the user's instructions, and the image content is adjusted by modifying the corresponding text prompt and combining the cross-attention map of the first image, and the pre-trained image generation model is used again to generate the image to finally generate a second image that meets the user's needs. The present application makes full use of the correspondence between the text description and the image content, and uses the cross-attention map to control the spatial layout and geometry in the image editing process, and guides the work of the image generation model by modifying the text prompt and combining the cross-attention map, thereby achieving more flexible and accurate image processing and generation capabilities.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0189] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0190] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An image processing method based on cross attention, It is characterized in that include: Receiving a first image generation instruction, and obtaining input text corresponding to the image to be generated; Based on the input text, generate a first image using a pre-trained image generation model to obtain a first image; receiving a second image generation instruction, and parsing the second image generation instruction to obtain image modification information; generating an initial text prompt of the first image, and modifying the initial text prompt based on the image modification information to obtain a modified text prompt; Obtain a cross-attention map corresponding to the first image, and based on the cross-attention map and the modified text prompt, use the image generation model to generate a second image to obtain a second image.

2. The image processing method based on cross attention as claimed in claim 1, It is characterized in that The image generation model is generated by pre-training a model architecture based on a cross-attention mechanism, and the step of generating a first image based on the input text using the pre-trained image generation model to obtain the first image specifically includes: Performing word segmentation processing on the input text to obtain text segmentation of the input text; Performing text embedding processing on the text segmentation of the input text to obtain a first text embedding vector; Using a preset text encoder to integrate the first text embedding vectors to obtain a text vector sequence; Based on the text vector sequence, generate image features of the first image using the cross-attention mechanism; Image features of the first image are synthesized to obtain the first image.

3. The image processing method based on cross attention as claimed in claim 2, It is characterized in that The step of generating an initial text prompt of the first image, and modifying the initial text prompt based on the image modification information to obtain a modified text prompt specifically includes: Performing part-of-speech tagging on the text segmentation to obtain a segmentation part-of-speech tagging result, and generating an initial text prompt for the first image based on the segmentation part-of-speech tagging result; Obtaining a modification keyword in the image modification information, and obtaining a part-of-speech tagging label corresponding to the modification keyword; Compare the part-of-speech tag label with the part-of-speech tag result to obtain a part-of-speech comparison result; Determine, according to the part-of-speech comparison result, a matching keyword in the text segmentation of the input text corresponding to the modification keyword; The modified keyword is used to replace the matching keyword in the input text to generate the modified text prompt.

4. The image processing method based on cross attention as claimed in claim 2, It is characterized in that After the step of generating the image features of the first image using the cross-attention mechanism based on the text vector sequence, the method further includes: Calculating an attention weight between the text vector sequence and the image features of the first image based on the cross attention mechanism; The attention weights are visualized to obtain a cross-attention map corresponding to the first image.

5. The image processing method based on cross attention as claimed in claim 4, It is characterized in that The step of calculating the attention weight between the text vector sequence and the image features of the first image based on the cross attention mechanism specifically includes: constructing a first vector matrix based on the text vector sequence, and constructing a second vector matrix based on the image features of the first image; Calculate the similarity between the first vector matrix and the second vector matrix to obtain an attention score; Convert the attention scores into a probability distribution of attention weights; Based on the probability distribution of the attention weights, a weighted summation method is used to calculate the attention weights between the text vector sequence and the image features of the first image.

6. The image processing method based on cross attention as claimed in claim 5, It is characterized in that The step of obtaining a cross-attention map corresponding to the first image, and generating a second image using the image generation model based on the cross-attention map and the modified text prompt to obtain the second image specifically includes: Obtaining a cross-attention map corresponding to the first image, and obtaining an attention weight in the cross-attention map; Performing word segmentation processing on the modified text prompt to obtain text segmentation of the text prompt; Performing text embedding processing on the text segmentation of the text prompt to obtain a second text embedding vector; Using the attention weight, weighted sum is performed on the second text embedding vector to obtain a vector weighted sum result; The attention constraint condition in the image modification information is obtained, and the weighted summation result and the attention constraint condition are imported as conditional input into the image generation model to generate a second image, thereby obtaining the second image.

7. The image processing method based on cross attention as claimed in claim 6, It is characterized in that The step of using the attention weight to perform weighted summation on the second text embedding vector to obtain a weighted summation result specifically includes: Determine a part-of-speech tagging label of the text word corresponding to the second text embedding vector to obtain a target part-of-speech tagging label; Determine a matching attention weight according to the target part-of-speech tag to obtain a target attention weight; The second text embedding vector is weighted using the target attention weight, and the weighted results of the second text embedding vector are summed to obtain the weighted summation result.

8. An image processing device based on cross attention, It is characterized in that include: A first instruction triggering module, used to receive a first image generation instruction and obtain input text corresponding to the image to be generated; A first image generation module, configured to generate a first image based on the input text using a pre-trained image generation model to obtain a first image; A second instruction triggering module, configured to receive a second image generation instruction and parse the second image generation instruction to obtain image modification information; A text prompt modification module, used for generating an initial text prompt of the first image, and modifying the initial text prompt based on the image modification information to obtain a modified text prompt; The second image generation module is used to obtain the cross-attention map corresponding to the first image, and based on the cross-attention map and the modified text prompt, use the image generation model to generate a second image to obtain a second image.

9. A computer device, It is characterized in that It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the cross-attention based image processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the cross-attention-based image processing method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • Three-dimensional costume design drawing generation method and device and electronic equipment

    CN120726232A

  • Three-dimensional garment design drawing generation method, device and electronic equipment

    CN120726232B