Theme image editing and generating method and system based on multi-party collaborative attention mechanism

Abstract

The invention discloses a subject image editing and generating method based on a multi-party collaborative attention mechanism, and the method comprises the steps: inputting a multi-mode instruction to a subject positioning module, positioning an action region, and extracting a subject image and a target image in the action region; inputting the subject image and the target image into a pre-trained deep learning model, and executing self-attention local query in a self-attention layer to integrate the background of the source image and the foreground of the subject image; and executing self-attention global injection to realize detail enhancement of the target image. And outputting an edited and generated target image in combination with results of the self-attention local query and the self-attention global injection. According to the method, the time required for generating the consistent image by the theme model is greatly shortened, and the practical application of the theme model in a wider range is promoted.

Description

Technical Field

[0001] The present application relates to the field of generative artificial intelligence, and in particular to a method and system for subject image editing and generation based on a multi-party collaborative attention mechanism. Background Art

[0002] In recent years, with the rapid development of generative artificial intelligence, text-to-image generative models (text-graph models) have been rapidly improved. These models can generate a variety of realistic images based on text prompts given by users. Through further extensions of text-graph models, some theme-driven image generation works have been proposed. A theme refers to a certain type of object, such as a schoolbag, a pet, a person, etc., and theme-driven image generation refers to placing a specific theme in an unused visual scene based on text prompts. Traditionally, these methods transform the visual representation of a given theme into a textual latent space and reconstruct a new theme image through placeholders.

[0003] However, traditional topic-based methods mainly focus on how to align the visual features of an image with the text features, thereby generating novel representations of a specific topic through feature recontextualization. However, these methods have the following three shortcomings:

[0004] 1. Existing methods usually require extensive fine-tuning or expensive optimization to learn the visual feature representation of a subject. This is extremely time-consuming and costly. If the feature representation of the subject can be directly obtained during the model inference process, it will greatly speed up the generation and reduce costs;

[0005] 2. Existing methods are usually unable to accurately extract and edit specified subjects in complex visual scenes (complex visual scenes referred to in this article are mainly divided into four categories: interactions between subjects, similar foreground and background features, multiple subjects in the image, and subjects are covered). However, natural scenes are inherently complex and usually contain multiple similar objects that may interact, so it is crucial to ensure the robustness of the model when dealing with complex scenes.

[0006] 3. Existing methods are usually text-driven. Most existing subject-driven editing models rely only on text prompts and then operate based on synthetic images. However, this approach introduces uncertainty in background, image layout, and other factors. Real scenarios often involve multimodal image-text instructions. For example, a user may want to locate a specific subject in a fixed scene, or copy the appearance of a subject in one image to a specified object in another image. The purpose is to perform subject editing and generation more accurately with consistent pose and layout.

[0007] In order to solve the above-mentioned technical problems, it is urgent to propose a topic image editing and generation method and system based on multi-party collaborative attention mechanism, so as to obtain a more controllable and lower-cost topic editing and generation method, which can greatly reduce the time required for topic models to generate consistent images and promote their practical application in a wider range. Summary of the invention

[0008] In order to solve the defects of the above-mentioned prior art that it is expensive and cannot accurately extract and edit specified subjects, the present invention proposes a subject image editing and generation method and system based on a multi-party collaborative attention mechanism.

[0009] In a first aspect, an embodiment of the present application provides a method for editing and generating a subject image based on a multi-party collaborative attention mechanism, the method comprising:

[0010] Theme localization step: inputting a multimodal instruction to the theme localization module, localizing the action area, and extracting the theme image and the target image in the action area, wherein the multimodal instruction includes: a user-specified theme image and a corresponding text description, and the localization module includes: a target detection model and a segmentation model;

[0011] Multi-party collaborative attention control steps: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image;

[0012] Editing and generating steps: Combine the results of self-attention local query and self-attention global injection to output the edited and generated target images.

[0013] As an optional implementation, in the first aspect of the present invention, the multi-party collaborative attention control step further includes:

[0014] The source image is input into the pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

[0015] As an optional implementation manner, in the first aspect of the present invention, the above-mentioned topic positioning step includes:

[0016] Target detection step: Use the target detection model to locate the specified area according to the input multimodal image and text information, and generate a positioning frame. The target detection model is composed of a DINO model;

[0017] Segmentation step: adopt the segmentation model to segment the user-specified topic based on the positioning box as conditional information, and generate a binary topic image layer and an editable image layer, wherein the segmentation model is composed of a SAM model.

[0018] As an optional implementation, in the first aspect of the present invention, the multi-party collaborative attention control step further includes:

[0019] Self-attention local query step: extract the appearance features in the subject image by injecting the content features of the source image and the subject image diffusion process into the target image generation process;

[0020] Self-attention global injection step: inject subject features and background features into the diffusion process of the target image, reconstruct the feature output of the diffusion of the target image, and replace and enhance the detailed information of the foreground and background in the target image.

[0021] As an optional implementation, in the first aspect of the present invention, the self-attention local query step includes:

[0022] Obtain the attention matrix of the target image to the global area of ​​the source image and the subject image; locally mask the attention matrix using the subject image layer and the editable image layer; obtain the queried foreground content features and background content features; and fuse the foreground content features and background content features.

[0023] As an optional implementation, in the first aspect of the present invention, the above-mentioned self-attention global injection step includes:

[0024] The attention matrices of the source image and the target image are calculated according to the equation, and the attention matrices are filtered using the editable image layer and the subject image layer. The injected features of the source image diffusion process and the subject image diffusion process are calculated, and the injected features are reconstructed by calculation.

[0025] In a second aspect, an embodiment of the present application provides a system for editing and generating a theme image based on a multi-party collaborative attention mechanism, using the above-mentioned method for editing and generating a theme image based on a multi-party collaborative attention mechanism, the system includes:

[0026] Theme localization module: input multimodal instructions to the theme localization module, locate the action area, and extract the theme image and target image in the action area, wherein the multimodal instructions include: user-specified theme image and corresponding text description, and the localization module includes: target detection model and segmentation model;

[0027] Multi-party collaborative attention control module: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image;

[0028] Editing and generation module: Combines the results of self-attention local query and self-attention global injection to output the edited and generated target images.

[0029] As an optional implementation, in the second aspect of the present invention, the multi-party collaborative attention control module further includes:

[0030] The source image is input into the pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

[0031] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for editing and generating subject images based on a multi-party collaborative attention mechanism.

[0032] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for editing and generating subject images based on a multi-party collaborative attention mechanism as described above are implemented.

[0033] Compared with the related prior art, it has the following outstanding beneficial effects:

[0034] 1) In order to achieve high-fidelity subject feature injection while maintaining background consistency, the method of the present invention proposes a two-stage multi-party collaborative attention control method: self-attention global injection and self-attention local query. Self-attention global injection enhances image details by injecting content features from the source image and subject image diffusion process into the target generation process, which is particularly important for editing and generating complex subjects. Self-attention local query actively queries and integrates the background of the source image and the foreground of the subject image through the query features of the target diffusion process;

[0035] 2) The method of the present invention introduces a multimodal topic localization module, which realizes the precise localization of the areas where features need to be extracted and the editable areas through multimodal image text detection in complex scenes, thereby reducing artifacts caused by query confusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0037] Figure 1 The following is a flow chart of the subject image editing and generation method according to an embodiment of the present invention. Figure 1 ;

[0038] Figure 2 The following is a flow chart of the subject image editing and generation method according to an embodiment of the present invention. Figure 2 ;

[0039] Figure 3 The following is a flow chart of the subject image editing and generation method according to an embodiment of the present invention. Figure 3 ;

[0040] Figure 4 A schematic diagram of a subject image editing and generating system according to an embodiment of the present invention;

[0041] Figure 5 Schematic diagram of computer hardware of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that the processor described in the present invention is the control center of the electronic device, which may be a processor or a general term for multiple processing elements. For example, it may be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0043] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0044] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single core processor or a multi-core processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0045] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0046] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0047] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0048] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0049] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0050] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0051] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0052] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0053] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0054] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0055] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0056] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0057] The present invention aims to propose a method for subject image editing and generation based on a multi-party collaborative attention mechanism. When conducting research on subject generation and editing methods, the present invention found that although the generation method based on a multimodal encoder can achieve a certain degree of low-cost subject editing and generation, there are still problems such as being only text-driven, poor editing effects in complex scenarios, and the need for fine-tuning of complex subjects to improve subject consistency.

[0058] The present invention directly obtains the visual features of the image by inverting and reconstructing the real image, thereby avoiding the training cost. According to the features obtained in the reconstruction process, the present invention can modify the feature area of ​​the source image or generate a new image specified by the text through the method of attention control. Therefore, the focus of the present invention is how to design an attention control method that can reasonably integrate the specified information in the real source image and the subject image, and can be compatible with various complex visual scenes.

[0059] The present invention is inspired by some attention control methods. By first generating an action area and then performing attention control within the action area, a more controllable and lower-cost topic editing and generation method is obtained. Using this method, the present invention can greatly reduce the time required for topic models to generate consistent images, promoting their practical application in a wider range.

[0060] This paper solves the problems of existing topic-based models, which are high in training cost, poor in editing effect in complex visual scenes, and most of them are text-driven only, and proposes a general method for editing and generating new contexts for specific topics in complex scenes without fine-tuning. This method is based on Stable Diffusion, and controls the generation of target images in the self-attention layer by manipulating the latent variables in the reconstruction process of the source image and the subject image.

[0061] The present invention outperforms existing untrained topic-based methods both qualitatively and quantitatively, and can achieve better or comparable performance to most fine-tuned topic-based methods.

[0062] Embodiment 1

[0063] like Figure 1 As shown, Figure 1 is a flowchart of a method for editing and generating a subject image based on a multi-party collaborative attention mechanism disclosed in an embodiment of the present invention. Figure 1 The subject image editing and generation method based on the multi-party collaborative attention mechanism shown can be applied to the subject image editing and generation system or device based on the multi-party collaborative attention mechanism, and the embodiment of the present invention is not limited. Figure 1 As shown, the method includes:

[0064] Theme localization step 101: input a multimodal instruction to a theme localization module, locate the action area, and extract the theme image and the target image in the action area, wherein the multimodal instruction includes: a user-specified theme image and a corresponding text description, and the localization module includes: a target detection model and a segmentation model;

[0065] Multi-party collaborative attention control step 102: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image;

[0066] Editing and generating step 103: combining the results of the self-attention local query and the self-attention global injection, outputting the edited and generated target image.

[0067] Embodiment 2

[0068] like Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 is a flowchart of another method for editing and generating a subject image based on a multi-party collaborative attention mechanism disclosed in an embodiment of the present invention. Figure 2 The subject image editing and generation method based on the multi-party collaborative attention mechanism shown can be applied to the subject image editing and generation system or device based on the multi-party collaborative attention mechanism, and the embodiment of the present invention is not limited. Figure 2 and Figure 3 As shown, the method includes:

[0069] Theme localization step 201: inputting a multimodal instruction to a theme localization module, localizing an action area, and extracting a theme image and a target image in the action area, wherein the multimodal instruction includes: a user-specified theme image and a corresponding text description, and the localization module includes: a target detection model and a segmentation model;

[0070] As an optional implementation manner, in the first aspect of the present invention, the above-mentioned topic positioning step includes:

[0071] Target detection step 211: using a target detection model to locate the designated area according to the input multi-modal image-text pair information and generate a positioning frame, wherein the target detection model is composed of a DINO model;

[0072] Segmentation step 2012: using a segmentation model, segmenting the user-specified subject based on the positioning box as conditional information, generating a binary subject image layer and an editable image layer, wherein the segmentation model is composed of a SAM model.

[0073] In a specific embodiment of the present invention, in order to prevent query confusion and the generation of subject feature artifacts in a complex visual scene containing multiple similar objects, the present invention introduces a subject positioning module to accurately locate the object specified by the user. The module consists of an object detection model DINO and a segmentation model SAM. DINO locates the specified area based on the input multimodal image-text pair information (such as the subject image and the prompt of the object to be extracted, the source image and the text description of the area to be edited), and generates a positioning frame. The SAM model uses the positioning frame generated by DINO as conditional information to segment the user-specified subject and generate a binary subject image layer M. R and an editable image layer M s In order to ensure that the edited area has enough space to blend with the background and avoid rigid transition, the present invention uses an expansion kernel of size 3×3 to expand the editable image layer to obtain

[0074] Multi-party collaborative attention control step 202: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image;

[0075] Furthermore, the source image is input into the pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

[0076] The technical route in the specific embodiment of the present invention includes three parallel diffusion processes: the subject image diffusion process B sub , Source image diffusion process B src (optional) and target image diffusion process B tgt .

[0077] Topic image diffusion process B sub Receive the real subject image and generate the diffusion initial features through DDIM (Deep Diffusion Implicit Model) inversion Source image diffusion process B src Depending on the situation, it receives the real source image or text prompt. When editing the real image, B src Receive the real source image, then the present invention uses and B sub The same party gets When generating a topic image, B src Receive the text prompt, then the present invention generates a random Gaussian distribution The subject image refers to the reference image (subject), while the source image refers to the image to be edited. tgt is a text-driven generation process that shares As the initial features of the target image, B tgt By default, "a photo of [subject]" is used as the text prompt. For the generation task, it uses the user-defined text prompt. In each diffusion step, the present invention selectively performs the following operations: 1) B sub and B src The foreground self-attention map and background self-attention map are injected into B tgt , which is called self-attention Global Injection (SAGI); 2) B tgt From B sub and B srcThe query of topic appearance and background content is called self-attention local query (SALQ).

[0078] The multi-party collaborative attention control step 202 further includes:

[0079] Self-attention local query step 221: extracting appearance features in the subject image by injecting content features in the source image and subject image diffusion process into the target image generation process;

[0080] Self-attention global injection step 222: inject subject features and background features into the diffusion process of the target image, reconstruct the feature output of the diffusion of the target image, and replace and enhance the detailed information of the foreground and background in the target image.

[0081] The above-mentioned self-attention local query step 2021 includes: obtaining the attention matrix of the target image to the global area of ​​the source image and the subject image; using the subject image layer and the editable image layer to locally mask the attention matrix; obtaining the queried foreground content features and background content features; and fusing the foreground content features and background content features.

[0082] In a specific embodiment of the present invention, the goal of the present invention is to extract the appearance features of the subject from the subject image and query the background content and semantic layout from the source image. src The target image can basically form a spatial layout similar to the source image. Therefore, the present invention focuses on the content query of the source image. The key features K and value features V of the self-attention layer can reflect the potential content representation of the image. Therefore, the present invention, at the denoising step t and layer l, B tgt The query feature Q through the self-attention layer T,t,l Query B sub and B src foreground and background content.

[0083] The present invention obtains the attention matrix A of the target image to the global area of ​​the source image and the subject image through the following formula: T,S,t,l ,A T,R,t,l . Where K S,t,l Indicates from B src The key feature of K R,t,l Indicates from B sub The key feature of . d represents the feature dimension.

[0084]

[0085] In order to limit the query area and avoid confusion, the present invention uses the subject image layer M R and editable image layers The attention matrix is ​​locally masked, that is, only the foreground content in the subject image and the background content in the source image are queried. Then, according to the following formula, the queried foreground content features can be obtained and background content characteristics Where V S,t,l Indicates from B src The value characteristics, V R,t,l Indicates from B sub The value feature.

[0086]

[0087] Finally, the present invention combines these two types of local query fusion features through the following formula:

[0088]

[0089] This operation has two purposes. First, Constraints are imposed on the editable image area to again ensure consistency with the layout of the source image, and secondly, foreground and background content are queried simultaneously to achieve replacement of the appearance of specific objects and enhance the alignment of background content with the source image.

[0090] The above-mentioned self-attention global injection step 2022 includes: calculating the attention matrix of the source image and the target image according to the equation, filtering the attention matrix using the editable image layer and the subject image layer, calculating the injection features of the source image diffusion process and the injection features of the subject image diffusion process, and reconstructing the injection features by calculation.

[0091] After the self-attention local query operation, the present invention finds that the generated images often have two problems: 1) various details lack authenticity, and 2) they are slightly confused with the original features during the query process. The present invention believes that this is because the query process is essentially a local fusion of the original features and the query features, which inevitably leads to feature crossover and confusion. Therefore, the present invention proposes a method of global attention hybrid injection to enhance the detail authenticity and content consistency of the foreground and background.

[0092] The present invention first calculates the attention matrix A of the source image and the image being viewed according to the equation S,t,l ,A R,t,l Different from self-attention local query, A S,t,l ,A R,t,l It is B src and B sub The original attention matrix in the reconstruction includes the mutual attention of all pixels in the image. Based on the objectives of the present invention, the present invention also uses and M R Filter A S,t,l and AR,t,l , the injection characteristics of the source image diffusion process are calculated And the injection characteristics of the topic image diffusion process To focus on the subject and background content. It should be noted that the self-attention global injection Does not interact with the foreground content of the source image.

[0093]

[0094] In a specific embodiment of the present invention, the feature output of the current step is reconstructed according to the above formula The detailed information of foreground and background in the target image is directly enhanced by replacement, reducing feature confusion.

[0095] Editing and generating step 203: combining the results of the self-attention local query and the self-attention global injection, outputting the edited and generated target image.

[0096] In a specific embodiment of the present invention, highly controllable and low-cost subject image editing and generation is ultimately achieved through a combination of self-attention local query and self-attention global injection operations.

[0097] Embodiment 3

[0098] like Figure 4 As shown, Figure 4 The embodiment of the present application provides a system for editing and generating a theme image based on a multi-party collaborative attention mechanism, and adopts the above-mentioned method for editing and generating a theme image based on a multi-party collaborative attention mechanism. The system includes:

[0099] Theme localization module 301: inputs a multimodal instruction to the theme localization module, locates the action area, and extracts the theme image and the target image in the action area, wherein the multimodal instruction includes: the user specifies the theme image and the corresponding text description, and the localization module includes: the target detection model and the segmentation model;

[0100] Multi-party collaborative attention control module 302: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image;

[0101] The source image is input into the pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

[0102] Editing and generating module 303: Combining the results of self-attention local query and self-attention global injection, output the edited and generated target image.

[0103] Embodiment 4

[0104] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for editing and generating subject images based on a multi-party collaborative attention mechanism.

[0105] Embodiment 5

[0106] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for editing and generating subject images based on a multi-party collaborative attention mechanism as described above are implemented.

[0107] In addition, combined Figure 1 The subject image editing and generation method based on the multi-party collaborative attention mechanism described in the embodiment of the present application can be implemented by an electronic device, such as a computer device. Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0108] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0109] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0110] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0111] The processor 81 reads and executes computer program instructions stored in the memory 82 to implement any one of the subject image editing and generation methods based on the multi-party collaborative attention mechanism in the above-mentioned embodiments.

[0112] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0113] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for editing and generating topic images based on a multi-party collaborative attention mechanism, characterized in that: The method comprises: Theme positioning step: inputting a multimodal instruction to the theme positioning module, positioning the action area, and extracting the theme image and the target image in the action area, wherein the multimodal instruction includes: the user specifies the theme image and the corresponding text description, and the positioning module includes: a target detection model and a segmentation model; Multi-party collaborative attention control step: input the subject image and the target image into a pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image; Editing and generating step: combining the results of the self-attention local query and the self-attention global injection, outputting the edited and generated target image.

2. According to claim 1, the method for editing and generating subject images based on the multi-party collaborative attention mechanism is characterized in that: The multi-party collaborative attention control step also includes: The source image is input into a pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

3. The method for editing and generating subject images based on a multi-party collaborative attention mechanism according to claim 1 or 2, characterized in that: The topic positioning step includes: Target detection step: using the target detection model to locate the designated area according to the input multimodal image-text pair information and generate a positioning frame, wherein the target detection model is composed of a DINO model; Segmentation step: using the segmentation model, segmenting the user-specified subject based on the positioning frame as conditional information, generating a binary subject image layer and an editable image layer, wherein the segmentation model is composed of a SAM model.

4. According to claim 3, the method for editing and generating subject images based on the multi-party collaborative attention mechanism is characterized in that: The multi-party collaborative attention control step also includes: Self-attention local query step: extracting appearance features in the subject image by injecting content features of the source image and the subject image during diffusion into the target image generation process; Self-attention global injection step: injecting subject features and background features into the diffusion process of the target image, reconstructing the feature output of the diffusion of the target image, and replacing and enhancing the detailed information of the foreground and background in the target image.

5. According to claim 4, the method for editing and generating subject images based on the multi-party collaborative attention mechanism is characterized in that: The self-attention local query step includes: Obtain an attention matrix of the target image to the global area of ​​the source image and the subject image; locally mask the attention matrix using the subject image layer and the editable image layer; obtain the queried foreground content features and background content features; and fuse the foreground content features and the background content features.

6. The method for editing and generating thematic images based on the multi-party collaborative attention mechanism according to claim 4, characterized in that: The self-attention global injection step includes: The attention matrices of the source image and the target image are calculated according to the equation, the attention matrices are filtered using the editable image layer and the subject image layer, the injection features of the source image diffusion process and the injection features of the subject image diffusion process are calculated, and the injection features are reconstructed by calculation.

7. A system for editing and generating a theme image based on a multi-party collaborative attention mechanism, using a method for editing and generating a theme image based on a multi-party collaborative attention mechanism as claimed in any one of claims 1 to 6, characterized in that: The system comprises: Theme positioning module: input multimodal instructions to the theme positioning module, locate the action area, and extract the theme image and the target image in the action area, wherein the multimodal instructions include: the user specifies the theme image and the corresponding text description, and the positioning module includes: a target detection model and a segmentation model; Multi-party collaborative attention control module: input the subject image and the target image into the pre-trained deep learning model, perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image; Editing and generating module: combining the results of the self-attention local query and the self-attention global injection, outputting the edited and generated target image.

8. The subject image editing and generation system based on multi-party collaborative attention mechanism according to claim 7, characterized in that: The multi-party collaborative attention control module also includes: The source image is input into a pre-trained deep learning model, and the subject image, target image and source image jointly perform self-attention local query in the self-attention layer to integrate the background of the source image and the foreground of the subject image; and perform self-attention global injection to achieve detail enhancement of the target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the subject image editing and generation method based on a multi-party collaborative attention mechanism described in any one of claims 1-6 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the subject image editing and generation method based on the multi-party collaborative attention mechanism are implemented as described in any one of claims 1 to 6.

Images