Image controllable generation method, electronic device, storage medium and program product

Abstract

The application provides an image controllable generation method, an electronic device, a storage medium and a program product. It relates to the technical field of image processing. The method comprises the following steps: acquiring a to-be-processed image, and dividing the to-be-processed image into at least two image blocks; extracting local detail features and global semantic features of the image blocks through a multi-path visual encoder to generate first visual features; determining controllable generation instructions according to the first visual features and image modification instructions; performing self-attention processing on the controllable generation instructions and the first visual features to obtain second visual features, wherein the self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instructions; and modifying the to-be-processed image according to the second visual features and the controllable generation instructions to obtain a target image. The application can improve the accuracy of the controllable generation scheme and the accuracy of the generated image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method for controllable image generation, electronic device, storage medium, and program product. Background Technology

[0002] In the fields of digital content creation and image processing, users' demands for controllability in image generation and editing are growing. In recent years, with the development of deep learning, image generation technology has made significant progress, especially with the emergence of diffusion models (such as Stable Diffusion) and controllable generation technologies (such as ControlNet), which have greatly improved the quality and controllability of images generated from text descriptions.

[0003] In related controllable image generation methods, there is usually a direct reliance on text prompts. The model directly generates the target image based on the text prompts. However, the above methods are insufficient in accurately controlling the generation of content in specific areas of the image. Especially in scenarios where it is necessary to keep the overall structure of the image unchanged and only modify local areas, it is difficult to achieve accurate and controllable generation. Summary of the Invention

[0004] This application provides a method, electronic device, storage medium, and program product for controlling image generation, in order to improve the accuracy of the controllable generation scheme and the accuracy of the generated images.

[0005] In a first aspect, this application provides a method for controllable image generation, comprising:

[0006] Obtain the image to be processed and divide it into at least two image blocks;

[0007] The first visual feature is generated by extracting local detail features and global semantic features of image patches through a multi-path visual encoder.

[0008] The controllable generation instructions are determined based on the first visual features and image modification instructions;

[0009] Self-attention processing is performed on the controllable generation instructions and the first visual features to obtain the second visual features. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instructions.

[0010] The image to be processed is modified based on the second visual features and controllable generation instructions to obtain the target image.

[0011] In one possible implementation, the multi-path visual encoder includes a local attention visual encoder and a global attention visual encoder. The multi-path visual encoder extracts local detail features and global semantic features of image patches to generate a first visual feature, including:

[0012] Using a local attention visual encoder, the image patch is processed through a window attention mechanism to extract local details and boundary features of the image to be processed, and to obtain initial visual features containing fine object information.

[0013] The initial visual features are convolutionally compressed to obtain the compressed visual features.

[0014] A global attention visual encoder is used to process the compressed visual features through a dense global attention mechanism to extract the semantic features of the image to be processed.

[0015] The initial visual features are fused with semantic features to obtain the first visual feature.

[0016] In one possible implementation, the controllable generation instructions are determined based on the first visual features and the image modification instructions, including:

[0017] Obtain the first prompt word, which is used to guide the hybrid expert model in parsing the image to be processed;

[0018] The first visual features and the first prompt words are analyzed by a hybrid expert model to generate the second prompt words. The second prompt words are semantic descriptive text used to describe the relationships between objects, attributes and scenes in the image to be processed.

[0019] The second prompt word is combined with the image modification instruction to obtain a controllable generation instruction. The controllable generation instruction is used to fully represent the original content of the image to be modified and the modification requirements to be performed.

[0020] In one possible implementation, self-attention processing is performed on the controllable generation instructions and the first visual features to obtain the second visual features, including:

[0021] The controllable generation instructions are vectorized to obtain a sequence of text feature vectors.

[0022] Convert the first visual feature into a sequence of visual feature vectors;

[0023] The first linear projection layer maps the text feature vector sequence to a feature space of a preset dimension to obtain the query feature vector sequence.

[0024] The visual feature vector sequence is mapped to a feature space of a preset dimension through the second linear projection layer to obtain the key feature vector sequence and the value feature vector sequence.

[0025] Using the query feature vector sequence as the query input, the key feature vector sequence as the key input, and the value feature vector sequence as the value input, multi-head attention computation is performed to obtain the second visual feature.

[0026] In one possible implementation, the image to be processed is modified based on the second visual features and controllable generation instructions to obtain the target image, including:

[0027] Obtain a controllable generative model;

[0028] The second visual features and controllable generation instructions are used as control conditions and input into the controllable generation model.

[0029] A target image is generated using a controllable generative model under controlled conditions, whereby the original content of non-specified regions in the image to be processed remains unchanged.

[0030] In one possible implementation, after modifying the image to be processed based on the second visual features and controllable generation instructions to obtain the target image, the method further includes:

[0031] Local detail features and global semantic features of the target image are extracted by a multi-path visual encoder to generate verification visual features;

[0032] The visual features for verification are analyzed using a hybrid expert model to generate verification prompts.

[0033] The verification prompts and image modification instructions are compared to obtain the loss function, which includes text semantic similarity loss and image perception loss.

[0034] The parameters of the multipath visual encoder, hybrid expert model, and controllable generative model are optimized based on the loss function.

[0035] In one possible implementation, the verification prompts are compared with the image modification instructions to obtain a loss function, including:

[0036] Calculate the textual semantic similarity loss between the verification prompt and the image modification instruction. The similarity loss is used to evaluate the semantic consistency between the target image and the image modification instruction.

[0037] Calculate the image perceptual loss between the target image and the image to be processed in the region not specified by the modification instruction. The perceptual loss is used to evaluate the degree to which the generation process preserves the content of the non-specified region of the image.

[0038] The loss function is obtained by weighted summation of text semantic similarity loss and image perception loss.

[0039] In one possible implementation, the method further includes, before acquiring the image to be processed:

[0040] Train the initial visual encoder and the initial hybrid expert model;

[0041] The initial controllable generative model is trained by fixing the model parameters of the initial visual encoder and the initial hybrid expert model.

[0042] The training images are modified using the trained initial visual encoder, initial hybrid expert model, and initial controllable generative model to obtain the target training image;

[0043] Determine the initial loss function based on the target training image;

[0044] By backpropagating the initial loss function, the parameters of the initial visual encoder, the initial hybrid expert model, and the initial controllable generative model are optimized respectively to obtain the multi-path visual encoder, the hybrid expert model, and the controllable generative model.

[0045] Secondly, this application provides an image controllable generation apparatus, comprising:

[0046] The acquisition module is used to acquire the image to be processed and divide the image to be processed into at least two image blocks;

[0047] The first generation module is used to extract local detail features and global semantic features of image patches through a multi-path visual encoder to generate the first visual features;

[0048] The first determining module is used to determine the controllable generation instructions based on the first visual features and the image modification instructions;

[0049] The second generation module is used to perform self-attention processing on the controllable generation instructions and the first visual features to obtain the second visual features. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instructions.

[0050] The second determining module is used to modify the image to be processed based on the second visual features and controllable generation instructions to obtain the target image.

[0051] In one possible implementation, the first generation module is specifically used for:

[0052] Using a local attention visual encoder, the image patch is processed through a window attention mechanism to extract local details and boundary features of the image to be processed, and to obtain initial visual features containing fine object information.

[0053] The initial visual features are convolutionally compressed to obtain the compressed visual features.

[0054] A global attention visual encoder is used to process the compressed visual features through a dense global attention mechanism to extract the semantic features of the image to be processed.

[0055] The initial visual features are fused with semantic features to obtain the first visual feature.

[0056] In one possible implementation, the first determining module is specifically used for:

[0057] Obtain the first prompt word, which is used to guide the hybrid expert model in parsing the image to be processed;

[0058] The first visual features and the first prompt words are analyzed by a hybrid expert model to generate the second prompt words. The second prompt words are semantic descriptive text used to describe the relationships between objects, attributes and scenes in the image to be processed.

[0059] The second prompt word is combined with the image modification instruction to obtain a controllable generation instruction. The controllable generation instruction is used to fully represent the original content of the image to be modified and the modification requirements to be performed.

[0060] In one possible implementation, the second generation module is specifically used for:

[0061] The controllable generation instructions are vectorized to obtain a sequence of text feature vectors.

[0062] Convert the first visual feature into a sequence of visual feature vectors;

[0063] The first linear projection layer maps the text feature vector sequence to a feature space of a preset dimension to obtain the query feature vector sequence.

[0064] The visual feature vector sequence is mapped to a feature space of a preset dimension through the second linear projection layer to obtain the key feature vector sequence and the value feature vector sequence.

[0065] Using the query feature vector sequence as the query input, the key feature vector sequence as the key input, and the value feature vector sequence as the value input, multi-head attention computation is performed to obtain the second visual feature.

[0066] In one possible implementation, the second determining module is specifically used for:

[0067] Obtain a controllable generative model;

[0068] The second visual features and controllable generation instructions are used as control conditions and input into the controllable generation model.

[0069] A target image is generated using a controllable generative model under controlled conditions, whereby the original content of non-specified regions in the image to be processed remains unchanged.

[0070] In one possible implementation, the image controllable generation device is also used for:

[0071] Local detail features and global semantic features of the target image are extracted by a multi-path visual encoder to generate verification visual features;

[0072] The visual features for verification are analyzed using a hybrid expert model to generate verification prompts.

[0073] The verification prompts and image modification instructions are compared to obtain the loss function, which includes text semantic similarity loss and image perception loss.

[0074] The parameters of the multipath visual encoder, hybrid expert model, and controllable generative model are optimized based on the loss function.

[0075] In one possible implementation, the image controllable generation device is also used for:

[0076] Calculate the textual semantic similarity loss between the verification prompt and the image modification instruction. The similarity loss is used to evaluate the semantic consistency between the target image and the image modification instruction.

[0077] Calculate the image perceptual loss between the target image and the image to be processed in the region not specified by the modification instruction. The perceptual loss is used to evaluate the degree to which the generation process preserves the content of the non-specified region of the image.

[0078] The loss function is obtained by weighted summation of text semantic similarity loss and image perception loss.

[0079] In one possible implementation, the image controllable generation device is also used for:

[0080] Train the initial visual encoder and the initial hybrid expert model;

[0081] The initial controllable generative model is trained by fixing the model parameters of the initial visual encoder and the initial hybrid expert model.

[0082] The training images are modified using the trained initial visual encoder, initial hybrid expert model, and initial controllable generative model to obtain the target training image;

[0083] Determine the initial loss function based on the target training image;

[0084] By backpropagating the initial loss function, the parameters of the initial visual encoder, the initial hybrid expert model, and the initial controllable generative model are optimized respectively to obtain the multi-path visual encoder, the hybrid expert model, and the controllable generative model.

[0085] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0086] The memory stores the instructions that the computer executes;

[0087] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0088] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0089] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0090] The image controllable generation method, electronic device, storage medium, and program product provided in this application extract local detail features and global semantic features of the image to be processed through a multi-path visual encoder to obtain a first visual feature. This first visual feature can simultaneously preserve image details and overall semantics, providing rich input for subsequent self-attention processing. A controllable generation instruction is determined through the first visual feature and the image modification instruction, and self-attention is applied to the controllable generation instruction and the first visual feature to obtain a second visual feature. This second visual feature can accurately focus on the target area that needs to be modified, avoiding interference from irrelevant areas. As a result, the target image generated based on the second visual feature can meet the user's intention, achieving precise modification of the target area and improving the accuracy of controllable generation. Attached Figure Description

[0091] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0092] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0093] Figure 2 A flowchart illustrating a controllable image generation method provided in this application embodiment. Figure 1 ;

[0094] Figure 3 for Figure 2 A flowchart illustrating the specific implementation of step S102 in the illustrated embodiment;

[0095] Figure 4 for Figure 2 A flowchart illustrating the specific implementation of step S103 in the illustrated embodiment;

[0096] Figure 5 A flowchart illustrating a controllable image generation method provided in this application embodiment. Figure 2;

[0097] Figure 6 This is a schematic diagram illustrating an example of controllable image generation provided in an embodiment of this application;

[0098] Figure 7 A schematic diagram of the image controllable generation device provided in this application;

[0099] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0100] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0101] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0102] In the fields of digital content creation and image processing, users' demands for controllability in image generation and editing are growing. For example, in advertising design, designers need to quickly adjust specific elements in an image (such as product color, background layout, or text content) according to client requirements without redesigning the entire image; in e-commerce, merchants need to dynamically modify product display images based on user feedback (such as changing clothing colors or adjusting product placement); and in virtual reality or game development, developers need to efficiently generate customized images that meet the needs of the scene (such as modifying building exteriors or adding specific decorations).

[0103] In related controllable image generation methods, there is usually a direct reliance on text prompts. The model directly generates the target image based on the text prompts. However, the above methods are insufficient in accurately controlling the generation of content in specific areas of the image. Especially in scenarios where it is necessary to keep the overall structure of the image unchanged and only modify local areas, it is difficult to achieve accurate and controllable generation.

[0104] Based on this, this application provides a method for controllable image generation. A first visual feature is obtained by extracting local detail features and global semantic features from the image to be processed using a multi-path visual encoder. This first visual feature can simultaneously preserve graphic details and overall semantics, providing rich input for subsequent self-attention processing. A controllable generation instruction is determined using the first visual feature and the image modification instruction. Self-attention is then applied to the controllable generation instruction and the first visual feature to obtain a second visual feature. This second visual feature can accurately focus on the target area that needs modification, avoiding interference from irrelevant areas. Therefore, the target image generated based on the second visual feature can meet the user's intent, achieving precise modification of the target area and improving the accuracy of controllable generation.

[0105] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0106] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, it includes the initial image and the target image.

[0107] The initial image is the image input by the user and needs to be modified. The target image is the image obtained after modifying the initial image using controlled generation technology. For example, if the initial image is an image of a fire hydrant, with the hydrant itself being red and the top cover being black, and the user needs to change the top cover from black to blue, then after modifying the initial image using controlled generation technology, the target image will include a red fire hydrant with a blue top cover. The initial image and the target image are identical in all areas except for the color of the hydrant cover.

[0108] Figure 2 A flowchart illustrating a controllable image generation method provided in this application embodiment. Figure 1 The execution subject of this method can be a computing device or a processor located within a computing device. For example, the computing device can be a server. For example, the processor within the computing device can be the central processing unit (CPU) of a server. For ease of understanding, the following explanation will use a computing device as the execution subject. Figure 2 As shown, the method includes:

[0109] S101. Obtain the image to be processed and divide the image to be processed into at least two image blocks.

[0110] The image to be processed is the image that needs to be modified. An image patch is a small region unit that is uniformly divided into fixed pixel sizes from the image to be processed; it is also called a patch. For example, dividing a 1024×1024 image into 16×16 pixel sizes results in 4096 image patches.

[0111] For example, the image to be processed can be uniformly gridded by determining the size of the image blocks, ensuring that each image block is non-overlapping and fully covers the image to be processed, thus obtaining at least two image blocks.

[0112] S102. Extract local detail features and global semantic features of image patches through a multi-path visual encoder to generate the first visual feature.

[0113] A multi-path visual encoder is an encoder used to extract image features. It can include multiple paths, such as path one for extracting local detail features of an image patch, path two for extracting global semantic features of an image patch, etc.

[0114] Local detail features are the fine features of a local area of ​​an image, such as boundaries, object outlines, and textures. They are the basis for achieving fine-grained modifications to a region, such as the shape of the top cover of a fire hydrant.

[0115] Global semantic features are macroscopic features of the entire image, such as scene information, object relationships, and overall scope. They are used to ensure the consistency between the modified image and the original scene. For example, the overall semantics of a fire hydrant located in a street scene with parked cars and trees around it.

[0116] The first visual feature is a set of visual tokens generated by integrating local detail features and global semantic features.

[0117] For example, a multi-path visual encoder may include a local attention visual encoder and a global attention visual encoder. The image patch can be processed sequentially by the local attention visual encoder and the global attention visual encoder to obtain the first visual feature.

[0118] S103. Determine the controllable generation instructions based on the first visual features and image modification instructions.

[0119] Image modification instructions are specific image modification requests submitted by the user, such as changing the top cover of a fire hydrant to blue. Controllable generation instructions are prompts used to guide the model's output.

[0120] For example, the first visual features can be input into a large language model, which generates a semantic understanding of the image to be processed and the corresponding text. Then, the image understanding text generated by the large language model and the image modification instructions provided by the user are combined to construct system prompt words as controllable generation instructions.

[0121] S104. Perform self-attention processing on the controllable generation instruction and the first visual feature to obtain the second visual feature. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instruction.

[0122] Self-attention processing is a cross-modal attention computation process used to establish a correlation between text instructions and image features, enabling the model to focus on the target region that needs modification. Secondary visual features are the set of visual tokens whose features in the target region are enhanced after self-attention processing; these are key features for achieving precise region modification. For example, secondary visual features might be the set of tokens that boost the feature weights of the area around the top cover of a fire hydrant while keeping the feature weights of other areas unchanged.

[0123] For example, performing self-attention processing on controllable generation instructions and first visual features to obtain second visual features includes: vectorizing the controllable generation instructions to obtain a text feature vector sequence; converting the first visual features into a visual feature vector sequence; mapping the text feature vector sequence to a feature space of a preset dimension through a first linear projection layer to obtain a query feature vector sequence; mapping the visual feature vector sequence to a feature space of a preset dimension through a second linear projection layer to obtain a key feature vector sequence and a value feature vector sequence; and performing multi-head attention computation using the query feature vector sequence as query input, the key feature vector sequence as key input, and the value feature vector sequence as value input to obtain the second visual features.

[0124] The text feature vector sequence is a one-dimensional vector sequence obtained by processing the controllable generation instruction through a text encoder, with each vector corresponding to a text token in the instruction. The visual feature vector sequence is a vector sequence obtained by format conversion of the first visual feature. The first linear projection layer is a linear transformation layer used to map the text feature vectors to a space with the same dimension as the visual features. The query feature vector sequence is a vector sequence obtained by projecting the text feature vectors and is used as the "query" input in the attention calculation. The second linear projection layer is a linear transformation layer used to map the visual feature vectors to a unified dimensional space. The key feature vector sequence and the value feature vector sequence are two sets of vector sequences obtained by projecting the visual feature vectors and are used as the "key" and "value" inputs, respectively, in the attention calculation.

[0125] The controllable generated instructions can be feature-extracted using a text encoder to obtain a text feature vector sequence. This text feature vector is then input into the first linear projection layer for linear transformation calculation to obtain a query feature vector sequence. The visual feature vector sequence is then input into the second linear projection layer for linear transformation calculation. The transformed vector sequence is copied into two sets, serving as the key vector feature sequence and the value vector feature sequence, respectively. The query feature vector sequence, key feature vector sequence, and value feature vector sequence are then input into a multi-head attention calculation module. The module processes each set of vectors separately, calculating the attention weights between the text token and the visual token. The attention weights are normalized, enhancing the weights of regions related to the modified instructions and weakening the weights of non-target regions. The results of the separate calculations are then concatenated and linearly transformed to output the weight-optimized visual feature vector sequence, i.e., the second visual feature.

[0126] In the above method, by projecting the text feature vector sequence corresponding to the controllable generation instruction and the visual feature vector sequence corresponding to the first visual feature into the same feature space, and establishing a deep association between text semantics and visual features through multi-head attention computation, the feature weights of the region corresponding to the image modification instruction can be accurately strengthened, the interference of non-target regions can be weakened, and the problem of insufficient interaction between text and visual features in related technologies can be effectively solved. This enables accurate positioning of the region to be modified in the image, avoids accidental modification of non-target regions, significantly improves the accuracy of image region control, provides accurate feature guidance for subsequent controllable image generation, and ensures that the generation result not only meets the user's modification instruction requirements, but also maintains the consistency of the overall image structure and scene.

[0127] S105. Modify the image to be processed according to the second visual features and the controllable generation instructions to obtain the target image.

[0128] The target image is the final output image that completes local modifications while maintaining the overall structure, such as changing the top cover of a fire hydrant from black to blue while keeping other areas completely unchanged.

[0129] Optionally, the image to be processed is modified according to the second visual features and the controllable generation instructions to obtain the target image, including: obtaining a controllable generation model; inputting the second visual features and the controllable generation instructions as control conditions into the controllable generation model; and generating the target image through the controllable generation model under the guidance of the control conditions, wherein the target image retains the original content of the non-specified area in the image to be processed unchanged.

[0130] The second visual features and controllable generation instructions are input as control conditions into the controllable generation module. The ControlNet model is used to control the Stable Diffusion image generation process, thereby controllably generating the modified image, i.e., the target image.

[0131] In the above method, by introducing a controllable generation model, the second visual features and controllable generation instructions are used as dual control conditions. This guides the model to accurately lock the designated area to be modified, effectively avoiding the problem of erroneous modification of non-target areas caused by insufficient interaction between text and visual features in related technologies. At the same time, relying on the precise constraints of the dual control conditions, the generated target image strictly maintains the original content and structure of non-designated areas while completing the modification of the content in the designated area. This ensures the overall semantic coherence and scene coordination of the image, making the target image more accurate in terms of content and layout.

[0132] The image controllable generation method provided in this application extracts local detail features and global semantic features of the image to be processed through a multi-path visual encoder to obtain a first visual feature. This first visual feature can simultaneously preserve graphic details and overall semantics, providing rich input for subsequent self-attention processing. The controllable generation instruction is determined by the first visual feature and the image modification instruction, and self-attention is applied to the controllable generation instruction and the first visual feature to obtain a second visual feature. This second visual feature can accurately focus on the target area that needs to be modified, avoiding interference from irrelevant areas. As a result, the target image generated based on the second visual feature can meet the user's intention, achieve precise modification of the target area, and improve the accuracy of controllable generation.

[0133] Optional, see Figure 3 , Figure 3 for Figure 2 The illustrated embodiment provides a flowchart of the specific implementation of step S102. A first visual feature is generated by extracting local detail features and global semantic features of image patches using a multi-path visual encoder, including:

[0134] S1021. Using a local attention visual encoder, the image patch is processed through a window attention mechanism to extract local details and boundary features of the image to be processed, and to obtain initial visual features containing fine object information.

[0135] Multiple image patches first pass through a local attention visual encoder, which is mainly window attention-based. By fusing image features, it focuses on local details and boundaries in the image to be processed, extracts detailed features of the image to be processed, efficiently segments any object in the image to be processed, obtains fine visual tokens of all objects in the image to be processed, and obtains initial visual features.

[0136] S1022. Perform convolution compression on the initial visual features to obtain compressed visual features.

[0137] The number of visual tokens can be reduced by inputting the initial visual features into a convolution module, and the convolution operation can be used to obtain compressed visual features. For example, using a 16x downsampling convolution module, the 4096 initial visual tokens of a fire hydrant image can be compressed to 256. The compressed 256 visual tokens still retain the feature information of key objects such as the fire hydrant cover and rivets, while discarding irrelevant pixel-level redundant data.

[0138] S1023. Use a global attention visual encoder to process the compressed visual features through a dense global attention mechanism to extract the semantic features of the image to be processed.

[0139] By inputting the compressed visual features into a global attention visual encoder, dense global attention processing can be performed to capture the global context of the image and extract the semantic features of the entire image to be processed.

[0140] S1024. The initial visual features and semantic features are fused to obtain the first visual features.

[0141] For example, feature splicing or weighted fusion can be used to align the dimensions of the initial visual features and semantic features, and then the fused features can be normalized to ensure a weight balance between local detail features and global semantic features, thus obtaining the first visual feature.

[0142] In the above method, the local attention visual encoder can accurately capture the details and boundary information of local objects in the image; convolutional compression processing can effectively reduce the dimensionality redundancy of the initial visual features, reduce computational resource consumption, and improve the overall inference efficiency of the model; the global attention visual encoder can fully explore the global semantic association and contextual information of the image, ensuring the overall style and scene coordination after image modification; finally, by fusing the initial visual features and semantic features, the advantages of local fine features and global semantic features are complemented, generating a first visual feature with both detail integrity and semantic consistency, providing high-quality feature support for subsequent cross-modal attention focusing and controllable image generation, thereby improving the regional control accuracy and generation effect stability of the entire controllable image generation system.

[0143] Optional, see Figure 4 , Figure 4 for Figure 2 The flowchart illustrates the specific implementation of step S103 in the illustrated embodiment. Determining the controllable generation instruction based on the first visual feature and image modification instruction includes:

[0144] S1031. Obtain the first prompt word. The first prompt word is used to guide the hybrid expert model to parse the image to be processed.

[0145] The first prompt word is provided by the user and input into a Mixture of Experts (MOE) model, which then generates a text description of the second prompt word. The MOE model is a large language model used to generate descriptive text for the image to be processed based on the first prompt word. For example, the first prompt word might be "describe this image".

[0146] S1032. The first visual features and the first prompt words are analyzed by a hybrid expert model to generate the second prompt words. The second prompt words are semantic descriptive text used to describe the relationships between objects, attributes and scenes in the image to be processed.

[0147] For example, the first visual feature and the first cue word are input into a hybrid expert model to obtain a second cue word, such as: "The image shows a red fire hydrant. There is a smiley face on the hydrant, which looks very friendly and welcoming. The top of the hydrant has a black cap with some metal rivets around it. At the bottom of the hydrant, there is a pink sticker that says 'bowtie'. In the background, a street can be seen with several parked cars and some trees. The overall picture gives a warm and friendly feeling."

[0148] S1033. Combine the second prompt word with the image modification instruction to obtain a controllable generation instruction. The controllable generation instruction is used to fully characterize the original content of the image to be modified and the modification requirements to be performed.

[0149] Controllable generation instructions, for example: "Based on the following image content: The image shows a red fire hydrant. There's a smiley face on the hydrant, which looks very friendly and welcoming. The top of the hydrant has a black cap with some metal rivets around it. At the bottom of the hydrant, there's a pink sticker that says 'bowtie.' In the background, you can see a street with several parked cars and some trees. The overall image gives a warm and friendly feeling. Perform the following modifications: Change the top cap of the fire hydrant to blue. Maintain the overall image structure and only modify the specified area."

[0150] In the above method, by setting a first cue word, the hybrid expert model can be accurately guided to focus on the direction of image analysis, avoiding deviation of the model's analysis from the core requirements. By combining the local details and global semantic information contained in the first visual features, the hybrid expert model generates a second cue word that can fully represent the image object, attributes, and scene relationships, achieving a deep semantic understanding of the image content and making up for the shortcomings of insufficient semantic analysis in traditional technologies. By combining the second cue word with the image modification instructions, a controllable generation instruction that has both complete semantic information of the original image and clear modification requirements is constructed. This provides precise text guidance for subsequent attention processing, ensuring that the model can accurately lock the area to be modified, and effectively associates the original image content with the modification requirements, avoiding the problem of erroneous modification of non-target areas due to ambiguous instructions. Ultimately, this improves the regional control accuracy and consistency of the generation results of the entire controllable image generation system.

[0151] Optional, see Figure 5 , Figure 5 A flowchart illustrating a controllable image generation method provided in this application embodiment. Figure 2 After obtaining the target image, the method also includes a closed-loop evaluation and optimization step, specifically including:

[0152] S106. Extract local detail features and global semantic features of the target image through a multi-path visual encoder to generate verification visual features.

[0153] The process of generating verification visual features is described in step S102, which involves inputting the target image into a multipath visual encoder to obtain verification visual features, which are used to verify the target image.

[0154] S107. Analyze the visual features of verification using a hybrid expert model to generate verification prompts.

[0155] The process of obtaining the verification prompts is described in steps S1031 and S1032. The verification prompts are texts generated by the hybrid expert model that describe the target image. In the verification prompts, the description of the fire hydrant cover color is "The fire hydrant has a blue cover on top," and the other descriptions are the same as the second prompt.

[0156] S108. Compare the verification prompts with the image modification instructions to obtain the loss function, which includes text semantic similarity loss and image perception loss.

[0157] The loss function is determined by comparing the verification prompts with the image modification instructions, such as comparing the difference between "the top of the fire hydrant has a blue cover" and "make the top cover of the fire hydrant blue", as well as the differences between the descriptions corresponding to other unmodified areas.

[0158] For example, the loss function can be determined as follows: calculate the textual semantic similarity loss between the verification prompt and the image modification instruction, whereby the similarity loss is used to evaluate the semantic consistency between the target image and the image modification instruction; calculate the image perceptual loss between the target image and the image to be processed in the non-specified region of the modification instruction, whereby the perceptual loss is used to evaluate the degree to which the generation process preserves the content of the non-specified region of the image; and sum the textual semantic similarity loss and the image perceptual loss by weight to obtain the loss function.

[0159] The loss function achieves a precise two-dimensional evaluation and constraint on the controllable image generation effect by weighted summing of text semantic similarity loss and image perception loss. Text semantic similarity loss effectively measures the semantic fit between the target image and the image modification instructions, ensuring that the generated result accurately matches the user's modification needs and solving the problem of semantic discrepancies between the generated content and the instructions in traditional models. Image perception loss focuses on the preservation of content in non-target areas, effectively constraining the model from destroying the structure and content of non-target areas in the original image during modification, avoiding unexpected image alterations, and compensating for the shortcomings of insufficient region control precision in traditional techniques. The weighted fusion of these two loss functions balances the dual requirements of semantic consistency and fidelity in non-target areas, making the evaluation dimensions of the loss function more comprehensive. By backpropagating this loss function, the model can be guided to simultaneously improve semantic matching accuracy and region control precision during iterative optimization, thereby significantly improving the overall quality and stability of controllable image generation.

[0160] S109. Optimize the parameters of the multipath visual encoder, the hybrid expert model, and the controllable generative model based on the loss function.

[0161] The backpropagation loss function is optimized by minimizing the loss function. The parameters of the multi-path visual encoder, the hybrid expert model, and the controllable generative model are modified to continuously optimize the performance of the entire system.

[0162] The closed-loop evaluation and optimization mechanism extracts visual features and performs semantic parsing on the target image again to obtain a complete semantic description of the generated image. It then compares this description with the modification instructions for the image to be processed to calculate the multi-dimensional consistency loss. Through backpropagation of the loss, the parameters of all modules in the process, such as the visual encoder, hybrid expert model, and controllable generation model, can be continuously optimized to reduce the deviation between the generated result and the user's expectations. This allows the entire controllable image generation system to form a virtuous cycle of "generation-evaluation-optimization," achieving continuous iterative improvement in generation quality and ensuring the stability of the target image in terms of content consistency and visual coordination.

[0163] Optionally, before acquiring the image to be processed, the method also includes a model training process, specifically including: training the initial visual encoder and the initial hybrid expert model; fixing the model parameters of the initial visual encoder and the initial hybrid expert model, and training the initial controllable generative model; using the trained initial visual encoder, the initial hybrid expert model, and the initial controllable generative model to modify and process the training image to obtain the target training image; determining the initial loss function based on the target training image; and optimizing the parameters of the initial visual encoder, the initial hybrid expert model, and the initial controllable generative model respectively by backpropagating the initial loss function to obtain the multi-path visual encoder, the hybrid expert model, and the controllable generative model.

[0164] In terms of training the model, the training optimization strategy is an end-to-end training optimization framework with multi-task learning objectives, and adopts a progressive training strategy: Stage 1: Pre-train the visual understanding module consisting of a dual-path visual encoder and an MOE model decoder; Stage 2: Fix the parameters of the visual understanding module and train the ControlNet image controllable generation module; Stage 3: End-to-end joint training, calculate the loss function, and optimize the overall performance through backpropagation of the loss function.

[0165] The image controllable generation method provided in this application embodiment is described below with a specific example. See also Figure 6 , Figure 6 This is a schematic diagram illustrating an example of controllable image generation provided in an embodiment of this application. It includes an image to be processed, an image segmented by a local attention visual encoder, and a target image.

[0166] The user-input image to be processed is a fire hydrant image, which is then divided into 16×16 pixel blocks. In this example, the fire hydrant image resolution is 1024×1024, so the image is divided into 1024 / 16×1024 / 16=4096 patchtokens.

[0167] A dual-path visual encoder is used to extract features from the image to be processed, obtaining the first visual features. A local attention visual encoder can efficiently segment objects in an image, obtaining fine-grained visual tokens for all objects. After segmentation by the local attention visual encoder, the image to be processed can be obtained as follows: Figure 6The image shown is segmented by a local attention visual encoder, which identifies and labels all objects in the image. The compression module uses a 16x downsampling convolution to obtain a reduced number of visual tokens. The global attention visual encoder captures the global context of the image, extracts its semantic features, and obtains the global visual tokens. The 4096 patch tokens first undergo local attention processing, then are compressed by a 16x convolution module, reducing the number to 4096 / 16 = 256 tokens, before proceeding to global attention processing.

[0168] Hybrid expert models can effectively balance model performance and computational efficiency. The first visual feature is input into the hybrid expert model, along with a first cue word, which is then used to generate a descriptive text for the image, i.e., a second cue word. In this embodiment, the first cue word is, for example, "Describe this image," and the second cue word is, for example, "The image shows a red fire hydrant. There's a smiley face on the hydrant, which looks very friendly and welcoming. The top of the hydrant has a black cap with some metal rivets around it. At the bottom of the hydrant, there's a pink sticker that says 'bowtie.' In the background, you can see a street with several parked cars and some trees. The overall image gives a warm and friendly feeling."

[0169] The second prompt and the user-provided image modification instruction are combined to form a controlled generation instruction. For example, if the image modification instruction is "change the top cover of the fire hydrant to blue," then the controlled generation instruction would be: "Based on the following image content: The image shows a red fire hydrant. There is a smiley face on the hydrant, which looks very friendly and welcoming. The top of the hydrant has a black cover with some metal rivets around it. At the bottom of the hydrant, there is a pink sticker that says 'bowtie.' A street can be seen in the background, with several parked cars and some trees. The overall image gives a warm and friendly feeling. Perform the following modification: Change the top cover of the fire hydrant to blue. Maintain the overall image structure and modify only the specified area."

[0170] The controllable generation instructions are combined with the first visual features to perform self-attention calculations to obtain the second visual features. After calculation, the calculated value for the fire hydrant top cover is the highest, and the model will focus on the area and object of the fire hydrant top cover.

[0171] The second visual feature and the controllable generation instruction are input into the controllable generation model to generate the modified image, i.e., the target image, in which the top cover of the fire hydrant has been changed to blue.

[0172] The target image is processed again using a dual-path visual encoder and a hybrid expert model to obtain the descriptive text for the generated image, i.e., the verification prompt. The verification prompt is: "The image shows a red fire hydrant. There is a smiley face on the hydrant, which looks very friendly and welcoming. The top of the hydrant has a blue cap with some metal rivets around it. At the bottom of the hydrant, there is a pink sticker that says 'bowtie'. In the background, a street can be seen with several parked cars and some trees. The overall picture gives a warm and friendly feeling."

[0173] The verification prompts are compared with the image modification instructions. The difference between the text generated by the model describing the fire hydrant cover, "There is a blue cover on the top of the fire hydrant," and the image modification instruction, "Make the cover on the top of the fire hydrant blue," is compared. The loss function is calculated, and the loss is backpropagated to continuously optimize the parameters of the overall model in a closed loop.

[0174] Figure 7 This is a schematic diagram of the structure of the image controllable generation device provided in this application, as shown below. Figure 7 As shown, the image controllable generation device 20 provided in this embodiment includes:

[0175] The acquisition module 21 is used to acquire the image to be processed and divide the image to be processed into at least two image blocks;

[0176] The first generation module 22 is used to extract local detail features and global semantic features of image patches through a multi-path visual encoder to generate first visual features;

[0177] The first determining module 23 is used to determine the controllable generation instruction based on the first visual features and the image modification instruction;

[0178] The second generation module 24 is used to perform self-attention processing on the controllable generation instruction and the first visual feature to obtain the second visual feature. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instruction.

[0179] The second determining module 25 is used to modify the image to be processed according to the second visual features and the controllable generation instructions to obtain the target image.

[0180] In one possible implementation, the first generation module 22 is specifically used for:

[0181] Using a local attention visual encoder, the image patch is processed through a window attention mechanism to extract local details and boundary features of the image to be processed, and to obtain initial visual features containing fine object information.

[0182] The initial visual features are convolutionally compressed to obtain the compressed visual features.

[0183] A global attention visual encoder is used to process the compressed visual features through a dense global attention mechanism to extract the semantic features of the image to be processed.

[0184] The initial visual features are fused with semantic features to obtain the first visual feature.

[0185] In one possible implementation, the first determining module 23 is specifically used for:

[0186] Obtain the first prompt word, which is used to guide the hybrid expert model in parsing the image to be processed;

[0187] The first visual features and the first prompt words are analyzed by a hybrid expert model to generate the second prompt words. The second prompt words are semantic descriptive text used to describe the relationships between objects, attributes and scenes in the image to be processed.

[0188] The second prompt word is combined with the image modification instruction to obtain a controllable generation instruction. The controllable generation instruction is used to fully represent the original content of the image to be modified and the modification requirements to be performed.

[0189] In one possible implementation, the second generation module 24 is specifically used for:

[0190] The controllable generation instructions are vectorized to obtain a sequence of text feature vectors.

[0191] Convert the first visual feature into a sequence of visual feature vectors;

[0192] The first linear projection layer maps the text feature vector sequence to a feature space of a preset dimension to obtain the query feature vector sequence.

[0193] The visual feature vector sequence is mapped to a feature space of a preset dimension through the second linear projection layer to obtain the key feature vector sequence and the value feature vector sequence.

[0194] Using the query feature vector sequence as the query input, the key feature vector sequence as the key input, and the value feature vector sequence as the value input, multi-head attention computation is performed to obtain the second visual feature.

[0195] In one possible implementation, the second determining module 25 is specifically used for:

[0196] Obtain a controllable generative model;

[0197] The second visual features and controllable generation instructions are used as control conditions and input into the controllable generation model.

[0198] A target image is generated using a controllable generative model under controlled conditions, whereby the original content of non-specified regions in the image to be processed remains unchanged.

[0199] In one possible implementation, the image controllable generation device 20 is further used for:

[0200] Local detail features and global semantic features of the target image are extracted by a multi-path visual encoder to generate verification visual features;

[0201] The visual features for verification are analyzed using a hybrid expert model to generate verification prompts.

[0202] The verification prompts and image modification instructions are compared to obtain the loss function, which includes text semantic similarity loss and image perception loss.

[0203] The parameters of the multipath visual encoder, hybrid expert model, and controllable generative model are optimized based on the loss function.

[0204] In one possible implementation, the image controllable generation device 20 is further used for:

[0205] Calculate the textual semantic similarity loss between the verification prompt and the image modification instruction. The similarity loss is used to evaluate the semantic consistency between the target image and the image modification instruction.

[0206] Calculate the image perceptual loss between the target image and the image to be processed in the region not specified by the modification instruction. The perceptual loss is used to evaluate the degree to which the generation process preserves the content of the non-specified region of the image.

[0207] The loss function is obtained by weighted summation of text semantic similarity loss and image perception loss.

[0208] In one possible implementation, the image controllable generation device 20 is further used for:

[0209] Train the initial visual encoder and the initial hybrid expert model;

[0210] The initial controllable generative model is trained by fixing the model parameters of the initial visual encoder and the initial hybrid expert model.

[0211] The training images are modified using the trained initial visual encoder, initial hybrid expert model, and initial controllable generative model to obtain the target training image;

[0212] Determine the initial loss function based on the target training image;

[0213] By backpropagating the initial loss function, the parameters of the initial visual encoder, the initial hybrid expert model, and the initial controllable generative model are optimized respectively to obtain the multi-path visual encoder, the hybrid expert model, and the controllable generative model.

[0214] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 30 provided in this embodiment includes at least one processor 301 and a memory 302. Optionally, the electronic device 30 further includes a communication component 303. The processor 301, memory 302, and communication component 303 are connected via a bus.

[0215] In a specific implementation, at least one processor 301 executes computer execution instructions stored in memory 302, causing at least one processor 301 to execute the above-described image controllable generation method embodiment.

[0216] The specific implementation process of processor 301 can be found in the above method embodiments, and its implementation principle and technical effect are similar, so it will not be repeated here. In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the image controllable generation method disclosed in the invention can be directly reflected as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0217] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0218] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0219] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described image controllable generation method.

[0220] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described image controllable generation method.

[0221] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0222] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0223] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0224] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0225] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0226] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0227] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0228] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for controllable image generation, characterized in that, The method includes: Obtain the image to be processed and divide the image to be processed into at least two image blocks; The first visual feature is generated by extracting local detail features and global semantic features of the image patch through a multi-path visual encoder. The controllable generation instruction is determined based on the first visual feature and the image modification instruction; Self-attention processing is performed on the controllable generation instruction and the first visual feature to obtain a second visual feature. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instruction. The image to be processed is modified according to the second visual feature and the controllable generation instruction to obtain the target image.

2. The image controllable generation method according to claim 1, characterized in that, The multi-path visual encoder includes a local attention visual encoder and a global attention visual encoder. The step of extracting local detail features and global semantic features of the image patch using the multi-path visual encoder to generate a first visual feature includes: Using the local attention visual encoder, the image patch is processed through the window attention mechanism to extract local details and boundary features of the image to be processed, and to obtain initial visual features containing fine object information. The initial visual features are subjected to convolutional compression to obtain compressed visual features; Using the global attention visual encoder, the compressed visual features are processed through a dense global attention mechanism to extract the semantic features of the image to be processed. The initial visual features are fused with the semantic features to obtain the first visual feature.

3. The image controllable generation method according to claim 1, characterized in that, The step of determining the controllable generation instruction based on the first visual feature and the image modification instruction includes: Obtain a first prompt word, which is used to guide the hybrid expert model to parse the image to be processed; The hybrid expert model is used to analyze the first visual feature and the first prompt word to generate a second prompt word. The second prompt word is a semantic descriptive text used to describe the relationship between objects, attributes and scenes in the image to be processed. The second prompt word is combined with the image modification instruction to obtain the controllable generation instruction, which is used to fully characterize the original content of the image to be modified and the modification requirements to be performed.

4. The image controllable generation method according to claim 1, characterized in that, The step of performing self-attention processing on the controllable generation instruction and the first visual feature to obtain the second visual feature includes: The controllable generation instructions are vectorized to obtain a text feature vector sequence. Convert the first visual feature into a sequence of visual feature vectors; The text feature vector sequence is mapped to a feature space of a preset dimension through the first linear projection layer to obtain the query feature vector sequence. The visual feature vector sequence is mapped to the feature space of the preset dimension through the second linear projection layer to obtain the key feature vector sequence and the value feature vector sequence. Using the query feature vector sequence as query input, the key feature vector sequence as key input, and the value feature vector sequence as value input, multi-head attention calculation is performed to obtain the second visual feature.

5. The image controllable generation method according to claim 1, characterized in that, The step of modifying the image to be processed according to the second visual feature and the controllable generation instruction to obtain the target image includes: Obtain a controllable generative model; The second visual feature and the controllable generation instruction are used as control conditions and input into the controllable generation model; The target image is generated by the controllable generation model under the guidance of the control conditions, wherein the target image retains the original content of the non-specified region in the image to be processed.

6. The image controllable generation method according to any one of claims 1-5, characterized in that, After modifying the image to be processed according to the second visual feature and the controllable generation instruction to obtain the target image, the method further includes: The multi-path visual encoder extracts local detail features and global semantic features of the target image to generate verification visual features. The verification visual features are analyzed using the hybrid expert model to generate verification prompts. The verification prompts are compared with the image modification instructions to obtain a loss function, which includes text semantic similarity loss and image perception loss. The parameters of the multipath visual encoder, the hybrid expert model, and the controllable generative model are optimized based on the loss function.

7. The image controllable generation method according to claim 6, characterized in that, The step of comparing the verification prompt with the image modification instruction to obtain the loss function includes: Calculate the text semantic similarity loss between the verification prompt and the image modification instruction, and use the text semantic similarity loss to evaluate the semantic consistency between the target image and the image modification instruction; Calculate the image perceptual loss between the target image and the image to be processed in the region not specified by the modification instruction. The perceptual loss is used to evaluate the degree to which the generation process preserves the content of the non-specified region of the image. The loss function is obtained by weighted summation of the text semantic similarity loss and the image perception loss.

8. The image controllable generation method according to any one of claims 1-5, characterized in that, Before acquiring the image to be processed, the method further includes: Train the initial visual encoder and the initial hybrid expert model; The initial controllable generative model is trained by fixing the model parameters of the initial visual encoder and the initial hybrid expert model. The initial visual encoder, the initial hybrid expert model, and the initial controllable generative model are used to modify the training images to obtain the target training images; Determine the initial loss function based on the target training image; By backpropagating the initial loss function, the parameters of the initial visual encoder, the initial hybrid expert model, and the initial controllable generative model are optimized respectively to obtain the multi-path visual encoder, the hybrid expert model, and the controllable generative model.

9. An image controllable generation device, characterized in that, include: The acquisition module is used to acquire the image to be processed and to segment the image to be processed into at least two image blocks; The first generation module is used to extract local detail features and global semantic features of the image patch through a multi-path visual encoder to generate a first visual feature; The first determining module is used to determine a controllable generation instruction based on the first visual feature and the image modification instruction; The second generation module is used to perform self-attention processing on the controllable generation instruction and the first visual feature to obtain a second visual feature. The self-attention processing is used to enhance the feature representation of the image region corresponding to the image modification instruction. The second determining module is used to modify the image to be processed according to the second visual feature and the controllable generation instruction to obtain the target image.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images