Image generation method and electronic equipment

Through the feature extraction and cross-attention mechanism of multiple reference images, the problem of difficulty in generating high-quality AI images for small and medium-sized merchants is solved, the user-friendliness and effect of image generation is improved, and the operation process is simplified.

CN118941655APending Publication Date: 2024-11-12阿里巴巴(上海)有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410765078.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-13
Publication Date
2024-11-12

Smart Images

  • Figure CN118941655A_ABST
    Figure CN118941655A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an image generation method and electronic equipment. The method comprises the following steps: receiving a request of generating image content initiated by a user; obtaining a plurality of reference images specified by the user, and determining control dimensions respectively corresponding to the plurality of reference images; processing the request by using an image generation model to generate image content; wherein in the process of generating the image content, feature extraction is performed according to the plurality of reference images, and after control conditions are generated on the corresponding control dimensions by using a control model, the extracted features and the generated control conditions are injected into a generative network of the image generation model based on a cross attention mechanism, so that the image content is generated. Therefore, the plurality of reference images play a role in the corresponding control dimensions respectively, and control the generation of the image content together. According to the embodiment of the invention, the image conforming to the expectation of the user can be produced without depending on the background knowledge of the user on the writing method of the prompt word.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to an image generation method and an electronic device. Background Art

[0002] For merchants in the product information service system, they often need to provide various product images, including the main product image, product images in the graphic details, advertising images when participating in marketing activities, etc. High-quality product images have a positive effect on improving the click-through rate and conversion rate of products. Some large merchants usually hire a dedicated team to be responsible for the production of various product images, but for some small and medium-sized merchants, they are usually unable to bear this part of the cost, and it is difficult to gain a competitive advantage in the quality of product images.

[0003] The emergence of large models such as "text-generated images" (content based on text-generated images) has completely ignited the trend of image AIGC (Artificial Intelligence Generated Content). With the help of AI image generation methods and tools, many products and gameplay have appeared on the market, such as AI painting, AI design, AI commercial photography (model pictures, product pictures), etc. If you want to generate more beautiful and practical pictures through these methods, users often need to have some corresponding background knowledge, such as how to write prompt words, how to control the content of the picture, etc. However, for merchants and other users on e-commerce platforms, due to the lack of such professional background knowledge, pictures generated by AI can only remain at the gameplay level and are difficult to be truly used in production. Summary of the invention

[0004] The present application provides an image generation method and an electronic device, which can produce an image that meets the user's expectations without relying on the user's background knowledge of how to write prompt words.

[0005] This application provides the following solutions:

[0006] A method for generating an image, comprising:

[0007] receiving a request initiated by a user to generate image content;

[0008] Obtaining a plurality of reference images specified by the user, and determining control dimensions corresponding to the plurality of reference images respectively;

[0009] The request is processed using an image generation model to generate image content; wherein, in the process of generating image content, feature extraction is performed according to the multiple reference images respectively, and control conditions are generated on corresponding control dimensions respectively using the control model, and then the extracted features and generated control conditions are injected into the generation network of the image generation model based on a cross-attention mechanism, so that the multiple reference images play a role on the corresponding control dimensions respectively and jointly control the generation of image content.

[0010] Among them, the control dimensions corresponding to the multiple reference images respectively include: layout dimension, style dimension, content dimension or lighting dimension.

[0011] Wherein, for the reference graph of the content dimension, during the feature extraction process, a semantic description text content of the reference graph is generated by a multimodal model and provided to the user for confirmation;

[0012] After receiving the user's confirmation message, text feature extraction is performed based on the semantic description text content for injection into the generation network.

[0013] The features required to be injected into the generative network of the image generation model include text features and image features, wherein the image features include image features generated by performing image visual encoding on the reference image; the text features include text features generated by performing text encoding on the text information after extracting the text information from the reference image, and / or text features generated by performing text encoding on the prompt text information additionally input by the user;

[0014] Before being injected into the generative network of the image generation model, it also includes:

[0015] Based on the cross-attention mechanism of the image generation model, the text features and the image features are processed so as to be injected into the generation network of the image generation model.

[0016] Among them, the cross-attention mechanism is a decoupled cross-attention mechanism, and the decoupled cross-attention mechanism is: on the basis of each original cross-attention layer of the image generation model, a cross-attention layer for injecting image features into the image generation model is added, wherein the original cross-attention layer is used to inject text features into the image generation model, so that the cross-attention layers of text features and image features are independent of each other.

[0017] Among them, reference images of different control dimensions correspond to weight information, and the weight information is adjustable to control the degree of influence of reference images on different control dimensions on the image content generation result.

[0018] Among them, it also includes:

[0019] An operation option for adjusting the weights of the reference graphs on the different control dimensions is provided to the user, so that the user can adjust the weights.

[0020] Among them, it also includes:

[0021] An operation option for editing the reference image is provided so as to control the generation of image content based on the edited reference image, wherein the editing includes: erasing or repositioning some elements in the reference image, or adding some elements based on the reference image.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described above.

[0023] An electronic device, comprising:

[0024] one or more processors; and

[0025] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of any of the methods described above.

[0026] A computer program product comprises a computer program / computer executable instructions, wherein the computer program / computer executable instructions implement the steps of any of the aforementioned methods when executed by a processor in an electronic device.

[0027] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0028] Through the embodiments of the present application, when a user needs to generate an image, multiple reference images and their corresponding control dimensions can be specified. Specifically, when the image is generated through the image generation model, feature extraction can be performed based on the multiple reference images respectively, and after the control conditions are generated on the corresponding control dimensions using the control model respectively, the extracted features and the generated control conditions are injected into the generation network of the image generation model based on the cross-attention mechanism of the image generation model, so that the multiple reference images can play a role on the corresponding control dimensions respectively, and jointly control the generation of the image content. In this way, image generation can be performed completely based on reference images without relying on high-quality prompt text input by the user, and multiple reference images can be supported to play a role on different control dimensions respectively, and jointly control the generation of the image. At the same time, it can be clarified which part of the information in each reference image plays a role, so that the generated image achieves the effect expected by the user.

[0029] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0032] Figure 2 is a flow chart of the method provided in the embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a device provided in an embodiment of the present application;

[0034] Figure 4 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0036] First of all, it should be noted that among some image generation products on the market, if you want to achieve a highly controllable AI image generation function, most of them need to rely on high-quality prompt words. As mentioned in the background technology section, high-quality prompt words require too high a level of user input, and only users who understand the AI ​​image generation function can express the content of the picture well through text. In addition, some products provide the function of uploading a reference image while entering the prompt word, but the specific role of this reference image is unclear, and the actual generation effect is difficult to meet the needs of users.

[0037] In view of the above problems, the embodiment of the present application provides a solution for generating image content by completely relying on reference images, that is, the user is no longer required to input high-quality prompt words, but only needs to input reference images to complete the generation of image content. Of course, since an image can have features in different dimensions such as layout, style, content, and lighting, when uploading a reference image, the user may only need the features of the reference image in a certain dimension to play a role, and the features in other dimensions may need to refer to other reference images. Therefore, in an embodiment of the present application, users can be supported to upload multiple reference images, and the corresponding control dimensions can be specified. For example, when specifically implemented, the user can be prompted to upload multiple reference images of different control dimensions respectively, so as to guide the user to create image content through multiple different reference images. After the user has uploaded multiple reference images for different control dimensions, when generating image content, these reference images can be made to play a role in the control dimensions corresponding to each other, and the generation of image content can be controlled together.

[0038] In order to achieve the above purpose, it may involve feature extraction of each reference image, and the extracted features may be injected into the generation network of the image generation model. In the embodiment of the present application, the specific extracted features may include image features and text features, and support the injection of features of two different vector spaces into the generation network. This will be introduced in detail later.

[0039] In addition, in order to enable each reference image to play a role in its corresponding control dimension, the control model (ControlNet) of the image generation model can also be used to generate control conditions on the corresponding control dimension based on each reference image. These control conditions can also be used to input into the generation network to control the generation of image content. There can be multiple ControlNets, which can be pre-trained independently and used to generate control conditions on different control dimensions.

[0040] From the perspective of system architecture, see Figure 1, the embodiment of the present application can provide a service for generating image content, which can be embedded in a comprehensive application such as an application of a commodity information service system, or can also provide an independent application or website, so that users can generate images based on the service. Specifically, the service may include a client and a server. The client is mainly used to interact with the user, so that the user can initiate a request to generate an image, and can guide the user to upload or select reference images on multiple control dimensions. In addition, in an optional manner, a function for editing the reference image can also be provided, including erasing some content in the reference image, or changing the position of some content in the reference image, drawing some image content, and so on. After determining the specific reference images of multiple control dimensions, they can be submitted to the server, and the server controls the generation of image content by calling the image generation model, extracting image features, generating control conditions, and injecting them into the generation network. Finally, the image generation result can be returned to the user. It should be noted here that in the embodiment of the present application, the degree of influence of the reference image on each control dimension on the generated image can be adjusted. In the specific implementation, a plurality of image generation results corresponding to different degrees of influence can be generated for the user to choose. Alternatively, the degree of influence may be abstracted into weights, etc., and opened to users for configuration, so that users can select the influence weight of the reference image on a specific control dimension according to their needs, so as to further improve the degree of freedom of image generation, etc.

[0041] The specific implementation scheme provided in the embodiments of the present application is described in detail below.

[0042] First, the embodiment of the present application provides an image generation method from the perspective of the server, see Figure 2 , the method may include:

[0043] S201: receiving a request initiated by a user to generate image content.

[0044] An option for initiating an image generation request may be provided on the client page, and the user may initiate an image generation request by clicking a specific option. Correspondingly, the client may forward the request to the server.

[0045] S202: Obtain multiple reference images specified by the user, and determine control dimensions corresponding to the multiple reference images respectively.

[0046] In an embodiment of the present application, after receiving a request for generating an image initiated by a user, the user can be guided to specify a plurality of reference images under different control dimensions. Among them, the user can specify the reference image by uploading or selecting from candidate images. In specific implementation, a plurality of different control dimensions and corresponding operation options for uploading or selecting reference images can be displayed on the client page, so that the user can upload or select reference images for each control dimension respectively. For example, the control dimensions corresponding to the plurality of reference images may include: layout dimension, style dimension, content dimension or lighting dimension. In addition, if the image to be generated includes content related to the human / animal image, and the human / animal image is required to have a certain posture, a posture reference image can also be provided, and so on. Of course, the user can also specify the reference image only on some control dimensions, for example, only the reference images of the layout dimension and the style dimension are specified, and the content dimension and the lighting dimension may not be specified, and so on.

[0047] In the specific implementation, in addition to uploading or selecting reference images, users can also be provided with the function of editing reference images. These editing functions can specifically include erasing some elements in the reference image, or adjusting the positions of some elements, or adding some elements by drawing some lines, etc. For example, in the scenario of a commodity information service system, a merchant obtains a picture from a certain channel and wants to refer to the style of this picture to generate a commodity picture for his own commodity; but this picture itself also includes the image of another commodity. At this time, before uploading the picture to the server, the above-mentioned editing function can be used to edit the picture, including erasing the image content of the original commodity, and then uploading it to the server as a reference picture in the style dimension, etc. Alternatively, an automatic erasing function can also be provided, that is, the erasing operation can be automatically performed without the user actively initiating a request. Alternatively, after erasing an element in the original reference image, it may also be necessary to add a replacement element to the reference image, for example, replace it with a commodity element in your own store, etc. Specifically, in the process of erasing or moving the position, the AI ​​large model can also be used to complete the image according to the image content of other parts of the image, etc.

[0048] S203: Processing the request using an image generation model to generate image content; wherein, in the process of generating the image content, feature extraction is performed based on the multiple reference images respectively, and control conditions are generated on corresponding control dimensions using the control model respectively, and then the extracted features and the generated control conditions are injected into the generation network of the image generation model based on a cross-attention mechanism, so that the multiple reference images play a role on the corresponding control dimensions respectively and jointly control the generation of the image content.

[0049] After determining the reference images on multiple different control dimensions, feature extraction can be performed according to the multiple reference images, and control conditions can be generated on the corresponding control dimensions. Among them, regarding the feature extraction part, the specific feature extraction can use a multimodal coding model (such as CLIP, which includes two independent encoders: a visual encoder for processing images (such as Vision Transformer, ViT), and another language encoder for processing text (such as BERT)) to directly extract image features from the reference image, or the image can be converted into text, and then the text features can be extracted from the text content. For example, for the reference image on the content dimension, since it is mainly necessary to refer to the content in the reference image, and "content" usually has a more concrete meaning, such as a certain type of item, etc.; therefore, when extracting features from such a reference image, the semantic description text content of the reference image can be generated by the multimodal model, and can even be provided to the user for confirmation. The user can also edit the semantic description text content, including adding or deleting a description, or modifying a description, etc. After confirmation, the multimodal model generates text features based on the semantic description text content, etc. In addition, in an optional manner, the embodiment of the present application also supports the user to output prompt text. In this case, text feature extraction can also be performed from the prompt text input by the user, and so on.

[0050] It should be noted here that, in the embodiment of the present application, the image generation model of the "text image" class can be used as the basic image generation model, and the traditional text-to-image generation model basically generates images based on text features. For example, the popular SD (Stable Diffusion) model is based on the text features extracted from the frozen CLIP text encoder. In the embodiment of the present application, since the input data also includes image features extracted from the reference image, therefore, in the specific implementation, an additional network can be inserted into the text-to-image diffusion model to guide image generation, and the network can be trained to map the image features extracted from the CLIP image encoder to new features, and then spliced ​​with the text features. The spliced ​​features can be used to replace the original text features and input into the generation network (e.g., UNet) of the diffusion model to guide image generation.

[0051] The above implementation process can be achieved through the cross-attention mechanism of the text-to-image diffusion model. However, the key and value projection weights of the cross-attention layer in the original diffusion model are trained to adapt to text features. Therefore, if the image features and text features are directly merged into the cross-attention layer, only the alignment of the image features and the text features in the vector space can be achieved, but this may miss some information unique to the image features, and ultimately lead to only coarse-grained controllable generation with the reference image.

[0052] Therefore, in an optional implementation, a decoupled cross-attention mechanism for text features and image features can also be used. That is, for each cross-attention layer acting on the diffusion model UNet, an additional cross-attention layer is added for the image features to inject the image features. This makes the cross-attention layers of text features and image features independent of each other. During the training phase, only the parameters of the new cross-attention layer can be trained, while the original UNet model can remain unchanged. In this way, image prompts can be made compatible with text prompts, thereby achieving multimodal image generation.

[0053] In addition, when the control conditions are generated based on each reference image, ControlNet can be used to achieve this. ControlNet is a method for controlling pre-trained large models with additional inputs for AI painting. It controls these models through additional inputs so that they can better meet some specific needs. During the painting process, ControlNet divides the network structure into two parts: trainable and locked. Among them, the trainable part learns the controllable part. The locked part retains the original data of basic models such as Stable-Diffusion. Therefore, using a small amount of data to guide can ensure that the pre-constraints can be fully learned while retaining the learning ability of the original diffusion model itself. The use of ControlNet can improve the quality and satisfaction of AI works. For example, when generating human postures, the human joint point features in the posture picture can be read through OpenPose. After inputting into ControlNet, it can be unaffected by the color and lines of the picture itself, and independent and interference-free precise control can be achieved.

[0054] Based on the above situation, in an embodiment of the present application, ControlNets on different control dimensions can be provided respectively. That is to say, assuming that there are four control dimensions in total, four different ControlNets can be pre-trained, which are used to generate control conditions for reference images under different control dimensions. After receiving the reference image specified by the user, the ControlNets corresponding to the specific control dimensions can be input respectively to generate corresponding control conditions. For example, for the reference image of the layout dimension, ControlNet can mainly generate control conditions based on the line drawing, depth map, etc. of the reference image, and so on.

[0055] Different ControlNets generate control conditions from different control dimensions and from different reference images, and then they can be injected into the generative network of the image generation model to work together and control the generation of images from different control dimensions.

[0056] It should be noted that, in the specific implementation, in the specific control process, the reference images of different control dimensions may correspond to weight information, and this weight information is adjustable to control the degree of influence of the reference images on different control dimensions on the image content generation results. In this way, for the same set of reference images, when each corresponds to a different weight, since each has a different degree of influence in the image generation process, multiple different image generation results can be obtained. In actual applications, the generated multiple different image results can be returned to the user for selection.

[0057] Alternatively, the client may provide the user with an operation option for adjusting the weights of the reference images on the different control dimensions so that the user can adjust the weights. That is to say, when the user has a certain amount of knowledge, or has a clearer idea or need for the image he needs, he can also manually specify the weights of each reference image, so as to control the degree of influence of each different reference image on the image generation. For example, a user uploads a layout reference image, a style reference image, and a content reference image, where a similar layout and style may be desired, but a lower degree of control may be exercised over the content. In this case, the weight of the content reference image may be appropriately reduced, and so on.

[0058] In summary, through the embodiments of the present application, when a user needs to generate an image, multiple reference images and their corresponding control dimensions can be specified. Specifically, when the image is generated through the image generation model, feature extraction can be performed based on the multiple reference images respectively, and after the control conditions are generated on the corresponding control dimensions using the control model respectively, the extracted features and the generated control conditions are injected into the generation network of the image generation model based on the cross-attention mechanism of the image generation model, so that the multiple reference images can play a role on the corresponding control dimensions respectively, and jointly control the generation of the image content. In this way, image generation can be performed completely based on reference images without relying on high-quality prompt text input by the user, and multiple reference images are supported to play a role on different control dimensions respectively, and jointly control the generation of the image. At the same time, it can be clarified which part of the information in each reference image plays a role, so that the generated image achieves the effect expected by the user.

[0059] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0060] Corresponding to the first embodiment, the present application embodiment also provides an image generating device, see Figure 3 , the device may include:

[0061] The request receiving unit 301 is used to receive a request initiated by a user to generate image content;

[0062] A control dimension determination unit 302, configured to obtain a plurality of reference images specified by the user, and determine control dimensions corresponding to the plurality of reference images respectively;

[0063] The image generation processing unit 303 is used to process the request using the image generation model to generate image content; wherein, in the process of generating image content, feature extraction is performed according to the multiple reference images respectively, and control conditions are generated on the corresponding control dimensions respectively using the control model, and then the extracted features and the generated control conditions are injected into the generation network of the image generation model based on the cross-attention mechanism, so that the multiple reference images play a role on the corresponding control dimensions respectively and jointly control the generation of image content.

[0064] Among them, the control dimensions corresponding to the multiple reference images respectively include: layout dimension, style dimension, content dimension or lighting dimension.

[0065] Specifically, for a reference graph of the content dimension, during the feature extraction process, a semantic description text content of the reference graph is generated by a multimodal model and provided to the user for confirmation;

[0066] At this time, after receiving the user's confirmation message, text feature extraction can be performed based on the semantic description text content to be injected into the generation network.

[0067] The features required to be injected into the generative network of the image generation model include text features and image features, wherein the image features include image features generated by performing image visual encoding on the reference image; the text features include text features generated by performing text encoding on the text information after extracting the text information from the reference image, and / or text features generated by performing text encoding on the prompt text information additionally input by the user;

[0068] The device may also include:

[0069] A feature processing unit is used to process the text features and image features based on the cross-attention mechanism of the image generation model before injecting them into the generation network of the image generation model, so as to inject them into the generation network of the image generation model.

[0070] Among them, the cross-attention mechanism is a decoupled cross-attention mechanism, and the decoupled cross-attention mechanism is: on the basis of each original cross-attention layer of the image generation model, a cross-attention layer for injecting image features into the image generation model is added, wherein the original cross-attention layer is used to inject text features into the image generation model, so that the cross-attention layers of text features and image features are independent of each other.

[0071] In addition, reference images of different control dimensions may correspond to weight information, and the weight information is adjustable to control the degree of influence of reference images of different control dimensions on the image content generation result.

[0072] Furthermore, the device may further include:

[0073] The weight adjustment unit is used to provide the user with an operation option for adjusting the weight of the reference graph on the different control dimensions, so that the user can adjust the weight.

[0074] An editing unit is used to provide operation options for editing the reference image so as to control the generation of image content based on the edited reference image, wherein the editing includes: erasing or changing the position of some elements in the reference image, or adding some elements based on the reference image.

[0075] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0076] And an electronic device, comprising:

[0077] one or more processors; and

[0078] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0079] A computer program product includes a computer program / computer executable instructions, which, when executed by a processor in an electronic device, implement the steps of the method described in the aforementioned method embodiment.

[0080] in, Figure 4The architecture of the electronic device is shown as an example, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a communication bus 430.

[0081] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit, processor), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0082] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the operation of the electronic device 400, and a basic input and output system (BIOS) for controlling the low-level operation of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and an image generation processing system 425, etc. can also be stored. The above-mentioned image generation processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0083] The input / output interface 413 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0084] The network interface 414 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0085] The bus 430 comprises a pathway for transmitting information between the various components of the device (eg, the processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420).

[0086] It should be noted that, although the above device only shows a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, a memory 420, a bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0087] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0088] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0089] The image generation method and electronic device provided by the present application are described in detail above. The principles and implementation methods of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An image generation method, characterized in that: include: receiving a request initiated by a user to generate image content; Obtaining a plurality of reference images specified by the user, and determining control dimensions corresponding to the plurality of reference images respectively; The request is processed using an image generation model to generate image content; wherein, in the process of generating image content, feature extraction is performed according to the multiple reference images respectively, and control conditions are generated on corresponding control dimensions respectively using the control model, and then the extracted features and generated control conditions are injected into the generation network of the image generation model based on a cross-attention mechanism, so that the multiple reference images play a role on the corresponding control dimensions respectively and jointly control the generation of image content.

2. The method according to claim 1, characterized in that The control dimensions corresponding to the multiple reference images include: layout dimension, style dimension, content dimension or lighting dimension.

3. The method according to claim 2, characterized in that For the reference graph of the content dimension, during the feature extraction process, a semantic description text content of the reference graph is generated through a multimodal model and provided to the user for confirmation; After receiving the user's confirmation message, text feature extraction is performed based on the semantic description text content for injection into the generation network.

4. The method according to claim 1, characterized in that: The features required to be injected into the generative network of the image generation model include text features and image features, wherein the image features include image features generated by performing image visual encoding on the reference image; the text features include text features generated by performing text encoding on the text information after extracting the text information from the reference image, and / or text features generated by performing text encoding on the prompt text information additionally input by the user; Before being injected into the generative network of the image generation model, it also includes: Based on the cross-attention mechanism of the image generation model, the text features and the image features are processed so as to be injected into the generation network of the image generation model.

5. The method according to claim 4, characterized in that The cross-attention mechanism is a decoupled cross-attention mechanism, which is: on the basis of each original cross-attention layer of the image generation model, a cross-attention layer for injecting image features into the image generation model is added, wherein the original cross-attention layer is used to inject text features into the image generation model, so that the cross-attention layers of text features and image features are independent of each other.

6. The method according to claim 1, characterized in that Reference images of different control dimensions correspond to weight information, and the weight information is adjustable to control the degree of influence of reference images of different control dimensions on the image content generation result.

7. The method according to claim 6, characterized in that: Also includes: An operation option for adjusting the weights of the reference graphs on the different control dimensions is provided to the user, so that the user can adjust the weights.

8. The method according to claim 1, characterized in that Also includes: An operation option for editing the reference image is provided so as to control the generation of image content based on the edited reference image, wherein the editing includes: erasing or repositioning some elements in the reference image, or adding some elements based on the reference image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 8.

11. A computer program product comprising a computer program / computer executable instructions, characterized in that: When the computer program / computer executable instructions are executed by a processor in an electronic device, the steps of the method according to any one of claims 1 to 8 are implemented.