Method and apparatus for generating image, computer program product and electronic device

Abstract

The embodiment of the invention discloses a method and device for generating an image, a computer program product and electronic equipment. The method comprises the following steps: based on user input, extracting relation characteristics between entities in the user input; and based on the user input, the relation features and the entity layout associated with the user input, determining fusion features of the target image. The method further includes generating the target image based on the fusion feature of the target image. Through the method, the fusion feature of the target image comprises the relationship feature between the entities in the user input and the entity layout associated with the user input, so that the relationship between the generated target image entities better conforms to the user intention.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image generation, and more particularly, to a method, an apparatus, a computer program product, and an electronic device for generating an image. Background Art

[0002] Controllable image generation refers to the use of computer algorithms and technologies to generate images with specific characteristics and attributes according to the control parameters and requirements set by the user. Controllable image generation technology can be used to generate various types of images, such as artistic creation, design, face generation and virtual scene construction.

[0003] When generating multiple objects, it is necessary to ensure the accuracy of the number, position, and other complex relationships of multiple entities in the generated image. Summary of the invention

[0004] In a first aspect of an embodiment of the present disclosure, a method for generating an image is provided. The method includes extracting, based on user input, relationship features between entities in the user input. Based on the user input, the relationship features, and the entity layout associated with the user input, a fusion feature of a target image is determined. In addition, the method also includes generating a target image based on the fusion feature of the target image.

[0005] In a second aspect of an embodiment of the present disclosure, a device for generating an image is provided. The device includes a relationship feature extraction module configured to extract relationship features between entities in the user input based on the user input. A fusion feature determination module configured to determine a fusion feature of a target image based on the user input, the relationship feature, and an entity layout associated with the user input. In addition, the device also includes an image generation module configured to generate a target image based on the fusion feature of the target image.

[0006] In a third aspect of the disclosed embodiment, a computer program product is provided, comprising a computer program, which implements the method provided according to the first aspect when executed by a processor.

[0007] In a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising one or more processors and a memory associated with the one or more processors, the memory being used to store program instructions, which, when read and executed by the one or more processors, execute the method provided according to the first scheme.

[0008] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flowchart of a method for generating an image according to some embodiments of the present disclosure is shown;

[0012] Figure 3 An example system architecture for generating images according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic training process of a model architecture for generating images according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic reasoning process for generating an image according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An example process of some embodiments of the present disclosure applied to a creative production platform is shown;

[0016] Figure 7 An example block diagram of an apparatus for generating an image according to some embodiments of the present disclosure is shown;

[0017] Figure 8 A schematic block diagram of an electronic device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below in combination with the embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0019] The terms "including" and "having" and any variations thereof in this specification and claims and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. Depending on the context, the word "if" as applied herein can be interpreted as "at..." or "when..." or "in response to determining" or "in response to detecting".

[0020] As mentioned above, when generating multiple objects, controllable image generation technology needs to ensure the accuracy of the number, position, and other complex relationships of multiple entities in the generated image. Vincent image technology usually adds semantic representation vectors as conditions for image generation to the generative adversarial network to perform semantic supervision for sample generation. However, the complex relationships between entities in the text description are not reflected or reflected incorrectly in the generated image.

[0021] To this end, an embodiment of the present disclosure proposes a method for generating an image. In an embodiment of the present disclosure, based on user input, relationship features between entities in the user input are extracted. On this basis, based on the user input, the relationship features, and the entity layout associated with the user input, a fusion feature of a target image is determined. Further, based on the fusion feature of the target image, a target image is generated.

[0022] In this way, the fused features of the target image include the relationship features between entities in the user input and the entity layout associated with the user input, so that the relationship between entities in the generated target image is more consistent with the user's intention.

[0023] Figure 1 1 shows an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, the environment 100 includes a computing unit 110, which may be any device with computing or processing capabilities. For example, the computing unit 110 may include but is not limited to a mobile phone, a tablet computer, a desktop computer, a server, etc. The method for generating an image may be executed in the computing unit 110. Figure 1As shown, the environment 100 includes a user input 111 and an entity layout 112 associated with the user input. The user input 111 may be a user description of a target image such as a voice input, a text input, or a text input converted from a voice input. For example, the voice or text content of the user input 111 may be converted into a text description of the target image "a girl holding a kite". The entity layout 112 associated with the user input may include one or more of entity-related information such as entity position, entity hierarchy, entity relationship, entity attribute, and comprehensive layout description. For example, the entity layout 112 marks the position and area size of "kite" and "girl" in the target image.

[0024] like Figure 1 As shown, the computing unit 110 may extract the relationship features 121 between entities in the user input based on the user input 111. The relationship features 121 between entities may be a series of features of properties such as interaction, position, functional connection, etc. between different entities described in the user input 111, for example, spatial relationship, functional relationship, semantic relationship, temporal relationship, causal relationship, physical relationship, and social relationship between entities. For example, the relationship between "girl" and "kite" in the text description "a girl holding a kite" is "holding".

[0025] After extracting the relationship features 121, the computing unit 110 can obtain the fused features 131 of the target image based on the user input 111, the relationship features 121, and the entity layout 112 associated with the user input. Since the fused features 131 of the target image include the relationship features 121 of the entities extracted based on the user input and the entity layout 112 associated with the user input on the basis of the user input 111, the understanding and representation capabilities of the relationship features 121 and the entity layout 112 of the entities in the target image are enhanced. The computing unit 110 generates the target image 141 based on the fused features 131 of the target image. Since the fused features include the relationship features 121 between the entities in the user input 111 and the entity layout 112 associated with the user input, the relationship between the entities in the target image generated by the computing unit 110 can be strengthened, so that the relationship between the entities in the target image 141 generated by the computing unit 110 is more in line with the user's intention.

[0026] Figure 2 1 shows a flow chart of a method 200 for generating an image according to some embodiments of the present disclosure. The method 200 may be executed by a server, or by a terminal device, or by the server and the terminal device cooperating with each other. Figure 2As shown, in box 202, method 200 can extract the relationship features between entities in the user input based on the user input. In the image generation task, the text description is the main basis for generating the target image. The user can issue an image generation instruction through voice or text input on the terminal device, and the text description of the target image that the user wants to generate can be obtained based on the voice or text input by the user. In order to ensure that the generated image meets the user's expectations as much as possible, the text description needs to be clear, specific and detailed.

[0027] In block 204, method 200 may determine the fusion features of the target image based on the user input, the extracted relationship features, and the entity layout associated with the user input. In the image generation task, the entity layout input by the user can specify the specific position and arrangement of each object or element in the image. By providing the entity layout information, the structure and composition of the target image can be more accurately controlled, so that the generated result is more in line with expectations.

[0028] At block 206, method 200 may generate a target image based on the fused features of the target image. The fused features of the target image, based on the user input, further consider the relationship features of entities extracted based on the user input and the entity layout associated with the user input, thereby enhancing the understanding and representation capabilities of the relationship between entities and the entity layout in the target image.

[0029] In this way, the fused features of the target image include the relationship features between entities in the user input and the entity layout associated with the user input, which can make the relationship between entities in the generated target image more consistent with the user's intention.

[0030] Figure 3 An example system architecture 300 for generating images according to some embodiments of the present disclosure is shown. Figure 3 As shown, in some embodiments of the present disclosure, the architecture 300 includes a scene graph parser 341, a text encoder 342, an image encoder 343, and an image decoder 344. Based on the scene graph parser 341, entities in the user input 311, the relationships between entities, and the overall structure of the scene can be identified, thereby obtaining the relationship features of the entities in the user input 311. The text encoder 342 can capture the semantic information, grammatical structure, and contextual relationship of the text, and output an embedding vector (i.e., text embedding) corresponding to the input text, so as to facilitate processing and understanding by the machine learning model. The relationship features of the user input 311 and the output of the scene graph parser 341 are processed by the text encoder 342 respectively, and then sent to the image encoder 343 together with the entity layout 312 as the generation condition of the image encoder, guiding the image encoder 343 to generate the fusion features of the target image, and then the image decoder 344 generates the target image 351 based on the fusion features.

[0031] In some embodiments of the present disclosure, the architecture 300 may be trained in an end-to-end manner based on a training data set. Figure 4 An example training process 400 of a model architecture for generating images according to some embodiments of the present disclosure is shown. Figure 4 As shown, the training data set includes sample user input 411, sample entity layout 412, and corresponding to sample input 411 and sample entity layout 412. The annotated target image 452 can be an image file in various formats, such as JPEG, PNG, BMP, etc. In addition, the annotated target image 452 can be a single channel (grayscale image) or a multi-channel (color image). In process 400, the architecture composed of scene graph parser 420, text encoder 430, image encoder 440 and image decoder 450 can be trained based on sample user input 411 and sample entity layout 412. By adjusting one or more model parameters of scene graph parser 420, text encoder 430, image encoder 440 and image decoder 450, the difference between the target image 451 generated by the model architecture and the annotated target image 452 is small enough.

[0032] Figure 5 An example reasoning process 500 for generating an image according to some embodiments of the present disclosure is shown. Figure 5 As shown, in some embodiments of the present disclosure, user input 511 can be a method of sampling natural language description. For example, user input 511 can include a simple sentence to describe the image content to be generated, or provide more details including color, action, background, etc., and can also add emotional elements to make the image more expressive. User input 511 can also use a structured description method. For example, user input 511 can divide the description into multiple parts, each part describes a specific object or scene, and can also list various entities and attributes in a list form. User input 511 can also use a multimodal description method. For example, user input 511 can add keywords to emphasize certain specific elements on the basis of natural language description, and can also combine natural language description and layout information to enable the model to better understand the positional relationship of objects. In some embodiments of the present disclosure, user input 511 is processed by text encoder 530-1 to obtain text embedding corresponding to user input 511. Text embedding can convert the text description of the user input into a high-dimensional vector that can capture text semantics and context information, which is convenient for fusion with other generation conditions input to image encoder 540, and guides the image encoder to generate fusion features 547 of the target image.

[0033] In some embodiments of the present disclosure, the user input 511 is also input to the scene graph parser 520 to extract the relational features 521 of the entities in the user input 511. For example, the scene graph parser 520 can integrate the extracted entities and relations into a triple relationship group as a relational feature output. Each triple relationship group contains a subject, a predicate, and an object, for example, based on the input text "a girl holding a kite". (girl, holding, kite). In some embodiments of the present disclosure, the scene graph parser 520 outputs the triple relationship group to the text encoder 530-2 to obtain a text embedding corresponding to the relational feature, which is convenient for fusion with other generation conditions input to the image encoder 540, and guides the image encoder 540 to generate a fusion feature 547 of the target image. In some embodiments of the present disclosure, the text encoder can be a pre-trained language model (for example, BERT, RoBERTa, CLIP, etc.) to extract high-quality text embeddings. The text encoder can also be a word embedding model (for example, Word2Vec, GloVe, etc.) to achieve simple conversion of text embedding vectors. The text encoder can also be a customized neural network model to meet some personalized needs in the text embedding conversion process.

[0034] In some embodiments of the present disclosure, the input method of the entity layout 512 can be a displayed layout annotation or an implicit layout information. For example, a bounding box can be provided for each entity to indicate its position in the image. The bounding box can accurately indicate the position and size of each entity in the image, which is suitable for tasks that require high-precision layout, and the annotation operation of the bounding box is simple and easy to adjust. When the user inputs the layout, he can have an intuitive feeling of the layout effect. A two-dimensional grid or a heat map can also be used to represent the layout of the entity, wherein the two-dimensional grid can better capture the spatial relationship such as the relative position and distance between entities, and the grid structure can be conveniently combined with models such as convolutional neural networks to improve the model's expression ability. The two-dimensional grid method is also easy to expand to different resolutions and sizes, and the heat map can smoothly represent the entity as a position, which is suitable for processing continuous spatial changes. The location information of the entity can also be included in the text description, and the text description and layout information can also be input at the same time. The text description can contain rich contextual information, such as the attributes, actions and relationships of the entity, and has better generalization ability. The entity layout 512 is input to the image encoder 540 and fused with other generation conditions of the image encoder 540 to guide the image encoder 540 to generate fusion features 547 of the target image.

[0035] In some embodiments of the present disclosure, the image encoder 540 generates a fusion feature 547 of the target image based on the text embedding corresponding to the user input 511, the text embedding corresponding to the relationship feature 521, and the entity layout 512 by means of addition fusion, multiplication fusion, concatenation fusion, attention fusion, and any combination thereof, and can combine features of different sources and scales to form a more accurate and comprehensive representation of the target image. In some embodiments of the present disclosure, the image encoder 540 determines a layout mask 546 of the target image based on the entity layout 512 associated with the user input, so as to facilitate fusion with other generation conditions input to the image encoder 540, and guide the image encoder 540 to generate the fusion feature of the target image. The layout mask 546 is generally a two-dimensional matrix, in which each element represents the state of a certain position in the image, such as whether there is an object, the shape, size and direction of the object, etc. The layout mask can have multiple forms, for example, it can be a two-dimensional matrix, each element of which represents whether the position contains a specific object. For example, the position information of each entity in the entity layout 512 can be expressed as (x1, y1, x2, y2), where (x1, y1) and (x2, y2) respectively represent the coordinates of the upper left corner and the lower right corner of the rectangular area where the entity is located. According to the position information of each entity, the area of ​​each entity is marked in the blank two-dimensional mask, and the layout mask 546 of the target image can be obtained.

[0036] In some embodiments of the present disclosure, the image encoder 540 processes the latent space image information matrix based on the text embedding output from the text encoders 530-1 and 530-2, the layout mask 546, and the randomly generated initialization image information matrix (i.e., the noise map 543 of the latent space), and outputs the final processed latent space image information matrix (i.e., the fusion feature 547). Specifically, the process of the image encoder 540 outputting the fusion feature 547 based on the generation condition is not completed in one step, but is an iterative process. Each iteration of the image encoder 540 removes part of the noise in the noise map 543, and adds more information related to the target image based on the generation condition to generate a new noise map. In some embodiments of the present disclosure, the image encoder 540 includes a noise predictor, in which there are a series of modules for converting latent space data (i.e., residual network, ResNet, Residual Network), each module and the previous module form a series of input and output, and the data is processed in the module. A part of the output of each module is sent directly to the final stage of the processing process in the form of a residual network. The noise predictor adds an attention mechanism module after each residual network module as a text adjustment mechanism, and integrates the two text embeddings of the input image encoder 540 and the layout mask 546 obtained based on the entity layout 512 into each processing stage of the image encoder 540. In each processing stage, the attention mechanism module merges these text embeddings and layout masks 546 into the data of the latent space and sends them to the next residual network module, so that the data module processed by the next residual network module can include more relevant information. Finally, the noise predictor will gradually remove the noise according to the generation conditions to obtain the fusion feature 547. In some embodiments of the present disclosure, the attention mechanism module can generate a cross attention matrix 444 through a cross attention mechanism 541 based on the text embedding and noise map 543 output by the text encoder 530-1. The attention preparation module can also generate a relationship attention matrix 545 through a relationship attention mechanism 542 based on the text embedding output by the text encoder 530-2. Among them, the cross attention mechanism 541 can establish the relationship between the text embedding for the user input and the noise map 543, and calculate the attention weight of each element in the text embedding to each element in the noise map. The relational attention mechanism 542 can model the relationship between entities in the text embedding for relational features, and calculate the attention weight of each element in the text embedding to other elements. On this basis, the image encoder 540 adds the cross attention matrix 544 and the relational attention matrix 545, and multiplies the added matrix with the layout mask 546. The result of the multiplication is merged into the data in the latent space and sent to the next residual network module.In this way, the fused features finally output by the image encoder will include the relationship features between entities in the user input and the entity layout associated with the user input.

[0037] In some embodiments of the present disclosure, the image decoder 550 can decode the operation result of the latent space into the actual picture dimension based on the fusion feature 547, so as to draw the target image 551. In this way, the relationship between the target image entities generated by the image decoder can be more consistent with the user's intention.

[0038] In some embodiments of the present disclosure, the method for generating images can be applied to a creative production platform that supports controllable material generation, so as to facilitate operations and merchants to produce and deliver materials. For example, operations and merchant users can input descriptions and layouts of control material generation in the creative production platform based on the characteristics of products or services, and the creative production platform can extract the relationship features of entities in the user input. Based on the relationship features, user input, and the entity layout associated with the user input, image materials that meet user needs are generated and delivered. Figure 6 The example process of applying some embodiments of the present disclosure to a creative production platform is shown. Figure 6 As shown, in some embodiments of the present disclosure, after the target image 651 is generated, the target image 651 is also saved to the material library 660. The material library can store various digital assets (for example, images, videos, audio texts, etc.) and perform centralized management (for example, classifying and tagging the materials in the material library). In some embodiments of the present disclosure, artificial intelligence technology can be used to analyze the materials, and suitable materials can be recommended according to the delivery goals and audience characteristics. The materials can also be distributed to different advertising platforms and channels to achieve multi-channel delivery. In some embodiments of the present disclosure, the user can initiate a material production request for the delivery page to the platform through the human-computer interaction interface. The delivery page generally refers to a specific web page in digital marketing or online marketing where advertising materials (such as pictures, texts, videos, etc.) are displayed to the target audience. The delivery page can be a specific page on a website, a post on a social media platform, the content of an email in email marketing, or any other place on an online platform for displaying advertising content. As Figure 6As shown, after receiving the user's request, the human-computer interaction interface 600 of the creative production platform first obtains the delivery page information requested by the user, wherein the delivery page information includes the resolution and editing area of ​​the delivery page. According to the delivery page information, the delivery page 620 of the material is displayed on the human-computer interaction interface, and the delivery page 620 generally includes a non-editable area 621 corresponding to the basic functions of the page and an editing area 622 that allows the user to display the advertising content. The user inputs the description 611 of the generated target image (i.e., the material) through the dialog box 610 of the human-computer interaction interface 600, and performs entity layout in the editing area 622 of the delivery page. On this basis, the method of some embodiments of the present disclosure generates a target image 651 based on the entity layout, user input, and the relationship features extracted from the user input. The generated target image 651 can be stored in the material library 660, and then delivered to the terminal devices 671, 672, 673 of the target audience group according to the predetermined operation strategy or delivery rules, thereby improving the quality and delivery efficiency of the material production of the operation and the merchant.

[0039] Figure 7 700 for generating an image according to some embodiments of the present disclosure is shown. The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Figure 7 As shown, the device 700 includes a relationship feature extraction module 701, which is configured to extract the relationship features between entities in the user input based on the user input. The device 700 also includes a fusion feature determination module 702, which is configured to determine the fusion features of the target image based on the user input, the relationship features and the entity layout associated with the user input. In addition, the device 700 also includes an image generation module 703, which is configured to generate a target image based on the fusion features of the target image.

[0040] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of this specification is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0041] Figure 8 8 is a block diagram of an electronic device 800 that can implement various embodiments of the present disclosure. Figure 8 As shown, the electronic device 800 includes a processor 810, a disk drive 820, an input / output interface 830, a network interface 840, and a memory 850. The processor 810, the disk drive 820, the input / output interface 830, the network interface 840, and the memory 850 can be communicatively connected via a communication bus 860.

[0042] Among them, the processor 810 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solution provided in this application.

[0043] The memory 850 can be implemented in the form of ROM (Read Only Memory), RAM (Read Access Memory), static memory, dynamic storage device, etc. The memory 850 can store an operating system 851 for controlling the operation of the electronic device 800 and a basic input and output system (BIOS) 852 for controlling the low-level operation of the electronic device 800. In addition, a web browser 853, a data storage management system 854, etc. can also be stored. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 850 and is called and executed by the processor 810.

[0044] The input / output interface 830 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, a prompt light, etc.

[0045] The network interface 840 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0046] The bus 860 comprises a pathway that transmits information between the various components of the device (eg, the processor 810, disk drive 820, input / output interface 830, network interface 840, and memory 850).

[0047] It should be noted that, although the above device only shows a processor 810, a disk drive 820, an input / output interface 830, a network interface 840, a memory 850, a bus 860, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may only include components necessary for implementing the method of the present application, and does not necessarily include all the components shown in the figure.

[0048] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0049] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0050] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for generating an image, comprising: Based on the user input, extracting relationship features between entities in the user input; Determining a fusion feature of a target image based on the user input, the relationship feature, and an entity layout associated with the user input; as well as Based on the fused features of the target image, the target image is generated.

2. The method according to claim 1, wherein determining the fusion feature of the target image based on the user input, the relationship feature, and the entity layout associated with the user input comprises: determining a layout mask for the target image based on the entity layout associated with the user input; as well as Based on the user input, the relationship feature and the layout mask, a fusion feature of the target image is determined.

3. The method according to claim 2, wherein determining the fusion feature of the target image based on the user input, the relationship feature and the layout mask comprises: Based on the user input, determining a first text embedding corresponding to the user input; Based on the relationship feature, determining a second text embedding corresponding to the relationship feature; as well as Based on the first text embedding, the second text embedding, and the layout mask, a fusion feature of the target image is determined.

4. The method of claim 3, wherein determining the fusion features of the target image based on the first text embedding, the second text embedding and the layout mask comprises: Determining a generation condition of the target image by fusing the first text embedding, the second text embedding, and the layout mask; as well as Based on the generation condition of the target image, a fusion feature of the target image is determined.

5. The method according to claim 4, wherein determining the generation condition of the target image by fusing the first text embedding, the second text embedding and the layout mask comprises: Determining a target text embedding of the target image based on the first text embedding and the second text embedding; as well as Based on the target text embedding and the layout mask, a generation condition of the target image is determined.

6. The method of claim 5, wherein determining a target text embedding of the target image based on the first text embedding and the second text embedding comprises: Based on the first text embedding, determining a first attention matrix corresponding to the first text embedding through an attention mechanism; Based on the second text embedding, determining a second attention matrix corresponding to the second text embedding through an attention mechanism; as well as Based on the first attention matrix and the second attention matrix, a target text embedding corresponding to the target image is determined.

7. The method of claim 3, wherein based on the user input, determining a first text embedding corresponding to the user input comprises: Based on the user input, a first text embedding corresponding to the user input is generated by a text encoder.

8. The method according to claim 7, wherein based on the user input, extracting the relationship features between entities in the user input comprises: Based on the user input, generating a ternary relationship group for the user input through a scene graph parser; And based on the relationship feature, determining a second text embedding corresponding to the relationship feature comprises: Based on the triple relationship group, a second text embedding corresponding to the triple relationship group is generated by the text encoder.

9. The method according to claim 8, wherein determining the fusion features of the target image based on the generation conditions of the target image comprises: Based on the generation condition of the target image, determining the fusion feature of the target image through an image encoder; And based on the fusion feature of the target image, generating the target image includes: Based on the fusion features of the target image, the target image is generated through an image decoder.

10. The method according to claim 1, further comprising: In response to generating the target image, saving the target image to a material library; as well as Based on the material library, the target image is delivered to a delivery page of the target image.

11. The method of claim 10, wherein determining the fusion feature of the target image based on the user input, the relationship feature, and the entity layout associated with the user input comprises: Acquire delivery page information of the target image, wherein the delivery page information includes resolution and editing area; Obtaining a physical layout associated with the user input in an editing area of ​​the delivery page, and determining a layout mask of the physical layout, wherein a resolution of the layout mask matches a resolution of the page; and Based on the user input, the relationship feature and the layout mask, a fusion feature of the target image is determined.

12. The method of claim 9, wherein the training of the text encoder, the scene graph parser, the image encoder, and the image decoder comprises: Based on the sample user input and the sample entity layout, a target image is generated through a text encoder, a scene graph parser, an image encoder, and an image decoder; The sample user input and the sample entity layout have corresponding annotated target images; as well as The text encoder, the scene graph parser, the image encoder and the image decoder are trained based on the generated target image and the annotated target image.

13. A device for generating an image, comprising: A relationship feature extraction module, configured to extract relationship features between entities in the user input based on the user input; a fusion feature determination module, configured to determine a fusion feature of a target image based on the user input, the relationship feature, and an entity layout associated with the user input; as well as The image generation module is configured to generate the target image based on the fusion features of the target image.

14. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 12.

15. Electronic equipment, including: one or more processors, and A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method according to any one of claims 1 to 12 are executed.

Images