Image generation method and device, computer device and storage medium

Abstract

The application discloses an image generation method and device, computer equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring object information of a plurality of objects and occlusion relationships between the plurality of objects; based on the object information of the plurality of objects, extracting first object features of the plurality of objects respectively; based on the occlusion relationships, determining upper-layer objects of the plurality of objects respectively, and based on the first object features of the upper-layer objects of the plurality of objects respectively, determining visible features of the plurality of objects respectively; according to the visible features of the plurality of objects respectively, adjusting the first object features of the plurality of objects respectively to obtain second object features of the plurality of objects respectively; and based on the second object features of the plurality of objects respectively, generating object images, wherein the object images comprise the plurality of objects with the occlusion relationships. The application can accurately control the occlusion relationships between the plurality of objects in the image generation process, improve the controllability of the occlusion relationships, and enhance the visual effect of the generated images.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of computer technology, the ways to process images are becoming increasingly diverse. Currently, images can be intelligently generated based on artificial intelligence technology. For example, by specifying an object, an image including the specified object can be intelligently generated.

[0003] In related technologies, when multiple objects are specified, the layout of multiple objects in the generated image can be controlled by providing the bounding boxes of each object. However, this method can only control the position of each object in the image and cannot control the occlusion relationship between objects, which has certain limitations and results in poor visual effects of the generated image. Summary of the Invention

[0004] This application provides an image generation method, apparatus, computer device, and storage medium, which can precisely control the occlusion relationships between multiple objects in an image during the image generation process, improving the controllability of occlusion relationships and enhancing the visual effect of the generated image. The technical solution is as follows:

[0005] On the one hand, an image generation method is provided, the method comprising:

[0006] Obtain object information of multiple objects and occlusion relationships between the multiple objects; based on the occlusion relationships, determine the parent objects of each of the multiple objects; the object information of the objects is used to describe the objects; the parent objects of the objects refer to other objects that occlude the objects.

[0007] Based on the object information of the multiple objects, extract the first object feature of each of the multiple objects;

[0008] Based on the first object features of the parent objects of each of the plurality of objects, the visual features of each of the plurality of objects are determined, and the visual features of the objects represent the visibility of the objects;

[0009] Based on the visual features of each of the multiple objects, the first object features of each of the multiple objects are adjusted to obtain the second object features of each of the multiple objects;

[0010] An object image is generated based on the second object features of each of the plurality of objects, the object image including the plurality of objects having the occlusion relationship.

[0011] Optionally, determining the normalization parameter based on the projection features, the visual features, and the second weighting parameter of the plurality of objects includes:

[0012] For each of the plurality of objects, the projection feature, the visual feature, and the second weighting parameter of the object are multiplied together to obtain the adjusted projection feature of the object;

[0013] The adjusted projection features of the multiple objects are added together to obtain the global projection feature;

[0014] The reciprocal of the global projection feature is determined as the normalization parameter.

[0015] Optionally, the second image generation model includes a second feature processing network and a decoding network, wherein the second feature processing network includes a cross-attention layer; the step of constructing the first image generation model based on the trained second image generation model includes:

[0016] Based on the cross-attention layer in the second feature processing network, M first feature processing networks are constructed, where M is an integer greater than 1;

[0017] The second feature processing network in the trained second image generation model is replaced with the M first feature processing networks to obtain the first image generation model.

[0018] On the other hand, an image generation apparatus is provided, the apparatus comprising:

[0019] The acquisition module is used to acquire object information of multiple objects and occlusion relationships between the multiple objects, and based on the occlusion relationships, determine the parent objects of each of the multiple objects. The object information of the object is used to describe the object, and the parent objects of the object refer to other objects that occlude the object.

[0020] The first feature processing module is used to extract the first object features of each of the multiple objects based on the object information of the multiple objects;

[0021] The second feature processing module is used to determine the visual features of each of the multiple objects based on the first object features of their respective parent objects, wherein the visual features of the objects represent the visibility of the objects.

[0022] The third feature processing module is used to adjust the first object features of each of the multiple objects according to their respective visual features to obtain the second object features of each of the multiple objects.

[0023] An image generation module is used to generate an object image based on the second object features of each of the plurality of objects, wherein the object image includes the plurality of objects having the occlusion relationship.

[0024] Optionally, the object information includes appearance information and location information; the first feature processing module includes:

[0025] The first extraction unit is used to extract the appearance features of the object based on the appearance information of the object;

[0026] The second extraction unit is used to extract the projection features of the object based on the object's position information;

[0027] The first fusion unit is used to fuse the appearance features of the object and the projection features of the object to obtain the first object feature of the object.

[0028] Optionally, the apparatus is executed through a first image generation model, which includes M first feature processing networks connected in sequence, where M is an integer greater than 1; the first extraction unit is used for:

[0029] The first feature processing network in the M first feature processing networks performs cross-attention encoding on the noise features and the appearance information of the object to obtain the first appearance feature of the object;

[0030] Cross-attention encoding is performed on the (i-1)th image feature and the appearance information of the object through the i-th first feature processing network among the M first feature processing networks to obtain the i-th appearance feature of the object, where i is an integer greater than 1 and not greater than M;

[0031] The (i-1)th image feature is obtained by fusing the (i-1)th second object features of each of the multiple objects, and the (i-1)th second object feature of the object is obtained through the (i-1)th first feature processing network in the M first feature processing networks.

[0032] Optionally, the first extraction unit is configured to:

[0033] The i-th first feature processing network maps the (i-1)-th image feature to the i-th query feature.

[0034] The appearance information of the object is mapped to the i-th key feature and the i-th value feature through the i-th first feature processing network;

[0035] The i-th appearance feature is obtained by fusing the i-th query feature, the i-th key feature, and the i-th value feature through the i-th first feature processing network.

[0036] Optionally, the second extraction unit is used for:

[0037] Convert the location information of the object into a location mask feature;

[0038] The i-th attention feature is obtained by multiplying the transpose of the i-th query feature and the i-th key feature through the i-th feature processing network.

[0039] The i-th attention feature and the location mask feature are multiplied by the i-th feature processing network to obtain the i-th projection feature of the object.

[0040] Optionally, the first fusion unit is configured to:

[0041] The i-th appearance feature and the i-th projection feature of the object are multiplied by the i-th feature processing network to obtain the i-th first object feature of the object.

[0042] Optionally, the second feature processing module includes:

[0043] The first processing unit is configured to determine the occlusion feature of each upper-level object of the object based on the opacity feature of the upper-level object and the first object feature of the upper-level object, wherein the occlusion feature of the upper-level object represents the occlusion status of the upper-level object;

[0044] The second processing unit is used to map the occlusion features of at least one upper-layer object of the object to the visible features of the object.

[0045] Optionally, the first processing unit is configured to:

[0046] Based on the opacity feature of the upper-layer object, determine the semantic density feature of the upper-layer object;

[0047] The occlusion feature of the upper-layer object is obtained by multiplying its semantic density feature with its projection feature.

[0048] Optionally, the image generation apparatus includes multiple iterative processes, wherein the iterative process refers to determining the second object features of the multiple objects in the current iteration based on the second object features and object information of the multiple objects obtained in the previous iteration; the first processing unit is configured to:

[0049] Map the opacity feature of the upper-layer object to the initial semantic density feature of the upper-layer object;

[0050] Determine the current iteration count and the total iteration count, and determine the first weighting parameter, which is negatively correlated with the current iteration count;

[0051] The initial semantic density features of the upper-level object are weighted based on the weighting parameters to obtain the semantic density features of the upper-level object.

[0052] Optionally, the second processing unit is configured to:

[0053] The occlusion features of at least one of the object's parent objects are added together to obtain the global occlusion features;

[0054] The visual features of the object are obtained by performing an exponential operation on the negative of the global occlusion feature.

[0055] Optionally, the first image generation model includes M first feature processing networks connected in sequence, where M is an integer greater than 1;

[0056] The second feature processing module is used to: determine the i-th visible feature of the object based on the i-th first object feature of the object's parent object through the i-th first feature processing network among the M first feature processing networks, where i is an integer greater than 1 and not greater than M;

[0057] The third feature processing module is used to: adjust the i-th first object feature of the object according to the i-th visible feature of the object through the i-th first feature processing network to obtain the i-th second object feature of the object, wherein the i-th first object feature is obtained through the i-th feature processing network.

[0058] Optionally, the image generation module includes:

[0059] The second fusion unit is used to fuse the second object features of the multiple objects to obtain image features;

[0060] The decoding unit is used to perform decoding operations on the image features to obtain the object image.

[0061] Optionally, the second fusion unit is used for:

[0062] For each of the plurality of objects, based on the opacity feature of the object, the semantic density feature of the object is determined, and the semantic density feature of the object is mapped to a second weighting parameter, wherein the second weighting parameter is positively correlated with the semantic density feature of the object;

[0063] Based on the second weighting parameters of each of the multiple objects, a weighted fusion operation is performed on the second object features of each of the multiple objects to obtain the image features.

[0064] Optionally, the second fusion unit is used for:

[0065] Based on the second weighting parameters of each of the multiple objects, the second object features of each of the multiple objects are weighted and summed to obtain the initial image features;

[0066] Based on the projection features, the visual features, and the second weighting parameter of the plurality of objects, a normalization parameter is determined;

[0067] The initial image features are multiplied by the normalization parameter to obtain the image features.

[0068] Optionally, the second fusion unit is used for:

[0069] For each of the plurality of objects, the projection feature, the visual feature, and the second weighting parameter of the object are multiplied together to obtain the adjusted projection feature of the object;

[0070] The adjusted projection features of the multiple objects are added together to obtain the global projection feature;

[0071] The reciprocal of the global projection feature is determined as the normalization parameter.

[0072] Optionally, the first image generation model includes M first feature processing networks and a decoding network connected in sequence, where M is an integer greater than 1;

[0073] The image generation module is used to: fuse the i-th second object features of each of the multiple objects through the i-th first feature processing network in the M first feature processing networks to obtain the i-th image feature, where i is an integer greater than 1 and not greater than M, and the i-th second object feature is obtained through the i-th feature processing network;

[0074] The image generation module is used to: perform a decoding operation on the Mth image feature through the decoding network to obtain the object image.

[0075] Optionally, the device further includes:

[0076] The model training module is used to acquire sample object information and sample label images, wherein the sample label images include the objects indicated by the sample object information;

[0077] The model training module is also used to generate a model from the second image and generate a sample target image based on the sample object information;

[0078] The model training module is also used to train the second image generation model based on the sample target image and the sample label image to obtain the trained second image generation model;

[0079] The model building module is used to build the first image generation model based on the trained second image generation model.

[0080] Optionally, the second image generation model includes a second feature processing network and a decoding network, wherein the second feature processing network includes a cross-attention layer; the model building module is used for:

[0081] Based on the cross-attention layer in the second feature processing network, M first feature processing networks are constructed, where M is an integer greater than 1;

[0082] The second feature processing network in the trained second image generation model is replaced with the M first feature processing networks to obtain the first image generation model.

[0083] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the image generation method as described above.

[0084] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the image generation method as described above.

[0085] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the image generation method as described above.

[0086] The solution provided in this application, for each of multiple objects, can determine its first object feature based on its object information. Based on the occlusion relationship between the multiple objects, the upper-level object occluding the object can be determined. Since the visibility of the object is determined by the upper-level object occluding it, the object's visual features can be determined based on the first object feature of the upper-level object. Then, by adjusting the object's first object feature based on its visual features, the remaining second object feature after occlusion can be obtained. Furthermore, based on the second object features of each of the multiple objects, an object image including all the objects can be generated, and the object image can present the occlusion relationship between the multiple objects. Therefore, the solution of this application can accurately control the occlusion relationship between multiple objects in the image generation process, improving the controllability of the occlusion relationship, overcoming the limitation of being unable to control the occlusion relationship during image generation, and enhancing the visual effect of the generated image. Attached Figure Description

[0087] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0088] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application;

[0089] Figure 2 This is a flowchart of an image generation method provided in an embodiment of this application;

[0090] Figure 3 This is a schematic diagram of the structure of a first image generation model provided in an embodiment of this application;

[0091] Figure 4 This is a schematic diagram of a first feature processing network provided in an embodiment of this application;

[0092] Figure 5 This is an occlusion relationship diagram with object information provided in an embodiment of this application;

[0093] Figure 6 This is a flowchart of another image generation method provided in the embodiments of this application;

[0094] Figure 7 This is a flowchart of another image generation method provided in the embodiments of this application;

[0095] Figure 8This is a flowchart of a model training method provided in an embodiment of this application;

[0096] Figure 9 This is a schematic diagram of an image provided in an embodiment of this application;

[0097] Figure 10 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0098] Figure 11 This is a schematic diagram of another image generation apparatus provided in an embodiment of this application;

[0099] Figure 12 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0100] Figure 13 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0102] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first object feature may be referred to as a second object feature, and similarly, a second object feature may be referred to as a first object feature.

[0103] "At least one" means one or more. For example, at least one object can be one object, two objects, three objects, or any integer number of objects greater than or equal to one. "Multiple" means two or more. For example, multiple objects can be two objects, three objects, or any integer number of objects greater than or equal to two. "Each" means each one of the at least one objects. For example, each object means each one of the multiple objects. If the multiple objects are three objects, then each object means each one of the three objects.

[0104] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0105] For example, the object information and occlusion relationships between multiple objects involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0106] For ease of understanding, the concepts involved in the embodiments of this application are explained below.

[0107] (1) Stable Diffusion: This is a variant of the diffusion model called the "Latent Diffusion Model" (LDM). The purpose of the diffusion model is to eliminate Gaussian noise applied continuously to the training image, which can be viewed as a series of denoising autoencoders. Stable Diffusion includes VAE (Variational Auto-Encoder) and U-Net (U-shaped network). VAE includes a VAE encoder and a VAE decoder, which are used to transform the image to a low-dimensional latent space. The process of adding and removing Gaussian noise is applied to the low-dimensional latent space, and then the denoised features are decoded to the pixel space. During forward diffusion, Gaussian noise is iteratively applied to the compressed latent features. Each denoising step is performed by a U-Net architecture containing a residual neural network, which obtains latent features by denoising from forward diffusion in the reverse direction. Finally, the VAE decoder generates the image by transforming the latent features back to the pixel space.

[0108] (2) U-Net: This is one of the algorithms for semantic segmentation using fully convolutional networks. It uses a symmetrical U-shaped structure that includes compression and expansion paths. U-Net is also the foundational backbone model of the Stable Diffusion model. A cross-attention module is added to U-Net to enable noise reduction. U-Net is divided into down blocks and up blocks. In fact, the U-Net in the Stable Diffusion model consists of three parts: down block, mid block, and up block. Each block in these three parts has the same composition, but the feature size and feature dimension direction differ, thus distinguishing the down block, mid block, and up block.

[0109] (3) Cross attention: This refers to the cross attention mechanism in the model. This cross attention mechanism is usually used to handle the association between multiple input sequences in order to capture important information interactions between different sequences in the model. In the Stable Diffusion model, the cross attention mechanism can help the model better understand the association between different parts, thereby more accurately predicting changes and effects during the diffusion process.

[0110] (4) Occlusion relationship: The occlusion and being occluded relationship between objects in the image caused by the spatial distribution of objects, the angle of the human eye or the camera.

[0111] (5) Occlusion graph: a directed graph structure where nodes represent objects and edges represent occlusion relationships, pointing from the occluding object to the occluded object.

[0112] (6) Semantic opacity: A value in the range [0,1) that represents the density or intensity of an object, such as the opacity of glass, the density of a forest, the intensity of light, the density of rain / snow / fog, the intensity of special effects, etc.

[0113] The image generation method provided in this application can be used in computer devices. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited to these. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0114] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application. See also... Figure 1 The computer system includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0115] Terminal 101 has a client 111 installed and running. This client 111 can be a social application client, online payment client, online shopping client, game client, medical service client, video client, etc. When terminal 101 runs client 111, the user interface of client 111 is displayed on the screen of terminal 101. Terminal 101 is the terminal used by user 121.

[0116] Optionally, terminal 101 can refer to one of a plurality of terminals. As those skilled in the art will know, the number of terminals can be more or less. For example, there may be only one terminal, or there may be six, eight or more terminals. This application does not limit the number of terminals or the type of device.

[0117] Figure 1 Only one terminal is shown in the diagram, but in different embodiments, multiple other terminals 103 can access the server 102. Optionally, one or more terminals 103 may also be terminals corresponding to developers, on which a client development and editing platform is installed. Developers can edit and update the client on the terminal 103 and transmit the updated client installation package to the server 102 via wired or wireless network. Terminal 101 can download the client installation package from the server 102 to update the client.

[0118] Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.

[0119] Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 102 is used to provide backend services to clients. Optionally, server 102 undertakes the primary computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0120] In this embodiment, terminal 101 sends an image generation request to server 102. The image generation request carries object information of multiple objects and occlusion relationships between the multiple objects. After receiving the image generation request, server 102 uses the method provided in this embodiment to generate an object image, which includes multiple objects with the occlusion relationships. Server 102 then returns the object image to terminal 101.

[0121] It should be noted that the above implementation environment is only an example. The method provided in this application embodiment can also be executed by terminal 101 or server 102 alone, or by other computer devices. This application embodiment does not limit this.

[0122] Figure 2 This is a flowchart of an image generation method provided in an embodiment of this application. This embodiment is executed by a computer device. See also... Figure 2 The method includes:

[0123] 201. The computer device obtains object information of multiple objects and occlusion relationships between multiple objects. Based on the occlusion relationships, it determines the parent objects of each of the multiple objects. The object information of the objects is used to describe the objects, and the parent objects of the objects refer to other objects that occlude the objects.

[0124] The computer device acquires object information of multiple objects and the occlusion relationships between the multiple objects. The multiple objects refer to the objects to be included in the image to be generated. For example, the multiple objects can be objects, shapes, plants, animals, etc. The types of the multiple objects are not limited in this embodiment.

[0125] For example, the object information of the multiple objects and the occlusion relationship between the multiple objects are uploaded to the computer device by the user.

[0126] The object information for multiple objects is used to describe the object. For example, the object information includes appearance information, which describes the appearance of the object, including its color, shape, etc. This appearance information may be descriptive text describing the object's appearance. For example, the object information also includes location information, which describes the object's position within the image. This position information may be the coordinates of the object's minimum bounding box.

[0127] The occlusion relationship between objects indicates which object occludes which object, or which object occludes which object. For example, if the occlusion relationship indicates that object A occludes object B, then the generated image should show the effect of object A occluding object B.

[0128] Computer devices determine the parent objects of multiple objects. Taking an object as an example, it may have one parent object, multiple parent objects, or no parent objects. If an object has one parent object, it means that the object is occluded by only one other object. If an object has multiple parent objects, it means that the object is occluded by multiple other objects. If an object has no parent objects, it means that the object is not occluded by any other object, and can be considered the topmost object.

[0129] 202. The computer device extracts the first object features of each of the multiple objects based on the object information of the multiple objects.

[0130] For each of the multiple objects, the computer device extracts the first object feature of the object based on the object information of that object, thereby obtaining the first object feature of each object.

[0131] 203. Computer devices determine the visual features of multiple objects based on the first object features of their respective parent objects. The visual features of an object represent the object's visibility.

[0132] For each of the multiple objects, the visual features of the object are determined based on the first object features of at least one parent object of that object.

[0133] 204. The computer device adjusts the first object features of each of the multiple objects based on their respective visual features to obtain the second object features of each of the multiple objects.

[0134] For each of the multiple objects, based on the object's visual features, adjust the object's first object feature to obtain the object's second object feature.

[0135] It can be understood that the first object feature is the feature of the object itself, and the second object feature is the feature of the object remaining after considering the occlusion situation, that is, the feature of the part that can be presented after occlusion.

[0136] In one possible implementation, in step 203, if the object has parent objects, the object's visual characteristics are determined based on the first object characteristics of at least one parent object. If the object does not have parent objects, then step 203 is not required to determine the object's visual characteristics. Accordingly, in step 204, the object's first object characteristics are used as the object's second object characteristics.

[0137] In one possible implementation, in step 203, if the object has a parent object, then the object's visual characteristics are determined based on the first object characteristics of at least one parent object. If the object does not have a parent object, then a preset characteristic is determined as the object's visual characteristics; for example, all values ​​in this preset characteristic are 1. Accordingly, in step 204, following normal processing logic, the first object characteristic is adjusted based on the object's visual characteristics to obtain the object's second object characteristic.

[0138] 205. A computer device generates an object image based on the second object features of each of the multiple objects. The object image includes multiple objects with occlusion relationships.

[0139] After acquiring the second object features of each object, the computer device generates an object image based on the second object features of multiple objects. Since the second object features of each object are the features of the remaining part that can be displayed after the object is occluded, the object image generated based on the second object features of multiple objects presents the remaining parts of each object after occlusion. Therefore, the object image can show the occlusion relationship between multiple objects.

[0140] The method provided in this application, for each of a plurality of objects, determines a first object feature of the object itself based on its object information. Based on the occlusion relationship between the plurality of objects, it determines the upper-level object occluding the object. Since the visibility of the object is determined by the upper-level object occluding it, the visual features of the object can be determined based on the first object feature of the upper-level object. Then, by adjusting the first object feature of the object itself based on its visual features, a second object feature remaining after the object is occluded can be obtained. Furthermore, based on the second object features of each of the plurality of objects, an object image including the plurality of objects can be generated, and the object image can present the occlusion relationship between the plurality of objects. Therefore, the solution of this application can accurately control the occlusion relationship between multiple objects in the image generation process, improving the controllability of the occlusion relationship, overcoming the limitation of being unable to control the occlusion relationship during image generation, and enhancing the visual effect of the generated image.

[0141] The above Figure 2 The embodiments described herein are merely brief illustrations of image generation methods. In some embodiments, the computer device may also perform the image generation process using a first image generation model.

[0142] In one possible implementation, the first image generation model can be a diffusion model, and the image generation method of the diffusion model can be of any type. For example, the diffusion model can be a text-to-image model or a layout-to-image model, etc. The mesh structure of the diffusion model can be of any type. For example, the diffusion model can be the aforementioned Stable Diffusion model, which includes VAE and U-Net, and U-Net includes a cross-attention network structure. Of course, the first image generation model can also be other types of models, and this application embodiment does not limit this.

[0143] Figure 3 This is a schematic diagram of the structure of a first image generation model shown in an embodiment of this application, such as... Figure 3As shown, the first image generation model includes M first feature processing networks 301 and a decoding network 302. The M first feature processing networks 302 are connected sequentially, and the last first feature processing network 301 is connected to the decoding network 302, where M is an integer greater than 1.

[0144] Among them, M first feature processing networks 301 are used to extract potential object features or image features, and decoding network 302 is used to generate images. For example, the network structures of the M first feature processing networks 301 are the same, and the network parameters of the M first feature processing networks 301 can be the same or different.

[0145] For example, the first image generation model is a diffusion model, such as Stable Diffusion. In addition to the aforementioned M first feature processing networks 301 and decoding network 302, this first image generation model also includes an encoding network. The encoding network can be a VAE encoder in a VAE network structure, and the decoding network can be a VAE decoder in a VAE network structure. The M first feature processing networks 301 can be U-Net structures, with the same network structure but different feature sizes. For example, each first feature processing network 301 includes a cross-attention network structure, that is, it includes cross-attention layers.

[0146] In one possible implementation, Figure 4 This is a schematic diagram of a first feature processing network provided in an embodiment of this application, as shown below. Figure 4 As shown, taking the l-th first feature processing network as an example, the output data of the (l-1)-th first feature processing network is the input data of the l-th first feature processing network, and the output data of the l-th first feature processing network is the input data of the (l+1)-th first feature processing network.

[0147] The l-th first feature processing network includes a cross-attention layer. The input data of the cross-attention layer is the appearance information in the object information and the R output of the (l-1)-th first feature processing network. (l-1) The output data of the cross-attention layer is the appearance features of the object. This appearance information includes descriptive text such as "yellow background," "blue triangle," "green circle," and "red rectangle." R... (l-1) This represents the image features output by the (l-1)th first feature processing network. This represents the appearance features of the first object output by the cross-attention layer. This represents the appearance features of the second object output by the cross-attention layer. This represents the appearance features of the third object output by the cross-attention layer. This represents the appearance features of the fourth object output by the cross-attention layer.

[0148] The first feature processing network also includes a first multiplication operator. The output data of the cross-attention layer also includes the object's attention features. Based on the positional information in the object information, the object's positional mask features can be determined. The object's attention features are multiplied by the object's positional mask features using the first multiplication operator to obtain the object's projection features. Figure 4 As shown, Attn.maps represents the attention features of the object, and Box.masks represents the position mask features of the object.

[0149] The first feature processing network also includes a second multiplication operator, which multiplies the appearance features and projection features of each object to obtain the first object feature for each object. For example... Figure 4 As shown, The first object characteristic represents the first object. The first object characteristic of the second object. The first object characteristic representing the third object. This represents the first object characteristic of the fourth object.

[0150] Based on the occlusion relationships between multiple objects, the first object features of these objects are mapped to image features, which are then the output data of the l-th first feature processing network. For example... Figure 4 As shown, R (l) This represents the image features output by the l-th first feature processing network.

[0151] Wherein, each of the first feature processing networks in the first image generation model is related to the above. Figure 4 The network architecture and processing logic shown are the same, and will not be described again here.

[0152] In one possible implementation, Figure 4 The object information and occlusion relationships shown can be represented by an occlusion relationship diagram containing object information. Figure 5 This application provides an occlusion relationship diagram with object information, such as... Figure 5 As shown, the occlusion relationship graph includes nodes 501, 502, 503, and 504. Each node carries object information, including descriptive text describing the object's appearance and the coordinates of the object's smallest bounding box. Nodes are connected by directed lines pointing to the occluded object. For example, if a directed line connects node 501 and node 502, pointing to node 502, it indicates that the object of node 501 occludes the object of node 502.

[0153] So, from Figure 5As can be seen, the red rectangle obscures the green circle, the blue triangle, and the yellow background; the green circle obscures the blue triangle and the yellow background; and the blue triangle obscures the yellow background. In other words, the red rectangle represents the topmost object, and the yellow background represents the bottommost object.

[0154] The image generation method provided in this application embodiment can be achieved through the above... Figure 3 and Figure 4 The first image generation model shown is used for execution. This first image generation model includes M first feature processing networks and a decoding network. For detailed procedures, please refer to the following... Figure 6 Examples of implementations. Figure 6 This is a flowchart of another image generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 6 The method includes:

[0155] 601. A computer device acquires object information of multiple objects and occlusion relationships between multiple objects. Based on the occlusion relationships, it determines the parent objects of each of the multiple objects. The object information of an object is used to describe the object, and the parent objects of an object refer to other objects that occlude the object.

[0156] In one possible implementation, the computer device also acquires the opacity characteristics of multiple objects.

[0157] For example, an object's opacity characteristic represents its opacity, which indicates the degree to which light is allowed to pass through. An object's opacity reflects the visibility of other objects it obscures. Higher opacity means a less transparent object, allowing less light to pass through, and thus lower visibility of other objects it obscures. Conversely, lower opacity means a more transparent object, allowing more light to pass through, and thus higher visibility of other objects it obscures.

[0158] For example, an object's transparency feature represents its semantic opacity. Semantic opacity represents the object's density or intensity, and it also reflects the visibility of other objects obscured by that object; it can be understood as opacity in a broad sense. For instance, if the object is rain, a higher semantic opacity indicates a higher rain density, and thus lower visibility of other objects within the rain. Similarly, if the object is an effect, a higher semantic opacity indicates a stronger effect, and thus lower visibility of other objects obscured by the effect.

[0159] In one possible implementation, the computer device sorts multiple objects according to their occlusion relationships, such that the sorted result satisfies the following condition: in any two adjacent objects, the preceding object occludes the following object, or the two objects do not occlude each other. That is, in any two adjacent objects, the preceding object is guaranteed not to be occluded by the following object.

[0160] For example, if object A occludes objects B, C, and D, object B occludes object C, and object C occludes object D, then the order of arrangement is: object A, object B, object C, and object D.

[0161] For example, if object A occludes objects B, C, and D, object B occludes object D, and object C occludes object D, then the order of arrangement is: object A, object B, object C, and object D, or object A, object C, object B, and object D.

[0162] 602. The computer device extracts the first first object feature of each of the M first feature processing networks based on noise features and object information of multiple objects.

[0163] For each of the multiple objects, the computer device inputs the noise features and the object features of that object into the first first feature processing network to obtain the first first object feature of that object, thereby obtaining the first first object feature of each object, which is the latent feature of the object.

[0164] In one possible implementation, the computer device acquires preset or randomly generated noise features, such as the noise features of random Gaussian noise.

[0165] In this embodiment of the application, the object information includes appearance information and location information. The computer device performs the following operations through the first feature processing network: extracting the first appearance feature of the object based on noise features and the appearance information of the object; extracting the first projection feature of the object based on the location information of the object; and fusing the first appearance feature and the first projection feature of the object to obtain the first object feature.

[0166] Appearance information describes the appearance of an object; for example, this appearance information may be descriptive text used to describe the object's appearance. Location information describes the location of an object; for example, this location information may be the coordinates of the object's smallest bounding box.

[0167] In one possible implementation, the process by which a computer device determines a first appearance feature of an object through a first feature processing network includes: performing cross-attention encoding on noise features and appearance information of the object through the first feature processing network to obtain the first appearance feature of the object.

[0168] like Figure 4 As shown, the first feature processing network includes a cross-attention layer. For each of the multiple objects, the computer device inputs the noise features and the appearance information of the object into the cross-attention layer, and the cross-attention layer outputs the first appearance feature of the object.

[0169] Optionally, the process of performing cross-attention encoding on the noise features and the appearance information of the object includes: mapping the noise features to a first query feature through a first feature processing network; mapping the appearance information of the object to a first key feature and a first value feature through a first feature processing network; and fusing the first query feature, the first key feature, and the first value feature through a first feature processing network to obtain a first appearance feature.

[0170] In this application embodiment, the key feature, value feature, and query feature belong to different feature spaces.

[0171] For example, a first mapping matrix is ​​used to map noise features to a first query feature, and a second mapping matrix is ​​used to map the appearance information of the object to a first key feature and a first value feature. The first and second mapping matrices are network parameters of the cross-attention layer of the first feature processing network.

[0172] For example, for each of the multiple objects, a normalization operation is performed on the product of the first query feature, the transpose of the first key feature, and the scaling factor to obtain the normalized feature. The normalized feature is then multiplied by the first value feature to obtain the first appearance feature.

[0173] In one possible implementation, the process by which a computer device determines the first projective feature of an object through a first feature processing network includes: converting the object's location information into a location mask feature; multiplying the transpose of the first query feature and the first key feature through the first feature processing network to obtain a first attention feature; and multiplying the first attention feature and the location mask feature through the first feature processing network to obtain the object's first projective feature.

[0174] This attention feature is also known as an attention map. The location mask feature is also known as a bounding box mask.

[0175] For example, the object's location information is the coordinates of its smallest bounding box. For instance, location information (0.2, 0.4, 0.7, 0.4) indicates that the object's smallest bounding box is a rectangle with coordinates (0.2, 0.4) and (0.7, 0.4) as its two diagonal points. This location mask feature is a feature matrix, where each value corresponds to a point in the image. The values ​​in the feature matrix include 0 and 1; a 1 indicates the region in the image containing the rectangle, and a 0 indicates other regions in the image besides the rectangle.

[0176] For example, the computer device performs a normalization operation on the first attention feature to obtain a normalized first attention feature, and multiplies the normalized first attention feature with the object's position mask feature to obtain the object's first projection feature.

[0177] In one possible implementation, the process by which a computer device determines a first first object feature of an object through a first first feature processing network includes: multiplying a first appearance feature of the object and a first projection feature of the object through the first first feature processing network to obtain a first first object feature of the object.

[0178] For each of the multiple objects, multiply the object's first appearance feature and the object's first projection feature to obtain the object's first first object feature, thus obtaining the first first object feature of each object.

[0179] 603. The computer device determines the first visible feature of each of the multiple objects based on the first first object feature of each of the upper-level objects through the first first feature processing network.

[0180] Among them, the visual characteristics of an object represent the object's visibility.

[0181] For each of the multiple objects, if the object has a parent object, the first visual feature of the object is determined based on the first first object feature of at least one parent object of the object; if the object does not have a parent object, a preset feature is determined as the first visual feature of the object.

[0182] The process of determining the visual features of an object based on the first object features of the upper-level object in step 603 is similar to the process in step 705 below, and will not be described in detail here.

[0183] 604. The computer device, through the first feature processing network, adjusts the first object features of each of the multiple objects based on their respective first visible features, and obtains the first object features of each of the multiple objects.

[0184] Here, the first object feature is the feature of the object itself, and the object's visual feature represents the object's visibility, or the state of being occluded. Then, the first object feature is adjusted according to the visual feature to obtain the second object feature. This second object feature is the remaining feature of the object after considering the occlusion situation, that is, the feature of the part that can still be displayed after being occluded.

[0185] In one possible implementation, the computer device combines the object's first visual feature with the object's first first object feature to obtain the object's first second object feature.

[0186] 605. The computer device, through the first feature processing network, fuses the first second object features of multiple objects to obtain the first image feature.

[0187] The process of determining the first image feature in step 605 is the same as the process of determining the image feature in step 707 below, and will not be described in detail here.

[0188] 606. A computer device extracts the i-th first object feature of each of the M first feature processing networks based on the (i-1)-th image feature and the object information of the multiple objects, where i is an integer greater than 1 and not greater than M.

[0189] For each of the multiple objects, the computer device inputs the (i-1)th image feature and the object feature of that object into the i-th first feature processing network to obtain the i-th first object feature of that object, thereby obtaining the i-th first object feature of each object, which is the latent feature of the object.

[0190] The (i-1)th image feature is obtained by fusing the (i-1)th second object features of multiple objects. The (i-1)th second object feature of an object is obtained through the (i-1)th first feature processing network in M ​​first feature processing networks.

[0191] In this embodiment, the object information includes appearance information and location information. The computer device performs the following operations through the i-th first feature processing network: based on the (i-1)-th image feature and the object's appearance information, extract the i-th appearance feature of the object; based on the object's location information, extract the i-th projection feature of the object; fuse the i-th appearance feature and the i-th projection feature of the object to obtain the i-th first object feature.

[0192] In one possible implementation, the process by which a computer device determines the i-th appearance feature of an object through the i-th first feature processing network includes: performing cross-attention encoding on the (i-1)-th image feature and the appearance information of the object through the i-th first feature processing network to obtain the i-th appearance feature of the object.

[0193] like Figure 4 As shown, the i-th first feature processing network includes a cross-attention layer. For each of the multiple objects, the computer device inputs the (i-1)-th image feature and the appearance information of the object into the cross-attention layer, and the cross-attention layer outputs the i-th appearance feature of the object.

[0194] Optionally, the process of performing cross-attention encoding on the (i-1)th image feature and the appearance information of the object includes: mapping the (i-1)th image feature to the ith query feature through the ith first feature processing network; mapping the appearance information of the object to the ith key feature and the ith value feature through the ith first feature processing network; and fusing the ith query feature, the ith key feature, and the ith value feature through the ith first feature processing network to obtain the ith appearance feature.

[0195] For example, a first mapping matrix is ​​used to map noise features to the i-th query feature, and a second mapping matrix is ​​used to map the appearance information of the object to the i-th key feature and the i-th value feature. Here, the first and second mapping matrices are the network parameters of the cross-attention layer of the i-th first feature processing network.

[0196] For example, for each of the multiple objects, a normalization operation is performed on the product of the i-th query feature, the transpose of the i-th key feature, and the scaling factor to obtain the normalized feature. The normalized feature is then multiplied by the i-th value feature to obtain the i-th appearance feature.

[0197] In one possible implementation, the process by which a computer device determines the i-th projective feature of an object through the i-th first feature processing network includes: converting the object's location information into a location mask feature; multiplying the i-th query feature and the transpose of the i-th key feature through the i-th feature processing network to obtain the i-th attention feature; and multiplying the i-th attention feature and the location mask feature through the i-th feature processing network to obtain the i-th projective feature of the object.

[0198] For example, the computer device performs a normalization operation on the i-th attention feature to obtain the normalized i-th attention feature, and multiplies the normalized i-th attention feature with the object's position mask feature to obtain the object's i-th projection feature.

[0199] In one possible implementation, the process by which a computer device determines the i-th first object feature of an object through the i-th first feature processing network includes: multiplying the i-th appearance feature of the object and the i-th projection feature of the object through the i-th feature processing network to obtain the i-th first object feature of the object.

[0200] For each of the multiple objects, multiply the i-th appearance feature of the object by the i-th projection feature of the object to obtain the i-th first object feature of the object, thus obtaining the i-th first object feature of each object.

[0201] 607. The computer device determines the i-th visual feature of each of the multiple objects based on the i-th first object feature of each of the multiple objects' upper-level objects through the i-th first feature processing network.

[0202] For each of the multiple objects, if the object has a parent object, the i-th visual feature of the object is determined based on the i-th first object feature of at least one parent object of the object; if the object does not have a parent object, the preset feature is determined as the i-th visual feature of the object.

[0203] The process of determining the visual features of an object based on the first object features of the upper-level object in step 607 is similar to the process in step 705 below, and will not be described in detail here.

[0204] 608. The computer device, through the i-th first feature processing network, adjusts the i-th first object feature of each of the multiple objects according to the i-th visible feature of each of the multiple objects, and obtains the i-th second object feature of each of the multiple objects.

[0205] In one possible implementation, the computer device combines the i-th visual feature of an object with the i-th first object feature of the object to obtain the i-th second object feature of the object.

[0206] 609. The computer device uses the i-th first feature processing network to fuse the i-th second object features of multiple objects to obtain the i-th image feature.

[0207] The process of determining the i-th image feature in step 605 is the same as the process of determining the image feature in step 707 below, and will not be described in detail here.

[0208] 610. The computer device performs a decoding operation on the Mth image feature through a decoding network to obtain the object image.

[0209] The Mth first feature processing network in the M first feature processing networks undergoes the operations described in steps 606-609 above to obtain the Mth image feature. The Mth first feature processing network is also the last first feature processing network.

[0210] like Figure 3 As shown, the first image generation model includes a decoding network. After obtaining the Mth image feature, the computer device inputs the Mth image feature into the decoding network, and the decoding network outputs the object image.

[0211] Since the second object feature of each object is the feature of the remaining part that can be presented after the object is occluded, the image feature generated based on the second object features of multiple objects includes the feature of the remaining part of each object after occlusion. Then, the decoding operation is performed on the Mth image feature to obtain the object image, which can ensure that the object image presents the remaining part of each object after occlusion. Therefore, the object image can present the occlusion relationship between multiple objects.

[0212] The method provided in this application, for each of a plurality of objects, determines a first object feature of the object itself based on its object information. Based on the occlusion relationship between the plurality of objects, it determines the upper-level object occluding the object. Since the visibility of the object is determined by the upper-level object occluding it, the visual features of the object can be determined based on the first object feature of the upper-level object. Then, by adjusting the first object feature of the object itself based on its visual features, a second object feature remaining after the object is occluded can be obtained. Furthermore, based on the second object features of each of the plurality of objects, an object image including the plurality of objects can be generated, and the object image can present the occlusion relationship between the plurality of objects. Therefore, the solution of this application can accurately control the occlusion relationship between multiple objects in the image generation process, improving the controllability of the occlusion relationship, overcoming the limitation of being unable to control the occlusion relationship during image generation, and enhancing the visual effect of the generated image.

[0213] In some embodiments, the first image generation model further includes an encoding network connected to each first feature processing network. The computer device first performs feature extraction on the object's appearance information through the encoding network to obtain initial appearance features, such as embedded features. Figure 6 In one embodiment, the appearance information input to each first feature processing network is replaced with the object's initial appearance features.

[0214] Figure 7 This is a flowchart of another image generation method provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 7 The method includes:

[0215] 701. A computer device acquires object information of multiple objects and occlusion relationships between multiple objects. Based on the occlusion relationships, it determines the parent objects of each of the multiple objects. The object information is used to describe the object. The parent objects of the object refer to other objects that occlude the object. The object information includes appearance information and position information.

[0216] Appearance information describes the appearance of an object; for example, this appearance information may be descriptive text used to describe the object's appearance. Location information describes the location of an object; for example, this location information may be the coordinates of the object's smallest bounding box.

[0217] The process of step 701 is the same as that of step 601 above, and will not be described again here.

[0218] 702. Computer devices extract the appearance features of objects based on their appearance information.

[0219] In one possible implementation, for each of a plurality of objects, the computer device performs a feature extraction operation on the appearance information of that object to obtain the appearance features of that object, thereby obtaining the appearance features of each object.

[0220] For example, a computer device performs self-attention encoding on the appearance information of the object to obtain the appearance features of the object.

[0221] In another possible implementation, the above step 602 is used to extract M appearance features of each object.

[0222] 703. Computer devices extract the projection features of objects based on their location information.

[0223] In one possible implementation, for each of a plurality of objects, the computer device performs a feature extraction operation on the object's location information to obtain the object's projected features, thereby obtaining the appearance features of each object.

[0224] In another possible implementation, the method described in step 602 above is used to extract M projected features for each object.

[0225] 704. Computer devices combine the appearance features and projection features of an object to obtain the object's first object features.

[0226] In one possible implementation, for each of a plurality of objects, the computer device multiplies the appearance features of the object with the projection features of the object to obtain the first object feature of the object, thereby obtaining the first object feature of each object.

[0227] 705. A computer device determines the visual characteristics of multiple objects based on the first object characteristics of their respective parent objects. The visual characteristics of an object represent the visual status of the object.

[0228] In one possible implementation, step 705 is achieved by steps 7051 and 7052 as described below.

[0229] 7051. For each parent object of an object, based on the opacity feature and the first object feature of the parent object, determine the occlusion feature of the parent object. The occlusion feature of the parent object represents the occlusion status of the parent object.

[0230] Optionally, the occlusion features of the upper-layer object are determined as follows: based on the opacity features of the upper-layer object, the semantic density features of the upper-layer object are determined; the semantic density features of the upper-layer object are multiplied with the projection features of the upper-layer object to obtain the occlusion features of the upper-layer object.

[0231] The semantic density feature of the upper-level object represents its semantic density, which refers to the amount of information contained within the object, or the complexity inherent in its expression. Higher opacity of the upper-level object indicates a greater degree of light obstruction, thus retaining more information. In other words, the semantic density feature of the upper-level object is positively correlated with its opacity feature. A higher opacity feature results in a higher semantic density feature, and vice versa.

[0232] Optionally, the image generation method includes multiple iterative processes. The iterative process refers to determining the second object features of multiple objects in the current iteration based on the second object features and object information of multiple objects obtained in the previous iteration.

[0233] For example, such as Figure 6As shown, the image generation method is executed by a first image generation model comprising M first feature processing networks. One iteration refers to executing steps 602 to 609 once. Specifically, in the first iteration, for the first first feature processing network, following step 602, the first first feature processing network extracts the first first object feature for each of the multiple objects based on noise features and object information. In each subsequent iteration, for the first first feature processing network, based on the last image feature output from the previous iteration and object information, the first first object feature for each of the multiple objects is extracted. This process continues until the preset number of iterations is reached. Then, the last image feature output from the last iteration is obtained, and a decoding operation is performed on the last image feature to obtain the object image.

[0234] In this implementation, the semantic density features of the upper-level object are determined as follows: the opacity features of the upper-level object are mapped to the initial semantic density features of the upper-level object; the current iteration number and the total iteration number are determined, and the first weighting parameter is determined, which is negatively correlated with the current iteration number; a weighting operation is performed on the initial semantic density features of the upper-level object based on the weighting parameter to obtain the semantic density features of the upper-level object.

[0235] For example, a computer device uses the following formula (1) to map opacity features to initial semantic density features.

[0236] α i =1-exp(-D i ); formula (1)

[0237] Where, α i D represents the opacity characteristic of the i-th object. i Let represent the initial semantic density feature of the i-th object, and exp(·) represent the natural exponential function.

[0238] For example, a computer device uses the following formula (2) to determine semantic density features.

[0239]

[0240] Where, σ i D represents the semantic density feature of the i-th object. i This represents the initial semantic density feature of the i-th object. Let T represent the first weighting parameter, T represent the total number of iterations, and t represent the current number of iterations.

[0241] 7052. Map the occlusion features of at least one parent object of an object to the visible features of the object.

[0242] After determining the occlusion features of each of the object's parent objects, the computer device maps the occlusion features of at least one parent object of the object to the visible features of the object.

[0243] Optionally, the occlusion features of the upper-level object can be mapped to the visible features of the object as follows: the occlusion features of at least one upper-level object of the object are added together to obtain the global occlusion features; the negative of the global occlusion features is then exponentially calculated to obtain the visible features of the object.

[0244] For example, a computer device uses the following formula (3) to determine the visual characteristics of an object.

[0245]

[0246] Among them, T i M represents the visual characteristics of the i-th object, where the first to (i-1)-th objects are the parent objects of the i-th object. j σ j M represents the occlusion feature of the j-th object. j σ represents the projective feature of the j-th object. j This represents the semantic density feature of the j-th object. represents the global occlusion feature, and exp(·) represents the natural exponential function.

[0247] 706. The computer device adjusts the first object features of each of the multiple objects based on their respective visual features to obtain the second object features of each of the multiple objects.

[0248] In one possible implementation, for each of the multiple objects, the object's visual features are multiplied by the object's first object features to obtain the object's second object features.

[0249] For example, a computer device uses the following formula (4) to determine the second object characteristics of an object.

[0250] Q i =T i M i R i ;Formula (4)

[0251] Among them, Q i T represents the second object characteristic of the i-th object. i M represents the visual features of the i-th object. i R i M represents the first object characteristic of the i-th object. iR represents the projective feature of the i-th object. i This represents the appearance characteristics of the i-th object.

[0252] 707. Computer devices fuse the secondary object features of multiple objects to obtain image features.

[0253] After obtaining the second object features of multiple objects, the computer device performs a fusion operation on the second object features of the multiple objects to obtain image features.

[0254] In one possible implementation, step 707 includes steps 7071 and 7072 as described below.

[0255] 7071. For each of the multiple objects, determine the semantic density feature of the object based on the opacity feature of the object, and map the semantic density feature of the object to a second weighted parameter. The second weighted parameter is positively correlated with the semantic density feature of the object.

[0256] For example, the process by which a computer device determines the semantic density features of an object is similar to the process of determining the semantic density features of the upper-level object in step 7051 above. For instance, the semantic density features of the object are determined using the formulas (1) and (2) above.

[0257] For example, the computer device uses the following formula (5) to map semantic density features to a second weighting parameter.

[0258] k i =1-exp(-σ i ); formula (5)

[0259] Where, k i σ represents the second weighted parameter of the i-th object. i Let represent the semantic density feature of the i-th object, and exp(·) represent the natural exponential function.

[0260] 7072. Based on the second weighting parameters of each of the multiple objects, perform a weighted fusion operation on the second object features of each of the multiple objects to obtain image features.

[0261] Optionally, based on the second weighting parameters of each of the multiple objects, a weighted summation operation is performed on the second object features of each of the multiple objects to obtain image features.

[0262] Optionally, the first object feature of the object includes the projection feature. Then, the image features are determined as follows: based on the second weighting parameters of each of the multiple objects, the second object features of each of the multiple objects are weighted and summed to obtain the initial image features; based on the projection features, visual features, and second weighting parameters of the multiple objects, the normalization parameters are determined; the initial image features are multiplied by the normalization parameters to obtain the image features.

[0263] The normalization parameter is determined as follows: For each of the multiple objects, the object's projective feature, visual feature, and second weighted parameter are multiplied to obtain the object's adjusted projective feature; the adjusted projective features of the multiple objects are added together to obtain the global projective feature; the reciprocal of the global projective feature is determined as the normalization parameter.

[0264] The computer device adjusts the projection features of the object based on its visual features and a second weighting parameter to obtain the adjusted projection features of the object.

[0265] For example, the computer device uses the following formula (6) to determine the adjusted projection characteristics and the following formula (7) to determine the global projection characteristics.

[0266] L i =T i (1-exp(-σ i ))M i ;Formula (6)

[0267]

[0268] Among them, L i T represents the adjusted projective feature of the i-th object. i Represents the visual features of the i-th object, 1-exp(-σ i ) represents the second weighted parameter of the i-th object, σ i Let M represent the semantic density feature of the i-th object, exp(·) represent the natural exponential function, and M i Let S represent the projection feature of the i-th object. Let S represent the global projection feature, and N represent the number of objects.

[0269] For example, combining the above formulas (4) to (7), the process of determining image features can be represented by the following formula (8).

[0270]

[0271] Where R represents image features and S represents global projection features. σ represents the normalization parameter, N represents the number of objects, and σ represents the normalization parameter. i T represents the semantic density feature of the i-th object.i M represents the visual features of the i-th object. i R represents the projective feature of the i-th object. i This represents the appearance characteristics of the i-th object. exp(·) represents the natural exponential function.

[0272] 708. The computer device performs a decoding operation on the image features to obtain the object image.

[0273] After acquiring image features, the computer device performs a decoding operation on the image features to obtain an object image, which includes multiple objects with occlusion relationships.

[0274] The method provided in this application, for each of a plurality of objects, determines a first object feature of the object itself based on its object information. Based on the occlusion relationship between the plurality of objects, it determines the upper-level object occluding the object. Since the visibility of the object is determined by the upper-level object occluding it, the visual features of the object can be determined based on the first object feature of the upper-level object. Then, by adjusting the first object feature of the object itself based on its visual features, a second object feature remaining after the object is occluded can be obtained. Furthermore, based on the second object features of each of the plurality of objects, an object image including the plurality of objects can be generated, and the object image can present the occlusion relationship between the plurality of objects. Therefore, the solution of this application can accurately control the occlusion relationship between multiple objects in the image generation process, improving the controllability of the occlusion relationship, overcoming the limitation of being unable to control the occlusion relationship during image generation, and enhancing the visual effect of the generated image.

[0275] Furthermore, when determining the visual characteristics of an object, the opacity characteristics of the objects above it are also considered, thereby controlling the visibility of other objects occluded by the object in the generated image. This is beneficial for creating special effects in the image, such as changing the transparency of glass, the density of trees, the concentration of fog, the intensity of sunlight, and the intensity of lens effects.

[0276] In one embodiment, the construction process of the first image generation model described above is as follows: Figure 8 Examples of implementations. Figure 8 This is a flowchart illustrating a model training method provided in an embodiment of this application. This embodiment is executed by a computer device. See also... Figure 8 The method includes:

[0277] 801. The computer device acquires sample object information and sample label images, wherein the sample label images include the objects indicated by the sample object information.

[0278] The sample object information is the same as above. Figure 6The object information in the embodiments is similar and will not be repeated here.

[0279] The sample label image includes the object indicated by the sample object information. For example, if the sample object information is a red circle, then the sample label image includes a red circle.

[0280] The sample object information can be for a single object or for multiple objects. If the sample object information is for a single object, then the sample label image includes the object indicated by the sample object information. If the sample object information is for multiple objects, then the sample label image includes the multiple objects indicated by the sample object information.

[0281] 802. The computer device generates a sample target image based on the sample object information using a second image generation model.

[0282] The second image generation model is used to generate an image including any object based on its object information. The computer device inputs sample object information into the second image generation model, and the second image generation model outputs a sample target image.

[0283] In one possible implementation, the second image generation model is a diffusion model, which includes an encoding network, a second feature processing network, and a decoding network. The encoding network encodes sample object information into latent features, the second feature processing network denoises the noisy latent features to obtain image features, and the decoding network decodes the image features into an image.

[0284] 803. The computer equipment trains a second image generation model based on the sample target image and the sample label image to obtain the trained second image generation model.

[0285] The sample label image is a real image indicating the sample object information, while the sample target image is obtained based on the second image generation model. The smaller the difference between the sample target image generated by the second image generation model and the sample label image, the more accurate the second image generation model is. Therefore, the computer device adjusts the model parameters of the second image generation model based on the sample target image and the sample label image to obtain a more accurate second image generation model. The training objective is to reduce the difference between the sample target image and the sample label image, thereby improving the image generation capability of the second image generation model.

[0286] 804. The computer device constructs a first image generation model based on the trained second image generation model.

[0287] After acquiring the trained second image generation model, the computer device adjusts the model structure of the second image generation model to obtain a first image generation model suitable for controlling occlusion relationships.

[0288] In one possible implementation, the second image generation model includes a second feature processing network and a decoding network, wherein the second feature processing network includes a cross-attention layer.

[0289] Then, the computer device constructs a first image generation model based on the trained second image generation model, including: constructing M first feature processing networks based on the cross-attention layer in the second feature processing network, where M is an integer greater than 1; replacing the second feature processing network in the trained second image generation model with the M first feature processing networks to obtain the first image generation model.

[0290] For example, the computer device copies the cross-attention layer in the second feature processing network, and combines the copied cross-attention layer, multiple multiplication operators, and multiple mapping functions to obtain a first feature processing network. The multiple multiplication operators are used to perform the multiplication operations mentioned in the above embodiments, and the multiple mapping functions include functions represented by at least one of the above formulas (1) to (8). The computer device copies the first feature processing network into M first feature processing networks.

[0291] For example, the second image generation model further includes an encoding network. Then, the second feature processing network in the second image generation model trained by the computer device is replaced with M first feature processing networks to obtain the first image generation model, which includes an encoding network, M first feature processing networks and a decoding network.

[0292] The encoding network can be an encoder of any type of network structure, and the decoding network can be a decoder of any type of network structure.

[0293] In this embodiment, a second image generation model is trained using sample object information and sample label images to improve its image generation capability. Then, based on the trained second image generation model, a first image generation model is constructed. This is equivalent to reusing the image generation capability of the second image generation model to construct a new first image generation model, which can be used to generate object images with occlusion relationships. Therefore, there is no need to additionally train the first image generation model for object images with occlusion relationships, saving time and resource costs associated with model training and improving processing efficiency and model versatility.

[0294] Furthermore, a new first feature processing network is constructed by using the network layers in the trained second image generation model. Based on the trained second image generation model and the first feature processing network, a new first image generation model is constructed, which reduces the time and resource cost required to train the new model and improves the construction efficiency of the image generation model.

[0295] Furthermore, by simply adjusting the network structure of the already trained image generation model, an image generation model suitable for controlling occlusion relationships and opacity can be obtained. There is no need to specially train an image generation model suitable for controlling occlusion relationships and opacity, nor is any additional data required. It can be easily applied to existing image generation models.

[0296] In one related technique, an image is generated through layout control. This involves providing bounding boxes or segmentation maps of multiple objects to control the layout of those objects within the generated image. While the bounding boxes and segmentation maps indicate the object positions, this method only controls the positions of the objects within the image and cannot control the occlusion relationships or opacity between them. For example, specifying the generation of an image including a cat, a dog, and tree branches, even with the positions of the cat, dog, and branches specified, still fails to control the occlusion relationships between them. Figure 9 As shown, the generated images include three cases: image (1), image (2) and image (3). In image (1), the tree branch obscures the cat and the dog. In image (2), the dog obscures the tree branch and the tree branch obscures the cat. In image (3), the cat obscures the tree branch and the tree branch obscures the dog.

[0297] In another related technique, images are generated through scene graphs. A scene graph is a directed graph structure where nodes represent objects and edges represent relationships between objects, such as "holding," "eating," and "sitting." It can also represent spatial relationships, such as "beside," "in front," and "behind." However, these spatial relationships do not represent occlusion relationships. For example, "a boy is standing in front of a girl" could mean that the two people are facing each other, one on the left and one on the right, and do not constitute an occlusion relationship. Therefore, it is still impossible to control the generation of images with occlusion relationships.

[0298] This application provides a training-free method for generating images with controlled occlusion relationships. Based on object information of multiple objects and their occlusion relationships, it can generate images of objects with occlusion relationships. This allows for precise control of the occlusion relationships between multiple objects in the image generation process, improving the controllability of occlusion relationships and overcoming the limitation of uncontrollable occlusion relationships during image generation, thus enhancing the visual effect of the generated image. Furthermore, this method does not require additional training of an image generation model suitable for controlling occlusion relationships. By simply adjusting the network structure of any trained image generation model that is not suitable for controlling occlusion relationships, an image generation model for controlling occlusion relationships can be obtained, saving the time and resource costs of model training and improving processing efficiency and model versatility.

[0299] The image generation method provided in this application can be deployed in any scenario that requires intelligent image generation, such as in applications with text-to-image generation functionality.

[0300] Users provide input data in this application, including: object information for multiple objects and occlusion relationships between them. Object information can include descriptive text about the object's appearance and its bounding box coordinates. Occlusion relationships between multiple objects can be represented using pairs, such as object pairs.<A,B> This represents object A occluding object B. The input data may also include the opacity features of multiple objects and background descriptive text, which describes the background of the image to be generated.

[0301] Applications with text-to-image generation capabilities provide output data, including: an object image comprising multiple objects with the occlusion relationship. Specifically, the application uses the method provided in this application embodiment to generate the object image based on object information of the multiple objects and the occlusion relationship between them.

[0302] Figure 10 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this application. See also... Figure 10 The device includes:

[0303] The acquisition module 1001 is used to acquire object information of multiple objects and occlusion relationships between multiple objects. Based on the occlusion relationships, the parent objects of each of the multiple objects are determined. The object information of the objects is used to describe the objects, and the parent objects of the objects refer to other objects that occlude the objects.

[0304] The first feature processing module 1002 is used to extract the first object features of each of the multiple objects based on the object information of the multiple objects.

[0305] The second feature processing module 1003 is used to determine the visual features of each of the multiple objects based on the first object features of their respective parent objects. The visual features of an object represent the visibility of the object.

[0306] The third feature processing module 1004 is used to adjust the first object features of the multiple objects according to their respective visual features to obtain the second object features of the multiple objects.

[0307] Image generation module 1005 is used to generate an object image based on the second object features of each of the multiple objects, wherein the object image includes multiple objects with occlusion relationships.

[0308] The image generation apparatus provided in this application, for each of a plurality of objects, can determine the first object feature of the object itself based on the object information of the object. Based on the occlusion relationship between the plurality of objects, the upper-layer object occluding the object can be determined. Since the visibility of the object is determined by the upper-layer object occluding it, the visual features of the object can be determined based on the first object feature of the upper-layer object. Then, by adjusting the first object feature of the object itself based on the visual features of the object, the remaining second object feature after the object is occluded can be obtained. Furthermore, based on the second object features of each of the plurality of objects, an object image including the plurality of objects can be generated, and the object image can present the occlusion relationship between the plurality of objects. Therefore, the solution of this application can accurately control the occlusion relationship between multiple objects in the image generation process, improve the controllability of the occlusion relationship, overcome the limitation of the inability to control the occlusion relationship during the image generation process, and enhance the visual effect of the generated image.

[0309] Optionally, see Figure 11 The object information includes appearance information and location information; the first feature processing module 1002 includes:

[0310] The first extraction unit 1012 is used to extract the appearance features of an object based on its appearance information.

[0311] The second extraction unit 1022 is used to extract the projection features of the object based on the object's location information;

[0312] The first fusion unit 1032 is used to fuse the appearance features and projection features of the object to obtain the first object feature of the object.

[0313] Optionally, see Figure 11 The device executes through a first image generation model, which includes M first feature processing networks connected in sequence, where M is an integer greater than 1; the first extraction unit 1012 is used for:

[0314] The first appearance feature of the object is obtained by performing cross-attention encoding on the noise feature and the appearance information of the object through the first first feature processing network among the M first feature processing networks;

[0315] By using the i-th first feature processing network among the M first feature processing networks, cross-attention encoding is performed on the (i-1)-th image feature and the appearance information of the object to obtain the i-th appearance feature of the object, where i is an integer greater than 1 and not greater than M;

[0316] The (i-1)th image feature is obtained by fusing the (i-1)th second object features of multiple objects. The (i-1)th second object feature of an object is obtained through the (i-1)th first feature processing network in M ​​first feature processing networks.

[0317] Optionally, see Figure 11 The first extraction unit 1012 is used for:

[0318] The i-th first feature processing network maps the (i-1)-th image feature to the i-th query feature.

[0319] The appearance information of an object is mapped to the i-th key feature and the i-th value feature through the i-th first feature processing network;

[0320] By using the i-th first feature processing network, the i-th query feature, the i-th key feature, and the i-th value feature are fused to obtain the i-th appearance feature.

[0321] Optionally, see Figure 11 The second extraction unit 1022 is used for:

[0322] Convert the object's location information into a location mask feature;

[0323] The i-th attention feature is obtained by multiplying the transpose of the i-th query feature and the i-th key feature through the i-th feature processing network.

[0324] The i-th attention feature and the location mask feature are multiplied by the i-th feature processing network to obtain the i-th projection feature of the object.

[0325] Optionally, see Figure 11 The first fusion unit 1032 is used for:

[0326] The i-th appearance feature and the i-th projection feature of the object are multiplied by the i-th feature processing network to obtain the i-th first object feature of the object.

[0327] Optionally, see Figure 11 The second feature processing module 1003 includes:

[0328] The first processing unit 1013 is used to determine the occlusion features of each upper object of the object based on the opacity features of the upper object and the first object features of the upper object. The occlusion features of the upper object represent the occlusion situation of the upper object.

[0329] The second processing unit 1023 is used to map the occlusion features of at least one upper-level object of the object to the visible features of the object.

[0330] Optionally, see Figure 11 The first processing unit 1013 is used for:

[0331] Based on the opacity features of the upper-level object, determine the semantic density features of the upper-level object;

[0332] Multiplying the semantic density features of the upper-level object with the projection features of the upper-level object yields the occlusion features of the upper-level object.

[0333] Optionally, see Figure 11 The image generation device includes multiple iterative processes. Each iterative process refers to determining the second object features of multiple objects in the current iteration based on the second object features and object information of multiple objects obtained in the previous iteration. The first processing unit 1013 is used for:

[0334] Map the opacity features of the upper-level object to the initial semantic density features of the upper-level object;

[0335] Determine the current iteration count and the total iteration count, and determine the first weighting parameter, which is negatively correlated with the current iteration count;

[0336] The initial semantic density features of the upper-level object are weighted based on the weighting parameters to obtain the semantic density features of the upper-level object.

[0337] Optionally, see Figure 11 The second processing unit 1023 is used for:

[0338] The global occlusion feature is obtained by adding the occlusion features of at least one of the object's parent objects;

[0339] The visual features of an object are obtained by performing an exponential operation on the negative of the global occlusion feature.

[0340] Optionally, see Figure 11 The first image generation model includes M first feature processing networks connected in sequence, where M is an integer greater than 1;

[0341] The second feature processing module 1003 is used to: determine the i-th visible feature of the object based on the i-th first object feature of the object's upper-level object through the i-th first feature processing network in the M first feature processing networks, where i is an integer greater than 1 and not greater than M;

[0342] The third feature processing module 1004 is used to: adjust the i-th first object feature of the object according to the i-th visible feature of the object through the i-th first feature processing network to obtain the i-th second object feature of the object, wherein the i-th first object feature is obtained through the i-th feature processing network.

[0343] Optionally, see Figure 11 Image generation module 1005 includes:

[0344] The second fusion unit 1015 is used to fuse the second object features of multiple objects to obtain image features;

[0345] The decoding unit 1025 is used to perform decoding operations on image features to obtain the object image.

[0346] Optionally, see Figure 11 The second fusion unit 1015 is used for:

[0347] For each of the multiple objects, the semantic density feature of the object is determined based on the object's opacity feature, and the semantic density feature of the object is mapped to a second weighting parameter, which is positively correlated with the semantic density feature of the object.

[0348] Based on the second weighting parameters of each of the multiple objects, a weighted fusion operation is performed on the second object features of each of the multiple objects to obtain image features.

[0349] Optionally, see Figure 11 The second fusion unit 1015 is used for:

[0350] Based on the second weighting parameters of each of the multiple objects, the second object features of each of the multiple objects are weighted and summed to obtain the initial image features;

[0351] The normalization parameter is determined based on the projection features, visual features, and second weighted parameters of multiple objects.

[0352] The initial image features are multiplied by the normalization parameters to obtain the image features.

[0353] Optionally, see Figure 11 The second fusion unit 1015 is used for:

[0354] For each of the multiple objects, the object's projective features, visual features, and second weighted parameters are multiplied to obtain the object's adjusted projective features;

[0355] The adjusted projection features of multiple objects are added together to obtain the global projection feature;

[0356] The reciprocal of the global projected feature is used as the normalization parameter.

[0357] Optionally, see Figure 11 The first image generation model includes M first feature processing networks and decoding networks connected in sequence, where M is an integer greater than 1;

[0358] Image generation module 1005 is used to: fuse the i-th second object features of multiple objects through the i-th first feature processing network in M ​​first feature processing networks to obtain the i-th image feature, where i is an integer greater than 1 and not greater than M, and the i-th second object feature is obtained through the i-th feature processing network;

[0359] The image generation module 1005 is used to: perform a decoding operation on the Mth image feature through a decoding network to obtain the object image.

[0360] Optionally, see Figure 11 The device also includes:

[0361] The model training module 1006 is used to acquire sample object information and sample label images. The sample label images include the objects indicated by the sample object information.

[0362] The model training module 1006 is also used to generate a model from the second image and generate a sample target image based on the sample object information.

[0363] The model training module 1006 is also used to train the second image generation model based on the sample target image and the sample label image to obtain the trained second image generation model.

[0364] Model building module 1007 is used to build a first image generation model based on the trained second image generation model.

[0365] Optionally, see Figure 11 The second image generation model includes a second feature processing network and a decoding network, the second feature processing network including a cross-attention layer; model building module 1007 is used for:

[0366] Based on the cross-attention layer in the second feature processing network, M first feature processing networks are constructed, where M is an integer greater than 1;

[0367] The second feature processing network in the trained second image generation model is replaced with M first feature processing networks to obtain the first image generation model.

[0368] It should be noted that the image generation apparatus provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image generation apparatus and the image generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0369] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the image generation method of the above embodiments.

[0370] Optionally, the computer device is provided as a terminal. Figure 12 A schematic diagram of the structure of a terminal 1200 provided in an exemplary embodiment of this application is shown.

[0371] Terminal 1200 includes a processor 1201 and a memory 1202.

[0372] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0373] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one computer program, which is used by the processor 1201 to implement the image generation method provided in the method embodiments of this application.

[0374] In some embodiments, the terminal 1200 may also optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, and a power supply 1208.

[0375] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0376] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0377] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of terminal 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of terminal 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0378] The camera assembly 1206 is used to acquire images or videos. Optionally, the camera assembly 1206 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1200, and the rear-facing camera is disposed on the back of the terminal 1200. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0379] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1201 for processing, or input to the radio frequency circuit 1204 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1200. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0380] Power supply 1208 is used to power the various components in terminal 1200. Power supply 1208 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1208 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0381] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on terminal 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0382] Optionally, the computer device is provided as a server. Figure 13This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memories 1302 store at least one computer program, which is loaded and executed by the processor 1301 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0383] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the image generation method of the above embodiments.

[0384] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the image generation method of the above embodiments.

[0385] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0386] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. An image generation method, characterized in that, The method includes: Obtain object information of multiple objects and occlusion relationships between the multiple objects; based on the occlusion relationships, determine the parent objects of each of the multiple objects; the object information of the objects is used to describe the objects; the parent objects of the objects refer to other objects that occlude the objects. Based on the object information of the multiple objects, extract the first object feature of each of the multiple objects; Based on the first object features of the parent objects of each of the plurality of objects, the visual features of each of the plurality of objects are determined, and the visual features of the objects represent the visibility of the objects; Based on the visual features of each of the multiple objects, the first object features of each of the multiple objects are adjusted to obtain the second object features of each of the multiple objects; An object image is generated based on the second object features of each of the plurality of objects, the object image including the plurality of objects having the occlusion relationship.

2. The method according to claim 1, characterized in that, The object information includes appearance information and location information; the step of extracting the first object feature of each of the multiple objects based on the object information of the multiple objects includes: Based on the appearance information of the object, extract the appearance features of the object; Based on the object's location information, extract the object's projection features; By fusing the appearance features and projection features of the object, a first object feature of the object is obtained.

3. The method according to claim 2, characterized in that, The method is executed through a first image generation model, which includes M first feature processing networks connected in sequence, where M is an integer greater than 1; the step of extracting the appearance features of the object based on its appearance information includes: The first feature processing network in the M first feature processing networks performs cross-attention encoding on the noise features and the appearance information of the object to obtain the first appearance feature of the object; Cross-attention encoding is performed on the (i-1)th image feature and the appearance information of the object through the i-th first feature processing network among the M first feature processing networks to obtain the i-th appearance feature of the object, where i is an integer greater than 1 and not greater than M; The (i-1)th image feature is obtained by fusing the (i-1)th second object features of each of the multiple objects, and the (i-1)th second object feature of the object is obtained through the (i-1)th first feature processing network in the M first feature processing networks.

4. The method according to claim 3, characterized in that, The step of performing cross-attention encoding on the (i-1)th image feature and the appearance information of the object through the i-th first feature processing network among the M first feature processing networks to obtain the i-th appearance feature of the object includes: The i-th first feature processing network maps the (i-1)-th image feature to the i-th query feature. The appearance information of the object is mapped to the i-th key feature and the i-th value feature through the i-th first feature processing network; The i-th appearance feature is obtained by fusing the i-th query feature, the i-th key feature, and the i-th value feature through the i-th first feature processing network.

5. The method according to claim 4, characterized in that, The step of extracting the projection features of the object based on its location information includes: Convert the location information of the object into a location mask feature; The i-th attention feature is obtained by multiplying the transpose of the i-th query feature and the i-th key feature through the i-th feature processing network. The i-th attention feature and the location mask feature are multiplied by the i-th feature processing network to obtain the i-th projection feature of the object.

6. The method according to claim 5, characterized in that, The process of fusing the appearance features and projection features of the object to obtain the first object feature includes: The i-th appearance feature and the i-th projection feature of the object are multiplied by the i-th feature processing network to obtain the i-th first object feature of the object.

7. The method according to claim 1, characterized in that, The step of determining the visual features of each of the plurality of objects based on the first object features of their respective parent objects includes: For each parent object of the object, an occlusion feature of the parent object is determined based on the opacity feature of the parent object and the first object feature of the parent object. The occlusion feature of the parent object represents the occlusion status of the parent object. Map the occlusion features of at least one upper-level object of the object to the visible features of the object.

8. The method according to claim 7, characterized in that, The step of determining the occlusion features of the upper-layer object based on the opacity feature and the first object feature of the upper-layer object includes: Based on the opacity feature of the upper-layer object, determine the semantic density feature of the upper-layer object; The occlusion feature of the upper-layer object is obtained by multiplying its semantic density feature with its projection feature.

9. The method according to claim 8, characterized in that, The image generation method includes multiple iterative processes. Each iterative process refers to determining the second object features of the multiple objects in the current iteration based on the second object features and object information of the multiple objects obtained in the previous iteration. The step of determining the semantic density features of the upper-layer object based on its opacity features includes: Map the opacity feature of the upper-layer object to the initial semantic density feature of the upper-layer object; Determine the current iteration count and the total iteration count, and determine the first weighting parameter, which is negatively correlated with the current iteration count; The initial semantic density features of the upper-level object are weighted based on the weighting parameters to obtain the semantic density features of the upper-level object.

10. The method according to claim 7, characterized in that, The step of mapping the occlusion features of at least one upper-layer object of the object to the visible features of the object includes: The occlusion features of at least one of the object's parent objects are added together to obtain the global occlusion features; The visual features of the object are obtained by performing an exponential operation on the negative of the global occlusion feature.

11. The method according to claim 1, characterized in that, The first image generation model includes M first feature processing networks connected in sequence, where M is an integer greater than 1; The step of determining the visual features of each of the multiple objects based on the first object features of their respective parent objects includes: determining the i-th visual feature of the object based on the i-th first object features of the parent objects of the object through the i-th first feature processing network of the M first feature processing networks, where i is an integer greater than 1 and not greater than M; The step of adjusting the first object features of each of the multiple objects according to their respective visual features to obtain the second object features of each of the multiple objects includes: adjusting the i-th first object feature of the object according to the i-th visual feature of the object through the i-th first feature processing network to obtain the i-th second object feature of the object, wherein the i-th first object feature is obtained through the i-th feature processing network.

12. The method according to claim 1, characterized in that, The step of generating an object image based on the second object features of each of the plurality of objects includes: By fusing the second object features of each of the multiple objects, image features are obtained; The image features are decoded to obtain the object image.

13. The method according to claim 12, characterized in that, The process of fusing the second object features of the multiple objects to obtain image features includes: For each of the plurality of objects, based on the opacity feature of the object, the semantic density feature of the object is determined, and the semantic density feature of the object is mapped to a second weighting parameter, wherein the second weighting parameter is positively correlated with the semantic density feature of the object; Based on the second weighting parameters of each of the multiple objects, a weighted fusion operation is performed on the second object features of each of the multiple objects to obtain the image features.

14. The method according to claim 13, characterized in that, The first object feature of the object includes projection features; The step of performing a weighted fusion operation on the second object features of each of the multiple objects based on their respective second weighting parameters to obtain the image features includes: Based on the second weighting parameters of each of the multiple objects, the second object features of each of the multiple objects are weighted and summed to obtain the initial image features; Based on the projection features, the visual features, and the second weighting parameter of the plurality of objects, a normalization parameter is determined; The initial image features are multiplied by the normalization parameter to obtain the image features.

15. The method according to claim 12, characterized in that, The first image generation model includes M first feature processing networks and decoding networks connected in sequence, where M is an integer greater than 1; The step of fusing the second object features of each of the multiple objects to obtain image features includes: fusing the i-th second object features of each of the multiple objects through the i-th first feature processing network of the M first feature processing networks to obtain the i-th image feature, where i is an integer greater than 1 and not greater than M, and the i-th second object feature is obtained through the i-th feature processing network; The step of performing a decoding operation on the image features to obtain the object image includes: performing a decoding operation on the Mth image feature through the decoding network to obtain the object image.

16. The method according to any one of claims 1 to 15, characterized in that, The method is executed through a first image generation model, the training process of which includes: Obtain sample object information and sample label images, wherein the sample label images include the objects indicated by the sample object information; Based on the sample object information, a sample target image is generated using a second image generation model. Based on the sample target image and the sample label image, the second image generation model is trained to obtain the trained second image generation model; The first image generation model is constructed based on the trained second image generation model.

17. An image generation apparatus, characterized in that, The device includes: The acquisition module is used to acquire object information of multiple objects and occlusion relationships between the multiple objects, and based on the occlusion relationships, determine the parent objects of each of the multiple objects. The object information of the object is used to describe the object, and the parent objects of the object refer to other objects that occlude the object. The first feature processing module is used to extract the first object features of each of the multiple objects based on the object information of the multiple objects; The second feature processing module is used to determine the visual features of each of the multiple objects based on the first object features of their respective parent objects, wherein the visual features of the objects represent the visibility of the objects; The third feature processing module is used to adjust the first object features of each of the multiple objects according to their respective visual features to obtain the second object features of each of the multiple objects. An image generation module is used to generate an object image based on the second object features of each of the plurality of objects, wherein the object image includes the plurality of objects having the occlusion relationship.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations of the image generation method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the image generation method as described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations performed by the image generation method as described in any one of claims 1 to 16.

Images