Image generation process optimization method and device, equipment, medium and program product

Abstract

The invention discloses an optimization method and device for an image generation process, equipment, a medium and a program product, and belongs to the field of artificial intelligence. The method comprises: in a process of generating a scene image through an image generation model, obtaining an operation vector participating in a first attention operation, the first attention operation being an attention operation executed based on a first image feature and a first text feature, the first image feature being an image feature generated in an operation process of the image generation model, and the first text feature being an image feature generated in an operation process of the image generation model; the first text feature is obtained based on the text feature of the first description text; and executing a region modulation operation on the operation vector of the first attention operation, wherein the region modulation operation is used for enhancing data corresponding to a scene region in the first design sketch in the operation vector. Through the image generation process optimized by the method, the generated scene image can accurately show the semantic category of the scene area in the first design sketch.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an optimization method, apparatus, device, medium, and program product for an image generation process. Background Technology

[0002] With the development of artificial intelligence technology, more and more generative models can generate scene images with diverse styles and rich content, such as scene images with urban themes, scene images with natural themes, and so on.

[0003] Related technologies include the Stable Diffusion model (a generative diffusion model for image generation), which can generate images that conform to the input text description. However, images generated by the Stable Diffusion model still suffer from viewpoint uncertainty. Related technologies also provide the ControlNet model, which is an improvement on the Stable Diffusion model. In addition to accepting text descriptions as constraints for image generation, the ControlNet model can also generate scene images that conform to the overall and partial composition of the input design sketch.

[0004] However, although the ControlNet model can respond to the contour features in the design sketch to a high degree, the generated scene image still cannot accurately represent the semantic categories on the design sketch. For example, a river in the design sketch may be mistakenly identified as a road, and a house may be mistakenly identified as a lake. Summary of the Invention

[0005] This application provides an optimization method, apparatus, device, medium, and program product for image generation. The optimized image generation process provided by this application results in a scene image that accurately represents the semantic category of the scene region in the first design sketch. The technical solution includes the following:

[0006] According to one aspect of this application, an optimization method for an image generation process is provided, the method comprising the following steps.

[0007] In the process of generating scene images through the image generation model, the operation vectors participating in the first attention operation are obtained. The first attention operation is an attention operation performed based on the first image features and the first text features. The first image features are image features generated during the operation of the image generation model, and the first text features are obtained based on the text features of the first descriptive text.

[0008] A region modulation operation is performed on the operation vector of the first attention operation. The region modulation operation is used to enhance the data corresponding to the scene region in the first design sketch in the operation vector.

[0009] The input to the image generation model includes a first descriptive text and a first design sketch. The first descriptive text describes the scene of the scene image to be generated, and the first design sketch is a sketch of the scene image to be generated.

[0010] According to another aspect of this application, an optimization apparatus for an image generation process is provided, the apparatus comprising the following modules.

[0011] The acquisition module is used to acquire the operation vectors participating in the first attention operation during the process of generating scene images through the image generation model. The first attention operation is an attention operation performed based on the first image features and the first text features. The first image features are image features generated during the operation of the image generation model, and the first text features are obtained based on the text features of the first descriptive text.

[0012] The modulation module is used to perform a region modulation operation on the operation vector of the first attention operation. The region modulation operation is used to enhance the data corresponding to the scene region in the first design sketch in the operation vector.

[0013] The input to the image generation model includes a first descriptive text and a first design sketch. The first descriptive text describes the scene of the scene image to be generated, and the first design sketch is a sketch of the scene image to be generated.

[0014] According to one aspect of this application, a computer device is provided, comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the optimized method of the image generation process as described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided, which stores a computer program that is loaded and executed by a processor to implement an optimized method for the image generation process described above.

[0016] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the optimized method of the image generation process described above.

[0017] The beneficial effects of the technical solutions provided in this application include at least the following:

[0018] In this application, by enhancing the data corresponding to the scene region in the first design sketch in the operation vector (the vector participating in the first attention operation), the data in non-scene regions is suppressed. Therefore, during the execution of the first attention operation, the model can accurately correspond the scene region in the first design sketch to the region description in the first descriptive text without interference from data in non-scene regions. Consequently, the generated scene image accurately represents the semantic category of the scene region in the first design sketch. Furthermore, when the image generation model is pre-trained, the image generation method supported by this application does not require retraining the pre-trained model; it only needs to perform region modulation operations on the first attention operation during inference. This improves image quality without consuming additional training resources. In addition, it fully utilizes the generalization ability of the pre-trained model, eliminating the need to consider overfitting on the inference data. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an image generation method provided in one embodiment of this application.

[0021] Figure 2 This is a schematic diagram of a region modulation operation provided in one embodiment of this application.

[0022] Figure 3 This is a flowchart of an optimized method for image generation process provided in one embodiment of this application.

[0023] Figure 4 This is a schematic diagram of a region modulation operation provided in another embodiment of this application.

[0024] Figure 5 This is a schematic diagram of an image generation method provided in one embodiment of this application.

[0025] Figure 6 This is a schematic diagram of a feature sharing method provided in one embodiment of this application.

[0026] Figure 7 This is a structural block diagram of an image generation process optimization device provided in one embodiment of this application.

[0027] Figure 8This is a structural block diagram of a computer device provided in one embodiment of this application.

[0028] Figure 9 This is a structural block diagram of a computer device provided in another embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described below with reference to the accompanying drawings.

[0030] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0031] Stable Diffusion (SDP) is a diffusion-based model for image generation. During image generation, SDP gradually removes noise from the original noisy image until a clear image is formed. Furthermore, SDP uses user-input text as a constraint, ensuring that the final generated image matches the user's input text description. For example, the user's input text description might be "Generate a terrain map of a scene with grass surrounding a lake in the center."

[0032] However, while the Stable Diffusion model can generate relatively detailed images based on user-input text descriptions, the camera viewpoint of the generated images is random. That is, the Stable Diffusion model cannot guarantee that the lake is located in the exact center of the grassland in the generated image. For a more detailed introduction to the Stable Diffusion model, please refer to the paper "High-resolution image synthesis with latent diffusion models."

[0033] The ControlNet model is a neural network model used to control generative diffusion models, similar to the Stable Diffusion model. Building upon the Stable Diffusion model, ControlNet adds a design sketch as a constraint. Users can input text descriptions and design sketches, constraining the ControlNet model to generate images that conform to both the text description and roughly satisfy the overall and local composition of the design sketch. Related technologies include various pre-trained models, each corresponding to different design sketches as constraints, such as edge maps, depth maps, line drawings, and semantic intent maps. However, while the ControlNet model can respond to contour features in the design sketch to a high degree, it still cannot accurately distinguish semantic categories in the design sketch. For example, a river in the design sketch might be incorrectly identified as a road, and a house might be incorrectly identified as a lake. For a more detailed introduction to the ControlNet model, please refer to the paper "Adding conditional control to text-to-image diffusion models."

[0034] Attention mechanism: Proposed by the paper "Attention Is All You Need", in a broad sense, the attention mechanism simulates the attention behavior of the human brain when reading and visualizing. When processing information, the human brain selectively focuses on the information that is worth paying attention to and ignores other information.

[0035] In related technologies, attention operations are based on Query (often abbreviated as Q, query vector), Key (often abbreviated as K, key vector), and Value (often abbreviated as V, value vector). Typically, attention operations are performed based on Q and K to calculate an attention map, which represents the degree of correlation between Q and K. Then, V is weighted based on the attention map to obtain the attention operation result.

[0036] The first attention operation refers to the cross-attention operation between the first text features and the first image features during the generation of scene images by the image generation model (optionally, the ControlNet model). The first text features are obtained based on the text features of the first descriptive text, and the first image features are the image features generated during the operation of the image generation model. During the operation of the image generation model, at least one first attention operation will be performed. The first image features used in each first attention operation are the image features generated by the image generation model during its operation, and the text features used in each first attention operation are the text features of the first descriptive text.

[0037] First design sketch: refers to a sketch of the scene image to be generated. The image generation model of this application supports generating scene images that conform to the first design sketch under the constraints of the first design sketch. Schematic, the first design sketch includes rivers, houses, paths, and hillsides, etc., of the scene image to be generated.

[0038] First descriptive text: Used to describe the scene to be generated. The image generation model of this application supports generating scene images that conform to the first descriptive text under the constraints of the first descriptive text. Schematic, the first descriptive text is "Generate scene images with rivers, houses, paths, and hillsides".

[0039] Image generation model: refers to a model capable of generating scene images based on a first design sketch and a first descriptive text. The scene images generated by the image generation model satisfy the scene description information indicated by the first descriptive text and conform to the overall and local composition on the first design sketch. Optionally, the image generation model is a generative diffusion model. Optionally, the image generation model is a ControlNet model.

[0040] Figure 1 A schematic diagram of an image generation method is shown. A first design sketch 101 and a first descriptive text 102 are obtained. These are then input into an image generation model 103, which generates a scene image 104. The first design sketch 101 is a sketch of the expected scene image 104, and the first descriptive text 102 describes the scene of the expected scene image 104.

[0041] like Figure 1 As shown, the first design sketch 101 illustrates the outlines of four scene regions: "hillside," "river," "path," and "house." The first descriptive text 102 is "Generate a scene image with a river, house, path, and hillside." Therefore, the scene image 104 generated by the image generation model 103 will not only include the river, house, path, and hillside, but the locations of the river, house, path, and hillside will also conform to the overall and local composition shown in the first design sketch 101. Optionally, the image generation model 103 is a generative diffusion model; alternatively, the image generation model 103 is a ControlNet model.

[0042] During the process of generating scene image 104 by image generation model 103, a first attention operation is performed. This first attention operation is a cross-attention operation based on the first image feature and the first text feature. The first image feature is the image feature generated during the operation of the image generation model, and the first text feature is the text feature of the first descriptive text 102. Image generation model 103 continuously generates image features during its operation, with subsequent image features obtained based on previous image features.

[0043] In the optimization method provided in this application, a region modulation operation is performed on the computation vectors participating in the first attention operation. The region modulation operation is used to enhance the data corresponding to the scene region in the first design sketch 101 in the computation vectors. The computation vector refers to the vector participating in the first attention operation. Optionally, the computation vector includes at least one of the query vector, key vector, value vector, and attention graph of the first attention operation.

[0044] Reference Figure 2 , Figure 2 This illustration shows a schematic diagram of performing a region modulation operation on the operation vector of the first attention operation according to an exemplary embodiment of this application. Specifically, Figure 2 A schematic diagram is shown showing the application of region modulation to both the attention map and the value vector.

[0045] Without performing region modulation on the first attention operation, a query vector is obtained based on the first image feature 201; a key vector is obtained based on the first text feature 202; and an attention map 203 is obtained based on the query vector and the key vector. Figure 2 (The complete process is not shown); and, based on the first text feature 202, a value vector 205 is obtained; based on the attention map 203 and the value vector 205, the attention operation result is obtained.

[0046] When performing region modulation on the first attention operation, a query vector is obtained based on the first image feature 201; a key vector is obtained based on the first text feature 202; and an attention map 203 is obtained based on the query vector and the key vector. Figure 2 (The complete process is not shown); Perform a region modulation operation on attention map 203 to obtain the modulated attention map 204;

[0047] Furthermore, a value vector 205 is obtained based on the first text feature 202. A region modulation operation is performed on the value vector 205 to obtain a modulated value vector 206. Based on the modulated attention map 204 and the modulated value vector 206, a modulated attention operation result 207 is obtained. The modulated attention operation result 207 will replace the attention operation result obtained without performing the region modulation operation and participate in the subsequent image generation process.

[0048] In this application, by enhancing the data corresponding to the scene region in the first design sketch in the operation vector (the vector participating in the first attention operation), the data in non-scene regions is suppressed. Therefore, during the execution of the first attention operation, the model can accurately correspond the scene region in the first design sketch to the region description in the first descriptive text without interference from data in non-scene regions. Consequently, the generated scene image accurately represents the semantic category of the scene region in the first design sketch. Furthermore, when the image generation model is pre-trained, the image generation method supported by this application does not require retraining the pre-trained model; it only needs to perform region modulation operations on the first attention operation during inference. This improves image quality without consuming additional training resources. In addition, it fully utilizes the generalization ability of the pre-trained model, eliminating the need to consider overfitting on the inference data.

[0049] In one embodiment, the optimized image generation process provided in this application is executed by a computer device, which includes at least one of a terminal and a server. The terminal device types include at least one of the following: smartphones, smartwatches, in-vehicle terminals, wearable devices, smart TVs, tablets, e-book readers, MP3 players, MP4 players, laptops, and desktop computers. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, and aircraft.

[0050] In some embodiments, the server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms.

[0051] It should be noted that the information (including but not limited to user device information, image generation models, descriptive text, design sketches), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0052] Furthermore, when dealing with relevant information, the relevant information processor will adhere to the principles of legality, legitimacy, and necessity, clearly define the purpose, method, and scope of the relevant information processing, obtain the consent of the relevant information subject, and take necessary technical and organizational measures to ensure the security of the relevant information.

[0053] Figure 3 A flowchart illustrating an optimized method for an image generation process provided in an exemplary embodiment of this application is shown, illustratively described as being performed by a computer device. The method includes:

[0054] Step 320: In the process of generating scene images through the image generation model, obtain the operation vectors that participate in the first attention operation;

[0055] In the image generation method supported by this application, a first design sketch and a first descriptive text are obtained, and then input into an image generation model to generate a scene image. During the operation of the image generation model, a first attention operation is performed.

[0056] The first design sketch refers to a sketch of the scene image to be generated. The first design sketch depicts the scene area of ​​the scene to be generated. Optionally, the first design sketch includes the overall and / or partial outlines of the scene image to be generated. Optionally, the scene area on the first design sketch can be a natural environment (such as a river, hillside, path), buildings (such as houses, bridges), human figures, animals, various objects, etc. Illustratively, the first design sketch depicts a river, houses, paths, and hillsides of the scene image to be generated.

[0057] Typically, the initial design sketch is not refined, only showing a blurry outline of the scene area. The purpose of this image generation is to produce a more polished image. In other cases, the initial design sketch can also be a well-crafted image, and the purpose of this image generation is to refine and polish that image.

[0058] In some embodiments, the first design sketch may be an image drawn by the user, an image obtained by the user from an image library, an image taken by the user, or an image generated by an artificial intelligence model. This application does not limit the acquisition method of the first design sketch. Optionally, the first design sketch may be at least one of edge map, depth map, line drawing, and semantic map.

[0059] In some embodiments, an image optimization operation is performed on the first design sketch, and the optimized first design sketch is input into the image generation model. The purpose of the image optimization operation is to improve the distinction between various scene regions on the first design sketch. Optionally, the image optimization operation includes removing redundant lines on the first design sketch, coloring various scene regions on the first design sketch, filling different shadows, etc. Optionally, the image optimization operation includes adjusting image contrast, brightness, scaling, etc. Typically, the first design sketch and the first descriptive text are provided by the user; therefore, there will be no major conflicts between the first design sketch and the first descriptive text. During image generation, the first design sketch and the first descriptive text can be input into the image generation model.

[0060] The first descriptive text describes the scene to be generated in the scene image. Optionally, the language of the first descriptive text can be Chinese, English, German, or other languages. In the image generation model of this application, the first descriptive text serves as a prompt to constrain the image generation model, ensuring that the image generated by the model conforms to the scene description information indicated by the first descriptive text. For example, the first descriptive text might be "Generate a scene image with a river, houses, paths, and hillsides."

[0061] The first attention operation is an attention operation performed based on the first image feature and the first text feature. The first image feature is the image feature generated during the operation of the image generation model, and the first text feature is obtained based on the text feature of the first descriptive text. In some embodiments, the image generation model continuously generates image features during its operation, with subsequent image features generated based on previous image features. The image generation model performs feature decoding based on the last generated image feature to output the final scene image.

[0062] Optionally, the first text feature is the text feature of the first descriptive text.

[0063] In some embodiments, the image generation model performs multiple first attention operations during its operation. For any one of these first attention operations, the attention operation outputs image features based on the input image features (first image features) and first text features (text features of the first descriptive text). Optionally, the image generation model is a generative diffusion model. The idea behind generative diffusion model image generation is to continuously remove noise from the original noisy image through inference at multiple time steps, given an original noisy image, and finally generate a clear image that meets the constraints. During the inference process at each time step, an Encoder-Decoder structure is also included. The Encoder structure includes multiple Encoder blocks, and the Decoder structure includes multiple Decoder blocks. At least one first attention operation is performed in each Encoder block, and at least one first attention operation is performed in each Decoder block.

[0064] Step 340: Perform a region modulation operation on the operation vector of the first attention operation. The region modulation operation is used to enhance the data corresponding to the scene region in the first design sketch in the operation vector.

[0065] During the process of generating scene images using the image generation model, a first attention operation is performed. This application performs a region modulation operation on the operation vector of the first attention operation, which is used to enhance the data corresponding to the scene region in the first design sketch within the operation vector.

[0066] The operation vector refers to the vector that participates in the first attention operation. Optionally, the operation vector includes at least one of the query vector, key vector, value vector, and attention map of the first attention operation.

[0067] In one embodiment, enhancing the data corresponding to the scene region in the first design sketch refers to the operation of distinguishing the data corresponding to the scene region in the first design sketch from other data. Optionally, enhancing the data corresponding to the scene region in the first design sketch includes at least one of the following operations: increasing the data corresponding to the scene region in the first design sketch, decreasing the data corresponding to the scene region in the first design sketch, increasing other data besides the data corresponding to the scene region in the first design sketch, and decreasing other data besides the data corresponding to the scene region in the first design sketch.

[0068] In one embodiment, the image generation model performs multiple first attention operations during its operation. Optionally, region modulation is performed on each of the multiple first attention operations. In this case, the generated scene image will achieve the best effect, but the inference process will consume more computational resources. Optionally, region modulation is performed on some of the multiple first attention operations. This can save some computational resources. The specific first attention operations to be performed with region modulation can be determined through experiments and conclusion analysis.

[0069] In this application, by enhancing the data corresponding to the scene region in the first design sketch in the operation vectors participating in the first attention operation, the data in non-scene regions is suppressed. Therefore, during the execution of the first attention operation, the model can accurately correspond the scene region in the first design sketch to the region description in the first descriptive text without interference from non-scene regions. Consequently, the generated scene image accurately represents the semantic category of the scene region in the first design sketch. Furthermore, when the image generation model is pre-trained, the image generation method supported by this application does not require retraining the pre-trained model; it only needs to perform region modulation operations on the first attention operation. This improves image quality without consuming additional training resources. In addition, it fully utilizes the generalization ability of the pre-trained model, eliminating the need to consider overfitting on inference data.

[0070] based on Figure 3 In the illustrated method embodiment, step 340 includes: performing a region modulation operation on the attention map and value vector of the first attention operation, wherein the attention map is obtained based on the first image features and the first text features, and the value vector is obtained based on the first text features.

[0071] In the image generation methods supported by this application, such as Figure 4 As shown, when performing the first attention operation, the query vector 403 for the first attention operation is obtained based on the first image features (image features generated during the operation of the image generation model) 401 and the pre-training parameters W1; the key vector 404 for the first attention operation is obtained based on the first text features (text features obtained based on the first descriptive text) 402 and the pre-training parameters W2; the attention map 406 for the first attention operation is obtained based on the query vector 403 and the key vector 404; and a region modulation operation is performed on the attention map 406 to obtain the modulated attention map 407.

[0072] Based on the first text feature 402 and the pre-trained parameter W3, the value vector 405 of the first attention operation is obtained. The value vector 405 is subjected to a region modulation operation to obtain the modulated value vector 408. Based on the modulated attention map 407 and the modulated value vector 408, the modulated attention operation result 409 is obtained.

[0073] Among them, the pre-trained parameters W1, W2, and W3 are the parameters that have been trained in the image generation model.

[0074] Region modulation operations performed on the attention map:

[0075] In one embodiment, the attention map of the first attention operation includes multiple semantic dimensions, each of which corresponds to a scene region. For the i-th semantic dimension, an i-th modulation operation is performed in the i-th semantic dimension. The i-th modulation operation includes at least one of enhancing the data corresponding to the i-th scene region in the first design sketch and suppressing the data corresponding to regions other than the i-th scene region. The i-th scene region is a scene region in the first design sketch, and the i-th semantic dimension is the semantic dimension corresponding to the i-th scene region.

[0076] like Figure 2 As shown, the attention map 203 has a shape of H×W×S, where H is the high dimension, W is the wide dimension, and S is the semantic dimension. In each semantic dimension's H×W region, the data corresponding to a scene region in the first design sketch is enhanced to obtain the modulated attention map 204. For example... Figure 2 As shown, S equals four. In the first semantic dimension, the data corresponding to "river" in the first design sketch is enhanced. In the second semantic dimension, the data corresponding to "road" in the first design sketch is enhanced. In the third semantic dimension, the data corresponding to "house" in the first design sketch is enhanced. In the fourth semantic dimension, the data corresponding to "tree" in the first design sketch is enhanced.

[0077] In one embodiment, the modulated attention map has a shape of H×W×S, and the value vector has a shape of S×C, where C is the channel dimension. Based on the modulated attention map and the value vector, the modulated attention operation result is obtained, and the shape of the modulated attention operation result is H×W×C. Figure 2 As shown, Figure 2 The channel dimensions are distinguished by different shades. It can be seen that each semantic dimension in the modulated attention map corresponds to the data of only one scene region. This makes the response of each scene region in the modulated attention operation result 207 reflected on the correct channel dimension.

[0078] In the above embodiments, the attention map represents the correlation between the first text features and the first image features. By performing a region modulation operation on the attention map, the correlation values ​​corresponding to the scene regions in the attention map are enhanced, while the correlation values ​​corresponding to the non-scene regions are suppressed. Therefore, when the value vector (obtained based on the text features of the first descriptive text) is weighted based on the attention map, the model can accurately correspond the correlation values ​​of the scene regions in the attention map to the region descriptions in the first descriptive text without being disturbed by the correlation values ​​of the non-scene regions. As a result, the final generated scene image can accurately represent the semantic category of the scene regions in the first design sketch.

[0079] Furthermore, in this embodiment, only one scene region's data will be enhanced in each semantic dimension of the attention map. That is, different scene regions on the first design sketch will be divided into different semantic dimensions for modulation enhancement, so that different scene regions on the first design sketch will be distinguished. The model can accurately correspond different scene regions with different region descriptions in the first descriptive text, so that the final generated scene image will not incorrectly represent one scene region as the semantic category of other scene regions.

[0080] Regional modulation operation performed on the value vector:

[0081] In one embodiment, the value vector of the first attention operation includes multiple channel dimensions. For each of the multiple channel dimensions, a distribution discretization operation is performed in each channel dimension. The distribution discretization operation includes at least one of increasing n maxima and decreasing m minima, where n and m are both positive integers.

[0082] Based on prior experiments and related analyses, it can be seen that in each channel dimension of the value vector, the maxima and minima often correspond to the scene regions in the first design sketch. Therefore, increasing the maxima and decreasing the minima is equivalent to enhancing the data corresponding to the scene regions in the first design sketch.

[0083] like Figure 2 As shown, the value vector 205 has a shape of S×C, where S represents the semantic dimension and C represents the channel dimension, which is optional. Figure 2 The value vector 205 shown has a shape of 4×4.

[0084] In this embodiment, the distribution discretization operation can increase the degree of distribution dispersion in the value vector and suppress the interference of values ​​other than the maximum and minimum values, that is, suppress the interference of data in non-scene areas. This enables the model to accurately correspond the region description in the first description text with the scene area in the first design sketch.

[0085] In this embodiment, the region description in the value vector (obtained based on the text features of the first descriptive text) will be enhanced, and correspondingly, the non-region description will be suppressed. Thus, when the value vector is weighted based on the attention map, the model can accurately correspond the region description in the value vector with the scene region data in the attention map without being disturbed by the non-region description. As a result, the final generated scene image can accurately represent the semantic category of the scene region in the first design sketch.

[0086] Figure 5 A schematic diagram of an image generation method is shown. A first design sketch 501 and a first descriptive text 502 are acquired and input into an image generation model 503. The image generation model 503 includes a first image generation channel 504 and a second image generation channel 505. The first image generation channel 504 generates a scene concept art 506 based on the acquired first design sketch 501 and the first descriptive text 502 as constraints. The second image generation channel 505 generates an empty topographic map 507 based on the acquired first design sketch 501 and the first descriptive text 502 as constraints. Optionally, the image generation model 503 is a ControlNet model with two image generation channels.

[0087] During the process of generating scene concept art 506 using the first image generation channel 504 and generating empty terrain map 507 using the second image generation channel 505, the attention maps of the first image generation channel 504 and the second image generation channel 505 are shared during the initial stage of model inference. The initial stage of model inference refers to the period when the number of inference steps of the model has not exceeded the step threshold. Assuming the step threshold is K, where K is a positive integer, K can optionally be 8.

[0088] A scene concept art refers to an image that contains all the scene elements indicated in the initial description text. An empty terrain map refers to an image that reserves the positions of the scene elements indicated in the initial description text. For example, if the initial description text is "generate a scene image with rivers, houses, paths, and hills," then the scene concept art will include rivers, houses, paths, and hills, while the empty terrain map will reserve the positions of rivers, houses, paths, and hills, but will not display or will not fully display them.

[0089] In this application embodiment, a method for generating scene concept art and empty terrain map is provided. The scene concept art and empty terrain map can achieve regional module consistency. Regional module consistency means that, apart from the scene elements indicated by the first descriptive text, other parts of the scene concept art and empty terrain map are consistent.

[0090] Attention Graph Sharing:

[0091] In this embodiment, the first image generation channel and the second image generation channel each need to perform multiple time-step inference. In the first image generation channel, each time-step inference removes some noise from the original noisy image. After multiple time-step inference, the first image generation channel can output a clear original scene image. In the second image generation channel, each time-step inference removes some noise from the original noisy image. After multiple time-step inference, the second image generation channel can output a clear empty terrain map.

[0092] In one embodiment, the first image generation channel will perform P first attention operations in each time step of inference, each first attention operation involving the generation of an attention map. Similarly, the second image generation channel will also perform P first attention operations in each time step of inference, each first attention operation involving the generation of an attention map.

[0093] Please refer to Figure 6 , Figure 6 A schematic diagram showing the first image generation channel and the second image generation channel sharing an attention map is shown. Figure 6 The example used is that each time step involves only one first attention operation. Figure 6 In the first step of inference, the first image generation channel generates attention map a, the second image generation channel generates attention map A, and based on attention map a and attention map A, attention map a' to be used in the first image generation channel is generated, and based on attention map a and attention map A, attention map A' to be used in the second image generation channel is generated.

[0094] In the second inference step, the first image generation channel generates attention map b, the second image generation channel generates attention map B, and based on attention map b and attention map B, attention map b' to be used in the first image generation channel is generated, and based on attention map b and attention map B, attention map B' to be used in the second image generation channel is generated.

[0095] And so on, without further explanation.

[0096] When performing the K+1 step of inference, attention map sharing is no longer performed. The attention map d generated by the first image generation channel is used as the attention map to be used in the first image generation channel, and the attention map D generated by the second image generation channel is used as the attention map to be used in the second image generation channel.

[0097] In this embodiment, the first image generation channel and the second image generation channel share a feature (attention map). For the first image generation channel, useful features for generating the empty topographic map can be utilized, enabling the second image generation channel to generate a higher quality empty topographic map. Similarly, useful features for generating the scene original image can be utilized, enabling the first image generation channel to generate a higher quality scene original image. Furthermore, the attention map is generated based on the first image features and the first text features. The attention map contains richer and more effective feature information. Choosing the attention map as the shared feature enables full feature sharing between the two image generation channels.

[0098] Shared attention map using a weighted summation method:

[0099] In one embodiment, during the initial stage of model inference, a third attention map to be used in the first image generation channel is obtained by using a first weighted summation method based on the first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel.

[0100] The first weighted summation method is for the first image generation channel, and the first image generation channel and the second image generation channel share attention maps. The first weighted summation method can enhance the correlation between the two image generation channels. Furthermore, the first weighted summation method is a relatively simple and easy-to-implement feature sharing method, which can fully realize feature sharing between the two image generation channels.

[0101] In the early stages of model inference, based on the first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel, a second weighted summation method is used to obtain the fourth attention map to be used in the second image generation channel.

[0102] The second weighted summation method is for the second image generation channel, which shares an attention map with the first image generation channel. The second weighted summation method can enhance the correlation between the two image generation channels. Furthermore, the second weighted summation method is a relatively simple and easy-to-implement feature sharing method, which can fully realize feature sharing between the two image generation channels.

[0103] Weighted summation method:

[0104] In one embodiment, a first intermediate attention map is obtained based on a first weight and a first attention map generated in a first image generation channel; and a second intermediate attention map is obtained based on a second weight and a second attention map generated in a second image generation channel; and a third attention map to be used in the first image generation channel is obtained based on the first intermediate attention map and the second intermediate attention map.

[0105] And, based on the second weight and the first attention map generated in the first image generation channel, a third intermediate attention map is obtained; and based on the first weight and the second attention map generated in the second image generation channel, a fourth intermediate attention map is obtained; and based on the third intermediate attention map and the fourth intermediate attention map, a fourth attention map to be used in the second image generation channel is obtained.

[0106] The sum of the first weight and the second weight is one.

[0107] To illustrate, the following formula is used:

[0108] f1_fuse = a*f1 + (1-a)*f2;

[0109] f2_fuse = (1-a)*f1 + a*f2;

[0110] Where f1 represents the first attention map generated in the first image generation channel, f2 represents the second attention map generated in the second image generation channel, a represents the first weight, (1-a) represents the second weight, a*f1 represents the first intermediate attention map, (1-a)*f2 represents the second intermediate attention map, f1_fuse represents the third attention map to be used in the first image generation channel; (1-a)*f1 represents the third intermediate attention map, a*f2 represents the fourth intermediate attention map, and f2_fuse represents the fourth attention map to be used in the second image generation channel.

[0111] In one embodiment, the first weight increases with the number of inference steps during the initial stage of model inference, while the second weight decreases with the number of inference steps during the initial stage of model inference. Optionally, the first weight gradually increases during the inference process from the first time step to the Kth time step, and correspondingly, the second weight gradually decreases during the inference process from the first time step to the Kth time step, that is, the discriminative power of the attention maps of the two image generation channels will gradually increase. When performing inference at the (K+1)th time step, the first weight is set to 1 and the second weight is set to 0. After that, the first and second image generation channels will no longer share attention maps, and the two image generation channels will perform their own inference processes.

[0112] By gradually changing the first and second weights, the distinguishability of the attention maps used by the first and second image generation channels gradually increases. In the later stages of inference, the two image generation channels will completely separate and execute their respective generation tasks. Since the two image generation channels share features in the early stages of inference, the scene concept art and empty terrain map generated by the two channels will be roughly consistent in overall image layout. However, because the two channels infer independently in the later stages of inference, the scene concept art and empty terrain map will differ in detail. Therefore, this application provides a method for generating scene concept art and empty terrain maps, and the generated scene concept art and empty terrain maps have a certain correlation.

[0113] In one embodiment, the optimization method provided in this application can be used to optimize the image generation process in various scenarios. For example, the optimized image generation process can be used to generate scene images of virtual environments in games, scene images of scenes presented in movies and TV shows, scene images rendered in AR (Augmented Reality) / VR (Virtual Reality) devices, scene images involved in navigation programs, scene images involved in music players and video players, scene images in instant messaging programs, and so on. The image generation process provided in this application can be widely used in various business scenarios and has universality.

[0114] It is worth noting that, since the image generation model of this application generates images based on the constraints of a first design sketch, and the first design sketch already provides the overall or partial structure of the scene region from a certain viewpoint, the viewpoint of the generated scene image is controllable through the optimized image generation process of this application. Controllable viewpoints are more conducive to subsequent creative work based on the generated scene image. Therefore, this application can be used to generate viewpoint-controllable scene images in various business scenarios. Optionally, the optimized image generation process of this application can generate 2D orthographic isometric projection maps.

[0115] Figure 7 This application shows a structural block diagram of an optimization apparatus for an image generation process provided in an exemplary embodiment. The apparatus includes:

[0116] The acquisition module 701 is used to acquire the operation vectors participating in the first attention operation during the process of generating scene images through the image generation model. The first attention operation is an attention operation performed based on the first image features and the first text features. The first image features are image features generated during the operation of the image generation model, and the first text features are obtained based on the text features of the first descriptive text.

[0117] The modulation module 702 is used to perform a region modulation operation on the operation vector of the first attention operation. The region modulation operation is used to enhance the data corresponding to the scene region in the first design sketch in the operation vector.

[0118] The input to the image generation model includes a first descriptive text and a first design sketch. The first descriptive text describes the scene of the scene image to be generated, and the first design sketch is a sketch of the scene image to be generated.

[0119] In an optional embodiment, the operation vector includes an attention map and a value vector; the modulation module 702 is further configured to perform a region modulation operation on the attention map and value vector of the first attention operation;

[0120] The attention map is obtained based on the first image features and the first text features, and the value vector is obtained based on the first text features.

[0121] In an optional embodiment, the attention map of the first attention operation includes multiple semantic dimensions, each of which corresponds to a scene region; the modulation module 702 is further configured to perform an i-th modulation operation in the i-th semantic dimension for the i-th semantic dimension, the i-th modulation operation including at least one of enhancing the data corresponding to the i-th scene region in the first design sketch and suppressing the data corresponding to regions other than the i-th scene region.

[0122] Here, the i-th scene region is a scene region in the first design sketch, and the i-th semantic dimension is the semantic dimension corresponding to the i-th scene region.

[0123] In an optional embodiment, the value vector of the first attention operation includes multiple channel dimensions; the modulation module 702 is further configured to perform a distribution discretization operation in each of the multiple channel dimensions, the distribution discretization operation including at least one of increasing n maxima and decreasing m minima, where n and m are both positive integers.

[0124] In an optional embodiment, the image generation model includes a first image generation channel and a second image generation channel, and the scene image includes a scene concept art and an empty terrain map. The apparatus also includes a sharing module 703, which is used to share the attention maps of the first and second image generation channels during the initial stage of model inference, in the process of generating the scene concept art through the first image generation channel and generating the empty terrain map through the second image generation channel.

[0125] The initial stage of model inference refers to the period when the number of inference steps of the model has not exceeded the step threshold.

[0126] In an optional embodiment, the shared module 703 is further configured to, in the early stage of model inference, obtain a third attention map to be used in the first image generation channel based on the first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel, using a first weighted summation method.

[0127] Furthermore, based on the first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel, a second weighted summation method is used to obtain the fourth attention map to be used in the second image generation channel.

[0128] In an optional embodiment, the sharing module 703 is further configured to obtain a first intermediate attention map based on the first weight and the first attention map generated in the first image generation channel; and to obtain a second intermediate attention map based on the second weight and the second attention map generated in the second image generation channel; and to obtain a third attention map to be used in the first image generation channel based on the first intermediate attention map and the second intermediate attention map.

[0129] Based on the second weight and the first attention map generated in the first image generation channel, a third intermediate attention map is obtained; and based on the first weight and the second attention map generated in the second image generation channel, a fourth intermediate attention map is obtained; based on the third intermediate attention map and the fourth intermediate attention map, a fourth attention map to be used in the second image generation channel is obtained.

[0130] The sum of the first weight and the second weight is one.

[0131] In an optional embodiment, the first weight increases with the number of inference steps in the early stage of model inference, and the second weight decreases with the number of inference steps in the early stage of model inference.

[0132] In this application, by enhancing the data corresponding to the scene region in the first design sketch in the operation vector (the vector participating in the first attention operation), the data of non-scene regions is suppressed. Therefore, during the execution of the first attention operation, the model can accurately correspond the scene region in the first design sketch to the region description in the first descriptive text without interference from non-scene regions. Consequently, the generated scene image accurately represents the semantic category of the scene region in the first design sketch. Furthermore, when the image generation model is pre-trained, the image generation method supported by this application does not require retraining the pre-trained model; it only needs to perform region modulation operations on the first attention operation. This improves image quality without consuming additional training resources. In addition, it fully utilizes the generalization ability of the pre-trained model, eliminating the need to consider overfitting on inference data.

[0133] Figure 8 This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 800 includes a Central Processing Unit (CPU) 801, a system memory 804 including Random Access Memory (RAM) 802 and Read-Only Memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The computer device 800 also includes a basic input / output system (I / O system) 806 to facilitate information transfer between various devices within the computer device, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0134] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for user input, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0135] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer device-readable media provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include computer device-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0136] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer device readable instructions, data structures, program modules, or other data. Computer device storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM, digital video disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer device storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0137] According to various embodiments of this disclosure, the computer device 800 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 800 can be connected to a network 811 via a network interface unit 812 connected to the system bus 805, or it can use the network interface unit 812 to connect to other types of networks or remote computer device systems (not shown).

[0138] The memory also includes one or more programs stored in the memory, and the central processing unit 801 executes the one or more programs to implement all or part of the steps of the optimization method applied to the image generation process.

[0139] Figure 9 This illustration shows a structural block diagram of a computer device 900 provided in an exemplary embodiment of this application. The computer device 900 may be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 900 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0140] Typically, computer device 900 includes a processor 901 and a memory 902.

[0141] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0142] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the optimized method of the image generation process provided in the method embodiments of this application.

[0143] In some embodiments, the computer device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. For example, the peripheral device may include at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0144] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0145] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0146] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, disposed on the front panel of computer device 900; in other embodiments, there may be at least two display screens 905, disposed on different surfaces of computer device 900 or in a folded design; in other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of computer device 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0147] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0148] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the computer device 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0149] Power supply 908 is used to supply power to the various components in computer device 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0150] In some embodiments, the computer device 900 further includes one or more sensors 909. The one or more sensors 909 include, but are not limited to, an accelerometer 910, a gyroscope 911, a pressure sensor 912, an optical sensor 913, and a proximity sensor 914.

[0151] Accelerometer 910 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 900. For example, accelerometer 910 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 910. Accelerometer 910 can also be used for games or for acquiring user motion data.

[0152] The gyroscope sensor 911 can detect the orientation and rotation angle of the computer device 900. The gyroscope sensor 911, in conjunction with the accelerometer sensor 910, can collect 3D motion data from the user on the computer device 900. Based on the data collected by the gyroscope sensor 911, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0153] The pressure sensor 912 can be disposed on the side bezel of the computer device 900 and / or on the lower layer of the display screen 905. When the pressure sensor 912 is disposed on the side bezel of the computer device 900, it can detect the user's grip signal on the computer device 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 912. When the pressure sensor 912 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0154] An optical sensor 913 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 913. For example, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 913.

[0155] A proximity sensor 914, also known as a distance sensor, is typically located on the front panel of a computer device 900. The proximity sensor 914 is used to detect the distance between the user and the front of the computer device 900. In one embodiment, when the proximity sensor 914 detects that the distance between the user and the front of the computer device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 914 detects that the distance between the user and the front of the computer device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0156] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0157] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the optimized method for image generation process provided in the above method embodiments.

[0158] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the optimized method for the image generation process provided in the above-described method embodiments.

[0159] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0160] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0161] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method of optimizing an image generation process, characterized by, The method comprises: In the process of generating a scene image by an image generation model, an operation vector participating in a first attention operation is obtained, the first attention operation is an attention operation performed based on a first image feature and a first text feature, the first image feature is an image feature generated in the running process of the image generation model, and the first text feature is a text feature obtained based on a first description text; A region modulation operation is performed on the operation vector of the first attention operation, and the region modulation operation is used to enhance data corresponding to a scene region in a first design sketch in the operation vector. The input of the image generation model comprises the first description text and the first design sketch, the first description text is used to describe a scene of the scene image to be generated, and the first design sketch is a sketch of the scene image to be generated.

2. The method of claim 1, wherein, The operation vector comprises an attention map and a value vector. The region modulation operation performed on the operation vector of the first attention operation comprises: The region modulation operation is performed on the attention map and the value vector of the first attention operation. The attention map is obtained based on the first image feature and the first text feature, and the value vector is obtained based on the first text feature.

3. The method of claim 2, wherein, The attention map of the first attention operation comprises a plurality of semantic dimensions, and each semantic dimension in the plurality of semantic dimensions corresponds to a scene region. The region modulation operation performed on the attention map of the first attention operation comprises: For an i-th semantic dimension in the plurality of semantic dimensions, an i-th modulation operation is performed in the i-th semantic dimension, and the i-th modulation operation comprises at least one of enhancing data corresponding to an i-th scene region in the first design sketch and suppressing data corresponding to a region other than the i-th scene region. The i-th scene region is a scene region in the first design sketch, and the i-th semantic dimension is a semantic dimension corresponding to the i-th scene region.

4. The method of claim 2, wherein, The value vector of the first attention operation comprises a plurality of channel dimensions. The region modulation operation performed on the value vector of the first attention operation comprises: For each channel dimension in the plurality of channel dimensions, a distribution dispersion operation is performed in the each channel dimension, and the distribution dispersion operation comprises at least one of increasing n maximum values and decreasing m minimum values, n and m are positive integers.

5. The method according to any one of claims 1 to 4, characterized in that, The image generation model comprises a first image generation channel and a second image generation channel, and the scene image comprises a scene layout and an empty terrain map. The method further comprises: In the process of generating the scene layout by the first image generation channel and generating the empty terrain map by the second image generation channel, an attention map of the first image generation channel and the second image generation channel is shared at a model inference initial stage. The model inference initial stage refers to a period when the number of inference steps of the model does not exceed a step threshold.

6. The method of claim 5, wherein, The attention map of the first image generation channel and the second image generation channel is shared at the model inference initial stage, comprising: In the initial stage of the model inference, a first attention map generated in the first image generation channel and a second attention map generated in the second image generation channel are used to obtain a third attention map to be used in the first image generation channel by using a first weighted summation method. In addition, the first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel are used to obtain a fourth attention map to be used in the second image generation channel by using a second weighted summation method.

7. The method of claim 6, wherein, The first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel are used to obtain the third attention map to be used in the first image generation channel by using the first weighted summation method, including: The first weight and the first attention map generated in the first image generation channel are used to obtain a first intermediate attention map, and the second weight and the second attention map generated in the second image generation channel are used to obtain a second intermediate attention map; and the first intermediate attention map and the second intermediate attention map are used to obtain the third attention map to be used in the first image generation channel. The first attention map generated in the first image generation channel and the second attention map generated in the second image generation channel are used to obtain the fourth attention map to be used in the second image generation channel by using the second weighted summation method, including: The second weight and the first attention map generated in the first image generation channel are used to obtain a third intermediate attention map, and the first weight and the second attention map generated in the second image generation channel are used to obtain a fourth intermediate attention map; and the third intermediate attention map and the fourth intermediate attention map are used to obtain the fourth attention map to be used in the second image generation channel. The sum of the first weight and the second weight is one.

8. The method of claim 7, wherein, The first weight increases with the increase of the inference step number in the initial stage of the model inference, and the second weight decreases with the increase of the inference step number in the initial stage of the model inference.

9. An optimization apparatus for an image generation process, characterized in that, The device includes: An acquisition module is configured to acquire an operation vector participating in first attention operation in a process of generating a scene image by using an image generation model, the first attention operation is an attention operation performed based on a first image feature and a first text feature, the first image feature is an image feature generated in a running process of the image generation model, and the first text feature is a text feature obtained based on a first description text. A modulation module is configured to perform a region modulation operation on the operation vector of the first attention operation, and the region modulation operation is configured to enhance data corresponding to a scene region in a first design sketch in the operation vector. The input of the image generation model includes the first description text and the first design sketch, the first description text is configured to describe a scene of the scene image to be generated, and the first design sketch is a sketch of the scene image to be generated.

10. The apparatus of claim 9, wherein, The operation vector includes an attention map and a value vector. The modulation module is further configured to perform the region modulation operation on the attention map and the value vector of the first attention operation. The attention map is obtained based on the first image feature and the first text feature, and the value vector is obtained based on the first text feature.

11. The apparatus of claim 10, wherein, The attention map of the first attention operation includes a plurality of semantic dimensions, and each semantic dimension of the plurality of semantic dimensions corresponds to a scene region. The modulation module is further configured to, for an i-th semantic dimension of the plurality of semantic dimensions, perform an i-th modulation operation in the i-th semantic dimension, the i-th modulation operation including at least one of enhancing data corresponding to an i-th scene region in the first design sketch and suppressing data corresponding to a region other than the i-th scene region. The i-th scene region is a scene region in the first design sketch, and the i-th semantic dimension is a semantic dimension corresponding to the i-th scene region.

12. The apparatus of claim 10, wherein, The value vector of the first attention operation includes a plurality of channel dimensions. The modulation module is further configured to, for each channel dimension of the plurality of channel dimensions, perform a distribution dispersion operation in the each channel dimension, the distribution dispersion operation including at least one of increasing n maximum values and decreasing m minimum values, n and m being positive integers.

13. A computer device, comprising: The computer device includes a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the optimization method of the image generation process according to any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program being loaded and executed by the processor to implement the optimization method of the image generation process according to any one of claims 1 to 8.

15. A computer program product, characterised in that, The computer program product stores a computer program, the computer program being loaded and executed by the processor to implement the optimization method of the image generation process according to any one of claims 1 to 8.

Images