Image generation method and device, electronic equipment and medium

Abstract

The invention provides an image generation method and device, electronic equipment and a medium, and relates to the technical field of computers. The method comprises the steps of obtaining a prompt text; target texts matched with the semantics of the prompt text are determined from multiple preset texts, and the preset texts are used for indicating image feature maps conforming to the semantics of the preset texts; obtaining a target image feature map indicated by the target text; and generating a target image corresponding to the prompt text by taking the target image feature map as an initial noise distribution map. According to the method, the relevance between the initial noise distribution diagram and the prompt text is improved, the quality of the generated image is improved, and the matching degree of the image and the prompt text is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, electronic device, and medium. Background Technology

[0002] Current text-to-image synthesis techniques utilize generative models, particularly diffusion models, to transform textual descriptions into vivid visual images. During image generation, a standard Gaussian distribution is typically used to randomly initialize noise, and this initial noise distribution map serves as the starting point for denoising by the diffusion model. However, this initial noise distribution map lacks semantic relevance to the input text description, resulting in poor image quality that fails to accurately reflect the text description's content, leading to a low degree of matching between the generated image and the text description. Summary of the Invention

[0003] This application provides an image generation method, apparatus, electronic device, and medium to improve the matching degree between generated images and text.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] In a first aspect, embodiments of this application provide an image generation method that uses an image feature map semantically associated with the prompt text as an initial noise distribution map to generate an image corresponding to the prompt text. This improves the quality of the generated image by enhancing the correlation between the initial noise distribution map and the prompt text. Specifically, first, the prompt text is obtained, and a target text that semantically matches the prompt text is determined from a plurality of preset texts; wherein the preset text is used to indicate an image feature map semantically consistent with the preset text. Then, the target image feature map indicated by the target text is obtained. Finally, the target image feature map is used as the initial noise distribution map to generate the target image corresponding to the prompt text.

[0006] In this embodiment, a target text that highly matches the prompt text semantically is found from multiple preset texts. The target image feature map indicated by the target text is used as the initial noise distribution map, thereby enhancing the correlation between the initial noise distribution map and the prompt text. Furthermore, based on the initial noise distribution map that is highly correlated with the prompt text, the model has more explicit and specific guidance in the image generation process. It can directly learn from these feature maps and retain visual information that is highly related to the text description, thereby generating an image that can more accurately reflect the content described by the prompt text, improving the consistency between the image and the prompt text, greatly reducing the deviation between the generated image and the content described by the prompt text, and improving the quality of the generated image.

[0007] In one possible implementation, determining the target text that semantically matches the prompt text from multiple preset texts includes: obtaining a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; based on the text embedding vector corresponding to each preset text, determining a second text embedding vector that has the highest similarity to the first text embedding vector; the second text embedding vector is used to represent the semantic information of the target text.

[0008] In this implementation, since the text embedding vector considers not only the literal meaning of the words in the prompt text, but also the meaning of the words in the context of the prompt text and their relationship with other words in the prompt text, it can handle the problems of polysemy and synonyms in language and distinguish the different meanings of the same word in different contexts. Therefore, the text embedding vector can capture the semantic information of the text more accurately. Thus, the target text found based on the text embedding vector has a higher semantic matching degree with the prompt text, and can accurately find target texts that are similar in meaning but different in literal sense.

[0009] In one possible implementation, obtaining the target image feature map of the target text indication includes: determining the image feature map corresponding to the second text embedding vector from a preset correspondence, as the target image feature map; the preset correspondence includes the correspondence between the text embedding vector of each preset text and the image feature map of the preset text indication.

[0010] In this implementation, the target image feature map is obtained directly by querying a preset correspondence, and the target image feature map is used to initialize the image generation process, avoiding the process of re-extracting features from the image, thereby significantly improving the processing speed and efficiency. Especially when dealing with a large number of similar or repeated queries, finding the target image feature map based on the preset correspondence has obvious advantages.

[0011] In one possible implementation, the target image corresponding to the prompt text is generated using the target image feature map as the initial noise distribution map. This includes: using a preset generation model, performing multiple iterations of denoising processing on the target image feature map based on the prompt text to obtain the target image; the preset generation model includes an encoder and a decoder; the encoder includes N sequentially connected encoding layers; the decoder includes N sequentially connected decoding layers; N is an integer greater than 1; each encoding layer and its corresponding decoding layer are skip-connected; the encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map; the first feature map is either the target image feature map or the feature map output by the previous encoding layer; the decoding layer is used to fuse the second and third feature maps based on the prompt text and according to preset factors corresponding to the second and third feature maps respectively, and output a fourth feature map; the third feature map is the feature map output by the previous decoding layer; the preset factors are used to scale the corresponding feature maps.

[0012] In this implementation, a preset factor is used to scale feature maps at different levels to varying degrees during feature fusion, fusing feature information from different levels. This helps the model focus more on important feature information when generating the target image, improving the quality and accuracy of the generated image. Furthermore, the preset factor can be adjusted according to implementation needs, enhancing the model's flexibility and adaptability.

[0013] In one possible implementation, in the decoding layer, a preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map; and a preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

[0014] In this implementation, by adjusting the contribution of different feature maps in the fusion process, the influence of feature information on the generated image can be controlled more precisely. Since the second feature map comes from the encoding layer and the third feature map comes from the deeper decoding layer, meaning the second feature map represents the shallow features of the image, containing background information and noise, while the third feature map represents the deep features, containing high-frequency information (also known as detail information), reducing the contribution of the second feature map to the fused fourth feature map and increasing the contribution of the third feature map enhances the detail information in the generated image, reduces noise and unnecessary background information, thereby improving the quality of the generated image, enhancing the model's denoising ability, and enabling the generated image to more accurately describe the content contained in the prompt text.

[0015] In one possible implementation, the value of the preset factor corresponding to the second feature map is less than the preset factor corresponding to the third feature map.

[0016] In this implementation, during image generation, the second feature map comes from the encoding layer and contains background information and noise, while the third feature map comes from a deeper decoding layer and contains detailed information. Assigning a larger preset factor (higher weight) to the third feature map ensures that the detailed information it represents is better preserved and emphasized during feature fusion. Conversely, assigning a smaller preset factor (lower weight) to the second feature map limits the influence of shallow information on the target image, reducing noise and unnecessary background information. By using a smaller preset factor for the second feature map and a larger preset factor for the third feature map, the dominance of deeper features is maintained during feature fusion, while shallow background information is appropriately preserved. This results in a target image that more accurately reflects the content of the prompt text, improving user satisfaction.

[0017] In one possible implementation, a preset generation model is used to perform multiple iterative denoising processes on the target image feature map based on the prompt text to obtain the target image. This includes: determining the number of iterations and the order of key denoising processes in the multiple iterations; for each iteration of denoising, if it is a key denoising process, the input feature map and the first text embedding vector corresponding to the prompt text are input to the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein, the input feature map is the target image feature map or the output feature map of the previous iteration of denoising; if it is not a key denoising process, the output of the first encoding layer and the output of the second-to-last decoding layer in the previous key denoising process are input to the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0018] In this implementation, the output of the encoding / decoding layer from the previous iteration is used in non-critical denoising processes, avoiding redundant computation of the entire model. This effectively reduces computational resource consumption, giving the model a significant advantage when processing large-scale data or real-time applications. Furthermore, the number of iterations and the order of critical denoising processes can be determined according to actual needs, adjusting the image generation process to adapt to different application scenarios and performance requirements.

[0019] In a second aspect, an image generation apparatus is provided, comprising: functional units for performing any of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented by hardware or by hardware executing corresponding software. For example, the image generation apparatus may include: a first acquisition module, a determining module, a second acquisition module, and a generation module; the first acquisition module is used to acquire prompt text; the determining module is used to determine target text from a plurality of preset texts that semantically matches the prompt text; the preset text is used to indicate an image feature map that semantically matches the preset text; the second acquisition module is used to acquire the target image feature map indicated by the target text; and the generation module is used to generate a target image corresponding to the prompt text using the target image feature map as an initial noise distribution map.

[0020] In one possible implementation, a determining module is used to obtain a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; based on the text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector is determined; the second text embedding vector is used to represent the semantic information of the target text.

[0021] In one possible implementation, the second acquisition module is used to determine the image feature map corresponding to the second text embedding vector from the preset correspondence, as the target image feature map; the preset correspondence includes the correspondence between the text embedding vector of each preset text and the image feature map indicated by the preset text.

[0022] In one possible implementation, a generation module is used to perform multiple iterations of denoising processing on the feature map of the target image based on the prompt text using a preset generation model to obtain the target image. The preset generation model includes an encoder and a decoder. The encoder includes N sequentially connected encoding layers. The decoder includes N sequentially connected decoding layers. N is an integer greater than 1. Each encoding layer is skipped from its corresponding decoding layer. The encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map. The first feature map is either the feature map of the target image or the feature map output by the previous encoding layer. The decoding layer is used to fuse the second and third feature maps based on the prompt text and according to preset factors corresponding to the second and third feature maps, and output a fourth feature map. The third feature map is the feature map output by the previous decoding layer. The preset factors are used to scale the corresponding feature maps.

[0023] In one possible implementation, in the decoding layer, a preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map; and a preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

[0024] In one possible implementation, the value of the preset factor corresponding to the second feature map is less than the preset factor corresponding to the third feature map.

[0025] In one possible implementation, a generation module is used to determine the number of iterations and the order of key denoising processes in multiple iterations of denoising processing. For each iteration of denoising processing, if it is a key denoising process, the input feature map and the first text embedding vector corresponding to the prompt text are input to the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model. The input feature map is either the target image feature map or the output feature map of the previous iteration of denoising processing. If it is not a key denoising process, the output of the first encoding layer and the output of the second-to-last decoding layer in the previous key denoising process are input to the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0026] Thirdly, an electronic device is provided, comprising: a processor and a memory, wherein the processor is connected to the memory. The memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions stored in the memory, thereby implementing any of the methods provided in the first aspect.

[0027] Fourthly, a chip is provided, comprising: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to execute the code instructions to perform any of the methods provided in the first aspect above.

[0028] Fifthly, a computer-readable storage medium is provided, storing computer-executable instructions that, when executed on a computer, cause the computer to perform any of the methods provided in the first aspect above.

[0029] In a sixth aspect, a computer program product is provided, including computer execution instructions that, when executed on a computer, cause the computer to perform any of the methods provided in the first aspect above.

[0030] The technical effects of any of the implementation methods in aspects two through six can be found in the technical effects of different implementation methods in aspect one, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating a text-to-image generation method based on a diffusion model, as provided in an embodiment of this application.

[0032] Figure 2 A system architecture diagram of an image generation system provided in this application embodiment;

[0033] Figure 3A flowchart illustrating an image generation method provided in this application embodiment;

[0034] Figure 4 A schematic diagram illustrating the construction of a noise feature library provided in an embodiment of this application;

[0035] Figure 5 A schematic diagram illustrating the second apparatus provided in this application for generating a target image using a preset generation model;

[0036] Figure 6 This is a schematic diagram of the structure of a preset generation model provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram illustrating the specific structure of a preset generation model provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram of the structure of another preset generation model provided in the embodiments of this application;

[0039] Figure 9 A schematic diagram illustrating the specific structure of another preset generation model provided in this application embodiment.

[0040] Figure 10 A flowchart illustrating the generation of an image by the second device provided in this application embodiment;

[0041] Figure 11 A schematic diagram illustrating a multi-iteration denoising process provided in an embodiment of this application;

[0042] Figure 12 A schematic diagram illustrating yet another multi-iteration denoising process provided in an embodiment of this application;

[0043] Figure 13 A schematic diagram illustrating another multi-iteration denoising process provided in an embodiment of this application;

[0044] Figure 14 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0045] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0047] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between the related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. A and B can be singular or plural.

[0048] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0049] Furthermore, to facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that "first" and "second" are not necessarily different. Meanwhile, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0050] The following describes the terminology used in the embodiments of this application.

[0051] Text-to-image synthesis (T2I) is an artificial intelligence (AI) technique that transforms textual descriptions into image content. Specifically, T2I uses deep learning models, such as diffusion models, generative adversarial networks (GANs), variational autoencoders (VAEs), and autoregressive models, to understand and parse the semantic information in textual descriptions, thereby generating images that match the textual descriptions.

[0052] The Unet model is a convolutional neural network architecture. It features a symmetrical U-shaped network structure that combines feature extraction (encoding) and feature reconstruction (decoding). The Unet model consists of an encoder and a decoder. The encoder extracts image features, while the decoder fuses these features. The encoder comprises multiple encoding layers that gradually reduce the spatial resolution of the image to extract high-frequency features (also known as detail features). The decoder comprises multiple decoding layers that gradually restore the spatial resolution of the image to reconstruct its features. Each decoding layer is skip-connected to its corresponding encoding layer. Each decoding layer performs feature fusion (usually a concatenation operation) based on the feature maps output by its corresponding encoding layer and its previous decoding layer, merging shallow features (containing background information) with deep features (containing detail information, also known as high-frequency information) to improve the quality of the generated image.

[0053] The diffusion model (DM) is a generative model based on probability theory. The core idea of ​​the diffusion model is to gradually add noise to the original data (such as an image) until the data becomes almost unrecognizable pure noise (called the forward diffusion process). Then, starting from the pure noise, a series of inverse operations are used to gradually remove the noise, ultimately generating a sample that approximates the original data distribution (called the reverse diffusion process). In this embodiment, the diffusion model is used for text-based image generation. Based on the core idea of ​​the diffusion model, noise is iteratively added to the low-dimensional space (also called the latent space) of the image, and then the noise is gradually removed to generate an image based on the text. In the process of generating an image based on text, the Unet model is used to predict noise multiple times, and the predicted noise is gradually removed to generate an image corresponding to the text.

[0054] First, the application scenarios of the embodiments of this application will be introduced by way of example.

[0055] In text-to-image (T2I) synthesis, diffusion models are often used to transform textual descriptions into realistic visual images. In this process, based on the core idea of ​​diffusion models, noise is randomly sampled from a standard Gaussian distribution to initialize the image generation process. Figure 1 This is a schematic diagram illustrating text-to-image generation based on a diffusion model. For example... Figure 1 As shown, the feature map corresponding to the noise randomly sampled from the standard Gaussian distribution is used as the initial noise distribution map. The Unet model is used to perform multiple iterations of denoising on the initial noise distribution map to obtain the image corresponding to the text.

[0056] However, the initial noise distribution map obtained in this way has almost no semantic connection with the input text description. This semantic disconnect often leads to the generated image being mismatched with the text description in terms of content, lacking accuracy and consistency, thus affecting the quality of the final generated image.

[0057] In view of this, embodiments of this application provide an image generation method that uses an image feature map semantically related to the prompt text as an initial noise distribution map to generate an image corresponding to the prompt text. This improves the quality of the generated image and enhances the matching degree between the image and the prompt text by increasing the correlation between the initial noise distribution map and the prompt text.

[0058] In some embodiments, the method specifically includes: first, acquiring a prompt text and determining a target text from a plurality of preset texts that semantically matches the prompt text. The preset texts are used to indicate an image feature map that semantically matches the preset text. Then, acquiring the target image feature map indicated by the target text. Finally, using the target image feature map as an initial noise distribution map, generating a target image corresponding to the prompt text.

[0059] The image generation method provided in this application identifies target text that semantically highly matches the prompt text from multiple preset texts. The target image feature map indicated by the target text is used as the initial noise distribution map, thereby enhancing the correlation between the initial noise distribution map and the prompt text. Furthermore, based on the initial noise distribution map highly correlated with the prompt text, the image generation process receives more explicit and specific guidance. It can directly learn and retain visual information highly relevant to the text description from these feature maps, thereby generating an image that more accurately reflects the content described in the prompt text, improving the consistency between the image and the prompt text, significantly reducing the deviation between the generated image and the content described in the prompt text, and improving the quality of the generated image.

[0060] The system architecture of the embodiments of this application will be described below by way of example.

[0061] This application provides an image generation system. Figure 2 This is a system architecture diagram of an image generation system provided in an embodiment of this application. The image generation system includes a first device and a second device. The first device is used to acquire prompt text and determine an initial noise distribution map corresponding to the prompt text. The second device is used to generate a target image based on the initial noise distribution map and the prompt text.

[0062] In one possible implementation, the first device uses an image feature map that is semantically associated with the prompt text as an initial noise distribution map.

[0063] Optionally, the first device determines a target image feature map that semantically matches the prompt text based on a pre-constructed information set, and uses the target image feature map as an initial noise distribution map.

[0064] like Figure 2 As shown, the information set contains first text embedding vectors of multiple preset texts, and image feature maps that correspond to the semantics of the preset texts. The preset texts are predefined text descriptions with clear semantics and image orientation, used to indicate image feature maps that correspond to the semantics of the preset texts.

[0065] The implementation method of the first device obtaining the target image features based on the information set will be described in the following embodiments, and will not be repeated here.

[0066] In one possible implementation, the second device uses a preset generation model to perform multiple iterations of denoising on the initial noise distribution map corresponding to the prompt text to obtain the target image corresponding to the prompt text.

[0067] exist Figure 2 In the process, for the i-th denoising process, the second device inputs the prompt text and the initial noise map into the preset generation model to obtain a noise feature map of the noise in the initial noise map, or a feature map after removing the noise feature map from the initial noise map, also known as the feature map after removing the noise feature map (or the feature map after noise removal). The initial noise map is either the initial noise distribution map or the feature map output from the previous denoising process. Specifically, when the preset generation model is used to predict the noise feature map of the noise in the input feature map, the preset generation model predicts the noise feature map of the noise in the initial noise map based on the input prompt text and the initial noise map. The second device removes the noise feature map from the initial noise feature map to obtain the output feature map of this denoising process. When the preset generation model is used to remove noise from the input feature map, the preset generation model obtains the feature map after removing the noise feature map (or the feature map after noise removal) based on the prompt text and the initial noise map, and uses the feature map after removing the noise feature map as the output feature map of this denoising process. The second device uses the feature map output from the last denoising process, i.e. the feature map after removing noise, as the feature map of the target image, and generates the target image based on the feature map of the target image.

[0068] The specific structure of the preset generation model will be described in the following embodiments, and will not be repeated here.

[0069] The first device can be implemented through software or hardware.

[0070] As an example of a software function, the first device may include code running on a computing instance. The computing instance may be at least one of a physical host, virtual machine, container, or other computing device. Furthermore, the aforementioned computing instance may be one or more. For example, the first device may include code running on multiple hosts / virtual machines / containers.

[0071] For example, the first device may be deployed as a software package on the computing device where the image generation system resides, or the first device may be deployed on a separate computing device. For instance, the first device may be deployed as a plug-in in the image generation system.

[0072] As an example of a hardware functional unit, the first device may include at least one computing device, such as a server. Alternatively, the first device may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0073] The server can be a single physical server, or it can be two or more physical servers that share different responsibilities. The physical servers work together to achieve the various functions of the server.

[0074] For example, the server can be a blade server, a high-density server, a rack server, or a tower server, etc.

[0075] The first device includes multiple computing devices that can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0076] Similar to the first device, the second device can be implemented in software or hardware, which will not be elaborated here.

[0077] It is understandable that, since a video is composed of multiple frames, the second device can also generate a target video corresponding to the prompt text based on multiple target images. This application does not limit the specific implementation method for generating a target video from multiple target images.

[0078] It should be noted that the form of the first device is not limited in the embodiments of this application. The following describes the system architecture of the image generation system provided in the embodiments of this application, taking the example that both the first device and the second device are implemented by software. Specifically, the system architecture of the image generation system provided in the embodiments of this application is described, taking the example that both the first device and the second device are deployed on a server.

[0079] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0080] The following embodiments of this application will be divided into three parts to exemplarily describe the image generation method.

[0081] Part One, Combination Figure 3 , Figure 4 , Figure 5 This paper introduces the image generation method provided in the embodiments of this application, aiming to illustrate the overall working process of generating images based on text.

[0082] Part Two, Combination Figure 6 , Figure 7 , Figure 8 , Figure 9 This paper introduces the image generation method provided in the embodiments of this application, aiming to illustrate the specific structure of the preset generation model.

[0083] Part Three, Combining Figure 10 , Figure 11 , Figure 12 , Figure 13 This paper introduces the image generation method provided in the embodiments of this application, aiming to illustrate the specific process of how to generate a target image using a preset generation model.

[0084] In some embodiments, an image feature map that is semantically associated with the prompt text is used as an initial noise distribution map to generate an image corresponding to the prompt text. In this way, the quality of the generated image is improved by increasing the correlation between the initial noise distribution map and the prompt text.

[0085] Figure 3 This is a flowchart illustrating an image generation method according to an exemplary embodiment, the method being performed by an image generation system. Exemplarily, the method includes the following steps S301-S304.

[0086] S301: The first device acquires the prompt text.

[0087] Optionally, the cue text can be descriptive text about various scenes, abstract concepts, and other objects. Specifically, the cue text can describe various elements of the object, including but not limited to the object's type, attributes, location, actions, emotions, atmosphere, etc. The cue text is an important basis for image generation, conveying to the model the content and style that the user wants the image to present.

[0088] For example, the type of an object can be a person, an animal, a building, etc., and the attributes of an object can be color, shape, size, material, etc. For example, "a red apple".

[0089] For example, the prompt text can also describe the positional relationship of objects in an image, such as "a red apple is placed on the dining table".

[0090] For example, prompt text can describe the action or state of an object, thereby increasing the vividness and storytelling of the image, such as "A little bird is singing happily on a branch".

[0091] S302: The first device determines the target text that matches the semantics of the prompt text from a plurality of preset texts.

[0092] The preset text is used to indicate the image feature map that matches the semantics of the preset text.

[0093] In one possible implementation, the preset text is a predefined text description with clear semantics and image-specific meaning. Thus, the preset text can be used to indicate an image feature map that semantically matches the preset text. That is, one preset text is associated with one image feature map that semantically matches the preset text. Based on the preset text, the image feature map indicated by the preset text can be obtained. Here, "matching" can be understood as the semantics of the preset text accurately describing part or all of the content of the image corresponding to the image feature map. One preset text can indicate multiple image feature maps. An image feature map can also be indicated by multiple preset texts.

[0094] Furthermore, in order to find target text that matches the prompt text entered by the user, multiple preset texts should cover a wide range of image themes, styles, and scenes to meet the needs of different users. For example, the preset texts, and the images indicated by the preset texts, can be obtained from computer vision datasets such as the ImageNet dataset.

[0095] In this embodiment, the prompt text is matched with multiple preset texts based on text embedding vectors. First, a first text embedding vector corresponding to the prompt text is obtained. The first text embedding vector is used to represent the semantic information of the prompt text. Then, based on the text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector is determined. The second text embedding vector is used to represent the semantic information of the target text.

[0096] Because text embedding vectors consider not only the literal meaning of words in the prompt text but also their meaning within the context of the prompt text and their relationship with other words in the prompt text, they can handle polysemy and synonyms in language. They can distinguish the different meanings of the same word in different contexts, and thus capture the semantic information of the text more accurately. Therefore, the target text found based on text embedding vectors has a higher semantic matching degree with the prompt text, accurately identifying target texts that, although literally different, have similar meanings.

[0097] Optionally, the text embedding vector corresponding to the text can be obtained by the text encoder in the contrastive language-image pre-training (CLIP) model.

[0098] Optionally, the similarity between the first text embedding vector and the second text embedding vector can be determined by Euclidean distance, Manhattan distance, Pearson correlation coefficient, etc. This application embodiment does not specifically limit the calculation method of the similarity between the first text embedding vector and the second text embedding vector.

[0099] S303: The first device acquires the target image feature map indicated by the target text.

[0100] In one possible implementation, the image feature map can be obtained through the encoder in the VAE model.

[0101] The VAE model comprises an encoder and a decoder. The encoder extracts image features, while the decoder reconstructs the image based on these features. The encoder in the VAE model is the one obtained after training. During training, the training data consists of images from an image dataset. The VAE model is trained by reducing the error between the images generated by the decoder and the original images.

[0102] Specifically, the image is input into the encoder of the trained VAE model to obtain the image feature map of the image output by the encoder of the VAE model. Based on this, for example, the image of the target text indicator is input into the VAE encoder to obtain the target image feature map.

[0103] In one possible implementation, a VAE encoder is used to pre-extract features from the image indicated by each preset text to obtain a corresponding image feature map. Then, relevant information about the preset text (such as text representation, text embedding vector, or the text itself) and the corresponding image feature maps are saved in a preset correspondence. Thus, in step S301, based on the relevant information of the target text, the image feature map corresponding to the target text can be determined from this preset correspondence and used as the target image feature map.

[0104] As can be seen from the above implementation, the preset correspondence includes, for example, the correspondence between the text embedding vector of each preset text and the image feature map indicated by the preset text. In this way, the target image feature map is obtained directly by querying the preset correspondence, and the image generation process is initialized with the target image feature map, avoiding the process of re-extracting features from the image. This significantly improves processing speed and efficiency, especially when dealing with a large number of similar texts or repeated queries; finding the target image feature map based on the preset correspondence has a clear advantage.

[0105] Optionally, an information set is constructed based on the text embedding vectors of multiple preset texts and the image feature maps of preset text indicators. This application embodiment does not specifically limit the data structure form of the information set. Taking a noise feature library as an example, in the noise feature library, text embedding vectors and image feature maps have a preset correspondence. The text embedding vector of the prompt text is compared with the text embedding vectors in the noise feature library, and the image feature map corresponding to the text embedding vector with the highest similarity is used as the target image feature map.

[0106] Figure 4 This is a schematic diagram illustrating the construction of a noise feature library as provided in an embodiment of this application. Figure 4 In this process, a pre-built CLIP model is used to input a pre-defined text from a computer vision dataset to obtain the text embedding vector of the pre-defined text. The image indicated by the pre-defined text is then input into a pre-built VAE encoder to obtain the image feature map. Based on the text embedding vector of the pre-defined text and the corresponding image feature map, a noise feature library is obtained.

[0107] Optionally, an information set can be constructed based on multiple preset texts and images corresponding to the preset texts. The prompt text is compared with the preset texts (e.g., based on text embedding vectors) to obtain the image corresponding to the prompt text. Feature extraction is performed on the image to obtain the image feature map of the image, and the image feature map of the image is used as the target image feature map.

[0108] Of course, you can also construct an information set by combining the text embedding vector of the preset text with the image indicated by the preset text, and establish a correspondence between the text embedding vector of the preset text and the image. Alternatively, you can construct an information set by combining the preset text with the image feature map, and establish a correspondence between the preset text and the image feature map. Through the correspondence, you can find the image feature map that matches the prompt text.

[0109] S304: The second device uses the feature map of the target image as the initial noise distribution map to generate the target image corresponding to the prompt text.

[0110] In one possible implementation, the second device uses a preset generation model to perform multiple iterations of denoising on the initial noise distribution map corresponding to the prompt text, thereby obtaining the target image corresponding to the prompt text. For example, the preset generation model is the Unet model.

[0111] Optionally, in each denoising process, a preset generative model is used to predict noise. The preset generative model includes an encoder and a decoder. The encoder consists of N sequentially connected encoding layers; the decoder consists of N sequentially connected decoding layers; N is an integer greater than 1; each encoding layer and its corresponding decoding layer are skipped connections. That is, the encoder and decoder present a U-shaped network structure.

[0112] The encoder is responsible for extracting image features, while the decoder is responsible for fusing these image features.

[0113] In the above S304, such as Figure 5 As shown, the second device inputs the initial noise map and the prompt text into a preset generation model to predict the noise feature map of the noise in the initial noise map. Then, it removes the predicted noise feature map from the initial noise map to obtain the noise-removed feature map. After multiple iterations, the final noise-removed feature map is used as the feature map of the target image. The second device then generates the target image corresponding to the prompt text based on the feature map of the target image. The initial noise map is either the initial noise distribution map or the output feature map obtained in the previous denoising process, i.e., the noise-removed feature map.

[0114] Optionally, a preset generation model is used for noise removal. In addition to an encoder and decoder, the preset generation model also includes a denoiser. The encoder and decoder are used to obtain noise feature maps of the noise in the initial noise map, and the denoiser is used to remove the noise feature maps from the initial noise map, obtaining a noise-removed feature map. In each denoising process, the second device inputs the initial noise distribution map into the preset generation model to directly obtain the noise-removed feature map. After multiple iterations of denoising processing, a target image feature map of the target image is obtained. The second device then generates the target image based on the target image feature map.

[0115] The structure of the preset generation model and the specific process of generating the target image using the preset generation model will be described in Parts 2 and 3 of the following embodiments, respectively, and will not be repeated here.

[0116] In the first part of this application's embodiments, target text that highly matches the prompt text semantically is found from multiple preset texts. The image feature map indicated by the target text is used as the initial noise distribution map, thereby enhancing the correlation between the initial noise distribution map and the prompt text. Furthermore, based on the initial noise distribution map that is highly correlated with the prompt text, there is more explicit and specific guidance in the image generation process. Visual information that is highly related to the text description can be learned directly from these feature maps and retained, thereby generating an image that can more accurately reflect the content described by the prompt text, improving the consistency between the image and the prompt text, greatly reducing the deviation between the generated image and the content described by the prompt text, and improving the quality of the generated target image.

[0117] The above is the first part of the embodiments of this application. The following, in conjunction with... Figure 6 , Figure 7 , Figure 8 , Figure 9 This section introduces the specific structure of the preset generative model.

[0118] In some examples, a pre-defined generative model is used to predict the noise feature map of the noise in the input feature map.

[0119] In some embodiments, the preset generative model includes an encoder and a decoder. Figure 6 This is a schematic diagram of a pre-defined generative model. For example... Figure 6 As shown, the encoder consists of n sequentially connected encoding layers, and the decoder consists of m sequentially connected decoding layers, where n and m are integers greater than 1. Each encoding layer is connected to its corresponding decoding layer in a skip connection. In the preset generation model, the input prompt text and initial noise map sequentially pass through multiple encoding layers in the encoder (encoding layer 1, encoding layer 2...encoding layer n) and multiple decoding layers in the decoder (decoding layer 1...decoding layer m-1, decoding layer m) to obtain the model's output, which is the noise feature map of the noise predicted by the model.

[0120] In this pre-defined generation model, the encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map. The first feature map can be either the target image feature map or the feature map output by the previous encoding layer.

[0121] In one possible implementation, the encoding layer includes, but is not limited to, convolutional layers, activation function layers, and pooling layers (also known as downsampling layers). Convolutional layers are used to perform convolution operations on the input feature map, thereby extracting local features from the image. The number of convolutional layers can be set according to the actual situation and is not specifically limited here. Activation function layers are used to introduce non-linear factors, enabling the encoding layer to learn more complex features and improving its generalization ability. Activation function layers can use functions such as Rectified Linear Unit (ReLU). Pooling layers are used to progressively reduce the size of the feature map, increasing the receptive field size.

[0122] The embodiments of this application do not limit the specific structure of the coding layer, and can be set according to actual needs.

[0123] In this pre-defined generation model, the decoding layer is used to fuse the second and third feature maps according to the prompt text and their respective pre-defined factors, outputting a fourth feature map. The third feature map is the feature map output by the previous decoding layer. The pre-defined factors are used to scale the corresponding feature maps.

[0124] In one possible implementation, the decoding layer includes, but is not limited to, upsampling layers, convolutional layers, and activation functions (such as sigmoid, Tanh, and ReLU activation functions). The upsampling layer can use deconvolution operations, interpolation operations, etc., to upsample the input feature map, thereby increasing the size of the feature map. After upsampling, one or more convolutional layers (such as 3x3 convolutional layers) are used in the decoding layer to further extract and fuse features. This application does not limit the specific construction of the decoding layer; it can be set according to actual needs.

[0125] For example, the decoding layer can use a convolutional layer to fuse the second and third feature maps according to the prompt text and preset factors corresponding to their respective values, resulting in a fourth feature map. By setting preset factors in the decoder, the pre-defined generation model scales feature maps from different levels (such as shallow features from the encoding layer and deep features from the decoding layer) to varying degrees during feature fusion. This fusion of feature information from different levels helps the model focus more on important feature information when generating the target image, improving the quality and accuracy of the generated image. Furthermore, the preset factors can be adjusted according to implementation needs to more finely control the impact of feature information on the generated image, thereby enhancing the model's flexibility and adaptability.

[0126] Figure 7 This is a schematic diagram of the specific structure of a pre-defined generative model. For example... Figure 7 As shown, the encoder in the preset generation model has three coding layers. Coding layer 1, as the first coding layer of the encoder, receives the initial noise distribution map and the prompt text, extracts features from the initial noise distribution map, and outputs a feature map (called the second feature map of coding layer 1). Coding layer 2 receives the feature map output by the previous coding layer, i.e., coding layer 1, downsamples the feature map output by coding layer 1 to obtain a downsampled feature map, extracts features from the downsampled feature map, and outputs a feature map (called the second feature map of coding layer 2). Coding layer 3 receives the feature map output by the previous coding layer, i.e., coding layer 2, downsamples the feature map output by coding layer 2, extracts features from the downsampled feature map, and outputs a feature map (called the second feature map of coding layer 3). Decoding layer 1, as the first coding layer in the decoder, receives the feature map output by coding layer 3, upsamples the feature map output by coding layer 3, and fuses the upsampled feature map with the feature map output by coding layer 2 according to their respective preset factors, outputting a feature map (called the fourth feature map of decoding layer 1). Decoding layer 2, as the last decoding layer, receives the feature map output by decoding layer 1 and upsamples it. It then fuses the upsampled feature map with the feature map output by encoding layer 1 according to their respective preset factors, outputting a feature map (called the fourth feature map of decoding layer 2). The feature map output by decoding layer 2 serves as the noise feature map for the predicted noise.

[0127] In one possible implementation, since the second feature map comes from the encoding layer and the third feature map comes from the deeper decoding layer, meaning the second feature map represents the shallow features of the image, containing background information and noise, while the third feature map represents the deep features of the image, containing high-frequency information (also known as detail information), the contribution of the second feature map to the fused fourth feature map is reduced, while the contribution of the third feature map to the fused fourth feature map is increased. This enhances the detail information in the generated image, reduces noise and unnecessary background information, thereby improving the quality of the generated image, enhancing the model's denoising ability, and enabling the generated image to more accurately describe the content contained in the prompt text.

[0128] Optionally, in the decoding layer, a preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map. A preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

[0129] For example, the preset factor corresponding to the second feature map has a smaller value than the preset factor corresponding to the third feature map. For instance, the preset factor for the second feature map is set to a value less than 1, and the preset factor for the third feature map is set to a value greater than 1. This way, assigning a larger preset factor (i.e., a higher weight) to the third feature map ensures that the detailed information represented by the third feature map is preserved and emphasized more during feature fusion. Assigning a smaller preset factor (i.e., a lower weight) to the second feature map limits the influence of shallow features on the target image, reducing noise and unnecessary background information. By using a smaller preset factor for the second feature map and a larger preset factor for the third feature map, it helps maintain the dominance of deep features during feature fusion while appropriately preserving shallow background information, resulting in a more accurate target image that reflects the content of the prompt text and improves user satisfaction.

[0130] Optionally, the preset factors corresponding to the second feature map and the third feature map can be determined during the training of the preset generative model.

[0131] For example, in the nth denoising process, the initial noise map is input into a preset generation model to obtain feature maps output by each coding and decoding layer in the preset generation model. Based on the frequency domain range of each feature map, a preset factor for each feature map is determined.

[0132] For example, if the frequency range of the second feature map a is 0-2MHz, and the preset frequency range corresponding to the second feature map is set to 0-3MHz, then the preset factor of the second feature map a can be set to 2 / 3. If the frequency range of the third feature map b is 0-5MHz, and the preset frequency range corresponding to the third feature map is set to 3-6MHz, then the preset factor of the third feature map b can be set to 5 / 3.

[0133] Understandably, since the frequency domain range of the second feature maps output by each coding layer in the encoder may differ, the preset factor of the second feature map output by each coding layer will also vary with the frequency domain range of its respective output second feature map. Similarly, since the frequency domain range of the third feature maps output by each decoding layer in the decoder will also differ, the preset factor of the third feature map output by each decoding layer will also vary with the frequency domain range of its respective output third feature map.

[0134] Of course, for the nth denoising process, the preset factor of the second feature map output by each encoding layer in the pre-generated model can be set to the same preset factor, and the preset factor of the third feature map output by each decoding layer can be set to another identical preset factor. This simplifies the number of parameters that need to be optimized in the model, accelerates the model's training speed, and reduces the model's computational complexity.

[0135] In other examples, a pre-defined generative model is used to remove noisy feature maps from the input feature maps.

[0136] In addition to the above, the preset generative models include Figure 6 The model includes an encoder and decoder, as well as a denoiser. The denoiser removes the feature map output by the decoder from the feature map input to the encoder.

[0137] Figure 8 This is a schematic diagram of the structure of another pre-defined generative model. Figure 8 In the preset generation model, there are an encoder, a decoder, and a denoiser. The initial noise map and the prompt text are input into the preset generation model, and then pass through multiple encoding layers in the encoder (encoding layer 1, encoding layer 2... encoding layer n), multiple decoding layers in the decoder (decoding layer 1... decoding layer m-1, decoding layer m), and the denoiser to obtain the feature map output by the model, that is, the feature map after removing noise.

[0138] Figure 9This is a schematic diagram of the specific structure of another preset generation model. The preset generation model includes an encoder, a decoder, and a denoiser. The encoder in the preset generation model has three encoding layers: encoding layer 1, encoding layer 2, and encoding layer 3. The decoder in the preset generation model contains two decoding layers: decoding layer 1 and additional decoding layer 2. The initial noise map and the prompt text are input into the preset generation model, and the encoder inputs the feature map (i.e.,...) Figure 9 The initial noise distribution map in the encoder and the feature map output by the decoder (i.e., the feature map output by the decoding layer 2) are subtracted in the denoiser, and the feature map output by the decoder is removed from the feature map input to the encoder to obtain the feature map after noise removal.

[0139] The embodiments of this application do not limit the structure of the preset generation model. In the third part of the following embodiments, the preset generation model including the encoder and the decoder is used as an example to explain how to obtain the target image using the preset generation model. The details will not be repeated here.

[0140] In the second part of this application's embodiments, by setting a preset factor, the contribution of different feature maps in the fusion process is adjusted, and feature maps at different levels are scaled to different degrees. This fusion of feature information from different levels helps the model focus more on important feature information when generating the target image, improving the quality and accuracy of the generated image. Furthermore, the preset factor can be adjusted according to implementation needs, enhancing the model's flexibility and adaptability.

[0141] The above is the second part of the embodiments of this application. The following, in conjunction with... Figure 10 , Figure 11 , Figure 12 , Figure 13 The specific process of generating a target image using a preset generation model.

[0142] In this embodiment, the second device uses a preset generation model to perform multiple iterations of denoising on the initial noise distribution map corresponding to the prompt text to obtain the target image corresponding to the prompt text.

[0143] In this application embodiment, a target image corresponding to the prompt text is generated using a preset generation model, including multiple implementation methods. Hereinafter, the following will be illustrated by way 1 to way 2.

[0144] Method 1: The denoising process is divided into critical denoising processes and non-critical denoising processes through multiple iterations.

[0145] During the critical denoising process, all encoding layers in the encoder and decoding layers in the decoder function normally. The second device inputs the input feature map and the prompt text (such as the first text embedding vector corresponding to the prompt text) to the first encoding layer of the preset generation model, obtaining the output feature map of the last decoding layer of the preset generation model. The input feature map is either the initial noise distribution map (i.e., the target image feature map) or the feature map output by the second device in the previous denoising process (i.e., the feature map output by the preset generation model obtained in the previous denoising process). In other words, each encoding layer in the encoder is used to extract features from the first feature map (i.e., the target image feature map or the feature map output by the previous encoding layer) based on the prompt text, and output the second feature map. Each decoding layer in the decoder is used to fuse the second and third feature maps according to the prompt text and their respective preset factors, outputting the fourth feature map.

[0146] Continue with Figure 7 Taking the preset generation model as an example, in the key denoising process, encoding layer 1 receives the initial noise distribution map and the prompt text, extracts features from the initial noise distribution map, and outputs a feature map (called the second feature map of encoding layer 1). Encoding layer 2 receives the feature map output by the previous encoding layer, i.e., encoding layer 1, downsamples the feature map output by encoding layer 1 to obtain a downsampled feature map, and extracts features from the downsampled feature map to output a feature map (called the second feature map of encoding layer 2). Encoding layer 3 receives the feature map output by the previous encoding layer, i.e., encoding layer 2, downsamples the feature map output by encoding layer 2, and extracts features from the downsampled feature map to output a feature map (called the second feature map of encoding layer 3). Decoding layer 1 receives the feature map output by encoding layer 3, upsamples the feature map output by encoding layer 3, and fuses the upsampled feature map and the feature map output by encoding layer 2 according to their respective preset factors to output a feature map (called the fourth feature map of decoding layer 1). Decoding layer 2, as the last decoding layer, receives the feature map output by decoding layer 1 and upsamples it. It then fuses the upsampled feature map with the feature map output by encoding layer 1 according to their respective preset factors, outputting a feature map (called the fourth feature map of decoding layer 2). The feature map output by decoding layer 2 serves as the noise feature map for the predicted noise.

[0147] For non-critical denoising processes, the pre-defined encoding and decoding layers in the generative model are not all active; only some or all of them are active. This has two advantages: First, when only some encoding or decoding layers are active, the amount of data the model needs to process and the number of computational steps are significantly reduced, thereby greatly decreasing computational complexity and accelerating the model's processing speed. This is particularly important in scenarios involving large-scale data or requiring rapid response, such as real-time image generation. Second, the partial activation of encoding and decoding layers means the model consumes less memory resources, contributing to more efficient resource utilization and reducing the consumption of computing resources. Furthermore, by flexibly adjusting the active encoding and decoding layers, the model can better adapt to different denoising tasks. For example, natural scene images require more encoding layers to capture image details compared to medical images.

[0148] In one possible implementation, during the non-critical denoising process, only the last decoding layer in the decoder of the preset generative model is operational. The second device inputs the output of the first encoding layer and the output of the penultimate decoding layer of the preset generative model from the previous critical denoising process into the last decoding layer of the preset generative model in this non-critical denoising process, obtaining the output feature map of the last decoding layer, which is the noise feature map of the predicted noise in this non-critical denoising process.

[0149] Optionally, in the non-critical denoising process, the second device inputs the output of the first encoding layer and the output of the penultimate decoding layer in the previous critical denoising process into the last decoding layer of the preset generation model. The last decoding layer performs feature fusion on the output of the first encoding layer and the output of the penultimate decoding layer in the previous critical denoising process according to their respective preset factors to obtain the noise feature map of the noise predicted in this non-critical denoising process.

[0150] For example, in the non-critical denoising process, the last decoding layer upsamples the penultimate decoding layer in the previous critical denoising process, and fuses the upsampled feature map with the output of the first encoding layer in the previous critical denoising process according to their respective preset factors to obtain the noise feature map of the noise predicted in this non-critical denoising process.

[0151] In one possible implementation, the second device utilizes a preset generation model to perform multiple iterative denoising processes on the feature map of the target image based on the prompt text, thereby obtaining the target image. During these multiple iterative denoising processes, if a critical denoising process is in progress, the second device uses all the encoding and decoding layers in the preset generation model to predict the noise feature map of the noise in this critical denoising process, and removes the predicted noise feature map from the input feature map of the preset generation model, obtaining the output feature map of this critical denoising process. If a non-critical denoising process is in progress, the second device uses some encoding or decoding layers in the preset generation model to predict the noise feature map of the noise in this non-critical denoising process, and removes the predicted noise feature map from the input feature map of the preset generation model, obtaining the output feature map of this non-critical denoising process. After multiple iterative denoising processes, the feature map of the target image is obtained, thus generating the target image.

[0152] Figure 10 This is a flowchart illustrating a second apparatus for generating an image according to an exemplary embodiment. Exemplarily, the method includes the following steps S3041-S3043:

[0153] S3041: The second device determines the number of iterations and the order of key denoising processes during multiple iterations of denoising processing.

[0154] The number of iterations refers to the number of times the preset generation model is used, which can also be understood as the number of times noise is removed.

[0155] For example, the number of iterations and the order of key denoising processes can be set according to actual needs, and no specific limitation is made here. For instance, if the number of iterations is 50, the order of key denoising processes can be 1, 2, 3, 5, 7, 9..., and the corresponding order of non-key denoising processes can be 4, 6, 8...

[0156] S3042: The second device performs multiple iterations of denoising on the target image feature map according to the number of key denoising processes, and obtains the feature map of the target image based on the feature map input to the encoder and the feature map output by the decoder during the last iteration of denoising.

[0157] In one possible scenario, for each iteration of the denoising process, if it is a critical denoising process, the second device inputs the input feature map and the first text embedding vector corresponding to the prompt text into the first encoding layer of the preset generation model, to obtain the output feature map of the last decoding layer of the preset generation model (i.e., the noise feature map of the noise predicted by the preset generation model). The input feature map is either the target image feature map or the output feature map of the previous iteration of the denoising process.

[0158] If it is not a critical denoising process, the second device will input the output of the first encoding layer and the output of the second-to-last decoding layer in the previous critical denoising process into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0159] Figure 11 This is a schematic diagram of a multi-iteration denoising process. For Figure 11 The process involves multiple iterations of denoising, with the key denoising processes occurring in the order of 1, 2, 3, 5, 7, 9... and the non-key denoising processes occurring in the order of 4, 6, 8... Figure 11 The diagram illustrates the third and fourth denoising processes. For the third denoising process, the second device inputs the output (feature map a) of the preset generation model decoder from the second denoising process into the first encoding layer (encoding layer 1) of the encoder in the preset generation model. Encoding layer 1 extracts features from feature map a, outputting feature map a1 to encoding layer 2. Encoding layer 2 downsamples and extracts features from feature map a1, outputting feature map a2 to encoding layer 3. Encoding layer 3 downsamples and extracts features from feature map a2, outputting feature map a3 to decoding layer 1. Decoding layer 1 upsamples feature map a3 and fuses the upsampled feature map a3 with feature map a2 according to their respective preset factors, outputting feature map a4 to decoding layer 2. Decoding layer 2 upsamples feature map a4 and fuses the upsampled feature map a4 with feature map a1 according to their respective preset factors, outputting feature map a5. The second device then removes feature map a5 from feature map a, obtaining feature map b. For the fourth denoising process, the second device inputs the output of the first encoding layer (encoding layer 1) (feature map a1) and the output of the second-to-last decoding layer (decoding layer 1) (feature map a4) from the third denoising process into the last decoding layer (decoding layer 2) of the preset generation model in the fourth denoising process. Decoding layer 2 upsamples feature map a4 and fuses feature map a1 and the upsampled feature map a4 according to their respective preset factors to obtain feature map b1. The second device then removes feature map b1 from feature map b. For the third denoising process, which is a critical denoising process, all decoding and encoding layers in the preset generation model function normally. For the fourth denoising process, which is a non-critical denoising process, all encoding layers of the encoder in the preset generation model (i.e.,...) function normally. Figure 11 The coding layers 1, 2, and 3 in the codec, as well as the decoding layer 1 in the decoder, are not working. Only the last decoding layer in the decoder (i.e., Figure 11 The decoding layer 2) in the middle is working.

[0160] Figure 12 This is a schematic diagram of yet another type of multi-iteration denoising process. For Figure 12 The process involves multiple iterations of denoising, with the key denoising processes ordered as 1, 2, 3, 6, 9... and the non-key denoising processes ordered as 4, 5, 7, 8... Figure 12 The image shows the third, fourth, and fifth noise reduction processes. Figure 12 The third noise reduction process in the process and Figure 11 The third denoising process is the same and will not be described in detail here. For the fourth denoising process, the second device inputs the input of the first encoding layer (encoding layer 1) and the output of the second-to-last decoding layer (decoding layer 1) from the third denoising process to the last decoding layer (decoding layer 2) of the preset generation model in the fourth denoising process, and the last decoding layer outputs feature map b1. For the fifth denoising process, the second device inputs the input of the first encoding layer (encoding layer 1) and the output of the second-to-last decoding layer (decoding layer 1) from the third denoising process to the last decoding layer (decoding layer 2) of the preset generation model in the fifth denoising process, and outputs feature map b2.

[0161] In one possible scenario, since the non-critical denoising process is based on the output of the first encoding layer and the penultimate decoding layer of the previous critical denoising process, the output feature map of the current non-critical denoising process is obtained. Therefore, the non-critical denoising process corresponding to a critical denoising process can be processed in parallel (also known as using a skip-step strategy). After the penultimate decoding layer of the critical denoising process outputs its feature map, the second device can input the output of the first encoding layer and the penultimate decoding layer of the critical denoising process into the last decoding layer of the non-critical denoising process corresponding to that critical denoising process. Here, the non-critical denoising process corresponding to the critical denoising process refers to the non-critical denoising process between the current critical denoising process and the next critical denoising process. In this case, the last decoding layer of the preset generation model in the critical denoising process and the last decoding layer of the preset generation model in the non-critical denoising process can be computed in parallel (also known as using a skip-step strategy).

[0162] In this way, since non-critical denoising processes can be performed in parallel with certain stages of critical denoising processes, the overall processing time is significantly reduced. This parallel processing is particularly effective when there are a large number of non-critical denoising processes, significantly improving processing efficiency and enabling real-time image generation from text, further enhancing the user experience. Furthermore, by separating critical and non-critical denoising processes, the second device can dynamically adjust the number of parallel processing tasks according to actual needs. For example, when the computing resources of the second device are limited, the number of non-critical denoising processes can be reduced to save resources; when resources are sufficient, the number of parallel processing tasks can be increased to improve processing speed.

[0163] Continue with Figure 12 For example, for the 3rd, 4th, and 5th denoising processes, the 4th and 5th denoising processes can begin after the second-to-last decoding layer (decoding layer 1) of the preset generation model outputs a feature map during the 3rd denoising process. In other words, the 4th denoising process does not need to wait for the last decoding layer (decoding layer 2) of the 3rd denoising process to output a feature map, and the 5th denoising process does not need to wait for the last decoding layer (decoding layer 2) of the 4th denoising process to output a feature map.

[0164] In one possible scenario, the second device removes the feature map output by the decoder during the last iteration of denoising from the feature map input to the encoder, thereby obtaining the feature map of the target image.

[0165] For example, the second device performs a subtraction operation on the feature map input to the encoder and the feature map output by the decoder during the last iteration of denoising processing to obtain the feature map of the target image.

[0166] Understandably, when the preset generation model includes a denoiser, the second device uses the feature map output by the preset generation model during the last iteration of denoising as the feature map of the target image.

[0167] S3043: The second device generates the target image based on the feature map of the target image.

[0168] In one possible scenario, the second device utilizes a decoder to generate the target image based on the feature map of the target image.

[0169] Tables 1 and 2 compare the results of image generation using the image generation method provided in this application with those not provided in this application. Table 1 compares the results of the second device generating images based on text using a graphics processing unit (GPU) in a computing device. Table 2 compares the results of the second device generating images based on text using a neural processing unit (NPU) in a computing device. The number of iterations in both Tables 1 and 2 is 50. As can be seen from Tables 1 and 2, parallel processing of non-critical denoising processes (also known as using a skip-step strategy) reduces latency without decreasing the accuracy of the generated image. The image generation method provided in this application incorporates a preset factor and employs a skip-step caching strategy for non-critical denoising processes, thereby reducing latency while improving the accuracy of the generated image.

[0170] Table 1

[0171]

[0172] Table 2

[0173]

[0174] Method 2: In the process of multiple iterations of denoising, each denoising process is the key denoising process mentioned above.

[0175] In other words, for each denoising process, all encoding layers in the encoder and all decoding layers in the decoder function normally. During multiple iterations of denoising, the preset generation model is used indiscriminately multiple times. For example, if the number of iterations is 50, then all encoding layers and all decoding layers in the preset generation model are used 50 times.

[0176] Figure 13 This is a schematic diagram of another iterative denoising process. For Figure 13 In the multiple iterations of denoising processing, the preset generation model was used multiple times without distinction. Figure 13 This demonstrates the first and second denoising processes. In the first denoising process, the second device first converts the target image feature map, i.e., the initial noise distribution map (…). Figure 13 Initial noise in Figure 1 The input is fed into the encoder of the preset generation model, passing through encoding layer 1, encoding layer 2, encoding layer 3, decoding layer 1, and decoding layer 2 in sequence. The decoder outputs feature map a, and then feature map a is separated from the initial noise distribution map (initial noise). Figure 1 Remove from ) to obtain the initial noise Figure 2 In the second noise reduction process, the second device first removes the initial noise. Figure 2 The input is fed into the encoder of the preset generation model, passing through encoding layer 1, encoding layer 2, encoding layer 3, decoding layer 1, and decoding layer 2 in sequence. The decoder outputs feature map b, which is then removed from the initial noise. Figure 2 Remove from the middle to obtain the initial noise. Figure 3 And so on. If the denoising process consists of 50 iterations, in the 50th denoising iteration, the second device first inputs the initial noise map 50 obtained in the 49th denoising iteration into the encoder of the preset generation model, outputs a feature map x through the decoder, removes the feature map x from the initial noise map 50, and obtains the feature map of the target image. The second device then obtains the target image based on the feature map of the target image.

[0177] In the third part of this application's embodiments, the multi-iteration denoising process is divided into critical denoising processes and non-critical denoising processes. For non-critical denoising processes, only some encoding or decoding layers are operational. This has two advantages: First, when only some encoding or decoding layers are operational, the amount of data and computational steps the model needs to process are significantly reduced, thereby greatly reducing computational complexity and accelerating the model's processing speed. This is particularly important in scenarios involving large-scale data or requiring rapid response, such as real-time image generation. Second, the operation of only some encoding and decoding layers means that the model consumes less memory resources, contributing to more efficient resource utilization and reducing the consumption of computing resources on the computing device. Furthermore, by flexibly adjusting the operating encoding and decoding layers, the model can better adapt to different denoising tasks. For example, natural scene images require more encoding layers to capture image details compared to medical images.

[0178] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the image generation apparatus includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] This application embodiment can, according to the above method, exemplarily divide the image generation apparatus into functional modules. For example, the image generation apparatus may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0180] For example, Figure 14 A possible structural schematic diagram of the image generation apparatus involved in the above embodiments is shown. The image generation apparatus includes: a first acquisition module 1401, a determination module 1402, a second acquisition module 1403, and a generation module 1404. The first acquisition module 1401 is used to acquire prompt text; the determination module 1402 is used to determine target text that semantically matches the prompt text from a plurality of preset texts; the preset text is used to indicate an image feature map that semantically matches the preset text; the second acquisition module 1403 is used to acquire the target image feature map indicated by the target text; and the generation module 1404 is used to generate a target image corresponding to the prompt text using the target image feature map as an initial noise distribution map.

[0181] Optionally, the determining module 1402 is specifically used to: obtain the first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; and determine the second text embedding vector with the highest similarity to the first text embedding vector based on the text embedding vector corresponding to each preset text; the second text embedding vector is used to represent the semantic information of the target text.

[0182] Optionally, the second acquisition module 1403 is specifically used to determine, from the preset correspondence, the image feature map corresponding to the second text embedding vector as the target image feature map; the preset correspondence includes the correspondence between the text embedding vector of each preset text and the image feature map indicated by the preset text.

[0183] Optionally, the generation module 1404 is specifically used to perform multiple iterations of denoising processing on the feature map of the target image based on the prompt text using a preset generation model to obtain the target image. The preset generation model includes an encoder and a decoder. The encoder includes N encoding layers connected in sequence. The decoder includes N decoding layers connected in sequence. N is an integer greater than 1. Each encoding layer is skipped from its corresponding decoding layer. The encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map. The first feature map is either the feature map of the target image or the feature map output by the previous encoding layer. The decoding layer is used to fuse the second and third feature maps according to the prompt text and preset factors corresponding to the second and third feature maps respectively, and output a fourth feature map. The third feature map is the feature map output by the previous decoding layer. The preset factors are used to scale the corresponding feature maps.

[0184] Optionally, in this device, in the decoding layer, a preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map; and a preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

[0185] Optionally, in this device, the value of the preset factor corresponding to the second feature map is less than the preset factor corresponding to the third feature map.

[0186] Optionally, the generation module 1404 is specifically used to determine the number of iterations and the order of key denoising processes in multiple iterations of denoising processing; for each iteration of denoising processing, if it is a key denoising process, the input feature map and the first text embedding vector corresponding to the prompt text are input to the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein, the input feature map is the target image feature map or the output feature map of the previous iteration of denoising processing; if it is not a key denoising process, the output of the first encoding layer and the output of the second-to-last decoding layer in the previous key denoising process are input to the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

[0187] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the image generation apparatuses provided above and the description of their beneficial effects can be found in the corresponding method embodiments described above, and will not be repeated here.

[0188] This application also provides an electronic device 100. For example... Figure 15As shown, the electronic device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The electronic device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the electronic device 100.

[0189] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 15 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the electronic device 100 (e.g., memory 106, processor 104, communication interface 108).

[0190] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0191] Memory 106 may include volatile memory, such as random access memory (RAM). Processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0192] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the aforementioned first acquisition module, determination module, second acquisition module, and generation module, thereby realizing the image generation method. That is, the memory 106 stores instructions for executing the image generation method.

[0193] The communication interface 108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the electronic device 100 and other devices or communication networks.

[0194] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform an image generation method, or instruct the computing device to perform an image generation method.

[0195] This application also provides a chip. The chip integrates a control circuit for implementing the functions of the aforementioned electronic device and one or more ports. Optionally, the functions supported by the chip can be referred to above, and will not be repeated here. Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, random access memory, etc. The aforementioned processing unit or processor can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0196] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform an image generation method.

[0197] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of this application, such as but not limited to the memory, computer-readable storage medium and communication chip, are all non-transitory.

[0198] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0199] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0200] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. An image generation method, characterized in that, The method includes: Get the prompt text; Determine the target text that semantically matches the prompt text from a plurality of preset texts; the preset text is used to indicate an image feature map that semantically matches the preset text; Obtain the target image feature map indicated by the target text; Using the feature map of the target image as the initial noise distribution map, the target image corresponding to the prompt text is generated.

2. The image generation method according to claim 1, characterized in that, The step of determining the target text that semantically matches the prompt text from a plurality of preset texts includes: Obtain the first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; Based on the text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector is determined; the second text embedding vector is used to characterize the semantic information of the target text.

3. The image generation method according to claim 2, characterized in that, The step of obtaining the target image feature map of the target text indication includes: From the preset correspondence, an image feature map corresponding to the second text embedding vector is determined as the target image feature map; the preset correspondence includes the correspondence between the text embedding vector of each preset text and the image feature map indicated by the preset text.

4. The image generation method according to any one of claims 1-3, characterized in that, The step of generating the target image corresponding to the prompt text by using the target image feature map as the initial noise distribution map includes: Using a preset generation model, the target image feature map is subjected to multiple iterations of denoising processing based on the prompt text to obtain the target image; the preset generation model includes an encoder and a decoder; the encoder includes N encoding layers connected in sequence; the decoder includes N decoding layers connected in sequence; N is an integer greater than 1; each encoding layer and its corresponding decoding layer are skipped connections. The encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map; the first feature map is the target image feature map or the feature map output by the previous encoding layer. The decoding layer is used to perform feature fusion on the second feature map and the third feature map according to the prompt text and the preset factors corresponding to the second feature map and the third feature map respectively, and output a fourth feature map; the third feature map is the feature map output by the previous decoding layer; the preset factors are used to perform feature scaling on the corresponding feature map.

5. The image generation method according to claim 4, characterized in that, In the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map; the preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

6. The image generation method according to claim 4 or 5, characterized in that, The value of the preset factor corresponding to the second feature map is less than the preset factor corresponding to the third feature map.

7. The image generation method according to any one of claims 4-6, characterized in that, The step of using a preset generation model to perform multiple iterations of denoising processing on the feature map of the target image based on the prompt text to obtain the target image includes: Determine the number of iterations and the order of key denoising processes in the multiple iterations of denoising processing; For each iteration of the denoising process, if it is the critical denoising process, the input feature map and the first text embedding vector corresponding to the prompt text are input to the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein, the input feature map is the target image feature map or the output feature map of the previous iteration of the denoising process; If it is not the critical denoising process, then the output of the first encoding layer and the output of the second-to-last decoding layer in the previous critical denoising process are input into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

8. An image generation apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the prompt text; A determination module is used to determine a target text that semantically matches the prompt text from a plurality of preset texts; the preset text is used to indicate an image feature map that semantically matches the preset text; The second acquisition module is used to acquire the target image feature map indicated by the target text; The generation module is used to generate the target image corresponding to the prompt text by using the feature map of the target image as the initial noise distribution map.

9. The image generation apparatus according to claim 8, characterized in that, The determining module is used to obtain a first text embedding vector corresponding to the prompt text; the first text embedding vector is used to represent the semantic information of the prompt text; based on the text embedding vector corresponding to each preset text, a second text embedding vector with the highest similarity to the first text embedding vector is determined; the second text embedding vector is used to represent the semantic information of the target text.

10. The image generation apparatus according to claim 9, characterized in that, The second acquisition module is used to determine, from the preset correspondence, the image feature map corresponding to the second text embedding vector as the target image feature map; the preset correspondence includes the correspondence between the text embedding vector of each preset text and the image feature map indicated by the preset text.

11. The image generating apparatus according to any one of claims 8-10, characterized in that, The generation module is used to perform multiple iterations of denoising processing on the feature map of the target image based on the prompt text using a preset generation model to obtain the target image; the preset generation model includes an encoder and a decoder; the encoder includes N encoding layers connected in sequence; the decoder includes N decoding layers connected in sequence; N is an integer greater than 1; each encoding layer and its corresponding decoding layer are connected in skip connections; The encoding layer is used to extract features from the first feature map based on the prompt text and output a second feature map; the first feature map is the target image feature map or the feature map output by the previous encoding layer. The decoding layer is used to perform feature fusion on the second feature map and the third feature map according to the prompt text and the preset factors corresponding to the second feature map and the third feature map respectively, and output a fourth feature map; the third feature map is the feature map output by the previous decoding layer; the preset factors are used to perform feature scaling on the corresponding feature map.

12. The image generation apparatus according to claim 11, characterized in that, In the decoding layer, the preset factor corresponding to the second feature map is used to reduce the contribution of the second feature map to the fused fourth feature map; the preset factor corresponding to the third feature map is used to enhance the contribution of the third feature map to the fused fourth feature map.

13. The image generating apparatus according to claim 11 or 12, characterized in that, The value of the preset factor corresponding to the second feature map is less than the preset factor corresponding to the third feature map.

14. The image generating apparatus according to any one of claims 11-13, characterized in that, The generation module is used to determine the number of iterations and the order of key denoising processes in the multiple iterations of denoising processing. For each iteration of the denoising process, if it is the critical denoising process, the input feature map and the first text embedding vector corresponding to the prompt text are input to the first encoding layer of the preset generation model to obtain the output feature map of the last decoding layer of the preset generation model; wherein, the input feature map is the target image feature map or the output feature map of the previous iteration of the denoising process; If it is not the critical denoising process, then the output of the first encoding layer and the output of the second-to-last decoding layer in the previous critical denoising process are input into the last decoding layer of the preset generation model to obtain the output feature map of the last decoding layer.

15. An electronic device, characterized in that, include: Processor and memory; The processor is connected to a memory for storing computer execution instructions, and the processor executes the computer execution instructions stored in the memory to enable the electronic device to implement the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The device stores computer instructions that, when executed on a computing device, cause the computing device to perform the method as described in any one of claims 1 to 7.

17. A computer program product, characterized in that, include: A computer executes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images