Image generation method and apparatus, and computing device cluster

Abstract

An image generation method, comprising: acquiring a first material; on the basis of the first material, generating at least one first feature map, and on the basis of the first material, generating a second feature map, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map; fusing the at least one first feature map into at least one target feature map to obtain a third first feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map; and decoding the third feature map to obtain a target image, wherein the target image aligns with image generation requirements represented by the first material. The method can significantly improve the effect of text-to-image or image-to-image generation technology.

Description

Image generation method, device and computing device cluster

[0001] The present application claims priority to the Chinese patent application No. 202411242684.6, filed on September 4, 2024, entitled "Image generation method, device and computing device cluster", the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to an image generation method, device and computing device cluster. BACKGROUND

[0003] With the continuous progress of AI technology, text-to-image and image-to-image technologies have made significant breakthroughs in the field of computer vision. Text-to-image technology is a technology that can generate corresponding images according to natural language descriptions. Image-to-image technology is a technology that can generate new images or transform images according to existing images. These technologies not only improve the efficiency of image generation and processing, but also greatly enrich the diversity and quality of visual content. However, these technologies still face some challenges, including low resolution of generated images, insufficient detail performance, poor authenticity, and slow inference speed, which results in poor actual application effect. SUMMARY

[0004] The present application provides an image generation method, device, computing device cluster, computer storage medium and computer product, which can significantly improve the effect of text-to-image and image-to-image technologies.

[0005] In a first aspect, the present application provides an image generation method, comprising: obtaining a first material; generating at least one first feature map based on the first material, and generating a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map; fusing the at least one first feature map into at least one target feature map to obtain a third feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map; and decoding the third feature map to obtain a target image, wherein the target image corresponds to the image generation requirement expressed by the first material.

[0006] In this way, by generating a high-resolution feature map in the latent space domain and guiding the feature maps obtained by different branches in the generation process, semantic information rich in the feature maps obtained by other branches can be fused into the high-resolution feature map, the structure and details of the high-resolution feature map are improved, and the generation difficulty of the high-resolution image is reduced. In addition, the method can directly obtain an image with high reality, high quality and high resolution through a single-stage operation, and significantly improves the effect of text / image generation technology.

[0007] In a possible implementation, the resolution of the first feature map is smaller than the resolution of the second feature map. The fourth feature map is any one of the at least one first feature map, and the first target feature map is a target feature map that is fused with the fourth feature map. At this time, the fusion of the fourth feature map into the first target feature map includes: upsampling the fourth feature map to obtain a feature enlarged map, the resolution of the feature enlarged map being the same as the resolution of the first target feature map; and fusing the feature enlarged map into the first target feature map. In this way, when the resolution of the first feature map is different from the resolution of the target feature map, the resolution of the first feature map can be improved through upsampling, thereby facilitating the fusion of the target feature map.

[0008] In a possible implementation, the upsampling of the fourth feature map to obtain the feature enlarged map includes: performing attention calculation on tokens of the fourth feature map and at least one first token to obtain at least one second token; obtaining weights of an implicit network based on the at least one second token; and inputting coordinates of a pixel point in the first target feature map into the implicit network to obtain the feature enlarged map. In this way, different weights of the implicit network can be used to obtain the corresponding feature enlarged map when upsampling different feature maps. Since the weights of the implicit network can be adaptively adjusted, different upsampling can be performed on different feature maps, thereby improving the upsampling effect.

[0009] In a possible implementation, the fusion of the feature enlarged map into the first target feature map includes: performing residual learning based on the feature enlarged map and the first target feature map to obtain a first residual; and adding the first residual to the first target feature map to fuse the feature enlarged map into the first target feature map. In this way, different feature maps can be fused together through residual learning.

[0010] In a possible implementation, after the feature enlarged map is fused into the first target feature map, the method further includes: adjusting the mean and standard deviation of the fused feature map according to a resolution specified by a user, wherein the fused feature map is a feature map obtained by fusing the feature enlarged map into the first target feature map. In this way, the details of the fused feature map can be optimized, and the display effect of a subsequent image is improved.

[0011] In a possible implementation, the first feature maps are multiple, the first first feature map is generated based on the first material, and the i-th first feature map is generated based on the (i-1)-th first feature map, i≥2.

[0012] In a second aspect, the present application provides an image generation device, comprising: an acquisition module and a processing module. The acquisition module is configured to acquire a first material. The processing module is configured to generate at least one first feature map based on the first material, and generate a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map. The processing module is further configured to fuse the at least one first feature map into at least one target feature map to obtain a third feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map. The processing module is further configured to decode the third feature map to obtain a target image, wherein the target image is consistent with the image generation requirement expressed by the first material.

[0013] In a possible implementation, the resolution of the first feature map is less than the resolution of the second feature map. The fourth feature map is any one of the at least one first feature map, and the first target feature map is the feature map in the at least one target feature map that is fused with the fourth feature map. When the processing module fuses the fourth feature map into the first target feature map, the processing module is specifically configured to: up-sample the fourth feature map to obtain a feature enlarged map, the resolution of the feature enlarged map being the same as the resolution of the first target feature map; and fuse the feature enlarged map into the first target feature map.

[0014] In a possible implementation, when the processing module up-samples the fourth feature map to obtain the feature enlarged map, the processing module is specifically configured to: perform attention calculation on a token of the fourth feature map and at least one first token to obtain at least one second token; obtain the weight of an implicit network based on the at least one second token; and input the coordinates of a pixel point in the first target feature map into the implicit network to obtain the feature enlarged map.

[0015] In a possible implementation, when the processing module fuses the feature enlarged map into the first target feature map, the processing module is specifically configured to: perform residual learning based on the feature enlarged map and the first target feature map to obtain a first residual; and add the first residual to the first target feature map to fuse the feature enlarged map into the first target feature map.

[0016] In a possible implementation, after the feature magnification map is fused into the first target feature map, the processing module is further configured to adjust the mean and standard deviation of the fused feature map according to a resolution specified by a user, where the fused feature map is a feature map obtained by fusing the feature magnification map into the first target feature map.

[0017] In a possible implementation, the first feature maps are multiple, where the first first feature map is generated based on the first material, and the ith first feature map is generated based on the (i-1)th first feature map, i≥2.

[0018] In a third aspect, the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can include one or more computing devices.

[0020] In a fifth aspect, the present application provides a computer program product including instructions, when the instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect, or, when the instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Illustratively, the computing device cluster can include one or more computing devices.

[0021] It can be understood that the beneficial effects of the above-mentioned second aspect to fifth aspect can refer to the related description in the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0022] FIG. 1 is a schematic diagram of a technical concept of image generation provided by an embodiment of the present application;

[0023] FIG. 2 is a schematic diagram of a structure of a neural network model provided by an embodiment of the present application;

[0024] FIG. 3 is a schematic diagram of a working process of an implicit expression module shown in FIG. 2;

[0025] FIG. 4 is a schematic diagram of a working process of the latent space guiding module shown in FIG. 2;

[0026] FIG. 5 is a schematic diagram of a working process of the resolution adaptation module shown in FIG. 2;

[0027] FIG. 6 is a schematic diagram of a flow of an image generation method provided by an embodiment of the present application;

[0028] FIG. 7 is a schematic diagram of a structure of an image generation apparatus provided by an embodiment of the present application;

[0029] FIG. 8 is a schematic diagram of a structure of a computing device provided by an embodiment of the present application;

[0030] FIG. 9 is a schematic diagram of a structure of a computing device cluster provided by an embodiment of the present application;

[0031] FIG. 10 is a schematic diagram of a structure of another computing device cluster provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] The term "and / or" used in this document is used to describe an association relationship between associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in this document represents an or relationship of associated objects, for example, A / B represents A or B.

[0033] The terms "first" and "second" and the like in the description and claims of this document are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, and are not used to describe a specific order of the response messages.

[0034] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0035] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0036] Generally, in order to generate a high-resolution image, whether it is a text-to-image technology or an image-to-image technology, one method is to divide a low-resolution image into multiple image blocks after the low-resolution image is obtained, and then generate a high-resolution image for each image block respectively, and finally splice the high-resolution image blocks to form a complete high-resolution image. However, in this method, since each image block is generated separately without considering the overall content, there will be problems of inconsistent generation, random generation, and insufficient authenticity between different image blocks, resulting in poor quality of the finally generated image.

[0037] Another method is to first generate a low-resolution image using a neural network model, and then enlarge the low-resolution image to a high-resolution image through an image super-resolution model. This method generates a high-resolution image through such a multi-stage operation. However, since multiple models are involved in serial inference, this can easily reduce inference efficiency and increase deployment costs. In addition, due to the limitations of the image super-resolution model, the high-resolution image generated by it may have a blur problem in the local area, thereby affecting the authenticity of the image, resulting in poor quality of the finally generated image.

[0038] From the above description of the two methods, it can be seen that both methods directly enlarge the image with low resolution to an image with high resolution in the image domain, that is, they directly enlarge the image. In this operation, regardless of the image generation efficiency, if the image with low resolution has defects, the high-resolution image obtained finally also has defects. Therefore, if a high-resolution image with high authenticity, high quality and high resolution can be directly obtained through a single-stage operation, the effect of the text / image generation technology will be significantly improved, and a high-resolution image with high authenticity, high quality and high resolution can be obtained.

[0039] Therefore, the embodiments of the present application provide an image generation method, which can generate a high-resolution feature map in a latent space domain and decode the feature map to obtain a high-resolution image, and the high-resolution image can be directly obtained through a single-stage operation in the process of generating the high-resolution image, thereby improving the effect of the text / image generation technology.

[0040] For example, FIG. 1 shows a schematic diagram of a technical concept of image generation provided by an embodiment of the present application. As shown in FIG. 1, the overall model architecture 100 for image generation can include a low-resolution branch 110, a high-resolution branch 120, and a cascaded decoder 130. The low-resolution branch 110 is mainly used to generate at least one feature map based on material from the outside. The material can be text, image, or a joint representation of text and image (i.e., containing both text and image). In addition, in the low-resolution branch 110, the first feature map is generated based on the material from the outside, and the i-th (i≥2) feature map is generated based on the (i-1)-th feature map. The high-resolution branch 120 is mainly used to generate a high-resolution feature map based on the material from the outside and the feature map from the low-resolution branch 110. The resolution of the feature map generated by the low-resolution branch 110 can be less than or equal to the resolution of the feature map generated by the high-resolution branch 120. In this embodiment, the feature map from the low-resolution branch 110 can be fused into the feature map generated by the high-resolution branch 120 during the generation of the high-resolution feature map, so as to obtain the high-resolution feature map. In this way, the generation of the high-resolution feature map can be guided by the feature map from the low-resolution branch 110, the semantic information rich in the feature map generated by the low-resolution branch 110 can be fused into the high-resolution feature map, and the structure and details of the high-resolution feature map can be improved. The cascaded decoder 130 is mainly used to decode the high-resolution feature map output by the high-resolution branch 120 to obtain a high-resolution image. The cascaded decoder 130 can be composed of at least one decoder. It should be noted that, for the convenience of description, the feature map generated by the low-resolution branch will be referred to as a "low-resolution feature map", and the feature map generated by the high-resolution branch will be referred to as a "high-resolution feature map".

[0041] Under the model architecture 100, in the image generation process, the low-resolution feature map generated by the low-resolution branch 110 guides the generation of the high-resolution feature map, so that the semantic information rich in the low-resolution feature map can be fused into the high-resolution feature map. Since the low-resolution feature map contains rich semantic information, the final high-resolution feature map also contains these semantic information. Therefore, in the image generation process, the semantic information can be accurately understood, and high-quality structures and details can be generated, thereby reducing the difficulty of generating high-resolution images. It should be understood that when the resolution of the feature map generated by the low-resolution branch 110 is the same as the resolution of the feature map generated by the high-resolution branch 120, since the low-resolution branch 110 is not guided in the process of generating the feature map, the feature map generated by the low-resolution branch 110 may be flawed, and the high-resolution branch 120 can improve the quality of the generated feature map by being guided by the feature map generated by the low-resolution branch 110 in the process of generating the feature map.

[0042] Under the technical concept shown in FIG. 1, the parts in the model architecture 100 can be trained using data of multiple different resolutions. Then, using the trained model, high-resolution images of any resolution can be generated under given any material.

[0043] In both the training phase and the use phase, when generating images, the high-resolution branch and the low-resolution branch in the model architecture 100 can work together to guide the generation of the feature map by the high-resolution branch through the feature map generated by the low-resolution branch; and finally, the high-resolution image can be obtained by decoding the feature map generated by the high-resolution branch. Through this collaborative construction method of the high-resolution branch and the low-resolution branch, high-resolution feature maps are generated in the latent space domain, so that the model can end-to-end generate high-quality high-resolution images, and the purpose of directly obtaining images with high reality, high quality and high resolution through single-stage operation is achieved, and the effect of text-to-image generation technology is significantly improved. It should be understood that the model architecture 100 shown in FIG. 1 can be applied to scenarios such as mobile phone photography, cloud camera, image editing and image quality enhancement of the gallery, and can also be applied to virtual reality products to provide realistic ultra-high-definition images and create immersive experiences.

[0044] The neural network model constructed based on the above technical concept is described below.

[0045] Exemplarily, FIG. 2 shows a structural schematic diagram of a neural network model provided by an embodiment of the present application. As shown in FIG. 2, the neural network model 200 can include a low-resolution branch 210, a high-resolution branch 220, and a cascaded decoder 230. Among them, the low-resolution branch 210 can correspond to the low-resolution branch 110 in FIG. 1, the high-resolution branch 220 can correspond to the high-resolution branch 210 in FIG. 1, and the cascaded decoder 230 can correspond to the cascaded decoder 130 in FIG. 1. For the use of the low-resolution branch 210, the high-resolution branch 220, and the cascaded decoder 230, refer to the description of the low-resolution branch 110, the high-resolution branch 120, and the cascaded decoder 130 in FIG. 1, which will not be repeated here.

[0046] In the neural network model 200, the low-resolution branch 210 mainly comprises at least one base network module 211 and at least one implicit representation module 212. Each base network module 211 can be used to generate a feature map, which can be but is not limited to a convolutional layer, an attention module, etc. The input of the i-th base network module 211 can be the output of the (i-1)-th base network module 211. When i = 1, the input of the i-th base network module 211 is the material from the outside. One implicit representation module 212 is associated with one base network module 211. Each implicit representation module 212 can be used to up-sample the feature map of the output of the base network module 211 associated therewith, to continuously enlarge the corresponding feature map (for example, to obtain a feature map with the same resolution as the feature map generated in the high-resolution branch 220), to obtain a feature enlarged map, so as to adapt to the high-resolution branch and prepare for the subsequent fusion with the high-resolution branch. For example, as shown in FIG. 3, the implicit representation module 212 can comprise an attention module 2221, a linear layer 2222 and an implicit network 2223. The attention module 2221 mainly performs attention calculation on the tokens of the feature map participating in the attention calculation (i.e., the feature map of the output of the base network module 211 associated with the implicit representation module 212, that is, a certain “low-resolution feature map”) and the pre-trained learnable tokens, so that the learnable tokens can fully understand the features of the tokens of the feature map participating in the attention calculation. Of course, the tokens of the feature map participating in the attention calculation can also learn the features of the learnable tokens. Wherein, the tokens of the feature map participating in the attention calculation can be output by the base network module 211, or can be obtained by segmenting the feature map output by the base network module 211, which is not limited here. The linear layer 2222 is mainly used to process the tokens output by the attention module 2221 to obtain the weights of the implicit network 2223. In some embodiments, the number of tokens output by the attention module 2221 can be the same as the pre-trained learnable tokens. In addition, the tokens output by the attention module 2221 can also be one-to-one corresponding to the pre-trained learnable tokens, wherein one learnable token can be converted into one token output by the attention module 2221. Of course, the tokens output by the attention module 2221 can also be related to the feature map participating in the attention calculation, or both related to the feature map participating in the attention calculation and the learnable tokens, which can be determined according to the actual situation, which is not limited here. The implicit network 2223 is mainly used to process the normalized coordinates (x, y) after up-sampling based on the weights output by the linear layer 2222, to obtain the high-resolution feature map at the coordinate position, thereby realizing the up-sampling of the feature map participating in the attention calculation.For example, the normalized coordinates (x, y) after upsampling can be understood as the coordinates of the pixels in the feature map generated in the high-resolution branch 220. The weights of the implicit network 2223 are calculated by the linear layer 2222, and the input of the implicit network 2223 is the normalized coordinates (x, y) after upsampling. From the working process of the implicit expression module 212, it can be seen that when the feature maps input into the attention module 2221 are different, the weights output by the linear layer 2222 are also different, so there is a parameter learning process, and the weights of the implicit network can be adaptively adjusted, which can make different upsampling for different feature maps, thereby improving the upsampling effect. It should be understood that the implicit expression module 212 can also be replaced by other modules that can perform upsampling on the feature map, such as: modules that can use bilinear interpolation, nearest neighbor interpolation, and other upsampling methods, and the replaced scheme is still within the protection scope of the present application. In some embodiments, the output of each basic network module 211 in the low-resolution branch 210 can be referred to as a “low-resolution feature map”, or the output of the last basic network module 211 can be referred to as a “low-resolution feature map”, and the outputs of the other basic network modules 211 can be referred to as “intermediate feature maps”, which are not limited here. In some embodiments, when the resolution of the feature map output by the basic network module 211 is the same as the resolution of the feature map generated in the high-resolution branch 220, the implicit expression module 212 can also be selected not to be added in the low-resolution branch 210, which can be determined according to the actual situation, which is not limited here.

[0047] The high-resolution branch 220 mainly comprises at least one base network module 221 and at least one latent space guiding module 222. One base network module 221 is associated with one latent space guiding module 222, and the two are in a one-to-one correspondence. One base network module 221 can be used to generate a feature map, which can be but is not limited to a convolutional layer, an attention module, etc. Among them, the resolution of the feature map generated by the base network module 221 can be greater than or equal to the resolution of the feature map generated by the base network module 211 in the low-resolution branch 210. One latent space guiding module 222 can be used to fuse a feature map from the low-resolution branch 210 and a feature map generated by the base network module 221 associated with the latent space guiding module 222 to obtain a fused feature map. It should be understood that it is difficult to generate a high-resolution feature map at one step, and it is a more feasible approach to use the "small first and then large" strategy to first generate a feature map with a lower resolution and then gradually guide the generation of a feature map with a higher resolution. Therefore, the main idea of the latent space guiding module 222 is to use the low-resolution feature map rich in semantic information to guide the generation of the high-resolution feature map, reduce the difficulty of generating the high-resolution feature map, and accurately understand the semantic information to generate high-quality structures and details. For example, as shown in FIG. 4, the workflow of the latent space guiding module 222 can be: first, splicing the feature map from the low-resolution branch 210 and the feature map generated by the base network 221 in the channel dimension; then, inputting the spliced feature into the linear layer 2221 for residual learning; and finally, adding the residual output by the linear layer 2221 to the feature map generated by the base network 221 to complete the fusion of the feature map, thereby obtaining a fused feature map. It should be understood that in addition to the fusion mode shown in FIG. 4, fusion can also be performed through addition (such as directly adding the feature map from the low-resolution branch 210 and the feature map generated by the base network 221), adaptive instance normalization, etc. The specific mode can be determined according to actual conditions, which is not limited here. Referring to FIG. 2, it can be seen from FIG. 2 that one implicit representation module 212 in the low-resolution branch 210 is associated with one latent space guiding module 222 in the high-resolution branch 220, and the two are in a one-to-one correspondence. In addition, in the high-resolution branch 220, a latent space guiding module 222 can be configured for each base network module 221, or only a part of the base network modules 221 can be configured with a latent space guiding module 222. The specific mode can be determined according to actual conditions, which is not limited here. In the high-resolution branch 220, the output of a certain latent space guiding module 222 can be used as the input of a base network module 221 or as the final output of the high-resolution branch 220.It should be understood that in FIG. 4, when the resolution of the feature map generated by the low-resolution branch 210 is the same as the resolution of the feature map generated by the high-resolution branch 220, the feature map from the low-resolution branch 210 and the feature map generated by the base network 221 can be directly spliced in the channel dimension without further upsampling the feature map from the low-resolution branch 210.

[0048] In the neural network model 200, considering that the feature maps of different resolutions usually have different distributions. If a single model is used to process feature maps of different resolutions without explicitly injecting resolution information into the model, the model will be difficult to converge due to the lack of resolution perception. To solve this problem, at least one resolution adaptive module 223 can also be added in the high-resolution branch 220. One resolution adaptive module 223 can be associated with one base network module 221. One resolution adaptive module 223 can dynamically adjust the mean and standard deviation of the feature map output from the base network module 221 associated therewith according to the resolution information specified by the user, to ensure that the model can generate high-quality details at any resolution. For example, when a base network module 221 is associated with both a latent space guiding module 222 and a resolution adaptive module 223, the base network module 221, the latent space guiding module 222 and the resolution adaptive module 223 can be connected in series. In this embodiment, the resolution adaptive module 223 can normalize features of different resolutions, and then use adaptive instance normalization to modulate the high-resolution feature map by the user-set resolution feature, thereby reducing the difficulty of model training, which is the main motivation of this module. Specifically, as shown in FIG. 5, the workflow of the resolution adaptive module 223 can be: processing the user-selected resolution feature (which can be obtained by feature extraction on the user-selected resolution) through a linear layer 2231 to obtain the mean and standard deviation of the resolution feature. And, normalizing the received high-resolution feature map through a normalization layer 2232. Then, the normalized high-resolution feature map can be multiplied by the standard deviation output by the linear layer 2231, and the multiplication result can be added to the mean output by the linear layer 2231, so that the mean and standard deviation of the high-resolution feature map can be converted to the mean and standard deviation of the target resolution feature (i.e. the user-selected resolution feature). That is, the resolution adaptive module 223 can dynamically calculate the mean and standard deviation of the normalization layer according to different resolutions, and then normalize the high-resolution feature map. Due to the presence of the resolution adaptive module 223, the model can generate high-quality details at any resolution. It should be understood that the resolution adaptive module 223 can also be added in the high-resolution branch 120 of the model architecture 100 shown in FIG. 1 to adjust the resolution of the high-resolution feature map generated thereby.

[0049] The following describes a specific implementation process of the above technical concept and neural network model.

[0050] For example, FIG. 6 shows a flowchart of an image generation method provided by an embodiment of the present application. It can be understood that the method can be executed by any device, equipment, platform, or cluster of equipment having computing and processing capabilities. For example, the method can be executed by an image generation device, which can be implemented by software and / or hardware, and can be configured in a computing device, typically, a mobile phone, a computer, or a server. For ease of description, the following will be described with the computing device as the execution subject. As shown in FIG. 6, the image generation method can include the following steps:

[0051] S601, obtaining first material.

[0052] In this embodiment, the first material can be text, image, or a combination of text and image (i.e., containing both text and image). The first material can be input by a user. For example, the user can input a piece of text on the terminal side, such as "a hearty traditional breakfast, including fluffy scrambled eggs, crispy bacon, golden hash browns, and buttered toast. The plate is decorated with fresh parsley, accompanied by a glass of freshly squeezed orange juice and a steaming cup of coffee. The background is a cozy kitchen environment, with morning sunlight shining through the window". The user can also input an image on the terminal side and provide corresponding text, such as "improve the resolution of the above image". The user can also input only an image on the terminal side, and so on. In addition, the first material can also be collected by the computing device. For example, during the user's process of taking a photo with a mobile phone, the image of a certain scene taken by the mobile phone can be the first image, and so on. It should be understood that the way to obtain the first material can be selected according to actual conditions, which is not limited here.

[0053] S602, generating at least one first feature map based on the first material, and generating a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map.

[0054] In this embodiment, after obtaining the first material from the outside, the computing device can use a certain branch network in the neural network model configured thereon to process the first material to generate at least one first feature map. When there are multiple first feature maps, the first first feature map is generated based on the first material, and the i-th first feature map is generated based on the (i-1)-th first feature map, i≥2. For example, the model architecture of the neural network model configured on the computing device can be, but is not limited to, the model architecture shown in FIG. 1. At this time, the branch network for generating the first feature map can be the low-resolution branch 110 shown in FIG. 1.

[0055] Meanwhile, the computing device can utilize another branch network in the neural network model configured thereon to process the first material to generate a second feature map. Each of the first feature maps has a resolution less than or equal to a resolution of the second feature map. Illustratively, the model architecture of the neural network model configured on the computing device can be, but is not limited to, the model architecture shown in FIG. 1. At this time, the branch network generating the second feature map can be the high resolution branch 120 shown in FIG. 1, and the second feature map can be the first feature map generated by the high resolution branch 120.

[0056] S603, fuse the at least one first feature map into at least one target feature map to obtain a third feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map.

[0057] In this embodiment, after obtaining the first feature map and the second feature map, the computing device can fuse at least one first feature map obtained by it into a target feature map to obtain a third feature map. When fusing the feature maps, different first feature maps can be fused with different target feature maps, that is, they are in a one-to-one correspondence. Of course, a plurality of first feature maps can also be fused together (such as addition, etc.) first, and then the fused feature map is fused into the target feature map, which can be determined according to actual conditions, and is not limited here. The target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map. Among them, the first feature map is mainly used to guide the generation of the third feature map. In this embodiment, the computing device can choose to fuse only one first feature map into the second feature map, or into a feature map obtained by processing the second feature map. The computing device can also choose to fuse a plurality of first feature maps into the second feature map and a certain (or certain) feature map obtained by processing the second feature map, or fuse one first feature map into the second feature map and fuse other first feature maps into a certain (or certain) feature map obtained by processing the second feature map. The computing device can also choose to fuse all first feature maps into the second feature map and a certain (or certain) feature map obtained by processing the second feature map, or fuse one first feature map into the second feature map and fuse other first feature maps into a certain (or certain) feature map obtained by processing the second feature map. In some embodiments, when there are a plurality of first feature maps, these feature maps can be fused with the corresponding target feature maps in turn. Among them, the target feature map used for the first fusion can be the second feature map, or a certain feature map obtained by processing the second feature map. The target feature map used for the kth (k≥2) fusion is obtained based on the fusion feature map obtained by the k-1th fusion. That is, when there are a plurality of first feature maps to be fused, one first feature map can be fused into a target feature map first to obtain a fusion feature map; then another first feature map is fused into another target feature map, and the target feature map at this time can be the fusion feature map obtained by the previous fusion, or a certain feature map obtained by processing the fusion feature map obtained by the previous fusion. In this embodiment, the third feature map can be the fusion feature map obtained by the last fusion, or a feature map obtained by processing the fusion feature map obtained by the last fusion. In some embodiments, after each time a first feature map is fused with a target feature map, the mean and standard deviation of the fused feature map can be adjusted according to the resolution specified by the user to optimize the details of the fused feature map and improve the display effect of the subsequent image.

[0058] As a possible implementation manner, when the resolutions of the first feature map and the second feature map are the same, when a certain first feature map is fused into a certain target feature map, the first feature map can be directly fused into the corresponding target feature map. For example, first, residual learning can be performed based on the first feature map and the corresponding target feature map (for example, processing the two through a certain neural network, etc.), to obtain a residual. Then, the obtained residual can be added to the target feature map.

[0059] As another possible implementation manner, when the resolution of the first feature map is less than the resolution of the second feature map, when a certain first feature map (hereinafter referred to as a "fourth feature map") is fused into a certain target feature map (hereinafter referred to as a "first target feature map", which is the target feature map that needs to be fused with the fourth feature map), the fourth feature map can be first up-sampled to obtain a feature enlarged map. The resolution of the feature enlarged map is the same as that of the first target feature map. For example, attention calculation can be first performed on the token of the fourth feature map and at least one first token, to obtain at least one second token. In some embodiments, the number of second tokens can be the same as that of first tokens. For example, the second tokens can correspond to the first tokens one by one. Wherein, after a certain first token learns the information in the fourth feature map, the first token can be converted into a second token. Of course, after a certain token of the fourth feature map learns the information of the first token, it can also be converted into a second token, which can be determined according to actual conditions, which is not limited here.

[0060] After obtaining the second token, the weight of the implicit network can be determined based on the obtained second token. For example, the obtained second token can be input into a certain linear layer in the neural network model, to obtain the weight of the implicit network. Finally, the coordinates of the pixel points in the first target feature map are input into the implicit network, and the implicit network can output the feature enlarged map. For the process of obtaining the feature enlarged map, please refer to the related description in the foregoing FIG. 3.

[0061] After obtaining the feature enlarged map, the feature enlarged map can be fused into the first target feature map. For example, first, residual learning can be performed based on the feature enlarged map and the first target feature map (for example, processing the two through a certain neural network, etc.), to obtain a residual. Then, the obtained residual can be added to the first target feature map.

[0062] S604, decoding the third feature map to obtain a target image, wherein the target image meets the image generation requirement expressed by the first material.

[0063] In this embodiment, after obtaining the third feature map, the third feature map can be decoded to obtain the target image. The target image is consistent with the image generation requirement expressed by the first material.

[0064] In this way, by generating a high-resolution feature map in the latent space domain and guiding the feature maps obtained by different branches during the generation process, the semantic information rich in the feature maps obtained by other branches can be fused into the high-resolution feature map, the structure and details of the high-resolution feature map are improved, and the difficulty of generating a high-resolution image is reduced. In addition, this method can directly obtain an image with high reality, high quality and high resolution through a single-stage operation, and significantly improves the effect of text / image generation technology.

[0065] It can be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, the technical features described in each of the above embodiments or related to the embodiments can be combined according to the actual situation, and the combined scheme is still within the protection scope of the present application.

[0066] Based on the method in the above embodiments, the present embodiment further provides an image generation device.

[0067] For example, FIG. 7 shows a structural schematic diagram of an image generation device provided by an embodiment of the present application. As shown in FIG. 7, the image generation device 700 includes an acquisition module 701 and a processing module 702. The acquisition module is configured to acquire a first material. The processing module is configured to generate at least one first feature map based on the first material, and generate a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map. The processing module is further configured to fuse at least one first feature map into at least one target feature map to obtain a third feature map, wherein different first feature maps are fused with different target feature maps, the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is the same as that of the second feature map. The processing module is further configured to decode the third feature map to obtain a target image, wherein the target image is consistent with the image generation requirement expressed by the first material.

[0068] In some embodiments, the resolution of the first feature map is less than the resolution of the second feature map; the fourth feature map is any one of the at least one first feature map, and the first target feature map is a target feature map that is fused with the fourth feature map. At this time, when the processing module fuses the fourth feature map into the first target feature map, it is specifically used for: up-sampling the fourth feature map to obtain a feature enlarged map, the resolution of the feature enlarged map being the same as the resolution of the first target feature map; and fusing the feature enlarged map into the first target feature map.

[0069] In some embodiments, when the processing module up-samples the fourth feature map to obtain the feature enlarged map, it is specifically used for: performing attention calculation on the token and the at least one first token of the fourth feature map to obtain at least one second token; obtaining the weight of the implicit network based on the at least one second token; and inputting the coordinates of the pixel points in the second feature map into the implicit network to obtain the feature enlarged map.

[0070] In some embodiments, when the processing module fuses the feature enlarged map into the first target feature map, it is specifically used for: performing residual learning based on the feature enlarged map and the first target feature map to obtain a first residual; and adding the first residual to the first target feature map to fuse the feature enlarged map into the first target feature map.

[0071] In some embodiments, after the processing module fuses the feature enlarged map into the first target feature map, it is further used for: adjusting the mean and the standard deviation of the fused feature map according to the resolution specified by the user, wherein the fused feature map is a feature map obtained by fusing the feature enlarged map into the first target feature map.

[0072] In some embodiments, the first feature map is a plurality of first feature maps, wherein the first first feature map is generated based on the first material, and the i-th first feature map is generated based on the (i-1)-th first feature map, i≥2.

[0073] In some embodiments, the acquisition module 701 and the processing module 702 shown in FIG. 7 can be implemented by software or by hardware. For example, the implementation of the acquisition module 701 is described below. Similarly, the implementation of the processing module 702 can refer to the implementation of the acquisition module 701.

[0074] As an example of a software functional unit, the obtaining module 701 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the obtaining module 701 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers for running the code can be distributed in the same availability zone (AZ), or in different AZs, each AZ including one data center or multiple data centers in close geographical proximity. Generally, one region can include multiple AZs.

[0075] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Generally, one VPC is set up within one region, and communication between two VPCs in the same region, or between VPCs in different regions, needs to be set up through a communication gateway in each VPC to achieve interconnection between VPCs.

[0076] As an example of a hardware functional unit, the obtaining module 701 can include at least one computing device, such as a server, etc. Alternatively, the obtaining module 701 can also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD can be implemented by a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0077] The plurality of computing devices included in the obtaining module 701 can be distributed in the same region or in different regions. The plurality of computing devices included in the obtaining module 701 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the obtaining module 701 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0078] It should be noted that in other embodiments, the obtaining module 701 can be configured to perform any of the steps of the image generation method described in the above embodiments, and the processing module 702 can be configured to perform any of the steps of the image generation method described in the above embodiments. In addition, the steps implemented by the obtaining module 701 and the processing module 702 can be specified as needed, and the entire function of the image generation apparatus 700 shown in FIG. 7 can be implemented by the obtaining module 701 and the processing module 702 implementing different steps of the image generation method described in the above embodiments, respectively.

[0079] The present application also provides a computing device 800. As shown in FIG. 8, the computing device 800 includes a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other through the bus 802. The computing device 800 can be a server or an electronic device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0080] The bus 802 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is shown in FIG. 8, but it does not mean that there is only one bus or only one type of bus. The bus 804 can include a path for transmitting information between various components (e.g., the memory 806, the processor 804, the communication interface 808) of the computing device 800.

[0081] The processor 804 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0082] The memory 806 can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM), floppy disks, or hard disks.

[0083] The memory 806 stores executable program code that the processor 804 executes to respectively implement the functions of the aforementioned acquisition module 701 and processing module 702 shown in FIG. 7, thereby implementing the image generation method described in the above embodiments. That is, the memory 806 stores instructions for executing the image generation method described in the above embodiments.

[0084] Alternatively, the memory 806 stores executable program code that the processor 804 executes to respectively implement the functions of the aforementioned image generation apparatus 700 shown in FIG. 7, thereby implementing the image generation method described in the above embodiments. That is, the memory 806 stores instructions for executing the image generation method described in the above embodiments.

[0085] The communication interface 803 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 800 and other devices or communication networks.

[0086] The embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a laptop computer, or an electronic device such as a smartphone.

[0087] As shown in FIG. 9, the computing device cluster includes at least one computing device 800. The memory 806 in one or more computing devices 800 in the computing device cluster can store the same instructions for executing the image generation method described in the above embodiments.

[0088] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster can also respectively store partial instructions for executing the image generation method described in the above embodiments. In other words, the combination of one or more computing devices 800 can collectively execute the instructions for executing the image generation method described in the above embodiments.

[0089] It should be noted that the memories 806 in different computing devices 800 in the computing device cluster can store different instructions for performing the partial functions of the image generation apparatus 700 shown in FIG. 7 respectively. That is, the instructions stored in the memories 806 in different computing devices 800 can implement the functions of one or more of the obtaining module 701 and the processing module 702.

[0090] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network, a local area network, or the like. FIG. 10 shows one possible implementation manner. As shown in FIG. 10, two computing devices 800A and 800B are connected through a network. Specifically, the computing devices are connected to the network through the communication interfaces in the computing devices. In this kind of possible implementation manner, the memory 806 in the computing device 800A stores instructions for performing the functions of the obtaining module 701. Meanwhile, the memory 806 in the computing device 800B stores instructions for performing the functions of the processing module 702.

[0091] It should be understood that the functions of the computing device 800A shown in FIG. 10 can also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B can also be completed by multiple computing devices 800.

[0092] Embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to FIG. 9 and FIG. 10. The difference is that the memories 806 in one or more computing devices 800 in the computing device cluster can store the same instructions for performing the method in the above-described embodiments.

[0093] In some possible implementation manners, the memories 806 in one or more computing devices 800 in the computing device cluster can also respectively store partial instructions for performing the data processing method described above. In other words, the combination of one or more computing devices 800 can collectively execute the instructions for performing the data processing method described above.

[0094] Based on the method in the above embodiments, the embodiments of the present application provide a computer readable storage medium including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiments; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiments. Illustratively, the computer readable storage medium can be any available medium or data storage device that can be accessed by the computing device or a data center including one or more available media or data storage devices. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc.

[0095] Based on the method in the above embodiments, the embodiments of the present application provide a computer program product including instructions, when the instructions are run by a computing device, the computing device executes the method in the above embodiments, or, when the instructions are run by a computing device cluster, the computing device cluster executes the method in the above embodiments.

[0096] It can be understood that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0097] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0098] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center through a wired (such as a coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0099] It can be understood that the various numerical numbers involved in the embodiments of the present application are only for the convenience of differentiation, and do not limit the scope of the embodiments of the present application.

[0100] Finally, it should be noted that: the above examples are used to illustrate the technical solutions of the present application, but not limited to them; although the present application is described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image generation method characterized by, The method comprises: acquiring a first material; generating at least one first feature map based on the first material, and generating a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map; fusing at least one first feature map into at least one target feature map to obtain a third feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and the resolution of the third feature map is greater than or equal to the resolution of the second feature map; decoding the third feature map to obtain a target image, wherein the target image meets the image generation requirement expressed by the first material.

2. The method of claim 1, wherein, The resolution of the first feature map is less than the resolution of the second feature map. The fourth feature map is any one of the at least one first feature map, and the first target feature map is the feature map in the at least one target feature map that is fused with the fourth feature map. Fusing the fourth feature map into the first target feature map comprises: upsampling the fourth feature map to obtain a feature magnification map, wherein the resolution of the feature magnification map is the same as the resolution of the first target feature map; fusing the feature magnification map into the first target feature map.

3. The method of claim 2, wherein, Upsampling the fourth feature map to obtain a feature magnification map comprises: performing attention calculation on the token and at least one first token of the fourth feature map to obtain at least one second token; obtaining the weight of the implicit network based on the at least one second token; inputting the coordinates of the pixel points in the first target feature map into the implicit network to obtain the feature magnification map.

4. The method according to claim 2 or 3, characterized in that, Fusing the feature magnification map into the first target feature map comprises: performing residual learning based on the feature magnification map and the first target feature map to obtain a first residual; adding the first residual to the first target feature map to fuse the feature magnification map into the first target feature map.

5. The method according to any of claims 2-4, characterized by, After fusing the feature magnification map into the first target feature map, it further comprises: adjusting the mean and standard deviation of the fused feature map according to the resolution specified by the user, wherein the fused feature map is the feature map obtained by fusing the feature magnification map into the first target feature map.

6. The method according to any one of claims 1 to 5, characterized in that, The first feature map is a plurality of first feature maps, wherein the first first feature map is generated based on the first material, and the i-th first feature map is generated based on the (i-1)-th first feature map, i≥2.

7. An image generation apparatus characterized by comprising: It comprises: an acquisition module for acquiring a first material; a processing module for generating at least one first feature map based on the first material, and generating a second feature map based on the first material, wherein the resolution of each first feature map is less than or equal to the resolution of the second feature map; The processing module is further configured to fuse the at least one first feature map into at least one target feature map to obtain a third feature map, wherein the target feature map is the second feature map or is obtained by processing the second feature map, and a resolution of the third feature map is greater than or equal to a resolution of the second feature map; The processing module is further configured to decode the third feature map to obtain a target image, wherein the target image meets an image generation requirement expressed by the first material.

8. The apparatus of claim 7, wherein, The resolution of the first feature map is less than the resolution of the second feature map; The fourth feature map is any one of the at least one first feature map, and the first target feature map is a feature map in the at least one target feature map that is fused with the fourth feature map; The processing module is configured to: perform upsampling on the fourth feature map to obtain a feature magnification map, wherein a resolution of the feature magnification map is the same as a resolution of the first target feature map; fuse the feature magnification map into the first target feature map.

9. The apparatus of claim 8, wherein, The processing module is configured to, when performing upsampling on the fourth feature map to obtain a feature magnification map: perform attention calculation on a token of the fourth feature map and at least one first token to obtain at least one second token; obtain a weight of an implicit network based on the at least one second token; input coordinates of a pixel point in the first target feature map into the implicit network to obtain the feature magnification map.

10. The apparatus of claim 8 or 9, wherein, The processing module is configured to, when fusing the feature magnification map into the first target feature map: perform residual learning based on the feature magnification map and the first target feature map to obtain a first residual; add the first residual to the first target feature map to fuse the feature magnification map into the first target feature map.

11. The apparatus of any of claims 8-10, wherein, The processing module is further configured to, after fusing the feature magnification map into the first target feature map: adjust a mean value and a standard deviation of a fused feature map according to a resolution specified by a user, wherein the fused feature map is a feature map obtained by fusing the feature magnification map into the first target feature map.

12. The apparatus of any of claims 7-11, wherein, The first feature maps are a plurality of first feature maps, wherein a first first feature map is generated based on the first material, and an i-th first feature map is generated based on an (i-1)-th first feature map, i≥2.

13. A cluster of computing devices, characterized in that, The at least one computing device each includes a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, The computer program instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-6, wherein the computing device cluster includes at least one computing device.

15. A computer program product comprising instructions, characterized in that, when the instructions are executed by the cluster of computing devices, cause the cluster of computing devices to perform the method of any of claims 1-6, wherein the cluster of computing devices comprises at least one computing device.

Images