Image generation method and device, equipment and medium

Abstract

The invention discloses an image generation method and device, equipment and a medium, and belongs to the technical field of image generation. The method comprises the following steps: obtaining quantitative sensitivity of each network layer in a first model, wherein the first model comprises at least one network layer; determining a first network layer in the at least one network layer according to the quantization sensitivity of each network layer and a first sensitivity threshold; receiving a first input to the first model; and under the condition that the first model performs image generation reasoning according to the first input and the at least one network layer, performing quantification processing on the first network layer to obtain a first image generated by the first model. According to the scheme, the quality of the generated image can be guaranteed in the process of generating the image, and meanwhile the image generation speed is increased.

Description

Technical Field

[0001] The present application belongs to the field of image generation technology, and specifically relates to an image generation method, device, equipment and medium. Background Art

[0002] Since images are more intuitive than text, users usually have a stronger demand for images. However, in real life, images can usually only be obtained by taking pictures with a camera, which greatly limits user demand.

[0003] In the prior art, there is a method of automatically generating images using a neural network model. This method mainly involves deploying a trained neural network model on an electronic device. The user then tells the user what kind of image they want, and the neural network model will automatically generate the image.

[0004] However, this method is heavily dependent on the hardware computing resources of electronic devices. The image generation process is prone to problems such as taking too long and poor output image quality, resulting in poor results for this automatic image generation method. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image generation method, device, equipment and medium, which can solve the technical problems in the related art that the automatic generation of images takes a long time and the output image quality is poor, resulting in poor automatic image generation effect.

[0006] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0007] Obtaining a quantized sensitivity of each network layer in a first model, wherein the first model includes at least one network layer;

[0008] Determining a first network layer in the at least one network layer according to the quantized sensitivity of each network layer and the first sensitivity threshold;

[0009] receiving a first input to the first model;

[0010] When the first model performs image generation inference based on the first input and the at least one network layer, the first network layer is quantized to obtain a first image generated by the first model.

[0011] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0012] A sensitivity acquisition module, used to acquire the quantitative sensitivity of each network layer in a first model, wherein the first model includes at least one network layer;

[0013] A network layer determination module, configured to determine a first network layer in the at least one network layer according to the quantized sensitivity of each network layer and a first sensitivity threshold;

[0014] An input receiving module, configured to receive a first input to the first model;

[0015] A quantization reasoning module is used to quantize the first network layer to obtain a first image generated by the first model when the first model performs image generation reasoning based on the first input and the at least one network layer.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0020] In an embodiment of the present application, by first determining which network layers in the model have less impact on the image quality, and then selectively quantizing these network layers, the quantized network layers can complete image generation faster, thereby improving the image generation speed. At the same time, it also ensures that the difference between the first image generated by the first model after the network layer is quantized and the second image generated by the second model without quantization of the network layer is within the set threshold range, which can avoid quality problems in the image generated by the first model after the network layer is quantized, and improve the model's image processing speed, ultimately achieving the purpose of improving the image generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of a scene of a Vincent diagram provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of an image generation method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the network layer structure in the model provided in the embodiment of the present application;

[0024] Figure 4 A schematic diagram of a quantitative sensitivity determination process provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a sensitivity data determination process provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a second image generation process provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of generating a second image provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of a third image generation process provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of generating a third image provided in an embodiment of the present application;

[0030] Fig.10 A schematic diagram of the Wensheng diagram process provided in the embodiment of the present application;

[0031] Fig.11 A schematic diagram of calculating the quantitative sensitive data provided in an embodiment of the present application;

[0032] Fig.12 A flowchart for determining the network layer that can be quantified and processed provided by an embodiment of the present application;

[0033] Fig.13 A schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0034] Fig.14 A schematic diagram of the framework of an electronic device provided in an embodiment of the present application;

[0035] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0037] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0038] The image generation method, device, equipment and medium provided in the embodiments of the present application are described in detail below in conjunction with specific embodiments and their application scenarios.

[0039] The image generation method provided in the embodiment of the present application can be applied to cultural image scenarios. For example, when a user is training a model, a large number of images may be required as training data. Thus, a large number of images can be generated by the image generation method, which avoids the user having to use a camera to take time-consuming and laborious photos of real objects in order to obtain a variety of images.

[0040] In simple terms, the text-generated image takes text information as the input of the model, and then the model generates an image that matches the text information through reasoning. For example, if the text information is "a black cat", then the model needs to infer and generate an image containing a black cat.

[0041] For example, Figure 1 A scene diagram of a Vincent diagram provided in an embodiment of the present application, such as Figure 1 As shown, the above models are usually deployed in electronic devices, and text information (such as Figure 1 The text prompt words in the text message may be obtained by recognizing the user's voice and then converting it. In addition, the text information may also be directly input into the electronic device by the user through an input device (such as a keyboard, a handwriting tablet, a touch screen, and a scanner, etc.). The electronic device may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wearable device, a vehicle-mounted terminal, etc.

[0042] Among them, the above model can be a diffusion model, and the core components of the diffusion model include the U-NET network. Performing multiple noise predictions through the U-NET network is the core process in the diffusion model reasoning and is the most time-consuming part in the reasoning. Taking the above model as a diffusion model as an example, the diffusion model can use text information (such as text prompt words) to infer and generate images, that is, to realize the Wensheng map. In the Wensheng map reasoning process, it is necessary to predict the sampling noise through multiple iterations and obtain the target image expected by the user through a step-by-step denoising method. Since it involves multiple iterations, the reasoning process of the Wensheng map is a computationally intensive task, which has very high requirements for the computing power of the reasoning hardware. In the related art, high-computing enterprise-level graphics processing unit (GPU) reasoning hardware is mainly used, but even so, the reasoning time of the Wensheng map is about 5 to 10 seconds, which is relatively long.

[0043] In response to the problems encountered in the above scenarios, the image generation solution provided by the embodiments of the present application can optimize the model, thereby effectively reducing the reasoning time of the text image and improving the image generation efficiency. At the same time, it can also ensure the quality of the final generated image and avoid generating low-quality images that do not meet user needs.

[0044] Figure 2 This is a flowchart of an image generation method provided in an embodiment of the present application. The method can be applied to an electronic device, and the electronic device may include Figure 1 Electronic equipment shown.

[0045] like Figure 2 As shown, the method may include the following steps:

[0046] Step S210: Obtain the quantized sensitivity of each network layer in the first model, wherein the first model includes at least one network layer.

[0047] Step S220: Determine a first network layer in at least one network layer according to the quantized sensitivity of each network layer and the first sensitivity threshold.

[0048] Step S230: Receive a first input to the first model.

[0049] Step S240: When the first model performs image generation inference based on the first input and at least one network layer, the first network layer is quantized to obtain a first image generated by the first model.

[0050] Among them, the image generation inference speed of the first network layer after quantization processing is greater than that of the first network layer before quantization processing, and the image quality inferred by the first network layer after quantization processing is higher than that of the first network layer before quantization processing.

[0051] The above steps are described in detail below:

[0052] In step S210, the first model may refer to the above-mentioned diffusion model, which includes a U-NET network, and the U-NET network structure includes a linear layer and a convolutional layer. The linear layer is a fully connected layer that accepts an input tensor and converts it into an output tensor. The mathematical formula is y=x*A T +b, where x is the input, A is the weight of the linear layer, b is the bias of the linear layer, and y is the output of the linear layer. The convolution layer is also called the Conv2d layer, which is a two-dimensional image convolution operation used to extract image features and convert the number of channels of input features. Exemplarily, the first model can also be various other text-based graph models, all of which include at least one network layer to realize the generation of images based on text information reasoning, which will not be introduced one by one here.

[0053] Among them, the quantization sensitivity is used to characterize whether the network layer is suitable for performing quantization processing during the Wensheng graph reasoning process. Specifically, the quantization sensitivity can refer to the influence of the network layer on the quality of the generated image during the Wensheng graph reasoning process using the network layer by the model. The higher the quantization sensitivity, the greater the influence of the network layer on the image quality, and the lower the quantization sensitivity, the smaller the influence of the network layer on the image quality.

[0054] Exemplarily, the higher the quantization sensitivity, the less suitable it is for performing quantization processing, and the lower the quantization sensitivity, the more suitable it is for performing quantization processing. A first sensitivity threshold can be configured in combination with actual conditions. If the quantization sensitivity of the network layer is greater than the first sensitivity threshold, the network layer is not suitable for performing quantization processing. Conversely, if the quantization sensitivity of the network layer is less than or equal to the first sensitivity threshold, the network layer is suitable for performing quantization processing.

[0055] In this embodiment, during the reasoning process of the first model for the Wensheng graph, the image generation reasoning speed of the network layer can be improved by quantizing the network layer, thereby shortening the time consumption. However, this is likely to lead to a decrease in image quality. In order to ensure the quality of the final generated image, the above-mentioned quantization sensitivity is used to measure whether the network layer is suitable for quantization processing.

[0056] In step S220, the first model generally includes several network layers, for example, the U-NET network described above includes at least a linear layer and a convolutional layer. In the process of reasoning the Wensheng graph, some network layers may be suitable for quantization processing, while some network layers may not be suitable for quantization processing, and for this purpose, the quantization sensitivity and the first sensitivity threshold are used for judgment.

[0057] For example, the network layer can be quantized, which means that in the process of image reasoning, if the network layer is quantized, the image generation reasoning speed is improved, the time consumption is shortened, and the network layer can still output high-quality images. If the quality of the output image decreases after the network layer is quantized, the network layer is not suitable for quantization.

[0058] In this embodiment, the number of first network layers is not limited, and the first model may include one first network layer or multiple first network layers. The more the number of first network layers, the shorter the time required for subsequent Wensheng graph reasoning. Exemplarily, the first network layer may refer to a linear layer and / or a convolutional layer in the above-mentioned U-NET network.

[0059] Figure 3 A schematic diagram of the network layer structure in the model provided in the embodiment of the present application, such as Figure 3 As shown in the figure, each network layer is arranged in sequence, and the input information is first input to the first network layer at the front of the sorting. After the image generation reasoning is performed through the first network layer, the output of the first network layer is used as the input of the next network layer. After reasoning is performed in sequence through each network layer, the reasoning result is finally output by the Nth network layer at the end of the sorting.

[0060] Exemplary, reference Figure 3 According to the arrangement order of the network layers, the first network layer may refer to the second network layer and the N-1th network layer.

[0061] In step S230, the first input may be triggered by a user, and the first input may refer to the above-mentioned text information.

[0062] Exemplarily, the above-mentioned first input includes, but is not limited to: the touch input of the electronic device by the user through a touch device such as a finger or a stylus, or the voice conversion of the voice command input by the user, or the conversion of a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and are not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a dragging gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the touch input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input. For example, the above-mentioned first input can be: the user utters a voice "I want a picture of a kitten", and then the electronic device converts the voice into text through voice recognition.

[0063] In step S240, the first input is used as the input of the first model, and the first model performs text-graph reasoning according to the first input to generate a first image.

[0064] Among them, the various network layers in the first model need to be used in the process of Wensheng graph reasoning, such as the above Figure 3 In the network, each network layer is arranged in sequence, and the output of the previous network layer is used as the input of the next network layer. The final output is obtained after reasoning through each network layer.

[0065] Exemplarily, quantizing the first network layer may refer to reducing the data accuracy of the parameters in the first network layer. The first model is obtained through training in advance, and each network layer contains optimized parameters. These optimized parameters can ensure that the first model generates a first image that matches the first input based on the first input. For example, assuming that the first model has not been trained in advance and the first input is "a kitten", the generated first image may contain a puppy, that is, the content in the first image does not match the first input. The quantization process is performed on the basis of the first model after training.

[0066] Exemplarily, taking the network parameters of the first network layer including weight values ​​and activation values ​​as an example, the quantization process may refer to low-bit quantization, that is, converting the original high-precision floating-point weight values ​​and activation values ​​into low-bit weight values ​​and activation values. For example, converting the original high-precision floating-point (FP32 / FP16) weights and activation values ​​(Weight and Activation, W&A) into low-bit specific numbers (such as INT8, FP8, INT4, etc.). Low-bit quantization can reduce the model's video memory overhead and computational complexity, reduce memory usage and computational complexity during model inference, thereby improving inference performance and shortening inference time.

[0067] Among them, FP32 and FP16 are floating-point representations used in deep learning models. FP32 represents single-precision floating-point numbers, each value occupies 32 bits; FP16 represents half-precision floating-point numbers, each value occupies 16 bits. These two formats provide high precision and a wide range of values, but also mean more memory usage and longer calculation time. INT8 represents 8-bit integers, each value occupies 8 bits; and FP8 represents 8-bit floating-point numbers, which are floating-point numbers with a smaller bit width than FP16. By using these low-bit-width data types, the size and computational complexity of the model can be significantly reduced while maintaining relatively high performance.

[0068] In the related art, although there is also a technique of using low-bit quantization to quantize the network layer to reduce the time consumption of inference, this will bring errors in calculation accuracy, and ultimately lead to a decrease in the quality of the generated image. In contrast, this embodiment selectively performs quantization, that is, firstly, according to the quantization sensitivity and the first sensitivity threshold, it determines which network layers are suitable for quantization from all network layers (that is, determines the first network layer), and then specifically performs quantization processing on the first network layer, and does not perform quantization processing on other network layers except the first network layer, which can ensure the quality of the generated image and shorten the time consumption.

[0069] The image quality includes at least one of image aesthetics and image-text consistency. Image-text aesthetics may mean that the generated image becomes blurred, distorted, lacks details, and characters are deformed. Image-text consistency may mean that the generated image does not comply with the text information input by the user, and sometimes even completely violates the text information (for example, the input text is "I want a cat image", and the output image contains a dog).

[0070] Exemplarily, in some embodiments, in step S240, when performing quantization processing, the following steps need to be performed to generate the first image:

[0071] 1. Reduce the parameters in the first network layer from the first data precision to the second data precision;

[0072] 2. Reduce the input data of the first network layer from the first data precision to the second data precision;

[0073] 3. Perform inference on the input data by reducing the parameters to the second data precision to obtain the inference result;

[0074] 4. Improve the data accuracy of the inference result to the first data accuracy;

[0075] 5. According to the inference result with improved accuracy, the first image is obtained.

[0076] Wherein, taking the diffusion model introduced above as the first model as an example, the network layer in the U-NET network in the diffusion model at least includes a linear layer and a convolutional layer. The first network layer may refer to any one or more network layers in the U-NET network.

[0077] Among them, the parameters of the first network layer may include weight values ​​and activation values. Exemplarily, taking the parameters of the first network layer as weight values ​​as an example, the weight values ​​of the first network layer can be quantized from FP32 or FP16 to low-bit INT8 or FP8. At the same time, if the input data input to the first network layer is FP32 or FP16, the corresponding quantization is low-bit INT8 or FP8, so as to facilitate the first network layer to perform inference calculations.

[0078] Among them, when the parameters and input data are quantized into low bits (for example, INT8 or FP8), the first network layer uses INT8 or FP8 to perform the forward calculation of the network layer (for example, including the forward calculation of the activation layer and / or the forward calculation of the convolution layer). This can greatly reduce the model memory overhead and computational complexity. The final calculation result is then dequantized into high-bit FP32 or FP16, which can further ensure the quality of the generated image.

[0079] In an embodiment of the present application, by first determining which network layers in the model have less impact on the image quality, and then selectively quantizing these network layers, the quantized network layers can complete image generation faster, thereby improving the image generation speed. At the same time, it also ensures that the difference between the first image generated by the first model after the network layer is quantized and the second image generated by the second model without quantization of the network layer is within the set threshold range, which can avoid quality problems in the image generated by the first model after the network layer is quantized, and improve the model's image processing speed, ultimately achieving the purpose of improving the image generation effect.

[0080] How to obtain the quantization sensitivity of each network layer in the first model is described in detail below through some embodiments.

[0081] Figure 4 A schematic diagram of a quantitative sensitivity determination process provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, it includes the following steps:

[0082] Step S410: Obtain a quantization test set, and select the Nth input information from the quantization test set, where the quantization test set includes at least one input information, wherein N is a positive integer.

[0083] Step S420: Obtain a second image generated by the first model through image generation inference based on the Nth input information and at least one network layer.

[0084] Step S430: Determine the i-th network layer from at least one network layer of the first model, where i is a positive integer;

[0085] Step S440: When the first model performs image generation reasoning according to the Nth input information and at least one network layer, quantization processing is performed on the i-th network layer to obtain a third image generated by the first model;

[0086] Step S450: determining sensitivity data of the i-th network layer under the N-th input information according to the second image and the third image;

[0087] Step S460: Determine the quantization sensitivity of the i-th network layer according to the sensitivity data of the i-th network layer under each input information.

[0088] For step S410, the quantitative test set may actually refer to a text image test set. The first input may refer to text description information, such as "I want a black cat image", which is used for model inference to generate an image. The more detailed the text description information is, the more accurate the corresponding generated image will be.

[0089] Exemplarily, the quantitative test set can be obtained from a training set or a validation set of a pre-trained text graph model, or typical input parameters of a portion of users can be extracted from online services to form a quantitative test set.

[0090] For step S420, the second image can be understood as a non-quantized accelerated reference image. Exemplarily, taking the first model as a diffusion model, in the process of generating the second image, all network layers of the U-NET network in the diffusion model are not quantized, that is, a single test parameter (i.e., the Nth input information) is first extracted from the quantization test set, and then in the Wensheng graph reasoning process, each network layer uses high-precision floating-point weights and activation values ​​for reasoning to obtain the second image.

[0091] For step S430 and step S440, after obtaining the second image, a network layer (i.e., the i-th network layer) is selected from the network layers of the U-NET network in the diffusion model, and then the extracted test parameter (i.e., the N-th input information) is input into the diffusion model for Wensheng graph reasoning. In the Wensheng graph reasoning process, the i-th network layer needs to be quantized, and the image finally generated is used as the third image.

[0092] In this embodiment, because there is usually more than one input information in the quantization test set, assuming that the test data set includes M input information, then M groups of data can be obtained, each group of data includes a second image and a third image, and N is less than or equal to M.

[0093] For the sake of distinction, the second image and the third image generated after the above steps S410-S440 are referred to as the Nith group of data. In this way, Ni groups of data can be obtained, namely the 1ith group of data, the 2ith group of data, the 3ith group of data, ..., the Mith group of data, totaling M groups of data.

[0094] For step S450, after obtaining M groups of data, the sensitivity data of the i-th network layer under the N-th input information can be determined according to the Ni-th group of input information. Specifically, the second image and the third image in the Ni-th group of data can be compared to determine the difference information, and then a sensitivity data can be calculated according to the difference information.

[0095] Exemplarily, the difference information may refer to the image display effect between the second image and the third image. For example, assuming that the image score of the generated second image is set to 100 points, and then another image score is evaluated based on the existence of distortion, blur, missing details, and deformed characters in the generated third image, and a sensitivity data is obtained by comparing the two scores.

[0096] For step S460, since there is usually more than one input information in the test data set in actual applications, it is necessary to count the sensitivity data of the i-th network layer under each input information, that is, to obtain M sensitivity data in total.

[0097] Exemplarily, the M sensitivity data may be averaged, and the final average value may be used as the quantized sensitivity of the i-th network layer.

[0098] In the embodiment of the present application, by comparing the difference between the images before and after quantization, the sensitivity of each network layer to quantization can be more accurately evaluated, which helps to understand which layers may lose important information after quantization, thereby facilitating the subsequent selection of quantization for network layers with lower sensitivity, while continuing to maintain floating-point precision for network layers with higher sensitivity, thereby balancing image generation efficiency and image generation quality.

[0099] Further, based on the above embodiments, the sensitivity data of the i-th network layer under the N-th input information can be determined by the following embodiments. Figure 5 A schematic diagram of a sensitivity data determination process provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method may include the following steps:

[0100] Step S510: obtaining structural similarity information between the second image and the third image;

[0101] Step S520: obtaining first matching information between the second image and the Nth input information and second matching information between the third image and the Nth input information;

[0102] Step S530: determining relative loss information according to the first matching information and the second matching information;

[0103] Step S540: Determine the sensitivity data of the i-th network layer under the N-th input information according to the structural similarity information and the relative loss information.

[0104] As for step S510, it is mentioned above that quantizing the network layer will mainly cause the following two problems in the image: (1) image aesthetics; (2) image-text consistency. For (1) image aesthetics, when measuring the aesthetics dimension, the Structural Similarity Index (SSIM) can be used as an evaluation index. SSIM is used as an index to measure the similarity between the second image and the third image:

[0105] ssim m =SSIM(image 0 , image m ) (1)

[0106] In the above formula, ssim i Represents the sensitivity data of the i-th network layer under the m-th input information, image 0 Represents the second image under the mth input information, image m represents the third image under the mth input information. m The value is between 0 and 1. The larger the value, the better the image. m Compared to image 0 The smaller the distortion, the smaller the loss of image quality after quantization.

[0107] For step S520 and step S530, the consistency of (2) text and image can be measured by relative loss information. First, an artificial intelligence model can be pre-trained to obtain the above-mentioned first matching information and second matching information. The artificial intelligence model can learn the correspondence between text and image through contrastive language-image pre-training (CLIP). The trained artificial intelligence model can be used to input the following two matching information:

[0108] CLIPScore 0 = CLIPScore(image 0 ,text) (2)

[0109] CLIPScore m = CLIPScore(image m ,text) (3)

[0110] In the above formula, CLIPScore 0 Indicates the first matching information, CLIPScore mrepresents the second matching information, and text represents the text prompt word, that is, the Mth input information. According to formulas (2) and (3), we can calculate the relative loss information clip of the i-th network layer i :

[0111]

[0112] Among them, chip i The larger the value, the better the consistency of the image and text before and after quantization.

[0113] For step S540, the sensitivity value of the i-th network layer can be obtained by combining the results of the two dimensions of image aesthetics and image-text consistency:

[0114]

[0115] In the above formula, α and β are configurable weight coefficients of the two dimensional parameters, and when S i The larger the value, the greater the loss of image quality after quantization of this network layer, that is, the more sensitive this network layer is to low-bit quantization and is not suitable for quantization.

[0116] In the embodiment of the present application, the structural similarity information reflects the degree of preservation of the image content before and after quantization, while the relative loss information provides the correlation between the generated image and the input information. By comprehensively considering the structural similarity information and the relative loss information, the sensitivity of each network layer to quantization can be more accurately evaluated. This further improves the efficiency of the Wensheng map and ensures the quality of the generated image.

[0117] Furthermore, the above step S420 can be specifically implemented through the following embodiments: Figure 6 A schematic diagram of the second image generation process provided in the embodiment of the present application is shown in FIG. Figure 6 As shown, the method may specifically include the following steps:

[0118] Step S610: Acquire a first input image;

[0119] Step S620: predicting a first noise according to the first input image, the Nth input information and the first model of the network layer that has not been quantized;

[0120] Step S630: De-noising the first input image according to the first noise, and updating the first input image;

[0121] Step S640: continue to perform noise prediction according to the updated first input image, the Nth input information and the first model that has not been quantized in the network layer, and continue to update the first input image according to the predicted noise until the first set condition is met, thereby obtaining the updated first input image;

[0122] Step S650: Generate a second image according to the updated first input image.

[0123] In this embodiment, a loop iteration condition (i.e., the first set condition) is configured, and N loop iteration denoising steps are used to gradually generate an image that conforms to the input information. That is, the first input image is denoised and updated through continuous loop iteration until the first set condition is met, and then the iteration is completed and the final image is output.

[0124] For step S610, a random number generator may be used to generate a Gaussian distributed random noise (called initial latents 0 ), as the initial input of the entire denoising process (i.e., the first input image).

[0125] In addition, in some implementations, a Vincent graph model may be used to preliminarily generate an image according to the first input as the first input image.

[0126] Exemplarily, taking the first model as a diffusion model as an example, N steps of cyclic denoising can be performed through the U-NET network. Figure 7 A schematic diagram of generating a second image provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the image generation process is mainly divided into two parts:

[0127] (1) Noise prediction. t (The first input image after the tth update), the input information and the time step corresponding to the current step are used as the input of the U-NET network in the diffusion model. Through the forward calculation of the U-NET network, the predicted noise of this step is obtained. t .

[0128] Among them, all network layers (including linear layers and convolutional layers, etc.) in the U-NET network at this time are not quantized, that is, the weights and calculated activation values ​​of each network layer are represented and calculated using high-bit FP32 or FP16.

[0129] (2) Remove noise. t From the current image latents t The process of removing latents is called denoising. t+1 =latents t -noise t At this point, latents t+1 represents the updated first input image.

[0130] After that, the denoised image latents t+1As the input of U-NET in the next iteration, the cycle is repeated, and finally the reference image image after N iterations is obtained 0 , as the second image.

[0131] Furthermore, the above step S440 can be specifically implemented through the following embodiments: Figure 8 A schematic diagram of a third image generation process provided in an embodiment of the present application, such as Figure 8 As shown, the method may specifically include the following steps:

[0132] Step S810: predicting a second noise according to the first model including the i-th network layer after the quantization process and other network layers that have not been quantized, the first input image and the N-th input information;

[0133] Step S820: De-noising the first input image according to the second noise, and updating the first input image;

[0134] Step S830: continuing to perform noise prediction according to the first model including the i-th network layer after the quantization process and other network layers that have not been quantized, the updated first input image and the N-th input information, and continuing to update the first input image according to the predicted noise until the second set condition is met, thereby obtaining the updated first input image;

[0135] Step S840: Generate a third image according to the updated first input image.

[0136] Similar to the above steps S610-S650, in the present embodiment, a loop iteration condition (i.e., the second set condition) is also configured, and N loop iteration denoising steps are used to gradually generate an image that conforms to the input information, that is, the first input image is denoised and updated through continuous loop iteration until the second set condition is met, and the iteration is completed to output the final image as the third image.

[0137] Among them, the difference from the above embodiment is that, in the process of performing Wensheng graph reasoning in this embodiment, the i-th network layer in the first model will perform quantization processing, while other network layers except the i-th network layer in the first model will not perform quantization processing, thereby detecting the quantization sensitivity of the i-th network layer.

[0138] For example, Fig. 9 The third image generation schematic diagram provided in the embodiment of the present application continues to take the first model as the diffusion model as an example, traverse all the linear layers and convolutional layers of the U-NET network in the diffusion model, select the network layer layeri one by one, and set the selected network layer as the only quantization layer for this text graph reasoning. Fig. 9As shown in the figure, when layeri is selected to perform the quantization operation and other network layers remain unchanged, the U-NET network obtains the corresponding reference image image after N steps of iteration. i , as the third image.

[0139] Among them, different from the above Figure 6 In this embodiment, when using the quantization operation of the layeri forward reasoning calculation in the U-NET network, at each step of the noise prediction, the layeri network layer of the U-NET performs forward calculation according to the low-bit data (the layeri network layer is as shown in FIG. Fig. 9 The specific steps of quantization processing at this layeri network layer are:

[0140] (1) Quantize the weight values ​​in the layeri network layer from FP32 / FP16 to low-bit INT8 / FP8;

[0141] (2) Quantize the input data of the layeri network layer from FP32 / FP16 to low-bit INT8 / FP8;

[0142] (3) Use INT8 / FP8 to perform Linear forward calculation or Conv forward calculation of the network layer

[0143] (4) Dequantize the calculation results into high-bit FP32 / FP16.

[0144] After multiple iterations of U-NET network to predict and remove noise, the reference image image_i of the quantized layeri network layer is obtained as the third image.

[0145] In the embodiment of the present application, through the iterative noise prediction and denoising process, the noise in the image can be gradually reduced and the quality of the final generated image can be improved. At the same time, in the denoising process, by continuously updating the image and re-predicting the noise, important details in the image can be retained to avoid the loss of details caused by over-smoothing.

[0146] The overall process of the Wensheng diagram is described in detail below through an embodiment. Taking the first model as a diffusion model as an example, Fig.10 The schematic diagram of the Wensheng diagram process provided in the embodiment of the present application is as follows: Fig.10 As shown, it includes the following steps:

[0147] Step S1010: construct a Wensheng graph quantization test set.

[0148] Step S1020: Generate a non-quantized accelerated reference image.

[0149] Step S1030: traverse the linear layers and convolutional layers in the U-NET network, perform a quantization operation on one of the layers, and obtain an inference acceleration image of the quantized layer.

[0150] Step S1040: Calculate the quantized sensitive data of this layer through the reference image and the inference acceleration image.

[0151] Step S1050: Obtain sensitivity data of each network layer under the test sample.

[0152] Step S1060: traverse all test samples in the quantization sample set.

[0153] Step S1070: Summarize the test results of all test samples to obtain the quantitative sensitivity of each network layer.

[0154] Step S1080: Receive a first input.

[0155] Step S1090: Load the quantization sensitivity of each network layer.

[0156] Step S1100: during inference, determine whether to perform quantization processing on the network layer according to the quantization sensitivity of each network layer.

[0157] Step S1110: Output image.

[0158] In this embodiment, steps S1030 and S1040 are repeated to complete the traversal of all linear layers and convolutional layers in the UNET, and the quantitative sensitivity data of each layer is obtained. Finally, the data of all network layers are summarized into a table with layer ID as index, and the quantitative sensitivity data of the diffusion model for the current sample is obtained. When determining the quantitative sensitivity data of the network layer, it can be calculated by the above formula (5). Exemplarily, Fig.11 The schematic diagram of the quantitative sensitive data calculation provided in the embodiment of the present application is as follows: Fig.11 As shown in Figure 1, by comparing the benchmark image and the inference acceleration image, and integrating the results of the two dimensions of image aesthetics and image-text consistency, we can get the sensitivity value of the image output of the i-th network layer.

[0159] For step S1050, in the quantitative test set constructed in step S1010, steps S1020 to S1050 are repeatedly executed to obtain the sensitivity data of each network layer. Then, for each network layer in the diffusion model, the sensitivity data of multiple samples are averaged. Finally, the quantitative sensitivity data DATA is persistently stored for easy use in subsequent steps.

[0160] In this embodiment, through steps S1010 to S1070, the measurement of the quantitative sensitivity of each network layer in the U-NET network structure in the diffusion model is completed. Steps S1090 to S1110 are the reasoning stage of the Wensheng graph. Fig.12 The network layer determination flow chart of the quantifiable processing provided in the embodiment of the present application is as follows: Fig.12 As shown in , if the quantization sensitivity of the network layer is lower than the first sensitivity threshold, the network layer can be quantized and accelerated; if the quantization sensitivity of the network layer is higher than the first sensitivity threshold, the network layer cannot be quantized and accelerated, and high-bit data is still required for calculation. After traversing all linear layers and convolutional layers in the U-NET network structure, multiple low-sensitivity layers (such as Fig.12 The filled boxes in the figure represent network layers that can be quantized), and quantization is used to accelerate the calculation of these less sensitive network layers during inference.

[0161] In this embodiment, in the Wensheng graph reasoning stage, the acceleration effect brought by low bits is obtained for the quantifiable network layer, and at the same time, some network layers that have a greater impact on the output quality are isolated, thereby retaining the output quality of the diffusion model while obtaining the acceleration effect.

[0162] In an embodiment of the present application, by measuring the impact of each network layer in the network on the final image output quality index, the image quality is ensured to be unaffected while the low-bit quantization inference speed is improved, thereby providing an excellent user experience.

[0163] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0164] Fig.13 This is a schematic diagram of the structure of the image generation device provided in the embodiment of the present application. Fig.13 As shown, the image generating device 1300 includes a sensitivity acquisition module 1310 , a network layer determination module 1320 , an input receiving module 1330 and a quantization reasoning module 1340 .

[0165] The sensitivity acquisition module is used to acquire the quantitative sensitivity of each network layer in the first model. The first model includes at least one network layer;

[0166] The network layer determination module is used to determine a first network layer in at least one network layer according to the quantized sensitivity of each network layer and a first sensitivity threshold.

[0167] The input receiving module is used for receiving a first input to the first model.

[0168] The quantization inference module is used to quantize the first network layer to obtain the first image generated by the first model when the first model performs image generation inference according to the first input and at least one network layer. The image generation inference speed of the first network layer after quantization is greater than that of the first network layer before quantization, and the image quality inferred by the first network layer after quantization is higher than that of the first network layer before quantization.

[0169] In an embodiment of the present application, by first determining which network layers in the model have less impact on the image quality, and then selectively quantizing these network layers, the quantized network layers can complete image generation faster, thereby improving the image generation speed. At the same time, it also ensures that the difference between the first image generated by the first model after the network layer is quantized and the second image generated by the second model without quantization of the network layer is within the set threshold range, which can avoid quality problems in the image generated by the first model after the network layer is quantized, and improve the model's image processing speed, ultimately achieving the purpose of improving the image generation effect.

[0170] In some possible implementations of the embodiments of the present application, the sensitivity acquisition module is specifically used to:

[0171] Obtain a quantization test set, and select Nth input information from the quantization test set, where the quantization test set includes at least one input information, and N is a positive integer;

[0172] Obtain a second image generated by the first model through image generation reasoning based on the Nth input information and at least one network layer;

[0173] Determine an i-th network layer from at least one network layer of the first model, where i is a positive integer;

[0174] When the first model performs image generation reasoning according to the Nth input information and at least one network layer, quantization processing is performed on the i-th network layer to obtain a third image generated by the first model;

[0175] Determine sensitivity data of the i-th network layer under the N-th input information according to the second image and the third image;

[0176] According to the sensitivity data of the i-th network layer under each input information, the quantized sensitivity of the i-th network layer is determined.

[0177] In some possible implementations of the embodiments of the present application, the sensitivity acquisition module is specifically used to:

[0178] obtaining structural similarity information between the second image and the third image;

[0179] Acquire first matching information between the second image and the Nth input information and second matching information between the third image and the Nth input information;

[0180] Determining relative loss information according to the first matching information and the second matching information;

[0181] According to the structural similarity information and the relative loss information, the sensitivity data of the i-th network layer under the N-th input information is determined.

[0182] In some possible implementations of the embodiments of the present application, the first input is text description information.

[0183] In some possible implementations of the embodiments of the present application, the sensitivity acquisition module is specifically used to:

[0184] Obtaining a first input image;

[0185] Predicting a first noise according to a first input image, an Nth input information, and a first model that is not quantized by a network layer;

[0186] Denoising the first input image according to the first noise, and updating the first input image;

[0187] Continue to perform noise prediction according to the updated first input image, the Nth input information, and the first model that has not been quantized in the network layer, and continue to update the first input image according to the predicted noise until the first set condition is met, thereby obtaining the updated first input image;

[0188] A second image is generated according to the updated first input image.

[0189] In some possible implementations of the embodiments of the present application, the sensitivity acquisition module is specifically used to:

[0190] Predicting a second noise according to a first model including an i-th network layer after quantization and other network layers that have not been quantized, a first input image, and an N-th input information;

[0191] Denoising the first input image according to the second noise, and updating the first input image;

[0192] Continue to perform noise prediction according to the first model including the i-th network layer after the quantization processing and other network layers that have not been quantized, the updated first input image and the N-th input information, and continue to update the first input image according to the predicted noise until the second set condition is met, thereby obtaining the updated first input image;

[0193] A third image is generated according to the first input image after the update is completed.

[0194] In some possible implementations of the embodiments of the present application, the quantization reasoning module is specifically used for:

[0195] Reducing parameters in the first network layer from a first data precision to a second data precision;

[0196] Reducing input data of the first network layer from a first data precision to a second data precision;

[0197] Inferring the input data by reducing the parameters to the second data precision to obtain an inference result;

[0198] Improving the data accuracy of the inference result to the first data accuracy;

[0199] A first image is obtained according to the inference result with improved accuracy.

[0200] The image generating device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0201] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0202] The device provided in the embodiment of the present application can be used to execute the method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0203] Fig.14 A schematic diagram of a framework of an electronic device provided in an embodiment of the present application, such as Fig.14As shown, the electronic device 1400 includes a processor 1401 and a memory 1402. The memory 1402 stores programs or instructions that can be executed on the processor 1401. When the program or instructions are executed by the processor 1401, the various steps of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0204] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0205] Fig.15 The hardware structure diagram of the electronic device provided in the embodiment of the present application is shown in FIG. Fig.15 As shown, the electronic device 1500 includes but is not limited to: a radio frequency unit 1501, a network module 1502, an audio output unit 1503, an input unit 1504, a sensor 1505, a display unit 1506, a user input unit 1507, an interface unit 1508, a memory 1509, and a processor 1510 and other components.

[0206] Those skilled in the art will appreciate that the electronic device 1500 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.15 The structure of the electronic device 1500 shown in the figure does not constitute a limitation of the electronic device 1500. The electronic device 1500 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0207] The user input unit is used to receive a first input to the first model. The processor is used to obtain the quantization sensitivity of each network layer in the first model; determine the first network layer in at least one network layer according to the quantization sensitivity of each network layer and the first sensitivity threshold; and perform quantization processing on the first network layer when the first model performs image generation reasoning according to the first input and at least one network layer to obtain a first image generated by the first model.

[0208] In an embodiment of the present application, by first determining which network layers in the model have less impact on the image quality, and then selectively quantizing these network layers, the quantized network layers can complete image generation faster, thereby improving the image generation speed. At the same time, it also ensures that the difference between the first image generated by the first model after the network layer is quantized and the second image generated by the second model without quantization of the network layer is within the set threshold range, which can avoid quality problems in the image generated by the first model after the network layer is quantized, and improve the model's image processing speed, ultimately achieving the purpose of improving the image generation effect.

[0209] In some possible implementations of the embodiments of the present application, the processor is specifically used to: obtain a quantization test set, select Nth input information from the quantization test set, the quantization test set includes at least one input information, and N is a positive integer;

[0210] Obtain a second image generated by the first model through image generation reasoning based on the Nth input information and at least one network layer;

[0211] Determine an i-th network layer from at least one network layer of the first model, where i is a positive integer;

[0212] When the first model performs image generation reasoning according to the Nth input information and at least one network layer, quantization processing is performed on the i-th network layer to obtain a third image generated by the first model;

[0213] Determine sensitivity data of the i-th network layer under the N-th input information according to the second image and the third image;

[0214] According to the sensitivity data of the i-th network layer under each input information, the quantized sensitivity of the i-th network layer is determined.

[0215] In some possible implementations of the embodiments of the present application, the processor is specifically configured to:

[0216] obtaining structural similarity information between the second image and the third image;

[0217] Acquire first matching information between the second image and the Nth input information and second matching information between the third image and the Nth input information;

[0218] Determining relative loss information according to the first matching information and the second matching information;

[0219] According to the structural similarity information and the relative loss information, the sensitivity data of the i-th network layer under the N-th input information is determined.

[0220] In some possible implementations of the embodiments of the present application, the processor is specifically configured to:

[0221] Obtaining a first input image;

[0222] Predicting a first noise according to a first input image, an Nth input information, and a first model that is not quantized by a network layer;

[0223] Denoising the first input image according to the first noise, and updating the first input image;

[0224] Continue to perform noise prediction according to the updated first input image, the Nth input information, and the first model that has not been quantized in the network layer, and continue to update the first input image according to the predicted noise until the first set condition is met, thereby obtaining the updated first input image;

[0225] A second image is generated according to the updated first input image.

[0226] In some possible implementations of the embodiments of the present application, the processor is specifically configured to:

[0227] Predicting a second noise according to a first model including an i-th network layer after quantization and other network layers that have not been quantized, a first input image, and an N-th input information;

[0228] Denoising the first input image according to the second noise, and updating the first input image;

[0229] Continue to perform noise prediction according to the first model including the i-th network layer after the quantization processing and other network layers that have not been quantized, the updated first input image and the N-th input information, and continue to update the first input image according to the predicted noise until the second set condition is met, thereby obtaining the updated first input image;

[0230] A third image is generated according to the first input image after the update is completed.

[0231] In some possible implementations of the embodiments of the present application, the processor is specifically configured to:

[0232] Reducing parameters in the first network layer from a first data precision to a second data precision;

[0233] Reducing input data of the first network layer from a first data precision to a second data precision;

[0234] Inferring the input data by reducing the parameters to the second data precision to obtain an inference result;

[0235] Improving the data accuracy of the inference result to the first data accuracy;

[0236] A first image is obtained according to the inference result with improved accuracy.

[0237] It should be understood that in the embodiment of the present application, the input unit 1504 may include a graphics processor (Graphics Processing Unit, GPU) 15041 and a microphone 15042, and the graphics processor 15041 processes the image data of the static image or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1506 may include a display panel 15061, and the display panel 15061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and at least one of other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 may include two parts: a touch detection device and a touch controller. Other input devices 15072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0238] The memory 1509 can be used to store software programs and various data. The memory 1509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1509 may include a volatile memory or a non-volatile memory, or the memory 1509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0239] The processor 1510 may include one or more processing units; optionally, the processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1510.

[0240] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0241] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0242] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0243] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0244] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0245] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0246] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0247] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: include: Obtaining a quantized sensitivity of each network layer in a first model, wherein the first model includes at least one network layer; Determining a first network layer in the at least one network layer according to the quantized sensitivity of each network layer and the first sensitivity threshold; receiving a first input to the first model; When the first model performs image generation inference based on the first input and the at least one network layer, the first network layer is quantized to obtain a first image generated by the first model.

2. The method according to claim 1, characterized in that The obtaining of the quantization sensitivity of each network layer in the first model includes: Acquire a quantization test set, and select Nth input information from the quantization test set, wherein the quantization test set includes at least one input information, and N is a positive integer; Obtain a second image generated by the first model through image generation reasoning based on the Nth input information and the at least one network layer; Determine an i-th network layer from at least one network layer of the first model, where i is a positive integer; When the first model performs image generation reasoning according to the Nth input information and the at least one network layer, performing quantization processing on the i-th network layer to obtain a third image generated by the first model; Determining sensitivity data of the i-th network layer under the N-th input information according to the second image and the third image; According to the sensitivity data of the i-th network layer under each input information, the quantization sensitivity of the i-th network layer is determined.

3. The method according to claim 2, characterized in that The determining, according to the second image and the third image, the sensitivity data of the i-th network layer under the N-th input information includes: Acquiring structural similarity information between the second image and the third image; Acquire first matching information between the second image and the Nth input information and second matching information between the third image and the Nth input information; Determining relative loss information according to the first matching information and the second matching information; According to the structural similarity information and the relative loss information, sensitivity data of the i-th network layer under the N-th input information is determined.

4. The method according to any one of claims 1 to 3, characterized in that The first input is text description information.

5. The method according to claim 2, characterized in that: The obtaining of a second image generated by the first model through image generation inference according to the Nth input information and the at least one network layer includes: Obtaining a first input image; Predicting a first noise according to the first input image, the Nth input information and a first model that is not quantized by the network layer; Denoising the first input image according to the first noise, and updating the first input image; Continue to perform noise prediction according to the updated first input image, the Nth input information, and the first model that has not been quantized by the network layer, and continue to update the first input image according to the predicted noise until the first set condition is met, thereby obtaining the updated first input image; The second image is generated according to the updated first input image.

6. The method according to claim 2, characterized in that The step of performing quantization processing on the i-th network layer to obtain a third image generated by the first model when the first model performs image generation reasoning according to the N-th input information and the at least one network layer, comprises: Predicting a second noise according to a first model including an i-th network layer after quantization and other network layers that have not been quantized, a first input image, and the N-th input information; Denoising the first input image according to the second noise, and updating the first input image; Continuing to perform noise prediction according to the first model including the i-th network layer after the quantization processing and other network layers that have not been quantized, the updated first input image and the N-th input information, and continuing to update the first input image according to the predicted noise until the second set condition is met, thereby obtaining the updated first input image; The third image is generated according to the first input image after the update is completed.

7. The method according to claim 1, characterized in that The step of performing quantization processing on the first network layer to obtain a first image generated by the first model when the first model performs image generation reasoning according to the first input and the at least one network layer, comprises: reducing parameters in the first network layer from a first data precision to a second data precision; Reducing the input data of the first network layer from a first data precision to a second data precision; Inferring the input data by reducing the parameters to the second data precision to obtain an inference result; Improving the data precision of the inference result to the first data precision; The first image is generated according to the inference result with improved accuracy.

8. An image generating device, characterized in that: include: A sensitivity acquisition module, used to acquire the quantitative sensitivity of each network layer in a first model, wherein the first model includes at least one network layer; A network layer determination module, configured to determine a first network layer in the at least one network layer according to the quantized sensitivity of each network layer and a first sensitivity threshold; An input receiving module, configured to receive a first input to the first model; A quantization reasoning module is used to quantize the first network layer to obtain a first image generated by the first model when the first model performs image generation reasoning based on the first input and the at least one network layer.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images