Picture generation model processing method and device, computer equipment and storage medium
By quantizing and fine-tuning the image generation model, the problems of high resource consumption and high deployment cost are solved, and high quality image generation is achieved in resource-constrained environments, improving the applicability and efficiency of the model.
Patent Information
- Application Number
- CN202411242670.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-27
AI Technical Summary
When processing high-resolution images or complex scenes, the existing image generation model consumes a lot of computing resources and has a long training time, making it difficult to balance image quality and fidelity, and has a high deployment cost, which limits its in resource-constrained environments. use.
By obtaining pre-trained generative models and training information, quantization processing and fine-tuning are performed, low-precision image generation models are generated, computational complexity is reduced, and selective fine-tuning is performed through key parameter tables and labels to ensure that high under low-precision conditions are maintained Generate quality.
The deployment cost of the image generation model is optimized, suitable for use in resource-constrained environments, while improving the fidelity and personalization of generated images to ensure image quality.
Smart Images
Figure CN119205953A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for processing an image generation model. Background Art
[0002] In recent years, image generation technology has received widespread attention and application in the field of artificial intelligence. In particular, through deep learning methods, technologies such as generative adversarial networks and diffusion models have been widely used in image generation tasks. The above models can generate high-quality images from random noise and show great potential in many fields such as art creation, game design and advertising.
[0003] However, existing image generation technologies still have some obvious shortcomings. In order to improve the subject fidelity, cue fidelity, and image quality of generated images, image generation models usually require a lot of computing resources and training time, especially when dealing with high-resolution images or complex scenes, which makes model training and reasoning expensive and time-consuming. Many models have difficulty balancing image quality and fidelity when generating images. In practical applications, they also face the problems of difficult deployment and high resource consumption, which limits their use in resource-constrained environments. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a method, apparatus, computer device and storage medium for processing an image generation model to reduce the deployment cost of the image generation model.
[0005] In order to solve the above technical problems, the present application embodiment provides a method for processing an image generation model, which adopts the following technical solution:
[0006] Obtaining a pre-trained first generation model and image generation training information;
[0007] When training the first generation model according to the training information generated by the picture, for a time step in training, obtaining a training configuration file of the time step, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step;
[0008] Quantize the first generation model according to the quantization parameter table to obtain a low-precision second generation model;
[0009] Based on the key parameter table and the label, fine-tune the second generation model, and obtain a picture generation model after completing fine-tuning of each time step of the second generation model;
[0010] Obtain image generation information, and input the image generation information into the image generation model to obtain a target image.
[0011] In order to solve the above technical problems, the embodiment of the present application further provides a picture generation model processing device, which adopts the following technical solution:
[0012] A training acquisition module, used to acquire a pre-trained first generation model and image generation training information;
[0013] A configuration acquisition module, configured to acquire, for a time step in training, a training configuration file of the time step when training the first generation model according to the image generation training information, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step;
[0014] A quantization processing module, used for performing quantization processing on the first generation model according to the quantization parameter table to obtain a low-precision second generation model;
[0015] A model fine-tuning module, used to fine-tune the second generation model based on the key parameter table and the label, and obtain a picture generation model after completing the fine-tuning of each time step of the second generation model;
[0016] The image generation module is used to obtain image generation information and input the image generation information into the image generation model to obtain a target image.
[0017] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0018] Obtaining a pre-trained first generation model and image generation training information;
[0019] When training the first generation model according to the training information generated by the picture, for a time step in training, obtaining a training configuration file of the time step, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step;
[0020] Quantize the first generation model according to the quantization parameter table to obtain a low-precision second generation model;
[0021] Based on the key parameter table and the label, fine-tune the second generation model, and obtain a picture generation model after completing fine-tuning of each time step of the second generation model;
[0022] Obtain image generation information, and input the image generation information into the image generation model to obtain a target image.
[0023] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0024] Obtaining a pre-trained first generation model and image generation training information;
[0025] When training the first generation model according to the training information generated by the picture, for a time step in training, obtaining a training configuration file of the time step, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step;
[0026] Quantize the first generation model according to the quantization parameter table to obtain a low-precision second generation model;
[0027] Based on the key parameter table and the label, fine-tune the second generation model, and obtain a picture generation model after completing fine-tuning of each time step of the second generation model;
[0028] Obtain image generation information, and input the image generation information into the image generation model to obtain a target image.
[0029] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining a pre-trained first generation model and image generation training information; when training the first generation model according to the image generation training information, for the time step in the training, obtaining the training configuration file of the time step, including the quantization parameter table, the key parameter table and the label within the time step; quantizing the first generation model according to the quantization parameter table to obtain a low-precision second generation model, reducing the computational complexity of the model, reducing resource consumption, and making the model more suitable for deployment in a resource-constrained environment; selectively fine-tuning the model parameters according to the key parameter table to ensure that a high generation quality can be maintained even under low-precision conditions; the label enables the model to flexibly adjust the generation strategy to ensure that the fidelity, personalization and image quality of the generated image are improved while meeting specific requirements; after fine-tuning the second generation model for each time step, the image generation model is obtained; obtaining the image generation information and inputting it into the image generation model can obtain the target image that meets the requirements; the present application optimizes the image generation model through quantization processing and selective fine-tuning, thereby reducing the deployment cost of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0032] Figure 2 A flowchart of an embodiment of a method for processing a picture generation model according to the present application;
[0033] Figure 3 is a structural schematic diagram of an embodiment of a picture generation model processing device according to the present application;
[0034] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0038] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0039] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0040] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.
[0041] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0042] It should be noted that the image generation model processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the image generation model processing device is generally set in the server.
[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for processing a picture generation model according to the present application. The method for processing a picture generation model comprises the following steps:
[0045] Step S201, obtaining a pre-trained first generation model and image generation training information.
[0046] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.
[0047] Specifically, a first generative model that has been pre-trained is obtained. The first generative model can be built based on a diffusion model; a diffusion model is a type of deep generative model used to generate data (such as images, audio, text, etc.). The core idea is to convert the original data (such as an image) into pure noise by gradually adding noise, and then gradually remove the noise during the generation process to generate new data samples. The diffusion model is generated through a reverse process (from noise to data), and a neural network is usually trained to learn how to gradually recover the original data from data with a small amount of noise. The diffusion model can be pre-trained for image generation tasks.
[0048] In addition, it is also necessary to obtain image generation training information, including the target requirements of image generation and the specific requirements of the generation task. It can include a description of the subject to be generated (involving the subject fidelity, such as a prompt requiring the generation of an image of a "white cat"), and input prompts (involving prompt fidelity, covering the details of the generated image, such as color, shape, scene and other details).
[0049] Furthermore, before the above step S201, the process may also include: obtaining a full-precision initial model; and pre-training the initial model to obtain a first generation model.
[0050] Specifically, during pre-training, a full-precision initial model is first obtained. The initial model is a basic model that has not undergone any quantization processing and usually contains complete high-precision parameter settings. "Full Precision" refers to the high-precision numerical representation used by the model when performing calculations, usually using 32-bit floating point numbers (FP32) to represent the model's parameters and perform calculations. In deep learning, full-precision models usually provide the highest computing accuracy and the best model performance because they use more bits to represent each parameter, which can more accurately capture and represent subtle differences in the data.
[0051] Subsequently, the full-precision initial model is pre-trained, that is, trained on a large-scale dataset so that it can learn rich image generation features and capabilities. This process ensures that the model has a broad base of knowledge, laying a solid foundation for subsequent application and fine-tuning in specific tasks. Through pre-training, the first generative model obtained has the initial ability to handle complex image generation tasks while retaining details and accuracy.
[0052] In this embodiment, the full-precision initial model retains complete parameter information without quantization processing, so that the model can fully learn and express complex features in the pre-training stage, thereby improving the initial performance of the model; in subsequent steps, even after quantization processing and fine-tuning, it can still maintain a high image generation quality and detail expression ability, providing support for the final generation of high-quality, personalized pictures.
[0053] Step S202, when training the first generation model according to the image generation training information, for the time step in training, obtain the training configuration file of the time step, the training configuration file includes the quantization parameter table, the key parameter table and the label in the time step.
[0054] Specifically, when the first generation model is trained according to the image generation training information, multiple time steps will be experienced, and each time step corresponds to the process of the model gradually generating an image.
[0055] Time step is a very important concept in diffusion model. The process of image generation by diffusion model is a step-by-step reverse denoising process. The model starts with a random noise image, then gradually reduces the noise to finally generate a clear image. Each step of the denoising process is called a "time step" (or "time stage"). Each time step corresponds to a specific denoising operation, which determines how fine the image details generated by this step are. For example, the earliest time step may mainly remove large blocks of noise and generate the general outline of the image; while the later time steps will gradually improve the image details, and the last time step may be the final modification and cleanup of the image.
[0056] In order to optimize the performance of the model, specific operations need to be performed on different time steps. This operation is achieved by obtaining the training configuration file for the corresponding time step. The training configuration file includes: a quantization parameter table for model quantization processing, a key parameter table for guiding fine-tuning, and labels for evaluating fine-tuning effects (such as fidelity, personalization, image quality, etc.).
[0057] Step S203: quantize the first generation model according to the quantization parameter table to obtain a low-precision second generation model.
[0058] Specifically, the quantization process is to reduce the computational complexity and resource consumption of the model. Through the quantization parameter table, some high-precision parameters in the model will be converted into low-precision parameters to obtain a low-precision second generation model. This process effectively reduces the storage and computational overhead of the model, and is particularly suitable for deployment scenarios with limited resources.
[0059] It should be noted that different time steps may have different quantization parameter tables, that is, in one time step, parameter a may be in a non-quantized state; in another time step, parameter a may be in a quantized state.
[0060] Step S204, fine-tuning the second generation model based on the key parameter table and the label, and obtaining the image generation model after completing the fine-tuning of the second generation model at each time step.
[0061] Specifically, the quantized model needs to be fine-tuned to ensure that it can generate high-quality images at low precision. Through the key parameter table, the model can identify the most important parameters at each time step and fine-tune them in a targeted manner; the labels are used to guide the direction of fine-tuning so that it can better meet user needs when generating images, such as improving the fidelity of the generated target image or meeting personalized requirements.
[0062] After fine-tuning the second generation model at each time step, the image generation model can be obtained.
[0063] Step S205, obtaining image generation information, and inputting the image generation information into an image generation model to obtain a target image.
[0064] Specifically, image generation information (such as descriptive text or specific image requirements) is obtained and input into the image generation model, and the model generates the final target image based on the previously fine-tuned parameters.
[0065] In this embodiment, a pre-trained first generation model and image generation training information are obtained; when the first generation model is trained according to the image generation training information, for the time step in the training, a training configuration file of the time step is obtained, including a quantization parameter table, a key parameter table and a label within the time step; the first generation model is quantized according to the quantization parameter table to obtain a low-precision second generation model, which reduces the computational complexity of the model and reduces resource consumption, making the model more suitable for deployment in a resource-constrained environment; the model parameters are selectively fine-tuned according to the key parameter table to ensure that a high generation quality can be maintained even under low-precision conditions; the label enables the model to flexibly adjust the generation strategy to ensure that the fidelity, personalization and image quality of the generated image are improved while meeting specific requirements; after fine-tuning the second generation model for each time step, a picture generation model is obtained; the picture generation information is obtained and input into the picture generation model to obtain a target picture that meets the requirements; the present application optimizes the picture generation model through quantization processing and selective fine-tuning, thereby reducing the deployment cost of the model.
[0066] Furthermore, the above-mentioned step of fine-tuning the second generation model based on the key parameter table and labels may include: obtaining the time step result of the current time step; calculating the model loss according to the time step result and the label; determining the adjustable parameters in the second generation model based on the key parameter table; and fine-tuning the determined adjustable parameters according to the model loss.
[0067] Specifically, at each training time step (T i me Step), the generative model generates the time step result of the current time step based on the current input and the generated intermediate results. It can be the intermediate state of the image, or the key information indicator in the generation process (such as the matching degree of specific features, the fidelity of the generated image, etc.).
[0068] Model loss is an indicator used to measure the gap between the current time step result and the expected target. The time step result is compared with the pre-set label to calculate the model loss. The label contains the expectations for the generated image, including requirements for personalization, fidelity, image quality, etc. Using labels to guide the calculation of model loss can ensure that the model is optimized in the direction that meets the requirements during the training process.
[0069] The key parameter table lists which parameters are most important or influential in generating results at the current time step. After quantizing the model, these key parameters need to be further adjusted. By identifying these adjustable parameters through the key parameter table, fine-tuning can focus on the most critical parts to improve training efficiency and the quality of generated results. The goal of fine-tuning is to reduce model loss by optimizing adjustable parameters so that the generated images are closer to expectations in terms of personalization, fidelity, and quality.
[0070] In this embodiment, at each time step of training, the generated time step result is obtained and compared with the preset label to calculate the current model loss; the key parameter table records the parameters that have the greatest impact on the generated result in each time step. During fine-tuning, the key adjustable parameters are adjusted according to the model loss without making large-scale adjustments to all parameters of the model, which reduces the complexity and computational burden of training, improves the model's ability to capture key features, and improves the pertinence and efficiency of fine-tuning, so that the generated images are more in line with user needs and expectations.
[0071] Furthermore, the above-mentioned step of calculating the model loss based on the time step results and labels may include: extracting personalized labels, fidelity labels and image quality labels in the labels; calculating the first loss based on the time step results and the personalized labels; calculating the second loss based on the time step results and the fidelity labels; calculating the third loss based on the time step results and the image quality labels; calculating the model loss based on the first loss, the second loss and the third loss.
[0072] Specifically, labels are predefined and used to measure whether the model-generated images meet the expected standards. Labels are divided into three categories: personalized labels, fidelity labels, and image quality labels. Personalized labels are related to the needs of specific users or specific applications, such as the accurate reproduction of style or specific objects; fidelity labels focus on the fidelity of the generated image to the prompt content; image quality labels are related to technical standards such as the clarity and visual effects of the generated image.
[0073] At a certain time step of training, the generative model outputs the current time step result. By comparing the time step result with the personalized label, the first loss can be calculated, which measures the extent to which the time step result deviates from the user or task-specific personalized requirements.
[0074] The fidelity label measures the consistency of the time step result with the input prompt content (including the subject description and the input prompt). The second loss determines the extent to which the generated image meets the prompt requirements by comparing the time step result with the fidelity label. This loss reflects the fidelity of the generated image.
[0075] The third loss is calculated by comparing the time step results with the image quality label, which measures the deviation of technical quality indicators such as image clarity, noise level, color accuracy, etc.
[0076] Finally, the overall model loss is calculated by combining the above three losses. For example, the first loss, the second loss, and the third loss can be weighted to obtain the model loss. The weights of different losses can be adjusted according to the needs of specific application scenarios to ensure that the model can be optimized in a balanced manner in multiple aspects.
[0077] In this embodiment, personalized labels, fidelity labels and image quality labels are extracted, and the corresponding loss values are calculated respectively, so that different problems of time step results can be accurately identified during the training process; the personalized loss ensures that the generated results meet the needs and styles of specific users, the fidelity loss ensures that the generated results are faithful to the prompt content, and the image quality loss ensures that the pictures meet high technical standards; by comprehensively considering the losses in different dimensions and fine-tuning the generated model, the model can be balanced and optimized in multiple dimensions, so as to generate pictures that meet the personalized requirements while maintaining high quality and high fidelity.
[0078] Furthermore, the above-mentioned step of obtaining the image generation model may include: obtaining a quantization parameter table and a key parameter table for each time step; obtaining the final parameters of each parameter when non-quantized and the final parameters when quantized in the second generation model; generating an application parameter table according to the quantization parameter table and the key parameter table for each time step, the final parameters of each parameter when non-quantized and the final parameters when quantized; and constructing the image generation model according to the application parameter table and the second generation model.
[0079] Specifically, each time step has its corresponding quantization parameter table and key parameter table. The quantization parameter table records the model parameters that need to be quantized in this time step, and the precision of these parameters is reduced to reduce the computational overhead. The key parameter table lists the parameters that need special attention and adjustment in this time step, which may have an important impact on the output of the model.
[0080] For each parameter, the system records its final parameter value in the non-quantized state and the quantized state. The parameters in the non-quantized state retain the full-precision calculation results of the model, while the parameters in the quantized state are processed with reduced precision to reduce the complexity and calculation amount of the model.
[0081] By combining the quantization parameters and key parameters of each time step, and combining the final parameter values of the non-quantized state and the quantized state, an application parameter table is generated. It is a summary of the parameter adjustment strategy for each time step, which indicates which parameters should be quantized at each time step in actual applications, what the parameter values are after the parameters are quantized, and what the parameter values are without quantization.
[0082] Based on the information in the application parameter table and the second generation model, the system will build a final image generation model, which combines the optimal parameter configurations at different time steps, taking into account both the conservation of computing resources and ensuring the quality and consistency of the generated images.
[0083] In this embodiment, specific quantization parameters and key parameters are set for each time step, which can flexibly balance the calculation accuracy and resource consumption; the model records the final values of the parameters in the quantized state and the non-quantized state, and in practical applications, the appropriate parameter state can be selected according to the needs, thereby improving the adaptability and flexibility of the model; the application parameter table integrates the optimal parameter configuration of each time step, ensuring that the image generation model can meet the requirements of limited computing resources while maintaining high-quality output effects.
[0084] Furthermore, the above-mentioned step of inputting the image generation information into the image generation model to obtain the target image may include: when generating the image according to the image generation information, for the time step in the application, obtaining an application parameter table; adjusting the parameters of the image generation model at the time step according to the application parameter table, and generating the image through the image generation model after the parameter adjustment until the last time step to obtain the target image.
[0085] Specifically, when starting image generation, first obtain the previously generated application parameter table, which contains the parameter settings that should be used in different time steps, including which parameters need to be quantized, and the parameter values when the parameters are quantized or not quantized.
[0086] At each time step, the relevant parameters of the model are adjusted according to the application parameter table. If the current time step requires certain parameters to be in a quantized state, these parameters will be processed with low precision to reduce the computational overhead. If the time step has special requirements for certain key parameters, these parameters will also be adjusted according to the application parameter table to ensure that the generated image meets the expected quality standards.
[0087] At each time step, the model generates a partial image output. As the time steps progress, the system will continue to adjust parameters and generate new image content until all time steps are processed. Finally, the system will output a complete target image.
[0088] In this embodiment, the application parameter table provides quantization parameters and values of each parameter for each time step, ensuring that the model can achieve the best balance between computing resources and image quality at each stage of the generation process. The model can adjust parameters according to different requirements and computing limitations to ensure that an intermediate image output that meets the requirements can be generated at each time step, ultimately forming an expected target image, thereby improving the accuracy and fidelity of image generation.
[0089] It should be emphasized that in order to further ensure the privacy and security of the above training configuration files, the above training configuration files can also be stored in a node of a blockchain.
[0090] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0091] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0092] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0093] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0094] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0095] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for processing a picture generation model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0096] like Figure 3 As shown, the image generation model processing device 300 described in this embodiment includes: a training acquisition module 301, a configuration acquisition module 302, a quantization processing module 303, a model fine-tuning module 304 and an image generation module 305, wherein:
[0097] The training acquisition module 301 is used to obtain the pre-trained first generation model and image generation training information.
[0098] The configuration acquisition module 302 is used to acquire a training configuration file of a time step in training when training the first generation model according to the training information generated by the picture. The training configuration file includes a quantization parameter table, a key parameter table and a label within the time step.
[0099] The quantization processing module 303 is used to perform quantization processing on the first generation model according to the quantization parameter table to obtain a low-precision second generation model.
[0100] The model fine-tuning module 304 is used to fine-tune the second generation model based on the key parameter table and the label, and obtain the image generation model after completing the fine-tuning of each time step of the second generation model.
[0101] The picture generation module 305 is used to obtain picture generation information and input the picture generation information into a picture generation model to obtain a target picture.
[0102] In this embodiment, a pre-trained first generation model and image generation training information are obtained; when the first generation model is trained according to the image generation training information, for the time step in the training, a training configuration file of the time step is obtained, including a quantization parameter table, a key parameter table and a label within the time step; the first generation model is quantized according to the quantization parameter table to obtain a low-precision second generation model, which reduces the computational complexity of the model and reduces resource consumption, making the model more suitable for deployment in a resource-constrained environment; the model parameters are selectively fine-tuned according to the key parameter table to ensure that a high generation quality can be maintained even under low-precision conditions; the label enables the model to flexibly adjust the generation strategy to ensure that the fidelity, personalization and image quality of the generated image are improved while meeting specific requirements; after fine-tuning the second generation model for each time step, a picture generation model is obtained; the picture generation information is obtained and input into the picture generation model to obtain a target picture that meets the requirements; the present application optimizes the picture generation model through quantization processing and selective fine-tuning, thereby reducing the deployment cost of the model.
[0103] In some optional implementations of this embodiment, the image generation model processing device 300 may further include:
[0104] The initial acquisition module is used to obtain the full-precision initial model.
[0105] The initial training module is used to pre-train the initial model to obtain the first generated model.
[0106] In some optional implementations of this embodiment, the model fine-tuning module 304 may include:
[0107] The result acquisition submodule is used to obtain the time step result of the current time step.
[0108] The loss calculation submodule is used to calculate the model loss based on the time step results and labels.
[0109] The adjustable determination submodule is used to determine the adjustable parameters in the second generation model based on the key parameter table.
[0110] The parameter adjustment submodule is used to fine-tune the determined adjustable parameters according to the model loss.
[0111] In some optional implementations of this embodiment, the loss calculation submodule may include:
[0112] The label extraction unit is used to extract the personalized label, the fidelity label and the image quality label from the label.
[0113] The first calculation unit is used to calculate the first loss according to the time step result and the personalized label.
[0114] The second computing unit is used to calculate the second loss based on the time step results and the fidelity labels.
[0115] The third computing unit is used to calculate the third loss according to the time step result and the image quality label.
[0116] The loss calculation unit is used to calculate the model loss according to the first loss, the second loss and the third loss.
[0117] In some optional implementations of this embodiment, the model fine-tuning module 304 may also include:
[0118] The parameter table acquisition submodule is used to obtain the quantitative parameter table and key parameter table of each time step.
[0119] The parameter acquisition submodule is used to obtain the final parameters of each parameter in the second generation model when it is not quantized and the final parameters when it is quantized.
[0120] The application generation submodule is used to generate an application parameter table according to the quantization parameter table and the key parameter table of each time step, the final parameters of each parameter when not quantized, and the final parameters when quantized.
[0121] The model building submodule is used to build a picture generation model according to the application parameter table and the second generation model.
[0122] In some optional implementations of this embodiment, the picture generation module 305 may include:
[0123] The acquisition submodule is used to obtain the application parameter table for the time step in the application when generating the image according to the image generation information.
[0124] The generation submodule is used to adjust the parameters of the image generation model at the time step according to the application parameter table, and generate images through the image generation model with the adjusted parameters until the last time step to obtain the target image.
[0125] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0126] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0127] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0128] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image generation model processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.
[0129] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the image generation model processing method.
[0130] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0131] In this embodiment, a pre-trained first generation model and image generation training information are obtained; when the first generation model is trained according to the image generation training information, for the time step in the training, a training configuration file of the time step is obtained, including a quantization parameter table, a key parameter table and a label within the time step; the first generation model is quantized according to the quantization parameter table to obtain a low-precision second generation model, which reduces the computational complexity of the model and reduces resource consumption, making the model more suitable for deployment in a resource-constrained environment; the model parameters are selectively fine-tuned according to the key parameter table to ensure that a high generation quality can be maintained even under low-precision conditions; the label enables the model to flexibly adjust the generation strategy to ensure that the fidelity, personalization and image quality of the generated image are improved while meeting specific requirements; after fine-tuning the second generation model for each time step, a picture generation model is obtained; the picture generation information is obtained and input into the picture generation model to obtain a target picture that meets the requirements; the present application optimizes the picture generation model through quantization processing and selective fine-tuning, thereby reducing the deployment cost of the model.
[0132] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image generation model processing method as described above.
[0133] In this embodiment, a pre-trained first generation model and image generation training information are obtained; when the first generation model is trained according to the image generation training information, for the time step in the training, a training configuration file of the time step is obtained, including a quantization parameter table, a key parameter table and a label within the time step; the first generation model is quantized according to the quantization parameter table to obtain a low-precision second generation model, which reduces the computational complexity of the model and reduces resource consumption, making the model more suitable for deployment in a resource-constrained environment; the model parameters are selectively fine-tuned according to the key parameter table to ensure that a high generation quality can be maintained even under low-precision conditions; the label enables the model to flexibly adjust the generation strategy to ensure that the fidelity, personalization and image quality of the generated image are improved while meeting specific requirements; after fine-tuning the second generation model for each time step, a picture generation model is obtained; the picture generation information is obtained and input into the picture generation model to obtain a target picture that meets the requirements; the present application optimizes the picture generation model through quantization processing and selective fine-tuning, thereby reducing the deployment cost of the model.
[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0135] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for processing an image generation model, characterized in that: The steps include: Obtaining a pre-trained first generative model and image generation training information; When training the first generation model according to the training information generated by the picture, for a time step in training, obtaining a training configuration file of the time step, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step; Quantize the first generation model according to the quantization parameter table to obtain a low-precision second generation model; Based on the key parameter table and the label, fine-tune the second generation model, and obtain a picture generation model after completing fine-tuning of each time step of the second generation model; Obtain image generation information, and input the image generation information into the image generation model to obtain a target image.
2. The image generation model processing method according to claim 1, characterized in that: Before the step of obtaining the pre-trained first generation model and the image generation training information, the method further includes: Get the initial model with full precision; The initial model is pre-trained to obtain a first generation model.
3. The image generation model processing method according to claim 1, characterized in that: The step of fine-tuning the second generation model based on the key parameter table and the label comprises: Get the time step result of the current time step; Calculate the model loss based on the time step result and the label; Based on the key parameter table, determining adjustable parameters in the second generation model; The determined adjustable parameters are fine-tuned according to the model loss.
4. The image generation model processing method according to claim 3, characterized in that: The step of calculating the model loss according to the time step result and the label comprises: Extracting personalized labels, fidelity labels and image quality labels from the labels; Calculate a first loss based on the time step result and the personalized label; Calculating a second loss based on the time step result and the fidelity label; Calculating a third loss based on the time step result and the image quality label; A model loss is calculated according to the first loss, the second loss and the third loss.
5. The image generation model processing method according to claim 1, characterized in that: The step of obtaining the image generation model comprises: Obtain the quantitative parameter table and key parameter table for each time step; Obtaining final parameters of each parameter in the second generation model when not quantized and final parameters when quantized; Generate an application parameter table according to the quantization parameter table and the key parameter table of each time step, the final parameters of each parameter when not quantized, and the final parameters when quantized; A picture generation model is constructed according to the application parameter table and the second generation model.
6. The image generation model processing method according to claim 5, characterized in that: The step of inputting the image generation information into the image generation model to obtain the target image comprises: When generating an image according to the image generation information, obtaining the application parameter table for a time step in an application; The parameters of the picture generation model at the time step are adjusted according to the application parameter table, and the picture generation model after the parameter adjustment is used to generate a picture until the last time step to obtain a target picture.
7. A picture generation model processing device, characterized in that: include: A training acquisition module, used to acquire a pre-trained first generation model and image generation training information; A configuration acquisition module, configured to acquire, for a time step in training, a training configuration file of the time step when training the first generation model according to the image generation training information, the training configuration file including a quantization parameter table, a key parameter table and a label within the time step; A quantization processing module, used for performing quantization processing on the first generation model according to the quantization parameter table to obtain a low-precision second generation model; A model fine-tuning module, used to fine-tune the second generation model based on the key parameter table and the label, and obtain a picture generation model after completing the fine-tuning of each time step of the second generation model; The image generation module is used to obtain image generation information and input the image generation information into the image generation model to obtain a target image.
8. The image generation model processing device according to claim 7, characterized in that: The model fine-tuning module includes: The result acquisition submodule is used to obtain the time step result of the current time step; A loss calculation submodule, used to calculate the model loss according to the time step result and the label; An adjustable determination submodule, used to determine the adjustable parameters in the second generation model based on the key parameter table; The parameter adjustment submodule is used to fine-tune the determined adjustable parameters according to the model loss.
9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the image generation model processing method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the image generation model processing method according to any one of claims 1 to 6 are implemented.