Picture generation model training method and device, storage medium and electronic equipment
The image generation model is reinforced through the Actor-Critic network, and the generated pictures are optimized by the reward model, which solves the problem of low matching of the image generation model in the existing technology, and achieves higher matching and accuracy.
Patent Information
- Application Number
- CN202310566961.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-22
AI Technical Summary
The pictures generated by the existing image generation model do not match the input text very well, and there are errors or unreasonable parts, which need further optimization.
By obtaining the basic image generation model and reward model, combining the Actor-Critic network for reinforcement learning, the Actor network is optimized to improve the accuracy and matching of image generation, and using the near-end strategy optimization algorithm and RLHF algorithm to generate more matching with the input text Target picture.
The matching degree between the pictures generated by the image generation model and the input text is improved, and the wrong or unreasonable parts are reduced. The generated pictures are more in line with user needs, have higher matching degree and higher accuracy.
Smart Images

Figure CN119007221A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, storage medium and electronic device for an image generation model. Background Art
[0002] With the continuous development of machine learning, image generation methods have gradually been upgraded and improved. Users can use text as input, and image generation models can intelligently generate a variety of images, showing great creativity. However, in related technologies, there are some errors or unreasonable parts in the images generated by the image generation models, resulting in the generated images not matching the input text well, which requires further optimization. Summary of the invention
[0003] The embodiments of the present application provide a method, device, storage medium and electronic device for training an image generation model, which can improve the matching degree between the image generated by the image generation model and the input text.
[0004] In a first aspect, an embodiment of the present application provides a method for training an image generation model, the method comprising:
[0005] Obtaining a basic image generation model, wherein the basic image generation model is used to generate an image according to an input text;
[0006] Obtaining a reward model, wherein the reward model is used to evaluate the image generated by the basic image generation model;
[0007] The basic image generation model is used as an Actor network, the reward model is used as a Critic network, and reinforcement learning is performed on the Actor network and the Critic network through an Actor-Critic network to obtain an optimized Actor network;
[0008] The optimized Actor network is used as a picture generation model, and the picture generation model generates a target picture according to the input text, and the target picture matches the input text.
[0009] In a second aspect, an embodiment of the present application provides a processing device for an image generation model, the device comprising:
[0010] A first acquisition module, used to acquire a basic image generation model, where the basic image generation model is used to generate an image based on an input text;
[0011] A second acquisition module is used to acquire a reward model, where the reward model is used to evaluate the image generated by the basic image generation model;
[0012] A first training module is used to use the basic image generation model as an Actor network and the reward model as a Critic network, and to perform reinforcement learning on the Actor network and the Critic network through an Actor-Critic network to obtain an optimized Actor network;
[0013] A generation module is used to use the optimized Actor network as a picture generation model, wherein the picture generation model generates a target picture according to the input text, and the target picture matches the input text.
[0014] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the method provided by the embodiment of the present application.
[0015] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the processor is used to execute the method provided in the embodiment of the present application by calling a computer program stored in the memory.
[0016] In the embodiment of the present application, a basic image generation model is obtained as an Actor network, a reward model is obtained as a Critic network, and reinforcement learning is performed on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network. Specifically, in the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network, the Actor network can continuously improve the accuracy of its own generated images according to the evaluation of the Critic network and the evaluation of the reward model, reduce some errors or unreasonable parts in the generated images, and tend to generate images with higher evaluations, thereby obtaining an optimized Actor network, and using the optimized Actor network as an image generation model, compared with the basic image generation model, the image generation model can generate a target image with a higher evaluation according to the input text, and the target image can be more matched with the input text, that is, the matching degree between the image generated by the image generation model and the input text is improved, so that the image generated by the image generation model is more in line with user needs. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation modes of the present application in conjunction with the accompanying drawings.
[0018] Figure 1 This is a first flow chart of the image generation model training method provided in the embodiment of the present application.
[0019] Figure 2A first structural schematic diagram of the first training stage of the image generation model provided in an embodiment of the present application.
[0020] Figure 3 A schematic diagram of the structure of the second training stage of the image generation model provided in an embodiment of the present application.
[0021] Figure 4 A second structural schematic diagram of the first training stage of the image generation model provided in an embodiment of the present application.
[0022] Figure 5 This is a second flow chart of the image generation model training method provided in an embodiment of the present application.
[0023] Figure 6 This is a first schematic diagram of a video memory optimization method provided in an embodiment of the present application.
[0024] Figure 7 A second schematic diagram of the video memory optimization method provided in an embodiment of the present application.
[0025] Figure 8 A first structural schematic diagram of a training device for an image generation model provided in an embodiment of the present application.
[0026] Fig. 9 A second structural schematic diagram of the image generation model training device provided in an embodiment of the present application.
[0027] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.
[0029] See also Figure 1 , Figure 1 This is a first flow chart of the image generation model training method provided in the embodiment of the present application.
[0030] In 101, a basic image generation model is obtained, where the basic image generation model is used to generate an image according to input text;
[0031] In 102, a reward model is obtained, where the reward model is used to evaluate the image generated by the basic image generation model;
[0032] In 103, the basic image generation model is used as the Actor network, the reward model is used as the Critic network, and the Actor network and the Critic network are reinforced through the Actor-Critic network to obtain the optimized Actor network;
[0033] In 104, the optimized Actor network is used as an image generation model, and the image generation model generates a target image according to the input text, and the target image matches the input text.
[0034] In the embodiment of the present application, a basic image generation model can be selected as the Actor network, such as a stable diffusion model, and the basic image generation model can generate images based on the input text, and a reward model can be selected as the Critic network, such as a Reword model, wherein the Critic network can evaluate the image generated by the Actor network based on the input text. Exemplarily, the reward model can evaluate the image more accurately based on the input text, and since the reward model is used as the initial model of the Critic network, the Critic network can also evaluate the image more accurately based on the input text in the initial state.
[0035] The Actor-Critic network can be an Actor-Critic network that adopts the Proximal Policy Optimization (PPO) algorithm, wherein the Proximal Policy Optimization (PPO) is a deep reinforcement learning algorithm used to train intelligent agents to learn and perform tasks in complex environments. In this algorithm, training an intelligent agent can maximize the cumulative reward to achieve the specified task goals. Therefore, the embodiment of the present application applies the proximal policy optimization algorithm to the image generation model, so that the image generation model can generate images that meet user needs based on the text input by the user.
[0036] Among them, the electronic device used to train the Actor network and the Critic network in the embodiment of the present application can be a computer, a server or a mobile terminal, etc., and the embodiment of the present application is not limited to this.
[0037] In the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network, the Actor network can continuously improve the accuracy of its own generated pictures according to the evaluation of the Critic network and the evaluation of the reward model, reduce some errors or unreasonable parts in the generated pictures, and tend to generate pictures with higher evaluations, thereby obtaining an optimized Actor network; moreover, the Critic network can also continuously improve the accuracy of its own evaluation of pictures according to the difference between its own evaluation and the evaluation of the reward model, thereby obtaining an optimized Critic network. In the embodiment of the present application, the optimized Actor network can be used as a picture generation model. Compared with the basic picture generation model, the picture generation model can generate a target picture with a higher evaluation according to the input text, and the target picture can better match the input text and better meet user needs.
[0038] In one embodiment, the reinforcement learning of the actor network and the critic network through the actor-critic network can be divided into a first training phase and a second training phase that are executed cyclically. Please refer to Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the structure of the first training stage of the image generation model provided in the embodiment of the present application. Figure 3 A schematic diagram of the structure of the second training stage of the image generation model provided in an embodiment of the present application.
[0039] The first training phase includes the following steps:
[0040] First, input text into the Actor network to generate images;
[0041] Second, associate the input text with the generated image to form a text-image combination;
[0042] Third, the text and picture combination is input into the Actor network, the Critic network and the reward model respectively, and the first picture feature information, the first evaluation and the second evaluation are generated accordingly;
[0043] Fourth, the first feature information, the first evaluation, and the second evaluation are processed to obtain a training intermediate data combination.
[0044] Specifically, the input text is input into the Actor network, and the Actor network can generate an image according to the input text; the input text is associated with the generated image to form a text-image combination; the text and image combination are input into the Actor network, and the Actor network extracts the features of the image according to the text-image combination to obtain first feature information; the text-image combination is input into the Critic network, and the Critic network can evaluate the image in the text-image combination to obtain a first evaluation; the text-image combination is input into the reward model, and the reward model can evaluate the image in the text-image combination to obtain a second evaluation; the first feature information and the second evaluation are processed to obtain a third evaluation; the first evaluation and the third evaluation are processed to obtain first evaluation difference information; in the first training stage, for a text-image combination, parameters such as the text-image combination, the first feature information, the first evaluation, the first evaluation difference information, and the third evaluation can be associated to form a set of training intermediate data combinations; in the first training stage, for multiple text-image combinations, multiple sets of corresponding training intermediate data combinations can be formed, such as 1000 sets of corresponding training intermediate data combinations.
[0045] The second training stage includes: inputting the training intermediate data combination into the Actor network and the Critic network for training to obtain an updated Actor network and an updated Critic network, triggering the execution of the first training stage, and replacing the Actor network and the Critic network in the first training stage with the updated Actor network and the updated Critic network respectively until the optimized Actor is obtained.
[0046] Specifically, the text-picture combination is input into the Critic network, and the Critic network can generate a new evaluation for the picture in the text-picture combination to obtain a fourth evaluation; the fourth evaluation and the first evaluation are processed to obtain second evaluation difference information; the second evaluation difference information and the third evaluation are processed to obtain a first loss value of the Critic network, and the Critic network is updated by the first loss value; the text-picture combination pair is input into the Actor network, and the Actor network re-extracts the features of the picture according to the text-picture combination to obtain third feature information; the first feature information and the third feature information are processed to obtain feature difference information; the feature difference information and the first evaluation difference information are processed to obtain a second loss value of the Actor network, and the Actor network is updated by the second loss value.
[0047] In the second training stage, the Critic network and the Actor network are trained multiple times in sequence using multiple training intermediate data combinations generated in the first stage. For example, each of the 1000 training intermediate data combinations generated in the first stage can be used to update the Critic network and the Actor network once. Then, the 1000 training intermediate data combinations generated in the first training stage can be used to update the Critic network and the Actor network 1000 times to obtain the updated Critic network and the updated Actor network.
[0048] It is understandable that after completing the second training stage, the updated Critic network can be used as the Critic network in the first training stage, and the updated Actor network can be used as the Actor network in the first training stage, and the first training stage can be repeated in a cycle to obtain the optimized Actor network after multiple iterations. It is understandable that the optimized Actor network can generate a target image based on the input text, and the target image has a high degree of match with the input text, that is, the target image matches the input text. Specifically, the optimized Actor network can output a target image with an evaluation higher than the evaluation threshold based on the input text. For example, the generated target image can reduce some errors or unreasonable parts, so that the generated image has a higher degree of match with the text and better meets the needs of users. For example, the Critic network can score the pictures generated by the Actor network between 0 and 5. Before the Actor network is optimized, the pictures generated by the Actor network are highly random and of varying quality. When the Critic network scores the pictures generated by the Actor network, the distribution of the scores is unstable, for example, there are 0 points, 1 point, 2 points, 3 points, 4 points, 5 points, etc. After the Actor network is optimized, the Actor network can generate pictures with higher scores. For example, when the optimized Critic network scores the pictures generated by the optimized Actor network, the distribution of the scores is relatively concentrated and the scores are relatively high. For example, the scores of the pictures can basically be between 4 and 5 points. Therefore, the optimized Actor network is used as a graphics generation model. The graphics generation model generates pictures with higher accuracy, the generated pictures have a higher match with the text, and are more in line with user needs.
[0049] It should be noted that in the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network, the Critic network is also constantly improving its accuracy in evaluating images. Exemplarily, the optimized Actor network and the optimized Critic network can also be used together as an image generation model. Specifically, the optimized Actor network generates multiple target images according to the input text, and the optimized Critic network is used to evaluate the multiple target images to select the target image with the highest evaluation, and output the target image with the highest evaluation to the user.
[0050] In one embodiment, in the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network, not only the reward model but also the basic image generation model is used. Figure 4 , Figure 4 A second structural schematic diagram of the first training stage of the image generation model provided in an embodiment of the present application.
[0051] In the first training stage, processing the first feature information and the second evaluation to obtain the third evaluation may include: inputting the text-image combination into the basic image generation model, and the basic image generation model extracts the second feature information of the image according to the text-image combination; sampling the first feature information and the second feature information by KL divergence to obtain KL divergence information, and processing the KL divergence information and the second evaluation to generate the third evaluation. Among them, the basic image generation model is the basic image generation model of the Actor network. During the training process of the Actor network and the Critic network, the parameters of the basic image generation model will not change. The second feature information extracted by the basic image generation model for the image can be used as a reference value in the training process. That is, the embodiment of the present application can improve the training effect of the Actor network by comprehensively considering the degree of deviation of the image features extracted by the basic image generation model and the Actor network.
[0052] See also Figure 5 , Figure 5 This is a second flow chart of the image generation model training method provided in the embodiment of the present application, and the process may include:
[0053] In 201, a basic image generation model is obtained, and a linear layer of the basic image generation model is numerically processed to convert the basic image generation model into a basic reward model;
[0054] In 202, a sample set is obtained, the sample set including training texts, training pictures corresponding to the training texts, and training evaluations corresponding to the training pictures, the training evaluations including picture quality evaluations that can be perceived by humans;
[0055] In 203, based on the sample set, the basic reward model is trained to obtain a reward model.
[0056] In one embodiment, the basic reward model in the embodiment of the present application can be obtained by converting the basic image generation model. Exemplarily, the basic image generation model is obtained, and the linear layer of the basic image generation model is numerically processed to convert the basic image generation model into a basic reward model. Among them, the basic image generation model can generate images based on the input text, such as a stable diffusion model or other models. The basic image generation model may include a text encoder, an image generator, and an image decoder. The text encoder can encode the text to generate a feature expression (text-embedding) of the text, and the image generator can generate a feature expression (image-embedding) of the image based on the feature expression (text-embedding) of the text. The image decoder can decode the feature expression (image-embedding) of the image into an image. In the embodiment of the present application, the linear layer of the basic image generation model can be numerically processed to obtain the basic reward model, so that the basic reward model does not output the image according to the feature expression (image-embedding) of the image, but outputs an evaluation according to the feature expression (image-embedding) of the image, such as a score of 0 to 5 points or sorting.
[0057] Exemplarily, in the embodiment of the present application, when training the basic reward model, the basic image generation model generates training images according to the input training text. It can be understood that although the basic image generation model can generate multiple training images according to the input training text, the quality of the training images is uneven. The generated training images may have a high degree of match with the training text in some parts and a low degree of match with the training text in others. Manual training evaluation can be performed on multiple training images, that is, the training evaluation includes a human-perceivable image quality evaluation. The training text, the training images corresponding to the training text, and the training evaluation corresponding to the training images are then input into the basic reward model for training to obtain a reward model, that is, the reward model is an optimized basic reward model, so that the reward model can be more appropriate and reasonable when evaluating images, and can tend to output evaluations close to humans.
[0058] In one embodiment, inputting a sample set into a basic reward model for training to generate a reward model includes: inputting the sample set into the basic reward model; obtaining keywords of the training text, the keywords including at least one of quantity keywords, attribute keywords, and background keywords; obtaining features of the training image, the image features including at least one of quantity features, attribute features, and background features; matching the features of the training image with the keywords of the training text, and training the basic reward model based on the matching results and training evaluation to obtain a trained reward model.
[0059] Exemplarily, for a training text, the basic image generation model can generate multiple training images. It is understandable that when humans conduct training evaluation on these training images, they generally judge whether the training images are reasonably matched with the training text. For example, when humans conduct training evaluation on training images, they will match the quantity features, attribute features of the subject, background features and other information in the training images with the quantity keywords, attribute keywords of the subject, background keywords and other information in the training text. The higher the matching degree, the higher the training evaluation. Among them, for example, the training text is: an Ultraman riding a horse on the water, then the quantity keyword of the subject in the training text is: one; the attribute keyword is: Ultraman riding a horse, wherein the attribute key also implies, for example, the body shape of Ultraman, the body shape of the horse, the positional relationship between Ultraman and the horse, etc.; the background keyword is: above.
[0060] The basic image generation model can be based on the training text: an Ultraman riding a horse on the water generates multiple training images. It is understandable that the training images generated by the basic image generation model are highly random, so the quality of the generated training images is uneven, and some training images may not satisfy users. These training images can be evaluated manually, exemplarily with a score of 0 to 5. For example, in training image 1, there is 1 Ultraman, and the positions of Ultraman and the horse are correct, the body shapes of Ultraman and the horse are reasonable, and the background is a river, which is equivalent to the water surface. Then it can be considered that training image 1 has a high match with the training text, and the corresponding training evaluation given by humans is: 4 points; for example, in training image 2, there is 1 Ultraman, the positions of Ultraman and the horse are correct, and the background is It is also a river, but some parts of Ultraman's body, such as the legs, are disordered. Then it can be considered that the matching degree between the attribute keywords of the training picture 2 and the training text is medium, and the corresponding training evaluation given by humans is: 3 points; for example, in training picture 3, there is 1 Ultraman and the background is also a river, but the positions of Ultraman and the horse are unreasonable, then it can be considered that the matching degree between the attribute characteristics of the training picture 3 and the attribute keywords of the training text is low, and the corresponding training evaluation given by humans is: 2 points; for example, in training picture 4, there is 1 Ultraman and the background is also a river, but Ultraman's body and the horse's body are disordered, and part of the horse's body becomes Ultraman's body, then it can be considered that the matching degree between the training picture 4 and the training text is very low, and the corresponding training evaluation given by humans is: 1 point.
[0061] For example, the training text is: 2 cats in the grass. If there are 2 cats in the training picture, the body shape and relative positions of the body parts of the cats are reasonable, and the background is grass, then the training picture is considered to match the training text, and a relatively high training evaluation is generally given. If there is 1 cat in the training picture, or the relative positions of the body parts of the cats are unreasonable, or the background is a river instead of grass, then the training picture is considered to not match the training text, and a relatively low training evaluation is generally given.
[0062] After the basic reward model receives the sample set, it can be trained according to the training text, training pictures and training scores to learn the human evaluation criteria from the sample set. For example, for the training text: 2 cats in the grass, the basic reward model obtains the keywords of the training text, where the keyword of the number of subjects in the training text is: 2; the keyword of the attribute of the subject is: cat, including the body shape of the cat and the relative position of each part of the body; the background feature is: grass. The features of the training picture are obtained, where the quantity feature is: the number of cats; the attribute feature is: the body shape of the cat and the relative position of each part of the body; the background features are: grass, forest, river, etc.
[0063] Match the features of the training images with the keywords of the training text, and train the basic reward model based on the matching results and the training evaluation to obtain a trained reward model. For example, the basic training model can also evaluate the training images corresponding to the training text, and process the evaluation with the manual training evaluation to obtain difference information. The basic reward model is trained based on the difference information so that the basic reward model can learn the rules of human evaluation based on a large number of sample sets. It should be noted that when manually scoring, in addition to considering quantitative characteristics, attribute characteristics, and background characteristics, the aesthetic angle of the picture will also be considered for scoring. When training the reward model, the embodiment of the present application incorporates the RLHF algorithm, and incorporates human training evaluation, that is, human feedback, into the training process, providing a natural and humanized interactive learning process, so that the trained reward model can tend to output evaluations close to humans.
[0064] It should be noted that the evaluation given by the reward model can be a score, a ranking, or other forms of evaluation, which is not limited in this embodiment of the present application.
[0065] In 204, the basic image generation model is used as the Actor network, the reward model is used as the Critic network, and the Actor network and the Critic network are reinforced through the Actor-Critic network to obtain an optimized Actor network;
[0066] In 205, the optimized Actor network is used as an image generation model, and the image generation model generates a target image according to the input text, and the target image matches the input text.
[0067] In the embodiment of the present application, the basic image generation model is used as an Actor network, the basic image generation model can generate images according to the input text, and the reward model can be used as a Critic network, wherein the Critic network can evaluate the images generated by the Actor network according to the input text. It can be understood that after the above training, the reward model learns the evaluation criteria of humans and can tend to output evaluations close to humans. Therefore, when the Critic network is in the initial state, when evaluating the images generated by the Actor network, it can also tend to output evaluations close to humans.
[0068] The Actor-Critic network in the embodiment of the present application can be an Actor-Critic network that adopts the Proximal Policy Optimization (PPO) algorithm, wherein the Proximal Policy Optimization (PPO) is a deep reinforcement learning algorithm used to train intelligent agents to learn and perform tasks in complex environments. In this algorithm, training an intelligent agent can maximize the cumulative reward to achieve the specified task goals. Therefore, applying the proximal policy optimization algorithm to the image generation model can enable the image generation model to generate images that meet user needs based on the text input by the user.
[0069] The embodiment of the present application comprehensively considers the evaluation of the Critic network and the evaluation of the reward model, that is, the embodiment of the present application incorporates the RLHF algorithm when training the Actor network, so that human feedback can be incorporated into the training process to guide the Actor network to generate pictures that meet human preferences. The embodiment of the present application combines the proximal strategy optimization algorithm and the RLHF algorithm to train the Actor network, so that in the multimodal usage scenario of generating pictures from input text, pictures with a high degree of match with the input text can be generated, and the aesthetic effect of the picture can be improved.
[0070] Please continue to refer to Figure 4 and Figure 5 , the optimized Actor network obtained by reinforcement learning of the Actor network and the Critic network through the Actor-Critic network can be divided into the first training stage and the second training stage of cyclic execution. The first training stage includes:
[0071] The first training phase includes the following steps:
[0072] First, input text into the Actor network to generate images;
[0073] Second, associate the input text with the generated image to form a text-image combination;
[0074] Third, the text and picture combination is input into the Actor network, the Critic network, the reward model and the basic picture generation model respectively, and the first picture feature information, the first evaluation, the second evaluation and the second picture feature information are generated accordingly;
[0075] Fourth, the first feature information, the first evaluation, the second evaluation and the second feature information are processed to obtain a training intermediate data combination.
[0076] Specifically, the input text is input into the Actor network, and the Actor network can generate an image according to the input text; the input text is associated with the generated image to form a text-image combination; the text and image combination are input into the Actor network, and the Actor network extracts the features of the image according to the text-image combination to obtain first feature information; the text-image combination is input into the Critic network, and the Critic network can evaluate the image in the text-image combination to obtain a first evaluation; the text-image combination is input into the reward model, and the reward model can evaluate the image in the text-image combination to obtain a second evaluation; the text-image combination is input into the basic image generation model, and the basic image generation model is used to generate the first feature information. The model extracts the second feature information of the image according to the text-image combination; the first feature information and the second feature information are sampled by KL divergence to obtain KL divergence information, and the KL divergence information is processed with the second evaluation to generate a third evaluation; the first evaluation and the third evaluation are processed to obtain first evaluation difference information; in the first training stage, for a text-image combination, parameters such as the text-image combination, the first feature information, the first evaluation, the first evaluation difference information, and the third evaluation can be associated to form a group of training intermediate data combinations; in the first training stage, multiple groups of corresponding training intermediate data combinations can be formed for multiple text-image combinations, for example, 1000 groups of corresponding training intermediate data combinations can be formed.
[0077] Among them, the second training stage includes: inputting the training intermediate data combination into the Actor network and the Critic network for training to obtain an updated Actor network and an updated Critic network. Specifically, inputting the text-picture combination into the Critic network, the Critic network can generate a new evaluation for the picture in the text-picture combination to obtain a fourth evaluation; processing the fourth evaluation and the first evaluation to obtain second evaluation difference information; processing the second evaluation difference information and the third evaluation to obtain a first loss value of the Critic network, and updating the Critic network through the first loss value; inputting the text-picture combination into the Actor network, the Actor network re-extracts the features of the picture according to the text-picture combination to obtain third feature information; processing the first feature information and the third feature information to obtain feature difference information; processing the feature difference information and the first evaluation difference information to obtain a second loss value of the Actor network, and updating the Actor network through the second loss value.
[0078] In the second training stage, the Critic network and the Actor network are trained multiple times in sequence using multiple training intermediate data combinations generated in the first stage. For example, each of the 1000 training intermediate data combinations generated in the first stage can be used to update the Critic network and the Actor network once. Then, the 1000 training intermediate data combinations generated in the first training stage can be used to update the Critic network and the Actor network 1000 times to obtain the updated Critic network and the updated Actor network.
[0079] It can be understood that after completing the second training stage, the first training stage can be triggered to execute, and the updated Actor network and the updated Critic network can replace the Actor network and the Critic network in the first training stage respectively, so as to obtain the optimized Actor network after multiple iterative updates.
[0080] During the Actor-Critic network training process, the Actor network continuously improves the accuracy of its generated images, reduces some errors or unreasonable parts in the generated images, and tends to generate images with higher evaluations. It is understandable that the higher the evaluation of the image, the more it tends to be in line with the user's preferences, that is, it can guide the Actor network to generate images that meet human preferences; and the Critic network can also continuously improve the accuracy of its own evaluation of images during the training process, thereby obtaining an optimized Actor network and an optimized Critic network.
[0081] Therefore, the optimized Actor network can output target images with evaluations higher than the evaluation threshold according to the input text. For example, the generated target image can reduce some errors or unreasonable parts, so that the generated image has a higher matching degree with the text and is more in line with the user's preferences. For example, the Critic network can score the images generated by the Actor network between 0 and 5. Before the Actor network is optimized, the images generated by the Actor network are highly random and of varying quality. When the Critic network scores the images generated by the Actor network, the distribution of the scores is unstable, such as 0, 1, 2, 3, 4, 5, etc. After the Actor network is optimized, the Actor network can generate images with higher scores. For example, when the optimized Critic network scores the images generated by the optimized Actor network, the distribution of the scores is relatively concentrated and the scores are relatively high. For example, the scores of the images can basically be between 4 and 5. Therefore, the images generated by the optimized Actor network are more accurate, and the generated images have a higher matching degree with the text, which can be more in line with the user's preferences. It should be noted that in the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network, the Critic network is also constantly improving its accuracy in evaluating images. Exemplarily, the optimized Actor network and the optimized Critic network can also be used together as an image generation model. Specifically, when there are multiple target images generated by the optimized Actor network according to the input text, the optimized Critic network can also be used to evaluate the multiple target images to select the target image with the highest evaluation, and output the target image with the highest evaluation to the user.
[0082] In one implementation, the embodiment of the present application can use a trainer controller to encapsulate the process of reinforcement learning of the Actor network and the Critic network through the Actor-Critic network. Specifically, the trainer controller provides a convenient function interface to encapsulate the basic image generation model, the reward model, the Actor network, the Critic network, the input text, and the parameter information, etc., to obtain a packaged training process, forming a reinforcement learning framework applied to the image generation model, so that developers can easily perform reinforcement learning on the Actor network and the Critic network. For example:
[0083] trainer=RLHFTrainer(
[0084] actor=generate_model,
[0085] critic = reward_model
[0086] reward_model=reward_model,
[0087] prompt=prompts,
[0088] config=config,
[0089] minibatch_size=20,
[0090] actor_lr=1e-6,
[0091] critic_lr = 1e-4;
[0092] Among them, generate_model is set as Actor network in the trainer controller, reward_model is used as Critic network, reward_model is used as reward model, and necessary parameters are set, so that the Actor network and the Critic network can be easily trained through the Actor-Critic network. After the training, the target image can be output according to the input text through the Actor network (generate_model) in the trainer controller, and the target image is the highest-rated image. The reinforcement learning framework of the embodiment of the present application can also be applied to various other basic image generation models for reinforcement learning to optimize various other basic image generation models, such as models that can be extended to natural language processing (Natural Language Processing, NLP) or computer vision (Computer Vision, CV) fields.
[0093] It should be noted that, although four models are involved in the training process of the Actor-Critic network in the embodiment of the present application, the user can only provide a basic image generation model, the basic image generation model can be converted into a basic reward model, the basic image generation model can also be used as the basic image generation model of the Actor network, and the basic reward model can be used as the basic image generation model of the Critic network, and the training process of the Actor network and the Critic network is encapsulated through the trainer controller to obtain a packaged training process, so that the reinforcement learning training of the Actor network and the Critic network can be easily completed to obtain the optimized Actor network and the optimized Critic network.
[0094] It is understandable that with the increase in hardware computing power, larger and larger models can be trained, and the parameters of each model have basically reached the scale of hundreds of billions. The introduction of reinforcement learning has also made the bottleneck of video memory more and more obvious. There is a problem of insufficient video memory. The current popular graphics cards such as V100 are generally 30G and A100 are generally 80GB. If V100 is used, a model with a parameter scale of about 1.5 billion can be stored; if A100 is used, a model with a parameter scale of about 3.5 billion can be stored. It can be seen that without the help of video memory optimization technology, the scale of models that can be trained is very limited and cannot support the current model training that reaches hundreds of billions. In the training of the image generation model of the embodiment of this application, at least the Actor network, the Critic network and the reward model are involved, and even the basic image generation model is included. Therefore, the graphics card required is 3 or 4 times that of a model, which is prone to insufficient video memory. Video memory optimization technology is crucial in reinforcement learning scenarios.
[0095] The embodiments of the present application may adopt a variety of video memory optimization methods. In one implementation, a tensor parallel video memory optimization method may be adopted. Specifically, the models involved in the training, such as the Actor network and the Critic network, may be split, and the model splitting technology may be used to distribute the model's memory usage to each graphics card. The model splitting method mainly cuts the Attention layer of the UNet in the Stable Diffusion model structure, or the MLP layer in the Transformer structure may also be split in this way.
[0096] In one implementation, a zero-redundancy optimizer (ZeRO optimizer) technology may be used. Specifically, during the model training process, the optimizer state, gradients, and model parameters will occupy a large amount of graphics card memory. The ZeRO optimizer technology may divide the optimizer state, gradients, and parameters into separate graphics cards for training, thereby enabling data-parallel training and increasing the training speed.
[0097] In one implementation, when training the Actor network and the Critic network, the embodiment of the present application can also simultaneously adopt tensor parallelism and zero-redundancy optimizer methods to perform reinforcement learning on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network.
[0098] For details, please refer to Figure 6 and Figure 7 , Figure 6 The first schematic diagram of the video memory optimization method provided in the embodiment of the present application is Figure 7 A second schematic diagram of the video memory optimization method provided in an embodiment of the present application.
[0099] For example, a model can be trained on two electronic devices at the same time. Then, the model needs to be loaded on both electronic devices, namely model 1 and model 2. For example, the two electronic devices each have 8 graphics cards (GPU cards), for a total of 16 graphics cards. Figure 6 As shown, they are GPU0 to GPU15 respectively. Here, the model may include the Actor network, the Critic network or the reward model in the Actor-Critic network. For example, the two Actor networks may be divided into 8 model slices respectively, and a total of 16 model slices are loaded into 16 graphics cards. Figure 6 As shown, model 1 is divided into GPU0, GPU1, GPU4, GPU5, GPU8, GPU9, GPU12, and GPU13, and model 2 is divided into GPU2, GPU3, GPU6, GPU7, GPU10, GPU11, GPU14, and GPU15. Moreover, the same shape represents the same model slice, for example, 1 and 2 are the same model slice, so the embodiment of the present application can not only divide the same model into different graphics cards, but also train different data for the same model slice in different graphics cards to form a data parallel group, for example, training the first data for 1 in GPU0, and training the second data for 2 in GPU2, and further, a ZeRO optimizer is further added to the data parallel group, and the optimizer state, gradient, and model parameters in the data parallel group are further divided. Taking GPU0 and GPU2 as an example, after adding the ZeRO optimizer technology, the video memory occupancy will be further evenly divided into two GPUs, as shown in FIG. Figure 7 As shown in the figure: the optimizer state (Pos), gradient (Pos+g) and model parameters (Pos+g+p) will be further evenly divided into GPU0 and GPU2. Similarly, other data parallel groups can also be divided in the same way, so that the memory usage of model training can be evenly distributed on each graphics card, which can not only optimize the usage of video memory, but also improve the speed of model training.
[0100] In one embodiment, in the first training stage, the reward model and the basic image generation model are loaded into the video memory; in the second training stage, the reward model and the basic image generation model are loaded into the memory, and the reward model is deleted from the video memory. In the embodiment of the present application, the Actor network and the Critic network are reinforced and learned to obtain an optimized Actor network, which includes a first training stage and a second training stage that are executed cyclically. In the first training stage, four models need to be loaded into the video memory, including the Actor network, the Critic network, the reward model, and the basic image generation model. Specifically, the text and image combination need to be input into the Actor network, the Critic network, the reward model, and the basic image generation model, respectively, to generate the first image feature information, the first evaluation, the second evaluation, and the second image feature information; the first feature information, the first evaluation, the second evaluation, and the second feature information are processed to obtain a training intermediate data combination.
[0101] However, in the second training phase, only the intermediate training data combination needs to be input into the Actor network and the Critic network for training to obtain the updated Actor network and the updated Critic network. Therefore, in the second training phase, the reward model and the basic image generation model can be loaded into the memory, and the reward model and the basic image generation model can be deleted from the video memory, thereby reducing the occupation of the video memory by the reward model and the basic image generation model.
[0102] For example, assuming that the number of model parameters is S, in the currently popular mixed precision training, the popular Adam optimizer state occupies 12S bytes of video memory, the gradient occupies 2S bytes, and the model weight occupies 2S bytes, a total of 16S bytes of video memory.
[0103] Using the current mainstream A100 GPU card for training, with a total of 80GB of video memory, if no video memory optimization method is used, then the maximum number of parameters that can be trained is 80GB / 16B=5G (5 billion). If hybrid parallel video memory optimization is used, theoretically 100G*16B / 80G=20 cards are required to expand to 100 billion parameters. This does not take into account the memory usage of activation values and other possible memory usage. If the memory usage of activation values is taken into account, more GPU cards are required.
[0104] During the training process of the Actor-Cirtic model, if the video memory optimization method is not used, four models need to be loaded into the GPU at the same time, and 20*4=80 cards are needed to complete the reinforcement learning training of the trillion-dollar model. By adopting the method of tensor parallelism and zero-redundancy optimizer in the embodiment of the present application, and unloading the reward model and the basic image generation model from the video memory during the second training stage, only two models need to be loaded into the video memory in the second training stage, and 20*2=40 cards can be used to complete the reinforcement learning training of the trillion-dollar model.
[0105] In one embodiment, the embodiment of the present application can also adopt the video memory optimization method of activation recomputation. Specifically, the principle of activation recomputation is to use the torch.utils.checkpoint package in pytorch to save the checkpoint. The checkpoint in PyTorch is realized by rerunning the forward propagation calculation of each check segment during the backward propagation process. Unlike the conventional method of saving all the intermediate activation values of the entire calculation graph for calculating the backward propagation, the intermediate activation values are no longer saved as part of the checkpoint, but the intermediate activation values are recalculated once in the backward propagation, that is, the forward propagation of the checkpoint part is rerun. Specifically, in the forward pass, the incoming function will run in the torch.no_grad mode, that is, the intermediate activation values are not saved. Instead, the forward pass saves the input tuple and function parameters. In the backward pass, the saved input tuple and function parameters will be retrieved, and the forward pass will be recalculated on the function, at which time the intermediate activation values will be tracked, and then the gradient will be calculated based on these intermediate activation values. Among them, the video memory optimization method of activating recalculation can be used in conjunction with any of the above video memory optimization methods, thereby further reducing the occupancy of video memory and improving the training speed of the model.
[0106] That is, the video memory optimization method of the embodiment of the present application can be combined with multiple video memory optimization methods such as tensor parallelism, zero-redundancy optimizer, activation recalculation, and unloading the reward model and the basic image generation model from the video memory during the second training phase, to build a unified architecture that can combine various video memory optimization methods. In addition, the embodiment of the present application uses a Trainer controller to encapsulate the training process. The Trainer controller can automatically perform hyperparameter search and automatically select the parallel mode with the best training performance through environmental perception, thereby achieving the ultimate video memory optimization effect while maintaining the optimal training speed. It can support the reinforcement learning process of the basic image generation model that is expanded to the scale of hundreds of billions. The video memory optimization method of the embodiment of the present application is not only suitable for the training of large-scale image generation models, but can also be applied to various other basic image generation models for reinforcement learning to optimize various other basic image generation models, such as models related to natural language processing (Natural Language Processing, NLP) or computer vision (Computer Vision, CV).
[0107] In order to facilitate better implementation of the training method of the picture generation model provided in the embodiment of the present application, the embodiment of the present application also provides a device for training the picture generation model based on the training method of the picture generation model. For specific implementation details, please refer to the description in the above method embodiment.
[0108] Please refer to Figure 8 and Fig. 9 , Figure 8 A first structural diagram of a training device for an image generation model provided in an embodiment of the present application, Fig. 9 A second structural diagram of a training device for a picture generation model provided in an embodiment of the present application. The training device 300 for a picture generation model may include a first acquisition module 301, a second training module 302, a first training module 303, and a generation module 303, wherein:
[0109] A first acquisition module 301 is used to acquire a basic image generation model, where the basic image generation model is used to generate an image according to an input text;
[0110] A second acquisition module 302 is used to acquire a reward model, where the reward model is used to evaluate the image generated by the basic image generation model;
[0111] The first training module 303 is used to use the basic image generation model as the Actor network and the reward model as the Critic network, and to perform reinforcement learning on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network;
[0112] The generation module 304 is used to use the optimized Actor network as a picture generation model, and the picture generation model generates a target picture according to the input text, and the target picture matches the input text.
[0113] In one embodiment, the training device 300 for generating a model may include a second training module 305, and the second acquisition module 302 is also used to: obtain a basic image generation model, and perform numerical processing on the linear layer of the basic image generation model to convert the basic image generation model into a basic reward model; the second training module 305 is used to: train the basic reward model to obtain a reward model.
[0114] In one embodiment, the second training module 305 is used to: obtain a sample set, the sample set includes training text, training pictures corresponding to the training text, and training evaluations corresponding to the training pictures, the training evaluations include human-perceivable picture quality evaluations; based on the sample set, train the basic reward model to obtain a reward model.
[0115] In one embodiment, the second training module 305 is used to: obtain keywords of the training text, the keywords include quantity keywords, attribute keywords, and at least one of background keywords; obtain features of the training image, the image features include quantity features, attribute features, and at least one of background features; match the features of the training image with the keywords of the training text to obtain matching results, and train the basic reward model based on the matching results and training evaluation to obtain a reward model.
[0116] In one embodiment, the first training module 303 is used to: perform reinforcement learning on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network, including a first training phase and a second training phase that are executed cyclically, and the first training phase includes: inputting input text into the Actor network to generate an image; associating the input text with the generated image to form a text-image combination; inputting the text and image combination into the Actor network, the Critic network, the reward model, and the basic image generation model, respectively, to correspondingly generate first image feature information, a first evaluation, a second evaluation, and second image feature information; processing the first feature information, the first evaluation, the second evaluation, and the second feature information to obtain a training intermediate data combination;
[0117] The second training stage includes: inputting the training intermediate data combination into the Actor network and the Critic network for training to obtain an updated Actor network and an updated Critic network, triggering the execution of the first training stage, and replacing the Actor network and the Critic network in the first training stage with the updated Actor network and the updated Critic network respectively until the optimized Actor network is obtained.
[0118] In one embodiment, the first training module 303 is used to: in the first training stage, load the reward model and the basic image generation model into the video memory; in the second training stage, load the reward model and the basic image generation model into the memory, and delete the reward model from the video memory.
[0119] In one implementation, the first training module 303 is used to: adopt tensor parallelism and zero-redundancy optimizer to perform reinforcement learning on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network.
[0120] The application embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the process in the thread scheduling method provided in this embodiment.
[0121] The present application also provides an electronic device, including a memory and a processor, wherein the processor executes the process in the thread scheduling method provided in the present embodiment by calling a computer program stored in the memory. The electronic device may be a computer, a server, or a mobile terminal, etc., which is not limited in the present application. Fig.10 , Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0122] The electronic device 400 may include components such as a memory 402 and a processor 401. Those skilled in the art will appreciate that Figure 5 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0123] The memory 402 may be used to store applications and data. The applications stored in the memory 402 include executable codes. The applications may be composed of various functional modules. The processor 401 executes various functional applications and data processing by running the applications stored in the memory 402. The processor 401 is electrically connected to the memory 402.
[0124] The processor 401 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.
[0125] In this embodiment, the processor 401 in the electronic device loads the executable code corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 runs the application program stored in the memory 402, thereby executing:
[0126] Get a basic image generation model, which is used to generate images based on input text;
[0127] Obtain a reward model, which is used to evaluate the images generated by the basic image generation model;
[0128] The basic image generation model is used as the Actor network, and the reward model is used as the Critic network. The Actor network and the Critic network are reinforced through the Actor-Critic network to obtain the optimized Actor network.
[0129] The optimized Actor network is used as an image generation model. The image generation model generates a target image based on the input text, and the target image matches the input text.
[0130] In one embodiment, in obtaining the reward model, the processor 401 executes: obtaining a basic image generation model, numerically processing the linear layer of the basic image generation model to convert the basic image generation model into a basic reward model; and training the basic reward model to obtain a reward model.
[0131] In one embodiment, in training a basic reward model to obtain a reward model, the processor 401 executes acquisition of a sample set, where the sample set includes training text, training images corresponding to the training text, and training evaluations corresponding to the training images, where the training evaluations include human-perceivable image quality evaluations; based on the sample set, the basic reward model is trained to obtain a reward model.
[0132] In one embodiment, in training a basic reward model based on a sample set to obtain a reward model, the processor 401 executes obtaining keywords of a training text, the keywords including at least one of quantity keywords, attribute keywords, and background keywords: obtaining features of a training image, the image features including at least one of quantity features, attribute features, and background features: matching the features of the training image with keywords of the training text, and training the basic reward model based on the matching results and the training evaluation to obtain a reward model.
[0133] In one embodiment, the optimized Actor network obtained by performing reinforcement learning on the Actor network and the Critic network through the Actor-Critic network includes a first training phase and a second training phase that are executed cyclically. In the first training phase, the processor 401 executes: inputting input text into the Actor network to generate an image; associating the input text with the generated image to form a text-image combination; inputting the text and the image combination into the Actor network, the Critic network, the reward model, and the basic image generation model, respectively, to generate first image feature information, a first evaluation, a second evaluation, and second image feature information; processing the first feature information, the first evaluation, the second evaluation, and the second feature information to obtain a training intermediate data combination;
[0134] In the second training stage, the processor 401 executes: inputting the training intermediate data combination into the Actor network and the Critic network for training to obtain an updated Actor network and an updated Critic network, triggering the execution of the first training stage, and replacing the Actor network and the Critic network in the first training stage with the updated Actor network and the updated Critic network respectively until an optimized Actor network is obtained.
[0135] In one embodiment, the processor 401 performs: in a first training phase, loading the reward model and the basic image generation model into a video memory; in a second training phase, loading the reward model and the basic image generation model into the memory, and deleting the reward model from the video memory.
[0136] In one embodiment, in training the Actor-Critic network using a reward model to obtain an optimized Actor-Critic network, the processor 401 executes: using tensor parallelism and zero-redundancy optimizer to perform reinforcement learning on the Actor network and the Critic network through the Actor-Critic network to obtain an optimized Actor network.
[0137] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the thread scheduling method above, which will not be repeated here.
[0138] The training device for the image generation model provided in the embodiment of the present application and the training method for the image generation model in the above embodiment belong to the same concept. Any method provided in the training method embodiment for the image generation model can be run on the training device for the image generation model. The specific implementation process is detailed in the training method embodiment for the image generation model, which will not be repeated here.
[0139] It should be noted that, for the training method of the picture generation model in the embodiment of the present application, a person of ordinary skill in the art can understand that all or part of the process of implementing the training method of the picture generation model in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the training method of the picture generation model. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), etc.
[0140] For the training device of the image generation model of the embodiment of the present application, its various functional modules can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.
[0141] The above is a detailed introduction to the training method, device, storage medium and electronic device of the image generation model provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A training method for an image generation model, It is characterized in that The method comprises: Obtaining a basic image generation model, wherein the basic image generation model is used to generate an image according to an input text; Obtaining a reward model, wherein the reward model is used to evaluate the image generated by the basic image generation model; The basic image generation model is used as an Actor network, the reward model is used as a Critic network, and reinforcement learning is performed on the Actor network and the Critic network through an Actor-Critic network to obtain an optimized Actor network; The optimized Actor network is used as a picture generation model, and the picture generation model generates a target picture according to the input text, and the target picture matches the input text.
2. The training method of the image generation model according to claim 1, wherein the reward model is obtained include: Acquire the basic image generation model, and perform numerical processing on the linear layer of the basic image generation model to convert the basic image generation model into the basic reward model; The basic reward model is trained to obtain the reward model.
3. The training method of the image generation model according to claim 2, wherein the basic reward model is trained to obtain the reward model include: Acquire a sample set, the sample set including training text, training pictures corresponding to the training text, and training evaluations corresponding to the training pictures, wherein the training evaluations include picture quality evaluations that can be perceived by humans; Based on the sample set, the basic reward model is trained to obtain the reward model.
4. The training method of the image generation model according to claim 2, It is characterized in that The step of training the basic reward model based on the sample set to obtain the reward model includes: Obtain keywords of the training text, the keywords include at least one of quantity keywords, attribute keywords, and background keywords: Obtain the features of the training image, which include at least one of quantity features, attribute features, and background features: The features of the training images are matched with the keywords of the training text to obtain a matching result, and the basic reward model is trained according to the matching result and the training evaluation to obtain the reward model.
5. The method for training a picture generation model according to claim 1, It is characterized in that The optimized Actor network obtained by performing reinforcement learning on the Actor network and the Critic network through the Actor-Critic network includes a first training phase and a second training phase that are executed cyclically, wherein the first training phase includes: Input text into the Actor network to generate an image; Associating the input text with the generated image to form a text-image combination; Inputting the text and picture combination into the Actor network, the Critic network, the reward model and the basic picture generation model respectively, and correspondingly generating first picture feature information, a first evaluation, a second evaluation and second picture feature information; Processing the first feature information, the first evaluation, the second evaluation, and the second feature information to obtain a training intermediate data combination; The second training phase includes: The intermediate training data combination is input into the Actor network and the Critic network for training to obtain an updated Actor network and an updated Critic network, triggering the execution of the first training phase, and replacing the Actor network and the Critic network in the first training phase with the updated Actor network and the updated Critic network, respectively, until the optimized Actor network is obtained.
6. The method for training a picture generation model according to claim 4, It is characterized in that The method further comprises: In the first training stage, the reward model and the basic image generation model are loaded into a video memory; In the second training stage, the reward model and the basic image generation model are loaded into the memory, and the reward model is deleted from the video memory.
7. The method for training a picture generation model according to claim 1, It is characterized in that The Actor network optimized by performing reinforcement learning on the Actor network and the Critic network through the Actor-Critic network includes: By adopting the tensor parallel and zero-redundancy optimizer method, the Actor network and the Critic network are subjected to reinforcement learning through the Actor-Critic network to obtain an optimized Actor network.
8. A device for generating a model from an image, It is characterized in that The device comprises: A first acquisition module, used to acquire a basic image generation model, where the basic image generation model is used to generate an image based on an input text; A second acquisition module is used to acquire a reward model, where the reward model is used to evaluate the image generated by the basic image generation model; A first training module is used to use the basic image generation model as an Actor network and the reward model as a Critic network, and to perform reinforcement learning on the Actor network and the Critic network through an Actor-Critic network to obtain an optimized Actor network; A generation module is used to use the optimized Actor network as a picture generation model, wherein the picture generation model generates a target picture according to the input text, and the target picture matches the input text.
9. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor, It is characterized in that The processor is configured to execute the method according to any one of claims 1 to 7 by calling the computer program stored in the memory.
Citation Information
Cited By
Visual content generation and target large model training method and device based on large model
CN120893504A
Method and device for generating graphic design drawing
CN122049121A
