Image generator training method, image generation method and device

Abstract

The embodiment of the present invention discloses a training method for an image generator, an image generation method and an apparatus. The training method for the image generator includes: generating an initial generative adversarial network model based on the sparsity of the generator and the discriminator; iteratively training the initial generative adversarial network model based on sample data, and during the iterative training process, updating the connection relationship of each network layer in the generator of the current iteration, continuing to iteratively train the updated generator and the discriminator of the current iteration until the end condition is met, and determining the trained generator as the target image generator. By updating the connections with low importance during the training process, pruning of the connections is achieved, so that an image generator that takes into account both processing accuracy and sparsity is obtained after the training is completed. At the same time, since the sparsity of the generator remains unchanged, the computational complexity of the training process is reduced, and since the training parameters are small, the training efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of deep learning technology, and in particular to a training method for an image generator, an image generation method, and an image generation device. Background Art

[0002] Generative Adversarial Networks (GANs) are a deep learning model that has emerged as one of the most promising approaches for unsupervised learning on complex distributions in recent years. The model produces robust outputs through a learning game involving (at least) two modules: a generator and a discriminator.

[0003] As the quality of image generation improves, the training cost of generative adversarial networks also increases. Currently, compressed generative adversarial networks can be obtained through model compression techniques such as pruning, distillation, and the lottery ticket hypothesis.

[0004] In the process of implementing the present invention, it was found that there are at least the following technical problems in the prior art: a dense generative adversarial network needs to be trained before model compression, and the model compression process includes iterative compression and retraining processes, resulting in an overall computational amount far exceeding the training of the dense model. Summary of the Invention

[0005] Embodiments of the present invention provide an image generator training method, an image generation method, and an apparatus to achieve a sparse image generator obtained through training, thereby reducing the storage space and computational complexity of the image generator.

[0006] In a first aspect, an embodiment of the present invention provides a method for training an image generator, comprising:

[0007] Generate an initial generative adversarial network model based on the sparsity of the generator and the discriminator;

[0008] The initial generative adversarial network model is iteratively trained based on the sample data, and during the iterative training process, when the model update conditions are met, the connection relationship between the network layers in the generator of the current iteration is updated, and the updated generator and the discriminator of the current iteration are continued to be iteratively trained until the end conditions are met, and the trained generator is determined as the target image generator.

[0009] In a second aspect, an embodiment of the present invention further provides an image generation method, comprising:

[0010] Get the image to be processed;

[0011] The image to be processed is input into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method for the image generator provided in any embodiment of the present invention.

[0012] In a third aspect, an embodiment of the present invention further provides a training device for an image generator, comprising:

[0013] The initial model generation module is used to generate the initial generative adversarial network model based on the sparsity of the generator and the discriminator;

[0014] The model training module is used to iteratively train the initial generative adversarial network model based on sample data, and during the iterative training process, when the model update conditions are met, the connection relationship of each network layer in the generator of the current iteration is updated, and the updated generator and the discriminator of the current iteration are continued to be iteratively trained until the end conditions are met, and the trained generator is determined as the target image generator.

[0015] In a fourth aspect, an embodiment of the present invention further provides an image generating device, comprising:

[0016] The image to be processed acquisition module is used to acquire the image to be processed;

[0017] A target image generation module is used to input the image to be processed into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method of the image generator provided in any embodiment of the present invention.

[0018] In a fifth aspect, an embodiment of the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image generator training method or an image generation method as provided in any embodiment of the present invention is implemented.

[0019] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an image generator training method or an image generation method as provided in any embodiment of the present invention.

[0020] The technical solution provided by the embodiment of the present invention constructs an initial generative adversarial network model based on the sparsity of the generator and the sparsity of the discriminator. During the iterative training of the initial generative adversarial network model, the connections in the initial generator are updated and optimized, and the updated generative adversarial network model is iteratively trained until the end condition is met, thereby obtaining the target image generator. By updating the connections with low importance during the training process, the connections are pruned, so that an image generator that takes into account both processing accuracy and sparsity is obtained after the training is completed. At the same time, since the sparsity of the generator remains unchanged, the computational complexity of the training process is reduced, and the training efficiency is improved due to the small number of training parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for training an image generator provided by an embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of an update process of a generator provided by an embodiment of the present invention;

[0023] Figure 3 is a flow chart of another method for training an image generator provided by an embodiment of the present invention;

[0024] Figure 4 A schematic flow chart of an image generation method provided by an embodiment of the present invention;

[0025] Figure 5 1 is a schematic structural diagram of a training device for an image generator provided by an embodiment of the present invention;

[0026] Figure 6 is a structural diagram of an image generating device provided by an embodiment of the present invention;

[0027] Figure 7 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0029] Figure 1This is a flow chart of a method for training an image generator provided by an embodiment of the present invention. This embodiment is applicable to the case of training a generative adversarial network model to obtain an image generator. The method can be performed by a training device for an image generator provided by an embodiment of the present invention. The training device for an image generator can be implemented by software and / or hardware. The training device for an image generator can be configured on an electronic computing device. The method specifically includes the following steps:

[0030] S110. Generate an initial generative adversarial network model based on the sparsity of the generator and the discriminator.

[0031] S120. Iteratively train the initial generative adversarial network model based on sample data, and execute step S130 if the model update condition is met.

[0032] S130. Update the connection relationship of each network layer in the generator of the current iteration, continue to perform the iterative training of step S120 on the updated generator and the discriminator of the current iteration until the end condition is met, and determine the trained generator as the target image generator.

[0033] The generator and discriminator each include multiple network layers, each of which includes multiple neurons. Each connection between any two neurons in each network layer is assigned a network parameter. Sparsity is used to characterize the ratio of network parameter settings in the model, and can be any value between 0 and 100%. The sparsity of a model can be the ratio of the number of inactive network parameters in the model to the initial number of network parameters in the corresponding fully connected model. Inactive network parameters are network parameters corresponding to two neurons that are not connected, i.e., empty network parameters. The higher the sparsity of the model, the smaller the number of actual network parameters in the model.

[0034] In some embodiments, the sparsity of the generator is greater than the sparsity of the discriminator. By training a discriminator with known sparsity to obtain a generator with high sparsity, the number of network parameters in the image generator is reduced while ensuring the processing accuracy of the trained image generator, thereby reducing the storage space and computational complexity of the obtained image generator, and reducing the storage space and computing power occupied by the image generator on the actual application platform.

[0035] The initial generative adversarial network model includes an initial generator and an initial discriminator. The initial generator is created based on the sparsity of the generator, and the initial discriminator is created based on the sparsity of the discriminator. Specifically, based on the sparsity of the generator, the connections between the network layers in the generator are randomly sampled, so that the connections obtained by random sampling meet the sparsity of the generator. It should be noted that each network layer includes at least one connection to ensure the connectivity of the generator. Similarly, based on the sparsity of the discriminator, the connections between the network layers in the discriminator are randomly sampled, so that the connections obtained by random sampling meet the sparsity of the discriminator. At the same time, the discriminator generated by random sampling meets the connectivity of the model.

[0036] Obtain sample data, and iteratively train the initial generative adversarial network model based on the sample data. The sample data is determined based on the application scenario of the initial generative adversarial network model. The generator in the generative adversarial network model is used to generate a new image based on the input image, and the discriminator is used to distinguish the authenticity of the input image. In this embodiment, the application scenarios of the initial generative adversarial network model may include but are not limited to image enhancement, image style transfer, image super-resolution, image denoising, etc. Accordingly, the image generator obtained by iteratively training the initial generative adversarial network model with sample data may include but is not limited to an enhanced image generator, a stylized image generator, a super-resolution image generator, a denoised image generator, etc. Exemplarily, if the application scenario is image enhancement, the sample data includes unenhanced images and enhanced images; or, if the application scenario is image style transfer, the sample data includes stylized images and base images; or, if the application scenario is image super-resolution, the sample data includes high-resolution images and low-resolution images; or, if the application scenario is image denoising, the sample images include noisy images and non-noisy images.

[0037] By obtaining corresponding sample data according to the application scenario of the initial generative adversarial network model, the initial generative adversarial network model is iteratively trained to obtain an image generator that meets the application scenario. The image generator has preset processing functions corresponding to the application scenario. For example, the preset processing functions can be image enhancement, image style transfer, image super-resolution, image denoising, etc.

[0038] The iterative training process of the initial generative adversarial network model can be: inputting an unprocessed image in the sample image (such as an unenhanced image, a low-resolution image, a noisy image, or a basic image without style) into the initial generator, obtaining a predicted image by the initial generator, inputting the predicted image or a standard image (such as an enhanced image, a high-resolution image, a noisy image, or a stylized image) into the initial discriminator, obtaining a result of the initial discriminator distinguishing the authenticity of the input image, generating a loss function based on the authenticity distinction result and the type of the input image, and adjusting the network parameters of the initial generative adversarial network model based on the loss function, wherein the network parameters include the network parameters in the initial generator and the network parameters in the initial discriminator, and iteratively performing the above training process to determine the generator that meets the training conditions as the target image generator.

[0039] During the training process, the network parameters of the initial GAN ​​model—that is, the connection weights corresponding to each connection in the initial generator and discriminator—are trained. No adjustments are made to the connections in these models. Due to the high sparsity of the initial generator, and to avoid the problem of low precision in the trained image generator due to the randomness of the connections in the randomly generated initial generator, the connections in the initial generator are optimized during the training process to remove low-importance connections in the generator and ensure the importance of the connections in the trained image generator.

[0040] During the above-mentioned iterative training process, it is determined in real time whether the model update condition is met. If so, the iterative training is interrupted, the connection update of the generator is performed, and after the connection update of the generator is completed, the iterative training of the updated generator is continued. If not, the above-mentioned iterative training process is continued. Among them, the model update condition can be a preset iteration step or a preset iteration time, etc. Exemplarily, the preset iteration step can be 2000 iteration steps. The model update condition can be set according to the training requirements. Exemplarily, the preset iteration step can be determined based on the total training step, which can be 10% of the preset iteration step, etc.

[0041] Among them, updating the connections of the generator can be to determine the importance of the connections in the current generator, determine the connections to be removed and the connections to be retained based on the importance, and update the connections to be removed. Specifically, determine the corresponding number of new connections to replace the connections to be removed to achieve the update of the generator.

[0042] Optionally, the connection relationship between each network layer in the generator of the current iteration is updated, including: determining the connection to be updated based on the connection parameters corresponding to each current connection in the generator of the current iteration, and eliminating the connection to be updated in the generator of the current iteration; for the generator after eliminating the connection to be updated, random sampling is performed in the inactivated connections to determine the new connection, and the connection parameters of the new connection are set as the initial connection parameters to obtain an updated generator.

[0043] In this embodiment, the importance of the connection can be determined based on the connection parameters corresponding to each current connection in the generator of the current iteration, wherein the connection parameters corresponding to each connection include a connection weight, and the size of the connection weight represents the importance of the corresponding connection, wherein the connection weight can be a positive number or a negative number, and the larger the numerical value (absolute value) of the connection weight, the greater the importance.

[0044] Based on the connection parameter determination rule, the connection parameters of each connection are determined to identify connections with lesser importance, i.e., connections to be updated. Alternatively, the connection parameter determination rule may be based on a determination threshold, with connections corresponding to connection parameters that meet the determination threshold being determined as connections to be updated. Alternatively, the connection parameter determination rule may be based on a ranking of the connection parameters, with connections to be updated determined based on the ranking.

[0045] In some embodiments, based on the connection parameters corresponding to each current connection in the generator of the current iteration, the connections to be updated are determined, including: sorting each of the connections based on the absolute value of the connection weight of each connection, and determining the connections in the corresponding sorting range in the sorting as the connections to be updated based on the expansion rate. Specifically, the absolute values ​​of the connection weights of the connections can be sorted by size, and from the end of the sorting forward, the connections that meet the sorting range corresponding to the expansion rate are determined as the connections to be updated. The expansion rate can be the proportion of the connections to be updated, which is a value between 0-100%. For example, the expansion rate can be 50%. Accordingly, the absolute values ​​of the connection weights are sorted by size, and the connections corresponding to the last 50% of the connection weights in the sorting are determined as the connections to be updated.

[0046] In some embodiments, during the overall training process of the image generator, the expansion rate is a fixed value, such as 50% or 60%. In some embodiments, during the overall training process of the image generator, the expansion rate may vary with the number of generator updates. For example, the expansion rate may gradually decrease as the number of generator updates increases.

[0047] The connections to be updated are removed from the current iteration's generator, and new connections are selected from the inactive connections. The number of new connections is the same as the number of connections to be updated, to ensure the sparsity of the generator. The removed connections to be updated are considered inactive, and the newly added connections can overlap with the connections to be updated. The new connections can be randomly sampled from the inactive connections, distributed across the network layers.

[0048] For example, see Figure 2 , Figure 2 This is a schematic diagram of an update process of a generator provided by an embodiment of the present invention. Figure 2 In the example, the sparsity of the generator is greater than that of the discriminator. During iterative training, the connection corresponding to the dotted line in the generator is determined to be the connection to be updated. The connection to be updated is removed, and the new connection is determined to form the updated generator.

[0049] In this embodiment, by continuously adding new connections to the generator, the total number of training parameters in the generator is increased, the problem of unbalanced sparsity between the initial generator and the initial discriminator is solved, and the smooth execution of the training process is ensured.

[0050] Initialize network parameters for newly added connections by setting the connection parameters of the newly added connections to preset parameters. For example, the preset parameters may be 0 or 0.5, etc., without limitation. In some embodiments, each connection may be set with different initialization parameters. When a connection is activated, that is, when it switches from an inactive state to an active state, the initialization parameters corresponding to the newly added connection are called to initialize the newly added connection to speed up the training efficiency of the network parameters.

[0051] An updated GAN model is formed based on the updated generator and the discriminator of the current iteration, and the updated GAN model is iteratively trained until a termination condition is met. The termination condition can be any one of the following: the GAN model has been trained a preset number of times, the GAN model has reached convergence, or the GAN model's training accuracy has reached a preset accuracy threshold.

[0052] The technical solution of this embodiment is to construct an initial generative adversarial network model based on the sparsity of the generator and the sparsity of the discriminator. During the iterative training of the initial generative adversarial network model, the connections in the initial generator are updated and optimized, and the updated generative adversarial network model is iteratively trained until the end condition is met, thereby obtaining the target image generator. By updating the connections with low importance during the training process, the connections are pruned, so that an image generator that takes into account both processing accuracy and sparsity is obtained after the training is completed. At the same time, since the sparsity of the generator remains unchanged, the computational complexity of the training process is reduced, and the training efficiency is improved due to the small number of training parameters.

[0053] Based on the above embodiments, Figure 3 It is a flow chart of another method for training an image generator provided by an embodiment of the present invention. Optionally, the generating of an initial generative adversarial network model based on the sparsity of the generator and the discriminator includes: randomly sampling the connections between the network layers in the generator based on the sparsity of the generator, determining a first connection relationship, and generating an initial generator based on the first connection relationship; randomly sampling the connections between the network layers in the discriminator based on the sparsity of the discriminator, determining a second connection relationship, and generating an initial discriminator based on the second connection relationship, and the initial generator and the initial discriminator constitute an initial generative adversarial network model. Specifically, the method specifically includes:

[0054] S210. Randomly sample the connections between the network layers in the generator based on the sparsity of the generator, determine a first connection relationship, and generate an initial generator based on the first connection relationship.

[0055] S220. Randomly sample the connections between the network layers in the discriminator based on the sparsity of the discriminator, determine a second connection relationship, and generate an initial discriminator based on the second connection relationship, wherein the initial generator and the initial discriminator constitute an initial generative adversarial network model.

[0056] S230. Iteratively train the initial generative adversarial network model based on the sample data, and execute step S240 if the model update condition is met.

[0057] S240 updates the connection relationship of each network layer in the generator of the current iteration, and continues to perform iterative training of step S230 on the updated generator and the discriminator of the current iteration until the end condition is met, and determines the trained generator as the target image generator.

[0058] In this embodiment, the generator and the discriminator may include multiple network layers, and adjacent network layers are interconnected to transmit information. Each network layer includes at least one neuron, and the connection between different network layers is realized through the connection between neurons in each network layer. Different neurons can process the information output by neurons in the previous network layer. For example, it can be convolution processing, pooling processing, etc. There is no limitation on this and it can be determined according to the function of the network layer.

[0059] In some embodiments, the infrastructure of the generator and discriminator can be directly called, and the infrastructure includes multiple network layers, each of which includes multiple neurons. Optionally, the infrastructure of the generator and discriminator of different depths can be called as needed. The infrastructure of the called generator and the infrastructure of the discriminator are randomly sampled to obtain the initial generator and the initial discriminator. It should be noted that steps S210 and S220 can be executed sequentially or synchronously, and there is no limitation on this.

[0060] In some embodiments, the connections between the network layers in the generator are randomly sampled based on the sparsity of the generator to determine a first connection relationship, including: obtaining a first basic structural parameter of the generator, wherein the first basic structural parameter includes the number of network layers of the generator and the number of neurons in each network layer; determining a first connection probability of each network layer in the generator based on the sparsity of the generator and one or more of the first basic structural parameters, and randomly sampling the connections between corresponding network layers based on the first connection probability of each network layer to determine a first connection relationship.

[0061] The first basic structural parameter of the generator may be a structural parameter of a dense model corresponding to the generator, wherein the dense model may be a fully connected model.

[0062] In some embodiments, a model database may be pre-created, which may store infrastructure parameters for multiple generators, wherein the infrastructure parameters for the multiple generators may be infrastructure parameters corresponding to generators of different depths and / or widths. The model database may be stored in the cloud, and a data request may be sent to the cloud-based model database. The data request may include one or more items including the network depth and width of the generator. The infrastructure parameters are received from the cloud-based model database. The model database may be stored locally, and the corresponding infrastructure parameters are called from the model database according to the model construction requirements.

[0063] The model database is an extensible database that receives infrastructure parameters uploaded by users. After matching the uploaded infrastructure parameters with the stored infrastructure parameters, it determines that the uploaded infrastructure parameters are new infrastructure parameters and stores them for the convenience of subsequent calls by users.

[0064] Optionally, the basic structural parameters of the generator include the number of network layers in the generator and the number of neurons in each network layer. The type of each network layer in the basic structural parameters can be pre-set.

[0065] The construction of the initial generator is based on the first connection relationship obtained by randomly sampling the connections based on the first connection probability of the network layer, where the first connection probability of any network layer is the first connection probability between any network layer and the previous network layer. The first connection probability satisfies: Among them, N is the initial parameter of the network parameter in the generator, s is the sparsity of the generator, and n l is the number of neurons in the l-layer network, n l-1 is the number of neurons in the l-1 network layer, and P(l) is the first connection probability between the l-1 network layer and the l-1 network layer.

[0066] In some embodiments, the first connection probabilities between different network layers may be the same. Optionally, determining the first connection probabilities of each network layer in the generator based on the sparsity of the generator and one or more of the first infrastructure parameters includes: determining the sparsity of the generator as the first connection probability of each network layer, illustratively, if the sparsity of the generator is 50%, then the first connection probability between each network layer is 50%, and sampling the connections between each network layer based on a uniform distribution.

[0067] In some embodiments, the first connection probability between different network layers may be different. Since the number of neurons in different network layers in the generator may be different, the initial number of network parameters between different network layers is different. In order to ensure the connectivity of the generator, the connection probability between network layers with fewer network parameters is increased. In order to ensure the overall sparsity of the generator, the connection probability between network layers with more network parameters is reduced. That is, the connection probability is negatively correlated with the initial number of network parameters between network layers.

[0068] Optionally, the first connection probability of each network layer in the generator is determined based on the sparsity of the generator and one or more of the first infrastructure parameters, including: determining the first probability parameter based on the sparsity of the generator and the total number of connections of each network layer; for any network layer, determining the first connection probability of the network layer based on the number of neurons in the network layer, the number of neurons in the previous network layer and the first probability parameter.

[0069] The first probability parameter is a parameter used to calculate the connection probability between each network layer in the generator, and is determined based on the number of network layers in the generator, the number of neurons in each network layer, and the sparsity. Specifically, the initial number of network parameters in the generator is determined based on the number of network layers in the generator and the number of neurons in each network layer, and the first probability parameter is determined based on the target number of network parameters in the generator and the number of neurons in each network layer. Specifically, the first probability parameter can be calculated based on the following formula:

[0070] Where σ is the first probability parameter, N is the initial parameter of the network parameter in the generator, s is the sparsity of the generator, and n l is the number of neurons in the lth layer of the generator, n l-1 is the number of neurons in the l-1th network layer in the generator,

[0071] In some embodiments, for any network layer in the generator, determining the first connection probability of the network layer based on the number of neurons in the network layer, the number of neurons in the previous network layer, and a first probability parameter may include: determining the sum of the number of neurons in the network layer and the number of neurons in the previous network layer, and determining the product of the number of neurons in the network layer and the number of neurons in the previous network layer, and determining the ratio of the product of the sum of the numbers and the first probability parameter to the product of the numbers as the first connection probability of the network layer. Exemplarily, it is determined based on the following formula:

[0072] In some embodiments, for any network layer in the generator, determining the first connection probability of the network layer based on the number of neurons in the network layer, the number of neurons in the previous network layer, and a first probability parameter may include: determining the ratio of the sum of the number of neurons in the network layer and the number of neurons in the previous network layer to the product of the number of neurons in the network layer and the number of neurons in the previous network layer, and determining the first connection probability of the network layer based on the product of the ratio and the first probability parameter. Exemplarily, it is calculated based on the following formula:

[0073]

[0074] On the basis of the above embodiment, in order to avoid the situation where the first connection probability determined based on the above method is greater than 1, optionally, the first connection probability of the network layer is determined based on the number of neurons in the network layer, the number of neurons in the previous network layer and the first probability parameter, including: determining a candidate probability based on the number of neurons in the network layer, the number of neurons in the previous network layer and the first probability parameter; and determining the minimum value of the candidate probability and 1 as the first connection probability.

[0075] Specifically, the candidate probability is determined according to the following formula:

[0076]

[0077] Wherein, σ is the first probability parameter, and n l is the number of neurons in the lth layer of the network, the n l-1 is the number of neurons in the l-1th network layer.

[0078] For the generator, the connection relationship between the corresponding adjacent network layers is determined based on the connection probability of each network layer. For example, the connections between the corresponding adjacent network layers are randomly sampled based on the connection probability to obtain the first connection relationship between the adjacent network layers. The sampled connections are activated, and the unsampled connections are inactivated. Optionally, based on the random sampling of the connections between the network layers in the generator, a connection matrix is ​​formed, in which the sampled connections are set to 1 and the unsampled connections are set to 0 to represent the first connection relationship between the network layers in the generator. The network layers are connected through the connection relationship of the network layers in the generator to form an initial generator.

[0079] The process of creating the initial discriminator is the same as the process of creating the initial generator. Optionally, based on the sparsity of the discriminator, the connections between the network layers in the discriminator are randomly sampled to determine the second connection relationship, including: determining the second connection probability of each network layer in the discriminator based on the sparsity of the discriminator and one or more of the second infrastructure parameters, and randomly sampling the connections between the corresponding network layers based on the second connection probability of each network layer to determine the second connection relationship. The second connection probability satisfies Among them, N is the initial parameter of the network parameter in the discriminator, s is the sparsity of the discriminator, and n l is the number of neurons in the l-layer network layer of the discriminator, n l-1 is the number of neurons in the l-1 network layer in the discriminator, and P(l) is the second connection probability between the l-1 network layer and the l-1 network layer.

[0080] The second connection probabilities between the network layers in the discriminator may be the same or different. Optionally, the second connection probability of each network layer in the discriminator is determined based on the sparsity of the discriminator and one or more of the second infrastructure parameters, including: determining the sparsity of the discriminator as the second connection probability of each network layer. Or, optionally, the second connection probability of each network layer in the discriminator is determined based on the sparsity of the discriminator and one or more of the second infrastructure parameters, including: determining the second probability parameter based on the sparsity of the discriminator and the total number of connections of each network layer; for any network layer, determining the second connection probability of the network layer based on the number of neurons in the network layer, the sparsity of the discriminator of the previous network layer, and the second probability parameter.

[0081] The connections between the network layers in the discriminator are randomly sampled based on the second connection probability to obtain a second connection relationship of the discriminator, and an initial discriminator is constructed based on the second connection relationship.

[0082] On the basis of the above embodiment, after calling the basic structural parameters of the generator or the basic structural parameters of the discriminator, the method further includes visually displaying the called basic structural parameters, wherein the visual display may include graphic display and text display. The graphic display may be formed by forming a model schematic diagram of the generator by each network layer and the neurons of each network layer in the generator according to the basic structural parameters of the generator, and displaying the graphic so that the user can intuitively understand the model structure of the generator to be constructed, and forming a model schematic diagram of the discriminator by each network layer and the neurons of each network layer in the discriminator according to the basic structural parameters of the discriminator, and displaying the graphic so that the user can intuitively understand the model structure of the discriminator to be constructed.

[0083] The text display may be to display the basic structural parameters of the generator and / or the basic structural parameters of the discriminator in text form. Optionally, for the generator or discriminator, the content of the text display includes the number of network layers in the basic structural parameters and the number of neurons in each network layer, as well as associated parameters determined based on the basic structural parameters, wherein the associated parameters include but are not limited to the initial number of network parameters in the model, the actual number of network parameters in the actually constructed model (generator and / or discriminator), the storage space of the actually constructed model, and the training calculation amount of the actually constructed model. The actual number of network parameters in the actually constructed model may be determined based on the initial number of network parameters and the currently selected sparsity, for example, it may be determined by multiplying the initial number of network parameters by the sparsity. The storage space and training computational load of the actual constructed model can be estimated based on the actual number of network parameters of the historical constructed model and the corresponding storage space and training computational load. For example, the actual number of network parameters of the actual constructed model can be matched with the actual number of network parameters of the historical constructed model, and the storage space and training computational load of the successfully matched historical model can be used as the storage space and training computational load of the model to be constructed. For example, a curve of changes in the number of network parameters and storage space, as well as a curve of changes in the number of network parameters and training computational load can be formed based on the actual number of network parameters, storage space, and training computational load of the historical model, and the corresponding storage space and training computational load can be determined in the above curves based on the actual number of network parameters of the actual constructed model. By displaying the text, the user can intuitively know the relevant parameters of the actual constructed model through the information displayed in the text.

[0084] Optionally, a user adjustment instruction for the invoked infrastructure parameters is received to update the infrastructure parameters of the generator and / or discriminator. The adjustment instruction for the infrastructure parameters includes, but is not limited to, instructions for adding or deleting network layers of the generator or discriminator, and instructions for adjusting neuron data in any network layer of the generator or discriminator. Accordingly, the visualization of the infrastructure parameters is updated simultaneously with the update of the infrastructure parameters.

[0085] The technical solution provided in this embodiment uses the obtained sparsity of the generator and discriminator to randomly sample the connections between the network layers in the generator and discriminator to construct an initial generator and an initial discriminator. This ensures that the constructed generative adversarial network model meets the sparsity requirements before training the model, thereby reducing the number of network parameters in the model. This reduces the storage space and computational complexity during the storage and training of the model, further simplifying the training process of the model. At the same time, the connections in the model are determined by randomly sampling the connection probabilities, which improves the generalization of the model.

[0086] Based on the above embodiment, the embodiment of the present invention further provides a preferred example. Obtain the initialization parameters of the generative adversarial network, including the basic structural parameters of the generator, the sparsity s of the generator, and the initialization parameters of the generator. g , the basic structural parameters of the discriminator, the sparsity s of the discriminator d , the time interval Δt of parameter expansion (i.e., model update condition, such as preset step size), expansion rate ρ, where the sparsity s of the discriminator d Less than the sparsity s of the generator g A sparse initial GAN ​​model is constructed based on the initialization parameters. Specifically, a uniformly distributed sampling method is used to sample the connections of each layer, resulting in a sparse model. After initialization, the sparsity of each layer of the sparse model is the same.

[0087] The initial generative adversarial network model is iteratively trained. For example, the initial generative adversarial network model can be trained based on the Adam (Adaptive Momentum Estimation) algorithm. After the initial generative adversarial network model is trained for a period of time and the model update conditions are met, such as when 2000 training steps are met, the connection of the generator is updated, unimportant parameters are pruned from the generator, and then the same number of parameters are redistributed to the current generator. The purpose of this is to optimize the existing sparse connection structure without increasing the number of parameters of the sparse generator. Specifically, the connection parameters with small values ​​in the connection parameters are removed, that is, the connections corresponding to the connection parameters with small values ​​in the connection parameters are eliminated. For example, see formula Specifically, the absolute values ​​of the connection parameters in the current generator are sorted, and the connection parameters to be retained and the connection parameters to be eliminated are determined based on the expansion rate ρ, that is, the connections to be updated are determined. Reserved connection parameters.

[0088] After removing the connections to be updated, the same number of new connections are randomly determined, and the connection parameters of each new connection are determined at the same time, that is, the same number of connection parameters are determined. For example, see formula:

[0089]

[0090] Among them, θ t+1 For the connection parameters in the updated generator, are the inactivated parameters in the current generator, wherein the inactivated parameters include the connection parameters that have been removed.

[0091] The updated generative adversarial network model formed by the updated generator and the discriminator of the current iteration continues to be iteratively trained. When the next model update condition is met, the generator is connected and updated again until the end condition is met to obtain a trained image generator, realizing end-to-end training of a generative adversarial network model with a very sparse generator and a dense discriminator.

[0092] By directly training a sparse generative adversarial network model, the number of model parameters and the amount of training computation are reduced. At the same time, generator parameter expansion can improve the expressiveness of the sparse generator without increasing the number of parameters, thereby balancing the sparsity imbalance between the generator and discriminator in the sparse generative adversarial network model.

[0093] Figure 4 This is a flow chart of an image generation method provided by an embodiment of the present invention. This embodiment is applicable to situations where image generation is performed based on an image generator. The method can be performed by an image generation device provided by an embodiment of the present invention. The image generation device can be implemented by software and / or hardware and can be configured on an electronic computing device. The method specifically includes the following steps:

[0094] S310: Obtain an image to be processed.

[0095] S310. Input the image to be processed into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method of the image generator provided in any of the above embodiments.

[0096] In this embodiment, the corresponding image to be processed can be called based on the processing method (or application scenario) of the image to be processed, where the processing method of the image to be processed includes but is not limited to image stylization migration, image enhancement, image denoising, image super-resolution, image compression, etc.

[0097] Correspondingly, the image generator is an enhanced image generator, and the target image is the enhanced image corresponding to the input image; or, the image generator is a stylized image generator, and the target image is the stylized image corresponding to the input image; or, the image generator is a denoising image generator, and the target image is the denoising image corresponding to the input image; or, the image generator is a super-resolution image generator, and the target image is the super-resolution image corresponding to the input image; or, the image generator is a compressed image generator, and the target image is the compressed image corresponding to the input image.

[0098] The called image generator is trained based on the training method of the image generator provided in any of the above embodiments. The image generator meets the preset sparsity, occupies a small storage space, and requires a small amount of computation in the processing of the data to be processed, thereby reducing the storage space and computing power occupied by the actual application platform.

[0099] Figure 5 : is a schematic diagram of the structure of a training device for an image generator provided by an embodiment of the present invention, the device comprising:

[0100] An initial model generation module 310 is used to generate an initial generative adversarial network model based on the sparsity of the generator and the discriminator;

[0101] The model training module 320 is used to iteratively train the initial generative adversarial network model based on sample data, and during the iterative training process, when the model update conditions are met, the connection relationship between the network layers in the generator of the current iteration is updated, and the updated generator and the discriminator of the current iteration are continued to be iteratively trained until the end conditions are met, and the trained generator is determined as the target image generator.

[0102] Optionally, the model training module 320 includes:

[0103] a connection-to-be-updated determining unit, configured to determine a connection to be updated based on connection parameters corresponding to each current connection in the generator of the current iteration, and remove the connection to be updated from the generator of the current iteration;

[0104] The generator updating unit is used to randomly sample the inactivated connections of the generator after the connection to be updated is eliminated, determine the newly added connection, set the connection parameters of the newly added connection as the initial connection parameters, and obtain the updated generator.

[0105] Optionally, the connection parameters corresponding to the connection include a connection weight;

[0106] The connection to be updated determining unit is used for:

[0107] The determining of the connection to be updated based on the connection parameters corresponding to each current connection in the generator of the current iteration includes:

[0108] The connections are sorted based on the absolute value of the connection weight of each connection, and the connections corresponding to the sorting range in the sorting are determined as connections to be updated based on the expansion rate.

[0109] Optionally, the initial model generation module 310 includes:

[0110] An initial generator generating unit, configured to randomly sample connections between network layers in the generator based on the sparsity of the generator, determine a first connection relationship, and generate an initial generator based on the first connection relationship;

[0111] An initial discriminator generation unit is used to randomly sample the connections between the network layers in the discriminator based on the sparsity of the discriminator, determine a second connection relationship, and generate an initial discriminator based on the second connection relationship. The initial generator and the initial discriminator constitute an initial generative adversarial network model.

[0112] Optionally, the initial generator generates a unit for:

[0113] Acquire first basic structural parameters of the generator, wherein the first basic structural parameters include the number of network layers of the generator and the number of neurons in each network layer;

[0114] Determine a first connection probability of each network layer in the generator based on the sparsity of the generator and one or more of the first infrastructure parameters, and randomly sample connections between corresponding network layers based on the first connection probability of each network layer to determine a first connection relationship;

[0115] The initial discriminator generation unit is used to:

[0116] Obtaining second basic structural parameters of the discriminator, wherein the second basic structural parameters include the number of network layers of the discriminator and the number of neurons in each network layer;

[0117] Based on the sparsity of the discriminator and one or more of the second infrastructure parameters, a second connection probability of each network layer in the discriminator is determined, and based on the second connection probability of each network layer, connections between corresponding network layers are randomly sampled to determine a second connection relationship.

[0118] Optionally, an initial generator generates units for:

[0119] Determine the sparsity of the generator as the first connection probability of each network layer; or,

[0120] Determining a first probability parameter based on the sparsity of the generator and the total number of connections of each network layer;

[0121] For any network layer, determining a first connection probability of the network layer based on the number of neurons in the network layer, the number of neurons in the previous network layer, and a first probability parameter;

[0122] The initial discriminator generation unit is used to:

[0123] Determine the sparsity of the discriminator as the second connection probability of each network layer; or,

[0124] Determining a second probability parameter based on the sparsity of the discriminator and the total number of connections of each network layer;

[0125] For any network layer, a second connection probability of the network layer is determined based on the number of neurons in the network layer, the sparsity of the discriminator of the previous network layer, and a second probability parameter.

[0126] Optionally, the sparsity of the generator is greater than the sparsity of the discriminator.

[0127] The image generator training device provided in an embodiment of the present invention can execute the image generator training method provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the image generator training method.

[0128] Figure 6 : is a schematic structural diagram of an image generating device provided by an embodiment of the present invention, the device comprising:

[0129] The image to be processed acquisition module 410 is used to acquire the image to be processed;

[0130] The target image generation module 420 is used to input the image to be processed into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method of the image generator provided in any of the above embodiments.

[0131] The image generation device provided by the embodiment of the present invention can execute the image generation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the image generation method.

[0132] Figure 7 This is a structural diagram of an electronic device provided in Example 4 of the present invention. Figure 7 A block diagram of an electronic device 12 suitable for implementing embodiments of the present invention is shown. Figure 7 The electronic device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. The device 12 is typically an electronic device that performs image classification functions.

[0133] like Figure 7 As shown, the electronic device 12 is implemented as a general-purpose computing device. Components of the electronic device 12 may include, but are not limited to, one or more processors 16, a storage device 28, and a bus 18 connecting various system components (including the storage device 28 and the processor 16).

[0134] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0135] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0136] The storage device 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a Compact Disc-Read Only Memory (CD-ROM), a Digital Video Disc-Read Only Memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Storage device 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0137] A program 36 having a set (at least one) of program modules 26 may be stored, for example, in a storage device 28. Such program modules 26 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a gateway environment. The program modules 26 generally perform the functions and / or methods of the embodiments described herein.

[0138] The electronic device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, camera, display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). This communication can occur via an input / output (I / O) interface 22. Furthermore, the electronic device 12 can communicate with one or more gateways (e.g., a local area network (LAN), a wide area network (WAN), and / or a public gateway, such as the Internet) via a gateway adapter 20. As shown, the gateway adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) systems, tape drives, and data backup storage systems.

[0139] The processor 16 executes various functional applications and data processing by running the programs stored in the storage device 28, such as implementing the image generator training method or image generation method provided by the above embodiments of the present invention.

[0140] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the training method for an image generator or the image generation method provided by the embodiment of the present invention is implemented.

[0141] Of course, the computer-readable storage medium provided by an embodiment of the present invention stores a computer program which is not limited to the method operation described above, but can also execute the image generator training method or image generation method provided by any embodiment of the present invention.

[0142] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0143] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable source code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0144] Source code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wire, electrical wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0145] The computer source code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The source code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of gateway, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A training method for an image generator, characterized in that: include: Generate an initial generative adversarial network model based on the sparsity of the generator and the discriminator; Iteratively training the initial generative adversarial network model based on the sample data, and during the iterative training process, not adjusting the connection of the initial discriminator in the initial generative adversarial network model; and, if a model update condition is met, updating the connection relationship of each network layer in the generator of the current iteration, continuing to iteratively train the updated generator and the discriminator of the current iteration until an end condition is met, and determining the trained generator as the target image generator; The updating of the connection relationship between each network layer in the generator of the current iteration includes: Determine the connection to be updated based on the connection parameters corresponding to each current connection in the generator of the current iteration, and remove the connection to be updated from the generator of the current iteration; The connections to be updated after being eliminated are treated as inactivated connections. For the generator after the connections to be updated are eliminated, random sampling is performed in the inactivated connections to determine new connections, and the connection parameters of the new connections are set as initial connection parameters to obtain an updated generator, wherein the number of the new connections is the same as the number of the connections to be updated.

2. The method according to claim 1, characterized in that The connection parameters corresponding to the connection include connection weights; The determining of the connection to be updated based on the connection parameters corresponding to each current connection in the generator of the current iteration includes: The connections are sorted based on the absolute value of the connection weight of each connection, and the connections corresponding to the sorting range in the sorting are determined as connections to be updated based on the expansion rate.

3. The method according to claim 1, characterized in that The initial generative adversarial network model is generated based on the sparsity of the generator and the discriminator, including: Randomly sampling connections between network layers in the generator based on the sparsity of the generator, determining a first connection relationship, and generating an initial generator based on the first connection relationship; Based on the sparsity of the discriminator, the connections between the network layers in the discriminator are randomly sampled to determine a second connection relationship, and an initial discriminator is generated based on the second connection relationship. The initial generator and the initial discriminator constitute an initial generative adversarial network model.

4. The method according to claim 3, characterized in that The randomly sampling the connections between the network layers in the generator based on the sparsity of the generator to determine the first connection relationship includes: Acquire first basic structural parameters of the generator, wherein the first basic structural parameters include the number of network layers of the generator and the number of neurons in each network layer; Determine a first connection probability of each network layer in the generator based on the sparsity of the generator and one or more of the first infrastructure parameters, and randomly sample connections between corresponding network layers based on the first connection probability of each network layer to determine a first connection relationship; as well as, The randomly sampling the connections between the network layers in the discriminator based on the sparsity of the discriminator to determine the second connection relationship includes: Obtaining second basic structural parameters of the discriminator, wherein the second basic structural parameters include the number of network layers of the discriminator and the number of neurons in each network layer; Based on the sparsity of the discriminator and one or more of the second infrastructure parameters, a second connection probability of each network layer in the discriminator is determined, and based on the second connection probability of each network layer, connections between corresponding network layers are randomly sampled to determine a second connection relationship.

5. The method according to claim 4, characterized in that Determining a first connection probability of each network layer in the generator based on the sparsity of the generator and one or more of the first infrastructure parameters includes: Determine the sparsity of the generator as the first connection probability of each network layer; or, Determining a first probability parameter based on the sparsity of the generator and the total number of connections of each network layer; For any network layer, determining a first connection probability of the network layer based on the number of neurons in the network layer, the number of neurons in the previous network layer, and a first probability parameter; And, determining a second connection probability of each network layer in the discriminator based on the sparsity of the discriminator and one or more of the second infrastructure parameters includes: Determining the sparsity of the discriminator as the second connection probability of each network layer; or, Determining a second probability parameter based on the sparsity of the discriminator and the total number of connections of each network layer; For any network layer, a second connection probability of the network layer is determined based on the number of neurons in the network layer, the sparsity of the discriminator of the previous network layer, and a second probability parameter.

6. The method according to any one of claims 1 to 5, characterized in that: The sparsity of the generator after training iterations is greater than the sparsity of the discriminator.

7. An image generation method, characterized in that: include: Get the image to be processed; The image to be processed is input into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method of the image generator described in any one of claims 1-6.

8. A training device for an image generator, characterized in that: include: The initial model generation module is used to generate the initial generative adversarial network model based on the sparsity of the generator and the discriminator; A model training module is configured to iteratively train the initial generative adversarial network model based on sample data, and during the iterative training process, not adjust the connection of the initial discriminator in the initial generative adversarial network model; and, if a model update condition is met, update the connection relationship of each network layer in the generator of the current iteration, continue iterative training on the updated generator and the discriminator of the current iteration, until an end condition is met, and determine the trained generator as the target image generator; The model training module includes: a connection-to-be-updated determining unit, configured to determine a connection to be updated based on connection parameters corresponding to each current connection in the generator of the current iteration, and remove the connection to be updated from the generator of the current iteration; A generator update unit is used to treat the removed connections to be updated as inactivated connections. For the generator after the removed connections to be updated, random sampling is performed among the inactivated connections to determine new connections, and the connection parameters of the new connections are set as initial connection parameters to obtain an updated generator, wherein the number of the new connections is the same as the number of the connections to be updated.

9. An image generating device, characterized in that: include: The image to be processed acquisition module is used to acquire the image to be processed; A target image generation module is used to input the image to be processed into a pre-trained image generator to obtain a target image output by the image generator, wherein the image generator is trained based on the training method of the image generator described in any one of claims 1-6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image generator training method according to any one of claims 1 to 6 or the image generation method according to claim 7 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the training method for an image generator as described in any one of claims 1 to 6 or the image generation method as described in claim 7 is implemented.

Images