Generative adversarial network architecture search method and system based on hybrid convolution operation

Abstract

The present invention provides a method and system for searching a generative adversarial network architecture based on hybrid convolution operations. The method first designs a broad search space, then constructs a supernet that integrates all candidate operations, and makes the training process of the supernet completely differentiable through continuous relaxation, so as to use the gradient descent method for efficient training. Finally, the best performing operation at each position is selected to obtain the optimal architecture. Different from the traditional architecture search method, the present invention introduces a novel hybrid convolution operation in the search space, and optimizes the search process by integrating a partial channel attention mechanism. The generative adversarial network architecture search based on hybrid convolution operations of the present invention enhances the network's ability to capture long-distance information and the architecture search speed. Compared with traditional search methods, this method has higher efficiency and better performance, and can find the optimal neural network architecture suitable for the task more quickly.

Description

Technical Field

[0001] The present invention relates to the field of deep neural network technology, and in particular to a method and system for searching a generative adversarial network architecture based on hybrid convolution operations. Background Art

[0002] Generative adversarial networks are a type of deep learning model that consists of a generator and a discriminator. They automatically learn the complex features of data through an adversarial process, which not only improves the quality of generated data, but also provides new ideas and tools for various applications. They have achieved remarkable results in computer vision, image generation, data enhancement, style transfer, and speech synthesis. Traditional generative adversarial network architectures are manually designed by scientific researchers and require constant trial and error and adjustment, which not only requires researchers to have rich expert experience, but also requires a lot of manpower costs. The emergence of neural architecture search technology has brought revolutionary changes to the design of generative adversarial network architectures.

[0003] Neural architecture search is a method for automatically designing neural network architectures. This technology uses search algorithms to automatically discover high-performance neural network architectures in a predefined search space. Through neural architecture search, labor costs and time investment can be greatly reduced, so that even non-experts can design efficient generative adversarial network architectures. Neural architecture search not only improves the design efficiency of generative adversarial network architectures, but also discovers innovative architectures that are difficult to explore with traditional methods, further improving the quality of generated data and the performance of the model. However, although traditional neural architecture search methods can achieve good results in generative adversarial network architecture search, they also have some problems, such as low search efficiency and huge consumption of computing resources.

[0004] In order to further improve the application of neural architecture search in the design of generative adversarial network architectures, many new improvement methods have emerged in recent years, such as introducing evolutionary computing into neural architecture search. Evolutionary computing simulates the biological evolution process and gradually optimizes the network architecture through operations such as selection, crossover and mutation. By combining evolutionary computing with neural architecture search, it is possible to efficiently discover generative adversarial network architectures with superior performance in a wider search space. However, this method still has some shortcomings, such as low efficiency in high-dimensional search space, easy to fall into local optimal solutions, and difficulty in finding the global optimal architecture. In addition, its search space is often composed of pure convolution operations, lacking the exploration of a wider search space, and failing to fully release the model's ability to capture long-distance information. Therefore, it is very necessary to propose a differentiable architecture search method that introduces hybrid convolution operations. This can improve the search efficiency of the architecture and enhance the ability to capture long-distance information. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a generative adversarial network architecture search method and system based on hybrid convolution operations in response to the shortcomings of the prior art.

[0006] The method comprises the following steps:

[0007] Step 1, designing a search space for a generative adversarial network architecture search, defining candidate operations, which include two types: ordinary candidate operations and upsampling candidate operations, and determining the scope of the architecture search;

[0008] Step 2: construct a generative adversarial network supernet containing all candidate architectures based on the search space designed in step 1, and randomly assign an architecture parameter to each candidate operation using a continuous value ranging from 0 to 1 to represent the probability of each candidate operation being selected;

[0009] Step 3: Add a partial channel attention mechanism to the edge where the common candidate operations of the supernet are located to calculate the attention weights of different channels;

[0010] Step 4: According to the feature map output of each candidate operation and the calculated channel attention weight, the important channels and features in the feature map are weighted and adjusted so that the outputs of different operations highlight the important features, thereby dynamically adjusting and optimizing the architecture parameters of the candidate operations;

[0011] Step 5: Use the gradient descent method to train the generated adversarial network supernet, and optimize the supernet architecture parameters and weight parameters according to the gradient information;

[0012] Step 6: Alternately execute steps 4 and 5 until the preset maximum number of cycles is reached, and select the candidate operation with the largest architecture parameter weight in each edge of the supernet to obtain the optimal network architecture.

[0013] In step 1, the search space includes N interconnected and skip-connected units, each unit is regarded as a directed acyclic graph consisting of M nodes, and the unit contains D upsampling candidate operation nodes and E ordinary candidate operation nodes, where , N, M, D, E are all natural numbers;

[0014] A hybrid convolution operation is added to the normal candidate operation; the hybrid convolution operation combines the convolution operation and the self-attention operation at the same time, so that the input feature map is projected through three shared 1×1 convolutions to generate three intermediate feature maps, and then the projected feature maps are used in the convolution path and the self-attention path respectively; in the self-attention path, each intermediate feature map is divided into h groups, each group contains three feature maps from each 1×1 convolution, which are used as query, key and value for multi-head attention calculation; in the convolution path with a convolution kernel size of k, k² feature maps are generated through a lightweight fully connected layer, and the input feature map is processed through translation and aggregation operations, similar to the traditional convolution operation; finally, the outputs from the convolution path and the self-attention path are summed according to the weights to obtain the final output :

[0015] ,

[0016] Where 𝛼 and 𝛽 represent the weights of the mixed convolution operation.

[0017] In step 2, the generative adversarial network supernet has edge, where The edges are skip connections, The edge is where the upsampling candidate operation is located. The strips are edges where common candidate operations are located; the skip connection and upsampling candidate operations have T operation options, and the common candidate operations have Y operation options;

[0018] The supernet includes S subnets, where S is a natural number. , a subnet represents a network with only one operation on all connected edges;

[0019] When the supernet is initialized, the architectural parameters of the candidate operations on each edge will be assigned values ​​using a random distribution. The value indicates the probability of the candidate operation being selected. For example, continuous data randomly distributed in the range of 0 to 1 is used to assign weights to each connection. This random initialization can break the symmetry of the network, allowing each node to learn the characteristics of the input data in a different way at the beginning of training, which helps to prevent the network from falling into a local optimal solution.

[0020] A supernet is a general term for all candidate architectures in a search space, and a subnet represents a specific architecture selected in a given search space, with each subnet being part of a supernet. Traditionally, input information is usually processed by a single operation (such as convolution) to generate a new feature map. However, in a supernet, the input information is processed by multiple operations, such as dynamic separable convolution, dilated convolution, and skip connections. The feature maps generated by each of these operations are fused by adding the elements bit by bit, and finally a new hybrid feature map is obtained. During the training process, through learning and optimization, the supernet can automatically select and adjust the weights of these candidate operations, thereby ultimately determining the optimal combination of operations and network structure.

[0021] In the present invention, there are T candidate operations on the upsampling and jump connection edges of the supernet, including nearest neighbor interpolation upsampling, bilinear interpolation upsampling, etc., and there are Y candidate operations in the ordinary candidate operations, including 1×1 convolution, 1×1 mixed convolution, 3×3 convolution, 3×3 separable convolution, 3×3 mixed convolution, 5×5 mixed convolution, etc.

[0022] The supernet described in the present invention is a network structure that includes multiple subnets, but the network itself is discrete and discontinuous, which directly hinders the use of gradient-based optimization methods for search. In order to overcome this problem, the present invention adopts a continuous relaxation strategy to convert the originally discrete candidate operations into a continuous and differentiable representation so that it can be searched using optimization algorithms such as gradient descent.

[0023] The specific continuous relaxation operation is as follows: For each node pair (i, j) in the generator supernet, all defined candidate operations are uniformly represented as a set O n , the range of n is determined by the number of candidate operations, and a set of n Corresponding architecture parameters , o represents an operation in the candidate operation set, The architecture parameters The size of represents the relative possibility of selecting operation o in the connection from node i to node j. The larger the value, the higher the probability that the operation is selected in the connection. Specifically, the architecture parameters are initialized as trainable continuous variables, and the architecture parameters are converted into probability distributions by applying the softmax function, where the probability of each candidate operation being selected is proportional to the architecture parameter value corresponding to the candidate operation. Specifically, for the output from node i to node j , the calculation formula is:

[0024] ,

[0025] in, Indicates that the architecture parameters Converted to probability distribution through softmax function, Indicates applying operation o to input As a result, is a temporary parameter, indicating the index of a specific operation, exp is an exponential function, , e represents a natural constant. Indicates that in the connection from node i to node j, a specific operation is selected relative likelihood.

[0026] Through such operations, the original discrete candidate operation selection problem is transformed into a continuous optimization problem, and then the optimal generative adversarial network architecture can be searched by optimizing the architecture parameters. By iteratively updating the architecture parameters and calculating the gradient of the loss function, the weight of each operation can be gradually adjusted so that the performance of the entire generator supernet on the training set is optimal. Finally, the optimal operation between each node pair is selected according to the value of the architecture parameters, and the optimal architecture searched can be obtained.

[0027] In step 3, after adding the partial channel attention mechanism to each ordinary convolution operation node, the search efficiency is improved while reducing the consumption of computing resources. Specifically, when calculating the feature map, the input feature map is first passed through an average pooling layer and a maximum pooling layer respectively to obtain two different feature maps and realize the compression of the number of channels and the extraction of important features; then the obtained feature map is passed through two fully connected layers FC 1 With FC 2 To further perform linear changes on the pooled features, the first fully connected layer FC 1 Reduce the number of channels to the original number of channels , and introduce nonlinearity through the ReLU activation function, and then through the second fully connected layer FC 2 The number of channels is restored to its original size; finally, the outputs from average pooling and maximum pooling are added, and the obtained output is normalized so that the sum of the weights of all channels is 1, and the attention weights of different channels are obtained.

[0028] In step 4, the attention weights of each channel in the partial channel attention mechanism are first summed to obtain a total vector representing the weight of each channel, where each element of the total vector represents the importance of the corresponding channel; then the weighted channels with the highest weights are selected from the total vector. The channels of participate in the weighted operation calculation, where q represents the proportional coefficient selected by the partial channel attention mechanism, and finally the weighted operation is calculated. The corresponding positions of the channel and the original feature map are updated to form a new output feature map. The input feature map is set to X, and the attention weight of the partial channel attention mechanism is set to A. represents the specific weight of the i-th channel, the number of channels is C, and the architectural parameter weight of each candidate operation is W. represents the architectural parameter weight of the jth operation, the operation list is O, represents the jth operation, is the calculation of the partial channel attention mechanism, topq() is the channel selection operation, , , , are all intermediate variables, then step 4 specifically includes the following steps:

[0029] Step 4.1, calculate the attention weights of the partial channel attention mechanism:

[0030] ;

[0031] Step 4.2, weighted input feature map:

[0032] ;

[0033] Step 4.3, calculate the total weight vector :

[0034] ;

[0035] Step 4.4, select the first channel according to the weight of each channel. The number of channels is C, so the parameters in the channel selection are :

[0036] ;

[0037] Step 4.5, extract important channel data, the indices parameter specifies the channel index to be extracted:

[0038] ;

[0039] The first symbol Indicates the selection of all samples, symbol Indicates that all features in the channel specified by indices are retained during data extraction to ensure complete feature information;

[0040] Step 4.6, weighting operation on important channels:

[0041] ;

[0042] Step 4.7, concatenate the calculated important channels with the original channels. The indices parameter specifies the channel index to be updated:

[0043] ,

[0044] The first symbol Indicates the selection of all samples, symbol Indicates that the previous value specified by indices is retained when the data is updated. All features in the channel to ensure that the updated channel is complete.

[0045] Through the above steps, the partial channel attention mechanism effectively reduces the consumption of computing resources and enhances the model's response to key information.

[0046] Step 5 includes:

[0047] Step 5.1, initialize the training parameters, which include the learning rates of the generator and the discriminator and network parameter weights , optimizer parameters, mixed convolution operation weights and a fixed noise vector;

[0048] Step 5.2: Conduct adversarial training on the generator and discriminator, and calculate the loss through the discriminator based on the images generated by the generator and the real images;

[0049] Step 5.3, back-propagate using the discriminator and generator losses, update the discriminator and generator weights, and optimize the architecture parameters of the candidate operation;

[0050] Step 5.4, adjust the weights of the mixed convolution operation based on the images generated during training and the feedback from the discriminator .

[0051] Step 5.1 includes:

[0052] Step 5.1.1, set the learning rates of the generator and discriminator to be and , and create an optimizer based on the preset parameters to ensure that the network parameters can be adjusted during the optimization process.

[0053] In step 5.1.2, initialize the weights α and β of the mixed convolution operation to 0.5 as the importance weights for controlling the output of each path of the mixed convolution to achieve flexible adjustment of network feature extraction.

[0054] Step 5.1.3, generate a fixed noise vector z , which is used to evaluate the performance of the generator during the training process and ensure the consistency of the evaluation results in each training iteration.

[0055] Step 5.2 includes:

[0056] Step 5.2.1: The generator randomly samples a noise vector z from the noise distribution. This process usually uses a uniform distribution or a normal distribution to ensure that the generator can explore a diverse potential space. The generator is then used to generate fake images. , so as to compare and distinguish with the real image later. The generative adversarial network consists of a generator and a discriminator. The generator is used to generate images and is a component of the generative adversarial network;

[0057] Step 5.2.2, the discriminator receives the real image x and the fake image , and output is calculated by forward propagation and , and They represent the discrimination results of real images and fake images respectively, and their values ​​are both [0,1]. The higher the value, the higher the possibility that the discriminator believes the input content is true. The goal of the discriminator is to identify real images and fake images as much as possible. It hopes to judge real images as real, that is, to improve The value of is close to 1, and the false image is judged as false, that is, the The value of is close to 0. The goal of the generator is to make the fake images it generates as much as possible to deceive the discriminator, that is, to improve The value of is close to 1. In this way, the generator and the discriminator are adversarially trained under these two conflicting objectives.

[0058] Step 5.2.3, calculate the discriminator loss function, where the discriminator loss function uses the logarithmic loss function , the formula is:

[0059] ,

[0060] Loss Function The value range is [0, +∞), where E represents the expectation, Indicates that the real image x follows the real data distribution , Indicates that random noise z follows the noise distribution ; The goal of the discriminator is to minimize the loss function The value of is to maximize the recognition rate of real data and the rejection rate of generated data;

[0061] Step 5.2.4, calculate the generator loss function, where the generator loss function uses the logarithmic loss function , the formula is:

[0062] ,

[0063] The loss function The value range is [0, +∞), and the goal of the generator is to minimize the loss function The value of As close to 1 as possible, that is, maximizing the probability that the generated image is considered a real image by the discriminator.

[0064] Step 5.3 includes:

[0065] Step 5.3.1, fix the network parameter weights of the generator , keeping the generator unchanged, according to the loss function of the discriminator , using the back-propagation method to calculate the gradient , and the discriminator learning rate is calculated by gradient descent Update the weight parameters of the discriminator , the calculation formula is:

[0066] ,

[0067] Then complete a weight update and update the frequency according to the discriminator Repeat the above steps for the settings Second-rate;

[0068] Step 5.3.2, fix the network parameter weights of the discriminator , keeping the discriminator unchanged, according to the loss function of the generator Calculate gradients using backpropagation , and use the gradient descent method according to the generator learning rate Update the weights of the generator , the calculation formula is:

[0069] ,

[0070] Then complete a weight update;

[0071] Step 5.3.3, optimize the candidate operation architecture parameter weight W, and the calculation formula for the output Output of each candidate operation is:

[0072] Output ,

[0073] Where X is the input feature map, tol is the total number of candidate operations, is the architectural parameter weight of the ith operation, is the output of the i-th operation;

[0074] Use the chain rule to calculate the loss function L for each candidate operation weight Gradient , update the candidate operation architecture parameter weights using gradient descent:

[0075] ,

[0076] Finally, the optimization of candidate operation architecture parameters is completed;

[0077] Step 5.3.4, according to the optimizer settings, use the adaptive moment estimation algorithm to dynamically adjust the learning rate of the discriminator and generator:

[0078] The parameters are updated by calculating the moving average of the first-order moment and the second-order moment. The update formula is:

[0079] ,

[0080] in Represents the weight parameters that need to be updated at the current time step, as shown above , the t in the subscript represents the current time step or number of iterations, and lr includes The default learning rate is is the exponentially weighted moving average of the current gradient, is the exponentially weighted moving average of the current squared gradient, It is a tiny constant to prevent division by zero errors. In this way, it ensures that the step size of weight update is dynamically optimized while maintaining the respective learning rates, and indirectly adjusts the learning rates, thereby improving the training effect and convergence speed of the generator and discriminator;

[0081] Step 5.3.5, monitor and record the perceptual similarity index IS (Inception Score) and Fréchet Inception Distance FID (Frechet Inception Distance) during the training process. IS and FID are commonly used performance indicators for evaluating generative adversarial networks. IS is used to evaluate the diversity and inter-class separation of generated images, measure the quality and diversity of generated images, and FID measures the similarity between generated images and real images in the feature space.

[0082] In step 6, the gradient descent method is used to continuously optimize and adjust the network weight parameters and architecture parameters, so that the supernet gradually approaches the optimal performance; finally, according to the distribution of the architecture parameter weights, the candidate operation with the largest weight is selected to construct the optimal network architecture.

[0083] At each iteration, the importance of different operations is dynamically adjusted based on the feature map output and corresponding attention weight of each candidate operation to ensure that the training process of the generative adversarial network supernet is more efficient. At the same time, the candidate operation with the largest weight in each edge of the supernet is selected to form the optimal network architecture. This process ensures that while optimizing the architecture, some attention mechanisms are fully utilized to improve the model's expressiveness and generation effect, thereby achieving better performance with lower computational complexity.

[0084] The present invention also provides a generative adversarial network architecture search system based on hybrid convolution operation, comprising:

[0085] A construction module, used to construct a generative adversarial network supernet, and assign an importance weight to each operation in the supernet using a random continuous numerical value ranging from 0 to 1, wherein the weight is used to represent the probability of each operation being selected as the optimal operation;

[0086] The hybrid convolution module is used to integrate the ordinary convolution operation with the self-attention mechanism to obtain a new operation that combines the advantages of both.

[0087] A training module for training the network weights of the supernet using training data and its labels;

[0088] The optimization module is used to optimize the parameters of the supernet architecture using the gradient descent method.

[0089] By introducing hybrid convolution operations, the present invention significantly expands the search space of traditional pure convolutional neural networks, enhances the model's ability to capture long-distance information, and improves the generation effect of generative adversarial networks. In addition, combined with the partial channel attention mechanism, the Matthew effect caused by the randomness of architecture parameter initialization during the architecture search process is effectively alleviated. This effect is manifested in that operations that initially obtain high random values ​​will continue to strengthen in subsequent processes, while operations with low random values ​​may be ignored in the architecture search, thereby missing out on potential excellent architectures and causing instability in the architecture search. This method achieves better results than existing methods.

[0090] The beneficial effect of the present invention is that the method of the present invention proposes a generative adversarial network architecture search algorithm based on hybrid convolution for neural architecture search for generative adversarial networks. The algorithm effectively expands the traditional convolutional neural network search space and improves the model's ability to capture long-distance information by introducing hybrid convolution that combines convolution and self-attention mechanisms into the search space. Before searching, a continuous relaxation method is used to make the originally discrete search space continuous so that gradient descent can be used directly for searching, which greatly improves the search efficiency. During the search process, a partial channel attention mechanism is introduced into ordinary candidate operations. By assigning attention weights to different candidate operations and selecting channels with high attention weight rankings to participate in subsequent calculations, the consumption of computing resources is reduced and the search efficiency is improved. This generative adversarial network architecture search algorithm based on hybrid convolution has brought new ideas and breakthroughs to the field of architecture search, and has sufficient practical value and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 This is the overall framework diagram of the search space in the present invention.

[0092] Figure 2 Schematic diagram of the hybrid convolution operation in the present invention.

[0093] Figure 3 Schematic diagram of the partial channel attention mechanism in the present invention.

[0094] Figure 4 IS score trend chart for architecture optimization in the present invention.

[0095] Figure 5 The FID score trend chart is optimized for the architecture in the present invention. DETAILED DESCRIPTION

[0096] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0097] like Figure 1 As shown, this embodiment provides a method for searching a generative adversarial network architecture based on a hybrid convolution operation, comprising the following steps:

[0098] Step 1: First design Figure 1 The search space of the generator architecture of the generative adversarial network shown in the figure consists of 3 interconnected and skip-connected units. The unit can be viewed as a directed acyclic graph with 5 interconnected nodes. The gray arrows represent upsampling operations, and the black arrows represent common operations. The upsampling candidate operations are set to {transposed convolution, nearest neighbor interpolation upsampling, bilinear interpolation upsampling}, and the common candidate operations are set to {1×1 convolution, 3×3 convolution, 5×5 convolution, 3×3 separable convolution, 5×5 separable convolution, 7×7 separable convolution, 1×1 mixed convolution, 3×3 mixed convolution, 5×5 mixed convolution}.

[0099] Among them, the mixed convolution operation is as follows Figure 2 As shown in the figure, the input feature map with length, width and number of channels H, R and C is first projected through three 1×1 convolutions to obtain three intermediate feature maps after projection, and then these three feature maps are divided into h groups, where h represents the number of attention heads. In this way, 3h intermediate feature maps are obtained, and then these projected feature maps are used in the convolution path and the self-attention path respectively. Figure 2 The upper part of shows the calculation process of the convolution path, where k represents the convolution kernel size in the convolution path. In this invention, it is 1, 3, and 5, respectively, corresponding to the hybrid convolution operation described above. In the convolution path, a lightweight fully connected layer is used to generate The feature map is processed by translating and aggregating the generated features in a convolution-like manner, and information is collected from the local receptive field like traditional convolution, and finally the feature map is output. . Figure 2 The lower part of the figure shows the operation process of the self-attention mechanism, which divides the intermediate features into h groups, each of which contains three features, one from each 1×1 convolution, and the corresponding three feature maps are used as query, key and value respectively, following the traditional multi-head self-attention calculation output Finally, the final output feature map y is calculated through two trainable parameters 𝛼 and 𝛽:

[0100] Step 2: According to Figure 1 The search space shown builds the generator supernet and randomly initializes the supernet, giving each candidate operation a random architecture parameter between 0 and 1 , and ensure that the sum of the architecture parameter values ​​of each node is 1. Randomly generate input noise with 128 channels to input into the supernet to generate images. The input direction of the supernet feature map is unidirectional from front to back.

[0101] Step 3: If Figure 3 As shown in the figure, a partial channel attention mechanism is added after the node where the common candidate operation is located, so that the input feature map passes through an average pooling layer and a maximum pooling layer respectively, and two different feature maps are obtained, and the number of channels is compressed and important features are extracted. Then the obtained feature map is passed through two fully connected layers with RELU linear units in the middle to further linearly change the pooled features, and finally the attention score is output after normalization.

[0102] Step 4: First, sum the architectural parameter weights of each channel in the attention mechanism to obtain a vector representing the total weight of each channel. Each element of this vector represents the total importance of the corresponding channel. Then select the top The channels participate in the weighted operation calculation, and finally the calculated part is The remaining channels and original feature maps The channels are concatenated to form a new output feature map. Set the input feature map to X, the channel attention weight to A, the number of channels to 128, the weight of each operation to W, and the operation list to O. Specifically, it can be expressed as:

[0103] Step 4.1, calculate the channel weight, ;

[0104] Step 4.2, weighted input feature map, ;

[0105] Step 4.3, calculate the total weight vector, ;

[0106] Step 4.4, select the most important channels, ;

[0107] Step 4.5, extract important channel data, ;

[0108] Step 4.6, weighting operation is performed on important channels. ;

[0109] Step 4.7, concatenate the calculated important channels with the original channels. .

[0110] Step 5: Based on the set supernet, search for the architecture of the generative adversarial network generator is carried out, which includes the following steps:

[0111] Step 5.1, initialize the training parameters, including the learning rate and network parameters of the generator and discriminator, optimizer parameters, and mixed convolution operation weights and a fixed noise vector:

[0112] Step 5.1.1: Set the learning rate of the generator and discriminator to 0.0002, and adjust the learning rate according to the parameters Create an adaptive moment estimation algorithm optimizer to ensure that network parameters can be adjusted during the optimization process.

[0113] Step 5.1.2: Initialize the weights α and β of the mixed convolution operation to 0.5 as the importance weights for controlling the output of each path of the mixed convolution to achieve flexible adjustment of network feature extraction.

[0114] Step 5.1.3: Generate a fixed noise vector of dimension 128 z , which is used to evaluate the performance of the generator during the training process and ensure the consistency of the evaluation results in each training iteration.

[0115] Step 5.2: Perform adversarial training on the generator and discriminator, and calculate the loss through the discriminator based on the images generated by the generator and the real images:

[0116] Step 5.2.1: The generator randomly samples a 128-dimensional noise vector z from the noise distribution. This process usually adopts uniform distribution or normal distribution to ensure that the generator can explore a diverse potential space. The generator network then maps the noise vector z through a multi-layer neural network to generate a fake image. , so as to be compared and identified with the real image later.

[0117] Step 5.2.2: The discriminator receives the real image x and the generated fake image , and output is calculated by forward propagation and .in, represents the discriminator's discrimination result on the real image, and It represents the discriminant result of the generated image. The goal of the discriminator is to distinguish the real image from the generated image as accurately as possible to enhance the model's understanding of the real data distribution.

[0118] Step 5.2.3: Calculate the loss function of the discriminator , which is a logarithmic loss function designed to measure the classification ability of the discriminator in distinguishing real images from fake images. The output of is logarithmic, and the discriminator is penalized to encourage it to better identify real data. Taking the logarithm of the output of , the discriminator is encouraged to enhance its recognition ability of generated images. The optimization of this loss function aims to improve the accuracy of the discriminator and ensure that it can effectively distinguish between real and generated samples during the training process.

[0119] Step 5.2.4: Calculate the loss function of the generator, using the logarithmic loss function , which aims to reflect the ability of fake images generated by the generator to deceive the discriminator. When the fake images generated by the generator can successfully make the discriminator think that they are from the real data distribution, the generator's loss will decrease. By optimizing this loss, the generator continuously adjusts its generation strategy to enhance the authenticity of the generated images, making them more visually recognizable, thereby improving the generation effect. In this process, the goal of the generator is to minimize its loss so that Try to keep it as close to 1 as possible to improve the quality of the generated image.

[0120] Step 5.3: Use the discriminator and generator losses for backpropagation, update the discriminator and generator weights, and optimize the network architecture parameters:

[0121] Step 5.3.1: Update the discriminator weight parameters, fix the generator parameters, keep the generator unchanged, calculate the gradient according to the discriminator loss function, use the back-propagation algorithm, calculate the gradient of the loss function to the discriminator weight by the chain rule, update the discriminator weight using the gradient descent method, multiply the calculated gradient value by the learning rate, and subtract it from the current weight, and repeat the above operation The weights are updated to ensure that the discriminative ability of the discriminator is fully trained and maintain a balance between the generator and the discriminator.

[0122] Step 5.3.2: Update the generator weight parameters, fix the discriminator parameters, keep the discriminator unchanged, calculate the gradient according to the generator loss function, use the back-propagation algorithm, calculate the gradient of the loss function to the generator weight by the chain rule, and use the gradient descent method to update the generator weight. Multiply the calculated gradient value by the learning rate and subtract it from the current weight to complete one weight update.

[0123] Step 5.3.3, optimize the network architecture parameters, and select the best operation by adjusting the weights of different candidate operations in the supernet. In this stage, the weights of each candidate operation in the supernet are dynamically adjusted according to the current training progress and network performance indicators. By evaluating the impact of different operations on the performance of the generator and discriminator, the best performing operation is selected, thereby further improving the model's generation and discrimination capabilities.

[0124] In step 5.3.4, according to the optimizer settings, the learning rates of the discriminator and generator are dynamically adjusted. According to the current training stage and performance, the learning rate is gradually adjusted to adapt to the convergence speed of the model.

[0125] Step 5.3.5, monitor and record the performance indicators and training parameters during the training process.

[0126] Step 5.4: Adjust the weights of the hybrid convolution operation based on the images generated during training and the feedback from the discriminator .

[0127] Step 6: The present invention adopts a differentiable search method, optimizes the supernet by gradient, and stops searching after 100 iterations. At this time, according to the architecture parameters of the candidate operation in each node Size, select the operation with the largest weight and use it as the final architecture. Specifically expressed as:

[0128] {Normal operations: [[1×1 convolution, 3×3 mixed convolution, 3×3 mixed convolution], [5×5 mixed convolution, 3×3 separable convolution, 3×3 separable convolution], [7×7 separable convolution, 3×3 separable convolution, 3×3 mixed convolution]], upsampling operations: [[transposed convolution, transposed convolution], [transposed convolution, transposed convolution], [nearest neighbor interpolation upsampling, nearest neighbor interpolation upsampling]], skip connections of length 1: [transposed convolution, bilinear interpolation upsampling], skip connections of length 2: [transposed convolution]}. 1×1, 3×3, etc. represent the size of the convolution kernel, and the content in each square bracket represents the operation combination of a unit. The units are arranged in numerical order, and the operations within the unit are arranged in order of node numbers. The length of a skip connection refers to the span of the connection: a skip connection of length 1 includes connections from unit 1 to unit 2 and from unit 2 to unit 3, and a skip connection of length 2 includes connections from unit 1 to unit 3.

[0129] This embodiment adds a hybrid convolution operation that integrates self-attention mechanism and convolution on the basis of traditional convolutional neural network, effectively expands the search space of the generative adversarial network architecture, and improves the network's ability to capture long-distance information. In addition, the partial channel attention mechanism introduced in this embodiment effectively reduces the consumption of computing resources and improves the ability of the supernet to select key operations. This embodiment conducts experiments and verifies the performance on two public data, CIFAR-10 and STL-10. CIFAR-10 contains 60,000 pictures of 10 different categories, of which 50,000 are used for network training and 10,000 are used for network testing. The size of the pictures is 32×32. STL-10 contains 13,000 pictures of 10 different categories, with a picture size of 96×96, which is larger than STL-10 and provides a higher resolution. In order to balance the performance in this embodiment, its size is set to 48×48. This embodiment only takes 2 hours on GPU4090 to search for an excellent network architecture, which is much faster than other non-gradient algorithms, and the search speed has been greatly improved. On this basis, the discovered architecture achieves better generation effects and can generate rich and realistic images. The performance of the network architecture searched by the present invention when optimized in the standard dataset CIFAR-10 is as follows: Figure 4 and Figure 5 As shown:

[0130] The Inception Score (IS) and Frechet Inception Distance (FID) in the figure are commonly used performance indicators in generative adversarial networks. Figure 4The IS score in is used to evaluate the diversity and inter-class separation of generated images. The size of the score reflects the quality and diversity of the generated images. Therefore, the higher the score, the better. A higher IS score indicates that the generated images are not only visually diverse, but also more attractive. At the same time, Figure 5 The FID score in measures the similarity between the generated image and the real image in the feature space, so the lower the score, the better. A lower FID score indicates that the feature distribution of the generated image is closer to that of the real image, indicating that the generated sample is more realistic at the feature level. The method of this embodiment can discover architectures with excellent performance in a very short time, and these architectures can quickly converge to the best performance when retraining.

[0131] The present invention provides a method and system for searching a generative adversarial network architecture based on a hybrid convolution operation. There are many methods and approaches to implement the technical solution. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A generative adversarial network architecture search method based on hybrid convolution operation, characterized in that: The following steps are involved: Step 1, designing a search space for a generative adversarial network architecture search, defining candidate operations, which include two types: ordinary candidate operations and upsampling candidate operations, and determining the scope of the architecture search; Step 2: construct a generative adversarial network supernet containing all candidate architectures based on the search space designed in step 1, and randomly assign an architecture parameter to each candidate operation using a continuous value ranging from 0 to 1 to represent the probability of each candidate operation being selected; Step 3: Add a partial channel attention mechanism to the edge where the common candidate operations of the supernet are located to calculate the attention weights of different channels; Step 4: According to the feature map output of each candidate operation and the calculated channel attention weight, the important channels and features in the feature map are weighted and adjusted so that the outputs of different operations highlight the important features, thereby dynamically adjusting and optimizing the architecture parameters of the candidate operations; Step 5: Use the gradient descent method to train the generated adversarial network supernet, and optimize the supernet architecture parameters and weight parameters according to the gradient information; Step 6, alternately execute step 4 and step 5 until the preset maximum number of cycles is reached, select the candidate operation with the largest architecture parameter weight in each edge of the supernet, and obtain the optimal network architecture; the optimal network architecture is used for speech synthesis.

2. The method according to claim 1, characterized in that In step 1, the search space includes N interconnected and skip-connected units, each unit is regarded as a directed acyclic graph consisting of M nodes, and the unit contains D upsampling candidate operation nodes and E ordinary candidate operation nodes, where M=D+E, and N, M, D, and E are all natural numbers; A hybrid convolution operation is added to the normal candidate operation; the hybrid convolution operation combines the convolution operation with the self-attention operation, so that the input feature map is projected through three shared 1×1 convolutions to generate three intermediate feature maps, and then the projected feature maps are used in the convolution path and the self-attention path respectively; in the self-attention path, each intermediate feature map is divided into h groups, each group contains three feature maps from each 1×1 convolution, which are used as query, key and value for multi-head attention calculation; in the convolution path with a convolution kernel size of k, k2 feature maps are generated through a lightweight fully connected layer, and the input feature map is processed through translation and aggregation operations; finally, the outputs from the convolution path and the self-attention path are summed according to the weights to obtain the final output y: y=αx conv +βx att , Where α and β represent the weights of the mixed convolution operation.

3. The method according to claim 2, characterized in that In step 2, the generative adversarial network supernet has edge, where The edges are skip connections, N×D edges are upsampling candidate operation edges, and N×E edges are common candidate operation edges. The skip connection and upsampling candidate operation have T operation options, and the common candidate operation has Y operation options. The supernet includes S subnets, where S is a natural number. A subnet represents a network with only one operation on all connected edges; When the supernet is initialized, the architectural parameters of the candidate operations on each edge will be assigned values ​​using a random distribution, and the value indicates the probability of the candidate operation being selected.

4. The method according to claim 3, characterized in that In step 3, after adding the partial channel attention mechanism to each ordinary convolution operation node, the search efficiency is improved while reducing the consumption of computing resources. Specifically, when calculating the feature map, the input feature map is first passed through an average pooling layer and a maximum pooling layer respectively to obtain two different feature maps and realize the compression of the number of channels and the extraction of important features; then the obtained feature map is passed through two fully connected layers FC1 and FC2 to further linearly change the pooled features, where the first fully connected layer FC1 reduces the number of channels to 1 / 2 of the original number of channels. Nonlinearity is introduced through the ReLU activation function, and the number of channels is restored to its original size through the second fully connected layer FC2; finally, the outputs from average pooling and maximum pooling are added, and the obtained output is normalized so that the sum of the weights of all channels is 1, and the attention weights of different channels are obtained.

5. The method according to claim 4, characterized in that In step 4, the attention weights of each channel in the partial channel attention mechanism are first summed to obtain a total vector representing the weight of each channel, where each element of the total vector represents the importance of the corresponding channel; then the weighted channels with the highest weights are selected from the total vector. The channels of participate in the weighted operation calculation, where q represents the proportional coefficient selected by the partial channel attention mechanism, and finally the weighted operation is calculated. The corresponding positions of the channel and the original feature map are updated to form a new output feature map. The input feature map is set to X, and the attention weight of the partial channel attention mechanism is A, A i represents the specific weight of the i-th channel, the number of channels is C, and the architectural parameter weight of each candidate operation is W, W j represents the architectural parameter weight of the jth operation, and the operation list is O, O j represents the jth operation, ChannelAttention(X) is the calculation of the partial channel attention mechanism, topq() is the channel selection operation, X′, indices, X temp , X′ temp are all intermediate variables, then step 4 specifically includes the following steps: Step 4.1, calculate the attention weights of the partial channel attention mechanism: A = ChannelAttention(X); Step 4.2, weighted input feature map: X′=X×A; Step 4.3, calculate the total weight vector Sa: Step 4.4, select the first channel according to the weight of each channel. The number of channels is C, so the parameters in the channel selection are Step 4.5, extract important channel data, the indices parameter specifies the channel index to be extracted: X temp =X′[:,indices,:.:]; The first symbol: indicates that all samples are selected, and the symbol:.: indicates that all features in the channel specified by indices are retained during data extraction; Step 4.6, weighting operation on important channels: Step 4.7, concatenate the calculated important channels with the original channels. The indices parameter specifies the channel index to be updated: X[:,indices,:.:]=X t ′ emp , The first symbol : indicates that all samples are selected, and the symbol :.: indicates that the previous samples specified by indices are retained when the data is updated. All features in the channel.

6. The method according to claim 5, characterized in that Step 5 includes: Step 5.1, initialize the training parameters, which include the learning rate lr of the generator and the discriminator G , lr D and network parameter weight w G , w D , optimizer parameters, mixed convolution operation weights α, β and fixed noise vector; Step 5.2: Conduct adversarial training on the generator and discriminator, and calculate the loss through the discriminator based on the images generated by the generator and the real images; Step 5.3, back-propagate using the discriminator and generator losses, update the discriminator and generator weights, and optimize the architecture parameters of the candidate operation; In step 5.4, the weights α and β of the mixed convolution operation are adjusted according to the images generated during training and the feedback from the discriminator.

7. The method according to claim 6, characterized in that Step 5.2 includes: Step 5.2.1, the generator randomly samples a noise vector z from the noise distribution and generates a fake image G(z) through the generator; the generative adversarial network consists of a generator and a discriminator. The generator is used to generate images and is a component of the generative adversarial network; Step 5.2.2, the discriminator receives the real image x and the fake image G(z), and calculates the output D(x) and D(G(z)) through forward propagation. D(x) and D(G(z)) represent the discrimination results of the real image and the fake image, respectively, and both have values ​​of [0,1]; Step 5.2.3, calculate the discriminator loss function, where the discriminator loss function uses the logarithmic loss function L D , the formula is: Loss function L D The value range is [0, +∞), where E represents the expectation, x~p data (x) indicates that the real image x follows the real data distribution p data (x), z~p z (z) indicates that the random noise z follows the noise distribution p z (z); The goal of the discriminator is to minimize the loss function L D The value of is to maximize the recognition rate of real data and the rejection rate of generated data; Step 5.2.4, calculate the generator loss function, where the generator loss function uses the logarithmic loss function L G , the formula is: The loss function L G The value range is [0, +∞), and the goal of the generator is to minimize the loss function L G The value of is to maximize the probability that the generated image is considered to be a real image by the discriminator.

8. The method according to claim 7, characterized in that Step 5.3 includes: Step 5.3.1, fix the network parameter weight w of the generator G , keep the generator unchanged, according to the loss function L of the discriminator D , using the back-propagation method to calculate the gradient And through the gradient descent method according to the discriminator learning rate lr D Update the weight parameter w of the discriminator D , the calculation formula is: Then complete a weight update and update the frequency n according to the discriminator critic Repeat the above steps n critic Second-rate; Step 5.3.2, fix the network parameter weight w of the discriminator D , keeping the discriminator unchanged, according to the loss function L of the generator G Calculate gradients using backpropagation And use the gradient descent method according to the generator learning rate lr G Update the weights w of the generator G , the calculation formula is: Then complete a weight update; Step 5.3.3, optimize the candidate operation architecture parameter weight W, and the calculation formula for the output Output of each candidate operation is: Where X is the input feature map, tol is the total number of candidate operations, and W i is the architectural parameter weight of the ith operation, O i (x) is the output of the i-th operation; Use the chain rule to calculate the loss function L for each candidate operation weight W i Gradient Update the candidate operation architecture parameter weights using gradient descent: Finally, the optimization of candidate operation architecture parameters is completed; Step 5.3.4, according to the optimizer settings, use the adaptive moment estimation algorithm to dynamically adjust the learning rate of the discriminator and generator: The parameters are updated by calculating the moving average of the first-order moment and the second-order moment. The update formula is: where θ t Represents the weight parameters that need to be updated at the current time step, including lr G , lr D , lr w The default learning rate, m t is the exponentially weighted moving average of the current gradient, v t is the exponentially weighted moving average of the current gradient square, ∈ is a constant; Step 5.3.5, monitor and record the perceptual similarity index IS and Fréchet distance FID during the training process, where IS is used to evaluate the diversity and inter-class separation of generated images, measure the quality and diversity of generated images, and FID measures the similarity between generated images and real images in the feature space.

9. The method according to claim 8, characterized in that In step 6, the gradient descent method is used to continuously optimize and adjust the network weight parameters and architecture parameters, so that the supernet gradually approaches the optimal performance; finally, according to the distribution of the architecture parameter weights, the candidate operation with the largest weight is selected to construct the optimal network architecture.

10. A generative adversarial network architecture search system based on hybrid convolution operations implemented by the method according to any one of claims 1 to 9, characterized in that: include: A construction module, used to construct a generative adversarial network supernet, and assign an importance weight to each operation in the supernet using a random continuous numerical value ranging from 0 to 1, wherein the weight is used to represent the probability of each operation being selected as the optimal operation; The hybrid convolution module is used to integrate the ordinary convolution operation with the self-attention mechanism to obtain a new operation that combines the advantages of both. A training module for training the network weights of the supernet using training data and its labels; The optimization module is used to optimize the parameters of the supernet architecture using the gradient descent method.

Images