Small sample image generation method and system thereof

Abstract

The application discloses a small sample image generation method and system, and specifically comprises the following steps: constructing and initializing a backbone network; in response to completing the construction and initialization of the backbone network, training the backbone network on a source domain; in response to completing the training of the plurality of backbone networks on the source domain, performing cross-domain training on the backbone network using a target domain based on meta-learning to obtain a backbone network adapted to the target domain; and generating and expanding small sample images according to the backbone network adapted to the target domain. The application designs a special backbone network and a source domain training unit, introduces the related idea of domain generalization, fully trains the backbone network on the source domain to extract domain generalization feature information, and proposes a special backbone network based on a style block to better extract feature information, thereby improving the image quality and quantity of the finally generated small sample images.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a small sample image generation method and system thereof. BACKGROUND

[0002] With the continuous development of machine learning related research, the related technology is gradually widely applied in the field of computer vision, natural language processing and other fields, and excellent methods and models are constantly being proposed. And in recent years, due to the continuous leap in the computing performance of hardware devices such as GPU and CPU, the emergence of advanced models and algorithms such as ResNet and BERT, machine learning methods have gradually moved from theory to practical application, empowering a large number of tasks in real-world scenarios, and actually changing everyone's life. However, the successful application of traditional machine learning methods depends on the detailed learning of large-scale data, and the model training of machine learning methods requires a large amount of data. In practical applications, for example, industrial product defect images, a large number of image samples that meet the requirements are often difficult to obtain, and due to data acquisition difficulties, image labeling costs and other problems, the scale of image data available for training is often not large, and even the training samples of each class are only a few. Therefore, the small sample problem has become one of the important research directions. In the face of small sample problems encountered in actual production, in addition to designing special machine learning methods in the case of limited samples, a more direct method is to directly expand the number of samples, so that a large number of machine learning methods for sufficient sample quantity conditions can be directly applied. In addition, considering that it is difficult to easily collect enough defect samples in actual scenarios, and it takes a lot of time for technical personnel with professional knowledge to complete the labeling of image samples, how to use the limited number of existing image samples to expand the number of image samples has become a key problem.

[0003] The existing image sample augmentation methods include a series of image sample enhancement methods and image generation methods. The image sample enhancement methods are mainly based on manual rules and divided into two methods, i.e., an image-level overall method without modifying the original image sample and a local method of modifying the original image. The method without modifying the original image sample operates on the whole image sample, such as rotation, flipping and mirroring, etc. The local method of modifying the original image mainly operates on local features and can cause certain damage to the whole image, including cutting and cropping, etc. The current image generation method is mainly based on the design of a generative adversarial network (GAN) to generate an algorithm. This kind of method mainly adopts an adversarial idea and contains two models: a generator and a discriminator. The two models are usually realized by a neural network, but can also be realized by any form of differentiable system that maps data from one space to another. The generator attempts to capture the distribution of real examples to generate new data examples. The discriminator is usually a binary classifier that distinguishes generated examples from real examples as accurately as possible.

[0004] With the continuous advancement of research on machine learning methods, the related technology is increasingly widely used in practical scenarios. However, the specific problems encountered in practical scenarios hinder the landing of related methods, and the most prominent one is the lack of available image samples for training. In the application tasks of machine learning methods in practical scenarios, such as target detection and image segmentation, in order to achieve satisfactory performance, the current machine learning method needs a large number of image sample data. However, compared with the commonly used public data set including thousands of samples for each class, there are only dozens or even only a few image samples for each class in most practical scenarios. This makes the small sample problem one of the key problems that hinder the landing of current machine learning methods in practical scenarios. One of the methods to solve this problem is to augment the number of image samples. The traditional image sample enhancement method can only augment a limited number of images and has different degrees of damage to the images. The method based on GAN can generate a large number of image samples, but it faces the problem of mode collapse and the pre-training of GAN requires a large amount of data, so its application in solving the small sample problem in practical scenarios is relatively small and the related research is relatively lacking.

[0005] Therefore, how to provide a method of not damaging the image and being able to generate a large number of image samples is a problem that those skilled in the art urgently need to solve. SUMMARY

[0006] The purpose of the present application is to propose a small sample image generation method based on domain generalization and meta-learning, which solves the problems of mode collapse and the need for a large amount of pre-training data faced by current GANs from two directions based on cross-domain angles, namely backbone network structure design and model training process. A special backbone network is designed, which can extract and transfer feature information in the source domain and apply it to the generation of target domain samples. At the same time, a special training process is designed, which enables the model to be trained on both the source domain and the target domain, so that the generalizable feature information in the source domain can be applied to the target domain, enabling the model to be fully trained in the target domain sample number extremely limited scene, thereby being able to generate a large number of images similar to the target domain image samples.

[0007] To solve the above problems, the present application provides a small sample image generation method, specifically comprising the following steps: constructing and initializing the backbone network; in response to completing the construction and initialization of the backbone network, training the backbone network on the source domain; in response to completing the training of the plurality of backbone networks on the source domain, using the meta-learning-based target domain to perform cross-domain training on the backbone network to obtain a backbone network adapted to the target domain; and generating and expanding the small sample image according to the backbone network adapted to the target domain.

[0008] As above, wherein the constructing and initializing the backbone network comprises the following sub-steps: designing the structure of the generator; in response to completing the structure design of the generator, designing the structure of the discriminator; and in response to completing the structure design of the discriminator, initializing the backbone network.

[0009] As above, wherein during the structure design of the generator, the input of the generator is the latent parameter z in the latent space z, which generally uses Gaussian noise with a distribution satisfying:

[0010]

[0011] Wherein μ and σ represent the set mean and variance parameters, respectively.

[0012] As above, wherein the discriminator comprises a convolution module, wherein the convolution module comprises a convolution layer, a down-sampling layer, and an activation function for increasing the nonlinearity of the network;

[0013] Wherein the activation function LeakyReLU(β) is generally represented as:

[0014]

[0015] Wherein a is a fixed parameter greater than 1, and β represents the input value of the activation function.

[0016] As described above, after obtaining the initialized backbone network, multiple backbone networks are trained on multiple data domains in the source domain. Let the source domain be R such that {r1, r2, ..., r...} N}, r i Let i represent the i-th image data domain. The source domain contains N data domains. A backbone network is trained on each data domain of the source domain.

[0017] As described above, training the backbone network in the source domain includes the following sub-steps: training the discriminator; and training the generator in response to completing the training of the discriminator.

[0018] As shown above, the loss function for training the discriminator... Specifically, it is expressed as follows:

[0019]

[0020] Where D(x) represents the discriminator's discrimination of the original image samples in the source domain, and during training, its discrimination result is made close to 1; G(z) represents the generated samples, and the discrimination result D(G(z)) of the trained discriminator is close to 0. Let represent the latent space, and z represent the latent parameters in the latent space. Represents the probability of real data. This indicates the probability of generating data.

[0021] As shown above, the loss function for training the generator... Specifically, it is expressed as follows:

[0022]

[0023] During training, the discrimination result D(G(z)) is made close to 1, meaning that the discriminator cannot distinguish between generated image samples and original image samples, thus minimizing the total value of the loss function.

[0024] As described above, the process of using a target domain based on meta-learning to train the backbone network across domains to obtain a backbone network adapted to the target domain includes the following sub-steps: dividing the backbone network and setting weight parameters; in response to completing the backbone network division and weight parameter setting, performing inner layer optimization of the backbone network; in response to completing the inner layer optimization of the backbone network, performing outer layer optimization of the backbone network; in response to completing the outer layer optimization of the backbone network, training the backbone network across domains as a whole in the target domain to obtain a backbone network adapted to the target domain.

[0025] The small sample image generation system specifically comprises an initialization unit, a source domain training unit, a target domain training unit and a generation unit; the initialization unit is configured to construct and initialize a backbone network; the source domain training unit is configured to train the backbone network on a source domain in response to completion of the construction and initialization of the backbone network; the target domain training unit is configured to perform cross-domain training on the backbone network using a target domain based on meta-learning in response to completion of training of the plurality of backbone networks on the source domain, to obtain a backbone network adapted to the target domain; and the generation unit is configured to generate and expand small sample images based on the backbone network adapted to the target domain.

[0026] The present application has the following beneficial effects:

[0027] (1) The present application designs a special backbone network and a source domain training unit, introduces the related idea of domain generalization, fully trains the backbone network on the source domain to extract domain generalization feature information, and proposes a special backbone network based on a style block to better extract feature information. More comprehensive source domain features can be extracted for transfer, providing a large amount of feature information for the subsequent model cross-domain training method on the target domain, laying an important foundation to improve the image quality and quantity of the final small sample image generation.

[0028] (2) The present application proposes a special model training method based on the idea of bottom freezing, in which the bottom layer of the model is trained on a source domain with a large number of samples, and the top layer is trained on a target domain with a small number of samples. Based on the idea of meta-learning, the target domain training unit is designed, the training process of the backbone network is divided into multiple episodes, each episode is divided into inner optimization and outer optimization for training, so that the backbone network can be fully trained even when the number of target domain samples is small. The proposed method has the feasibility of generating a large number of high-quality images similar to the target domain images in actual small sample scenarios. Unlike current GAN methods, this method does not require a large amount of training data, so it can be better applied to actual scenarios and expand the scope of current research. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0030] Figure 1 is an internal structure diagram of the small sample image generation system provided by the embodiment of the present application;

[0031] Figure 2is a flowchart of a small sample image generation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0033] The purpose of the present application is to propose a small sample image generation method based on domain generalization and meta learning, which introduces domain generalization and meta learning methods into the small sample machine learning method. The "domain" refers to a field composed of different data distributions, and in the field of computer vision, a data set of images can be simply regarded as an image data domain. A set of multiple image data domains available for training is called a source domain, and a data domain finally used for generating images is called a target domain. Meta learning is generally understood as learning to learn, which refers to the process of improving the learning algorithm in multiple learning scenarios. Considering the extreme lack of samples in the small sample scenario, the present method introduces the related idea of domain generalization, pre-trains the model on the source domain to extract domain generalization feature information, and proposes a special backbone network model to better extract feature information. At the same time, based on the bottom freezing idea, a hierarchical freezing method is proposed to make the bottom layer of the model trained on the source domain with a larger number of samples to obtain generalizable feature information, and the top layer trained on the target domain with a smaller number of samples to obtain domain-specific information that determines the final generated image. At the same time, based on the idea of meta learning, a scenario training method is used to divide the model training process into multiple scenarios, each scenario is divided into inner optimization and outer optimization, and source domain samples are randomly extracted to form a support set to train the bottom layer in the inner optimization, and target domain samples are combined as a query set to train the top layer in the outer optimization. The inner and outer optimization interact with each other so that the model can also be fully trained when the number of target domain samples is relatively small. The method proposed in the present application can generate images similar to the target domain under the condition of extremely limited target domain data in the small sample scenario, and expand the number of samples.

[0034] Embodiment one

[0035] As Figure 1 shown, it is a small sample image generation system provided by the present application, which can introduce domain generalization and meta learning methods into the small sample machine learning method, and finally obtain an overall backbone network that meets the target domain to generate a large number of image samples similar to the target domain image.

[0036] The system of the application specifically comprises: an initialization unit 110, a source domain training unit 120, a target domain training unit 130, and a generation unit 140.

[0037] The initialization unit 110 is configured to construct and initialize the backbone network.

[0038] The source domain training unit 120 is connected to the initialization unit 110 and configured to, in response to completion of the construction and initialization of the backbone network, train the backbone network on the source domain.

[0039] The target domain training unit 130 is connected to the source domain training unit 120 and configured to, in response to completion of the training of the plurality of backbone networks on the source domain, perform cross-domain training on the backbone network using a target domain based on meta-learning to obtain a backbone network adapted to the target domain.

[0040] The generation unit 140 is connected to the target domain training unit 130 and configured to generate and augment small sample images according to the backbone network adapted to the target domain.

[0041] Embodiment Two

[0042] In this example, the NEU surface defect dataset is selected for the experiment. The dataset is six typical defects of hot-rolled steel strips in actual production, i.e., rolling scale (RS), patch (Pa), cracking (Cr), pitting surface (PS), inclusion (In), and scratch (Sc), with a total of 300 samples for each type of defect. Referring to the experimental setup of small sample machine learning of the reference standard, 5 samples of each type are taken to form a training set. For the data domain in the source domain, Guangdong Tianchi, Crack Forest, Magnetic-tile, and dagm-2007 are collected and selected, which are denoted as r1, r2, r3, and r4, respectively. Among them, Guangdong Tianchi and Magnetic-tile are similar to NEU and are industrial product defect datasets. The Guangdong Tianchi dataset is composed of aluminum defect samples, and the Magnetic-tile dataset is composed of ceramic tile defect samples. At the same time, in order to enrich the source domain information and reflect the generalization ability of the method, the Crack Forest dataset composed of road defect samples and the artificially generated texture defect dataset dagm-2007 are selected. The number of emotions N episode is set to 1000, the outer layer optimization training number N outer is set to 50, and the inner layer optimization training number N inner is set to 500. In the experiment, the size of all image samples is uniformly set to 256x256 (unit: pixels).

[0043] As Figure 2 shown, it is a small sample image generation method provided by the application. Specifically, it comprises the following steps:

[0044] Step S210: constructing and initializing the backbone network.

[0045] wherein the network initialization includes constructing the backbone network and initializing the backbone network using random parameters.

[0046] Specifically, each backbone network contains information of its training data domain, including domain generalization information that can be generalized and domain-specific information specific to the domain. In order to better extract these features, a special backbone network structure needs to be designed, so that it can highlight the features specific to each domain while ensuring that sufficient underlying features are carried.

[0047] wherein the structure of the backbone network of the present application includes designing the structure of the generator and the structure of the discriminator.

[0048] wherein step S210 includes the following sub-steps:

[0049] Step S2101: designing the structure of the generator G.

[0050] The initial input of the generator is the hidden parameter z in the hidden space with a Gaussian noise, whose distribution satisfies:

[0051]

[0052] wherein μ and σ represent the set mean and variance parameters, respectively.

[0053] The generator proposed in the present application does not directly use the hidden parameter z as the input, but maps it through a multilayer perceptron (MLP), wherein the MLP is composed of 8 fully connected (FC) layers. Each neuron node in the FC layer is connected to all nodes of the previous layer. Without changing the dimension of the input parameter, the mapping relationship between the hidden space and the training sample can be further learned during training. This operation provides additional depth for the network, thereby improving the perception path length and image generation quality. Especially in the case of large differences between the source domain and the target domain, the FC can maintain the learned large model capability to ensure the migration of the model's "representation ability".

[0054] After the processing of the MLP, the nonlinear mapping of the hidden parameter z can be obtained, denoted as w:

[0055] w = MLP(z)

[0056] and further use w as the input of the remaining part of the generator.

[0057] In the training process, the MLP gradually learns the "style" of the source domain, i.e., the domain-specific features, and embodies it in the parameter w. After obtaining the nonlinear mapping w for the input noise, the generator needs to be trained in combination with w containing the "style" information in the source domain as the training effect, so that the generator learns the domain-specific features in the source domain as the training effect.

[0058] Specifically, the style block proposed in the StyleGAN method in the GAN method is used as the basic module of the generator in this embodiment to construct the overall generator structure, where the generator G is composed of multiple style blocks.

[0059] The generator receives the latent parameter z and Gaussian noise as input in the latent space and obtains the output generated image through the processing of multiple layers of style blocks. At the same time, unlike the original StyleGAN, the output of the latent parameter z after the FC layer is used as the input of the first layer of the style block instead of a constant set by humans to improve the learning ability of the network. The last convolutional layer converts the feature map into a 3-channel color image, i.e., the generated image x.

[0060] Step S2102: In response to completing the structure design of the generator G, the structure design of the discriminator D is performed.

[0061] Specifically, the inputs of the discriminator are the image x generated by the generator and the original image used for training, and the discriminator discriminates the inputs. Specifically, the ability of the discriminator to determine whether the input image is the original image of the source domain or the image generated by the generator is determined by training the discriminator with the original image, and the classification result is returned to the generator for joint training.

[0062] In order to improve the ability of the discriminator to distinguish images and learn, the discriminator of the present application includes a special convolutional module, which includes a convolutional layer, a down-sampling layer, and an activation function for increasing the nonlinearity of the network.

[0063] Specifically, "conv" represents the convolutional layer, and the convolutional kernel is uniformly set to 3x3; "downsample" is the down-sampling layer, which is a special convolutional layer structure for dimensionality reduction operation on the feature map; "LeakyReLU" is the activation function, which is used to increase the nonlinearity of the network, where the activation function LeakyReLU(β) is generally represented as:

[0064]

[0065] where a is a fixed parameter greater than 1, β represents the input value of the activation function, and LeakyReLU as the activation function retains some values on the negative axis when β<0, so that the information is not completely lost.

[0066] Therefore, the discriminator D designed in the present application is composed of multiple convolution modules conv block, a first layer convolution layer "conv", and a linear layer "linear". The input of the discriminator D is the image sample x generated by the generator. The first layer convolution layer "conv" is used to convert the input image sample (including the generated image x and the original image used for training) into a multi-channel feature map. Multiple conv blocks can be stacked according to specific experimental settings to gradually reduce the feature map dimension and learn the image features. The linear layer "linear" is a special fully connected layer, which is used to directly reduce the final feature map processed by multiple conv blocks to 1 as the output, i.e., whether the image is the original image.

[0067] Step S2103: In response to completing the structure design of the discriminator D, the backbone network is initialized.

[0068] Specifically, the construction of the backbone network is completed after the design of the generator G and the discriminator D, wherein the backbone network is a network structure composed of the generator G and the discriminator D, and the backbone network is initialized using random parameters.

[0069] Step S320: In response to completing the construction and initialization of the backbone network, the backbone network is trained on the source domain.

[0070] After obtaining the initialized backbone network, the backbone network is trained on multiple data domains in the source domain. Let the source domain be R = {r1, r2,..., rN}, wherein ri represents the i-th image data domain, and the source domain contains N data domains. A backbone network is trained on each data domain in the source domain, thereby obtaining multiple backbone networks trained on multiple data domains. N}r i i represents the i-th image data domain, and the source domain contains N data domains. A backbone network is trained on each data domain in the source domain, thereby obtaining multiple backbone networks trained on multiple data domains.

[0071] For example, the step S320 specifically includes the following sub-steps in the case of training on each domain:

[0072] Step S3201: Training the discriminator D.

[0073] Specifically, in the case of training the backbone network on r i , the discriminator D is first trained, wherein the loss function of training the discriminator D is Specifically, the loss function is

[0074]

[0075] wherein D(x) represents the discrimination of the original image sample in the source domain by the discriminator D, and the discrimination result is close to 1 during training. G(z) represents the generated sample, and the discrimination result D(G(z)) of training the discriminator D is close to 0, wherein ri represents the i-th image data field, represents the latent space, z represents the latent parameter in the latent space, represents the real data probability, represents the generated data probability.

[0076] Step S3202: In response to completing the training of the discriminator D, the training of the generator G is performed.

[0077] Specifically, the loss function for training the generator G is Specifically represented as:

[0078]

[0079] During training, the result D(G(z)) of the discrimination is close to 1, that is, the discriminator D cannot distinguish between the generated image sample and the original image sample, so that the total value of the loss function is minimized.

[0080] In summary, in the r i The overall loss function for training the backbone network in the domain is represented as:

[0081]

[0082] By training the backbone network in each domain of the source domain, the generator G parameters for each domain can be obtained, represented as {G1, …, G N These parameters contain domain-specific features and domain generalization features that can be used for generalization and migration, that is, the above features are collectively referred to as source domain feature information.

[0083] Step S330: In response to completing the training of the multiple backbone networks in the source domain, the backbone network is trained across domains using the target domain based on meta-learning, obtaining a backbone network adapted to the target domain.

[0084] Specifically, the source domain feature information obtained in step S320 is applied to the generation of image samples in the target domain. The present application is based on the idea of meta-learning in small sample generation and combined with the specific scene of cross-domain.

[0085] The meta-learning method is mainly applied to classification tasks in the field of small samples. Unlike traditional meta-learning methods, this embodiment does not divide the overall process into a meta-training phase and a meta-testing phase, but uses the idea of double-layer optimization in meta-learning to train the backbone network, with the sample set of the source domain as the support set and the sample set of the target domain as the query set.

[0086] To illustrate the idea of double-layer optimization, first define a general machine learning task as, given a training set train a backbone network parameterized by θ on Loss function of training backbone network parameterized by θ * As shown below:

[0087]

[0088] Wherein ω is a condition artificially preset in general machine learning, which can be the selection of an optimizer, the selection of a network model, the number of training steps, or initialization parameters and various hyperparameters, and the condition is added to explicitly indicate the dependence of the solution on various factors. In meta-learning, ω is learned between multiple tasks, and ω is used as cross-task knowledge or meta-knowledge to generalize to new tasks, which is called a meta-parameter.

[0089] There are many training methods and ideas for meta-parameters ω in meta-learning. Based on the idea of double-layer optimization, the target domain model is trained. The idea of double-layer optimization divides the training process into inner-layer optimization and outer-layer optimization, respectively trains θ and ω, and performs multiple cycle training called episode training. It is worth noting that in addition to the cycle training between the inner and outer layers, the inner-layer optimization and the outer-layer optimization are each trained for several rounds in each episode to ensure that the most suitable θ and ω for the current task are obtained.

[0090] Based on the above idea, the application proposes a special meta-learning method combined with the hierarchical freezing method mentioned in step S330 to better combine source domain samples and extremely limited target domain samples to train the model. In the specific training process, the discriminator bottom layer parameters in the backbone network with a large number of parameters are trained using the support set composed of the source domain sample set with a sufficient amount of samples in the inner-layer optimization; the meta-parameters with a small number of parameters are trained using the query set composed of the target domain samples with a small number of samples in the outer-layer optimization, specifically the weight parameters combined from the outputs of multiple bottom layer source domain generators and the top layer parameters of the discriminator and generator; the parameters are transmitted between the outer-layer optimization and the inner-layer optimization, and the backbone network is trained together.

[0091] Step S330 specifically includes the following sub-steps:

[0092] Step S3301: Divide the backbone network and set the weight parameters.

[0093] The backbone network in this step is the backbone network trained on the source domain. In this step, the generator parameters are trained on the target domain samples and their limited small sample scenarios to obtain a generator G suitable for the target domain that can generate samples similar to the target domain samples. T .

[0094] Specifically, the generator and discriminator are divided into bottom and top layers, respectively. The number of style blocks and convolution blocks in the bottom and top layers can be set according to specific requirements. Furthermore, the parameters of the bottom or top layer are frozen at different stages of training, i.e., gradient propagation and parameter updates are not performed on them. This method controls the number of model parameters required for training, ensuring that the backbone network can still fit the data even when the training samples are extremely limited. In the figure, x represents the image generated by the generator, and y... pred This represents the discriminator's prediction result.

[0095] The discriminator D is divided into a lower layer with a large number of parameters—denoted as D0. L And the top layer with a smaller number of parameters—denoted as D H Similarly, the generator G is divided into a top-level section with a relatively small number of parameters—denoted as Gtop. H However, at this point, the generator's underlying structure is not composed of a single model. Instead, it extracts the N source domain generators {G1, ..., G...} obtained from training on the N source domain training datasets in the steps described above. N The underlying {G} L1 , ..., G LN}, combining the domain generalization idea, the weighted combination of the model's input and the output after these underlying processing steps is used as G. H Input.

[0096] Furthermore, a weight parameter p is defined. p is a 1×N vector, where N dimensions correspond to N source domain generators, used to adjust the weights of different data domain feature information in the source domain. Specifically, this operation is represented as follows:

[0097]

[0098] in G represents H Input, G represents Li The output of {G} should be noted. L1 , ..., G LN The parameters are frozen throughout the entire training process and are not updated.

[0099] Step S3302: In response to the completion of backbone network partitioning and weight parameter setting, perform inner layer optimization of the backbone network.

[0100] In this step, the backbone network is trained using source domain data.

[0101] Specifically, N are randomly selected from each domain in the source domain. sdata The n samples form the source domain training dataset R in the i-th episode. i Make {r i1r iN}, wherein r ij represents the jth image dataset in the ith episode, and the source domain includes N source domain training datasets. At this time, D H , G H and p are frozen, D L is trained, and it is noted that the parameters frozen at this time are from the previous episode. The objective function for r ij in the inner optimization is specifically represented as:

[0102]

[0103] In the formula, D represents the D L corresponding to the jth image data domain in the ith episode. At this time, the generator G is completely frozen, and only the discriminator D is trained. In the formula, D represents the D H in the i-1th episode, and p i-1 represents the p H in the i-1th episode. represents the G inner in the i-1th episode, represents element multiplication.

[0104] Step S3303: In response to completing the inner optimization of the backbone network, outer optimization of the backbone network is performed.

[0105] After the inner optimization is performed N inner times on the data domain in each source domain, the plurality of D L trained is transmitted into the outer optimization in the same episode.

[0106] In the outer optimization, other parts of the backbone network are trained in combination with the target domain data T. At the time of training, D L is frozen, G H , D H and p are trained using the target domain data T. It is noted that, in order to train the generalization and learning ability of the network, the source domain data in the inner optimization is different in different episodes, but the target domain data is completely used for training due to the definition of small sample machine learning and the lack thereof, and the target domain data in each episode of the overall training process is the same. Specifically, at this time, D L is frozen, G H , D H and p are trained, and the objective function of the GAN is combined. For the D L trained in the data domain of each source domain in the inner optimization, the objective function in the outer optimization is specifically represented as:

[0107]

[0108] It should be noted that due to the extremely limited number of samples in the target domain, if the loss function is designed for p alone, it may cause overfitting, and if the loss function for p is calculated separately, it will cause excessive calculation, so p is regarded as an internal parameter of the generator G, and the generator G is trained as a whole, that is, the meaning of the formula .

[0109] The overall loss function of the outer optimization can be obtained as .

[0110]

[0111] Similar to the inner optimization, the overall loss function is also needed in the outer optimization for N outer round training.

[0112] Step S3304: In response to completing the outer optimization of the backbone network, the backbone network is trained across the domain on the target domain to obtain a backbone network adapted to the target domain.

[0113] The above steps are analyzed for the model training process in a single episode in episode training. Before training starts, G H , D L , D H and p are randomly initialized, denoted as G and p0, as the input of the first episode. In each episode, the backbone network first receives the model parameters obtained by training in the last episode, and then performs inner optimization training and outer optimization training after the inner optimization training, and the model parameters obtained by training are passed into the next episode. It should be noted that D L in the figure is a set containing N elements, and each element in the set is the bottom layer of the discriminator D corresponding to each data domain in the source domain. It should be noted that there is also parameter transmission within an episode, and the D L obtained by training in the inner optimization is passed into the outer optimization and frozen to train G H , D H and p.

[0114] Wherein, steps S3302-3303 are repeated N episode times until the model converges, and the G H and p most suitable for the target domain are obtained, combined with the generated bottom layer {G L1 , …, G LN} obtained in the source domain pre-training, to obtain the final backbone network most suitable for the target domain, that is, the generator G T most suitable for the target domain is obtained.

[0115] Step S340: generating and augmenting the small sample images according to the backbone network adapted to the target domain.

[0116] Specifically, the generator G most suitable for the target domain is used to generate the generated images similar to the target domain images. T The input of Gaussian noise can generate a large number of generated images similar to the target domain images, thereby augmenting the number of image samples of the target domain.

[0117] The present application has the following beneficial effects:

[0118] (1) The present application designs a special backbone network and a source domain training unit, introduces the related idea of domain generalization, fully trains the backbone network on the source domain to extract domain generalization feature information, and proposes a special backbone network based on the style block to better extract feature information. More comprehensive source domain features can be extracted for transfer, providing a large amount of feature information for subsequent model cross-domain training methods on the target domain, laying an important foundation to improve the image quality and quantity of the final small sample image generation.

[0119] (2) The present application proposes a special model training method based on the idea of bottom freezing, which trains the bottom layer of the model on the source domain with a large number of samples and trains the top layer on the target domain with a small number of samples. Based on the idea of meta-learning, the target domain training unit is designed, and the training process of the backbone network is divided into multiple episodes, each episode is divided into inner optimization and outer optimization for training, so that the backbone network can be fully trained even when the number of target domain samples is small. The proposed method has the feasibility of generating a large number of high-quality images similar to the target domain images in actual small sample scenarios. This method does not require a large amount of training data like current GAN methods, so it can be better applied to actual scenarios and expand the scope of current research.

[0120] Although the examples referred to in the present application are described, they are only for the purpose of explanation and not limitation of the present application. Changes, additions and / or deletions to the embodiments can be made without departing from the scope of the present application.

[0121] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A small sample image generation method, characterized in that, Specifically comprising the following steps: Performing construction and initialization of the backbone network; In response to completing the construction and initialization of the backbone network, training the backbone network on the source domain; In response to completing the training of the plurality of backbone networks on the source domain, performing cross-domain training of the backbone network using the target domain based on meta-learning to obtain the backbone network adapted to the target domain; According to the backbone network adapted to the target domain, generating and expanding the small sample image; The target domain training unit uses the target domain based on meta-learning to perform cross-domain training of the backbone network to obtain the backbone network adapted to the target domain, including the following sub-steps: Divide the backbone network and set the weight parameters; Set the weight parameter p, p is a vector with a dimension of 1xN, and the N dimension corresponds to N source domain generators, used to adjust the weight of the feature information of different data domains in the source domain, and the operation here is represented as: wherein represents G H the input, represents G Li the output; In response to completing the backbone network division and weight parameter setting, inner-layer optimization of the backbone network is performed; in the inner-layer optimization, the target function of r ij is expressed as: represents the D corresponding to the jth image data field in the ith episode L At this time, the generator G is completely frozen, and only the discriminator D is trained, represents the D in the i-1th episode H , similarly p i-1 represents p in the i-1th episode, represents the G in the i-1th episode H , represents element multiplication; In response to completing the inner-layer optimization of the backbone network, the outer-layer optimization of the backbone network is performed; the objective function in the outer-layer optimization Specifically represented as: Outer layer optimized overall loss function is: In response to completing the outer layer optimization of the backbone network, training the backbone network on the target domain to obtain the backbone network adapted to the target domain.

2. The small sample image generation method of claim 1, wherein, Performing construction and initialization of the backbone network, including the following sub-steps: Performing structure design of the generator; In response to completing the structure design of the generator, performing structure design of the discriminator; In response to completing the structure design of the discriminator, initializing the backbone network.

3. The small sample image generation method of claim 2, wherein, In the process of structural design of the generator, the input of the generator is the latent parameter z in the latent space with a Gaussian noise whose distribution satisfies: Where μ and σ represent the set mean and variance parameters respectively.

4. The small sample image generation method of claim 2, wherein, The discriminator includes a convolution module, wherein the convolution module includes a convolution layer, a down-sampling layer, and an activation function for increasing the nonlinearity of the network; Where the activation function LeakyReLU(β) is represented as: Where a is a fixed parameter greater than 1, and β represents the input value of the activation function.

5. The small sample image generation method of claim 1, wherein, After obtaining the initialized backbone network, a plurality of backbone networks are trained on a plurality of data domains in the source domain, and the source domain is represented as R = {r1, r2, …, rN} N} and r i represents the i-th image data domain, and the source domain includes N data domains in total, and one backbone network is trained on each data domain in the source domain.

6. The small sample image generation method of claim 5, wherein, Training the backbone network on the source domain, including the following sub-steps: Training the discriminator; In response to completing the training of the discriminator, training the generator.

7. The small sample image generation method of claim 6, wherein, Loss function for training the discriminator Specifically represented as: where D(x) represents the discriminator to distinguish the original image sample in the source domain, and the discrimination result is close to 1 during training, G(z) represents the generated sample, and the discrimination result D(G(z)) of the discriminator is close to 0 during training, represents the latent space, and z represents the latent parameter in the latent space, represents the probability of real data, represents the probability of generated data.

8. The small sample image generation method of claim 7, wherein, Loss function for training the generator Specifically represented as: During training, the result of discrimination D(G(z)) is close to 1, that is, the discriminator cannot distinguish between the generated image sample and the original image sample, so as to make the total value of the loss function minimum.

9. A small sample image generation system, characterized by, Specifically, the initialization unit, the source domain training unit, the target domain training unit, and the generation unit are included; The initialization unit is used to perform construction and initialization of the backbone network; The source domain training unit is used to train the backbone network on the source domain in response to completing the construction and initialization of the backbone network; The target domain training unit is used to perform cross-domain training of the backbone network using the target domain based on meta-learning to obtain the backbone network adapted to the target domain in response to completing the training of the plurality of backbone networks on the source domain; The generation unit is used to generate and expand the small sample image according to the backbone network adapted to the target domain; The target domain training unit uses the target domain based on meta-learning to perform cross-domain training of the backbone network to obtain the backbone network adapted to the target domain, including the following sub-steps: Divide the backbone network and set the weight parameters; Set the weight parameter p, p is a vector with a dimension of 1xN, and the N dimension corresponds to N source domain generators, used to adjust the weight of the feature information of different data domains in the source domain, and the operation here is represented as: wherein represents G H the input, represents G Li the output; In response to completing the backbone network division and weight parameter setting, inner-layer optimization of the backbone network is performed; in the inner-layer optimization, the target function of r ij is expressed as: D represents the D corresponding to the jth image data field in the ith episode L At this time, the generator G is completely frozen, and only the discriminator D is trained, D represents the D in the i-1th episode H , similarly p i-1 p represents the p in the i-1th episode, G represents the G in the i-1th episode H , represents element multiplication; In response to completing the inner-layer optimization of the backbone network, the outer-layer optimization of the backbone network is performed; the objective function in the outer-layer optimization Specifically represented as: Outer layer optimized overall loss function is: In response to completing the outer layer optimization of the backbone network, training the backbone network on the target domain to obtain the backbone network adapted to the target domain.

Images