Method and system for autonomous modification of data

Abstract

The present invention relates to a method and system for autonomous modification of data. A computer-implemented method for modifying patterns in a data set using a generative adversarial network is provided. The method includes providing a data sample pair. Each pair includes a base data sample and a modified data sample. The modified pattern is determined by applying a random modification to the base data sample. The method includes using an adversarial training method and using the data sample pairs as input to train the generator to build a model of the generator, wherein the discriminator receives a data set pair of the data set as input, each of the data set pairs including the predicted output of the generator based on the base data sample and the corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, and predicting an output data set for an unknown data sample as input to the generator in the absence of the discriminator.

Description

Technical Field

[0001] The present invention generally relates to autonomously modifying data patterns, and more particularly, to a computer-implemented method for modifying patterns in a data set using a generative adversarial network. The present invention also relates to a corresponding machine learning system and a computer program product for modifying patterns in a data set using a generative adversarial network. Background Art

[0002] Artificial intelligence (AI), in the form of machine learning specifically, is being widely introduced into enterprise deployments and as part of enterprise applications. Currently, software development is undergoing a transformation from linear programming to machine learning (ML) model training. However, it turns out that training a machine learning system is not easy, it is a highly complex process, and success or failure depends on the availability of training data. The predictions of a machine learning system are only as good as the results of the training data. However, good training data usually requires good annotations or labeling in order to be correctly interpreted by the machine learning system in order to develop a successful model.

[0003] Therefore, programming is no longer the most time-consuming part of the process today. With the rise of machine learning, labeling has become an essential component of the development of new tools. In fact, the number of samples required for machine learning-based processes scales with the complexity of the input. For example, the LSVRC-2010 ImageNet training set includes 1.3 million images organized into 1,000 categories (Sutskever, Hinton, & Krizhevsky, 2012).

[0004] In this context, generative adversarial networks (GANs) have begun to gain attention as a way to capture the inherent distribution of a dataset (Goodfellow et al., 2014), leading to applications such as data augmentation (Antoniou, Storkey, and Edwards, 2018), where synthetically generated samples can be used to train other AI models. Summary of the invention

[0005] According to one aspect of the present invention, a computer-implemented method for modifying patterns in a data set using a generative adversarial network may be provided. The generative adversarial network may include a generator and a discriminator. The method may include providing pairs of data samples. Each pair may include a base data sample having a pattern and a modified data sample having a corresponding modified pattern. The modified pattern may be determined by applying at least one random modification to the base data sample.

[0006] The method may further include training the generator using an adversarial training method and using the data sample pairs as input to build a model of the generator. Thus, the discriminator may receive a dataset pair of datasets as input, wherein each dataset pair may include a predicted output of the generator based on a base data sample and the corresponding modified data sample, thereby enabling optimization of a joint loss function for the generator and the discriminator.

[0007] Furthermore, the method may include predicting an output data set for an unknown data sample as input to the generator without the discriminator (ie, the discriminator may be removed).

[0008] According to another aspect of the present invention, a machine learning system for modifying patterns in a data set using a generative adversarial network may be provided. The generative adversarial network may include a generator network system and a discriminator network system. The machine learning system may include a receiving unit adapted to provide data sample pairs. Each pair may include a base data sample having a pattern and a modified data sample having a corresponding modified pattern. The modified pattern may be determined by applying at least one random modification to the base data sample.

[0009] The system may further include a training module adapted to control the use of an adversarial training method and to train the generator network system using the data sample pairs as input to construct a model of the generator network system. Thus, the discriminator network system may receive a data set pair of data sets as input. Each of the data set pairs may include a predicted output of the generator based on a base data sample and the corresponding modified data sample. Thus, a joint loss function for the generator and the discriminator can be optimized.

[0010] The system may further comprise a prediction unit adapted to predict an output data set for unknown data samples as input to the generator in the absence of the discriminator.

[0011] Furthermore, embodiments may take the form of an associated computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in conjunction with a computer or any instruction execution system. For purposes of this specification, a computer-usable or computer-readable medium may be any device that may contain means for storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method claims, while other embodiments are described with reference to apparatus claims. However, those skilled in the art will understand from the above and below descriptions that, unless otherwise indicated, any combination of features relating to different subject matters (particularly features of method claims and features of apparatus claims) in addition to any combination of features belonging to one type of subject matter is considered disclosed herein.

[0013] The aspects defined above and further aspects of the invention are apparent from the examples of embodiment to be described hereinafter and are explained with reference to the examples of embodiment but to which the invention is not limited.

[0014] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings, which are:

[0015] Figure 1 A block diagram illustrating an embodiment of a computer-implemented method of the present invention for modifying patterns in a data set using a generative adversarial network;

[0016] Figure 2 shows the general setup of the system proposed in this paper used by the proposed method;

[0017] Figure 3 The main processing steps are shown;

[0018] Figure 4 An example of a pattern to be modified and ground truth data of the modified output and ground truth of the modified output are shown;

[0019] Figure 5 An example of transforming a corrupted input image into modified output 1 and modified output 2 is shown;

[0020] Figure 6 is an example 600 of a reconstruction of a phase map;

[0021] Figure 7 Another example 700 involving a form is shown;

[0022] Figure 8 and 9 Other examples of challenging datasets of very noisy scanned documents are shown;

[0023] Fig.10 and 11 shows an example involving another generator in series with a generator that is trained to "separate" noise from documents;

[0024] Fig.12 A block diagram showing an embodiment of a machine learning system of the present invention for modifying patterns in a data set using a generative adversarial network;

[0025] Fig.13 Shown including according to Fig.12 Block diagram of the computing system of a machine learning system. DETAILED DESCRIPTION

[0026] In the context of this description, the following conventions, terms and / or expressions may be used:

[0027] The term "generative adversarial network" (GAN) refers to a class of machine learning systems. Two neural networks can compete with each other in a zero-sum game framework. The technique can generate, for example, photos with many realistic features that appear at least superficially real to a human observer. It can represent a form of unsupervised learning.

[0028] A generative network, or generator network, can generate candidates, while a discriminator network evaluates these candidates. The competition can be conducted with respect to a data distribution. Typically, a generative network can learn a mapping from a latent space to a data distribution of interest, while a discriminator network can distinguish the candidates generated by the generator from the true data distribution. The training goal of the generative network can be to increase the error rate of the discriminator network (i.e., "fool" the discriminator network) by generating new candidates that the discriminator may consider to be non-synthetic (i.e., part of the true data distribution).

[0029] A known dataset can be used as initial training data for the discriminator. Training involves presenting patterns from the training dataset until an acceptable accuracy is achieved. The generator can be trained based on whether it successfully fools the discriminator. Typically, the generator can be seeded with random inputs sampled from a predefined latent space (e.g., a multivariate normal distribution). Thereafter, the discriminator can evaluate the candidates synthesized by the generator. Backpropagation can be applied in both networks so that the generator produces better images, while the discriminator may become more skilled in labeling synthesized images. The generator can typically be a deconvolutional neural network, while the discriminator is a convolutional neural network.

[0030] The term "neural network" may refer to a computing system inspired by the biological neural networks that make up animal brains. A neural network may not only be an algorithm in itself, but also serve as a framework that enables many different ML algorithms to work together and process complex data inputs. Such a system can "learn" to perform a task by considering examples, generally without being programmed with any task-specific rules. A neural network may include multiple nodes as an input layer, multiple hidden layers, and multiple nodes at an output layer. The nodes may be connected layer by layer, and each node may include an activation function that uses input values ​​from the previous layer nodes. The number of nodes in the hidden layer may be smaller than the number of nodes in the input and / or output layer deconvolutional neural networks.

[0031] The term "generator" herein may refer to a neural network having multiple layers, wherein the number of nodes in the input layer may be equal to the number of nodes in the output layer. In this way, a generator or a generator network or a generator network system may generate output data of the same complexity (i.e., resolution) as the input data, i.e., a deconvolutional network.

[0032] The term "discriminator" may refer to an artificial neural network that contains the same number of input nodes as the number of output nodes that the generator has. Therefore, the discriminator cannot distinguish whether its input is an original data sample or a data sample generated by the generator based solely on the resolution parameter. The number of output nodes of the discriminator may be two. This is to distinguish between the original data sample and the artificially generated data sample output by the generator.

[0033] The term "data sample" may refer to, for example, images, sound data, text data, or any kind of other unstructured data. Unstructured data may refer to data that does not fit into a typical structured data scheme such as in a relational database.

[0034] The term "base data sample" may refer to an unmodified data sample used for training and / or also used as input to a trained generator.

[0035] The term "modified data sample" may refer to a data sample related to a base data sample having at least one modified feature compared to the base data sample. A base data sample may always have an associated modified data sample, thereby constructing a pair of data samples for training.

[0036] The term "modify mode" may exemplarily mean that (in the case of an image as a data sample) dotted lines may be completed as solid lines, colored lines may be converted to black and white lines, or lines may be completely removed from the data sample so that form data and content data may be separated.

[0037] The term "joint loss function" can mean a function that measures the content loss between the base data sample and the modified data sample. However, it is also possible that their joint loss function involves the added content. What matters is that the content has changed.

[0038] The term "Wasserstein loss function" (also denoted as Kantorovich-Rubinstein metric or distance) can refer to a distance function defined between probability distributions on a given metric space. Intuitively, if each distribution is considered to be a unit amount of "dirt" piled on M, then the metric is the minimum "cost" to turn one pile into another, which is assumed to be the amount of dirt that needs to be moved multiplied by the average distance the dirt moves. Because of this analogy, the metric is known in computer science as the bulldozer distance.

[0039] The term “PatchGAN” may refer to a convolutional neural network that processes input data (e.g., an image) in patches identically and independently, which makes the processing cheap in terms of required parameters, time, and memory.

[0040] The term "VGG19" can refer to a pre-trained 19-layer neural network developed by the VGG group in the ILSVRC-2014 competition. For more information, see the following arXiv paper: "Very Deep Convolutional Networks for Large-Scale Image Recognition (K. Simonyan, A. Zisserman published on arXiv:1409.1556)".

[0041] The term "ResNet-50" can refer to a 50-layer convolutional neural network used for residual learning. In residual learning, instead of trying to learn some features, you can try to learn some residuals. The residual can be simply understood as subtracting the features learned from the input of this layer. ResNet does this using shortcut connections (connecting the input of the nth layer directly to the (n+x)th layer). It turns out that training this form of network is easier than training a simple deep convolutional neural network, and it also solves the problem of reduced accuracy.

[0042] The proposed computer-implemented method for modifying patterns in a dataset using a generative adversarial network can provide multiple advantages and technical effects:

[0043] The proposed super-resolution GAN improves training convergence and allows for a separate architecture for the generator and discriminator. By choosing to implement Wasserstein loss instead of adversarial loss, the proposed method and system can lead to better results than any individual component in the standard architecture.

[0044] The generator can be similar to U-NET, so low-level information can be passed directly to deeper layers, making training more efficient because the skipped layers facilitate gradient backpropagation and overcome the gradient vanishing problem of deeper networks. In addition, the absence of dense layers can provide flexibility for testing under input shapes different from those used for training.

[0045] An optionally modified discriminator implemented as PatchGAN allows reducing the number of required power meters, which can save memory and time, and enables applying the architecture to arbitrary input shapes.

[0046] In summary, the proposed method and the related system can allow training neural networks to process different kinds of input data, separate different patterns as output data, handle difficult and noisy input data, and can also be applied to other data types besides images. The main advantage is that no labeled or annotated data is required; therefore, the proposed concept can represent a special form of unsupervised learning, which significantly reduces the manpower of the training phase.

[0047] In the following, other embodiments applicable to the method and the corresponding system will be described:

[0048] According to an advantageous embodiment of the method, the joint loss function can be a Wasserstein loss function, in particular a loss function with a gradient penalty. This enables particularly efficient training of a joint system comprising a generator and a discriminator.

[0049] According to a permitted embodiment, the method may further comprise training different models of the generator network using an adversarial training method and using pairs of data samples as input. Thus, the modified data samples can be modified according to different aspects. Thus, reconstruction of the complete pattern, extraction of forms from the input data, and optimization of noisy documents can be achieved. In some cases, it can be suggested to serialize generators trained in different ways.

[0050] According to a preferred embodiment of the method, the generator can be a neural network with as many output nodes as input nodes and fewer hidden layer nodes than the number of input nodes. This can be called a deconvolutional neural network. Therefore, output data (e.g., images) with the same resolution as the input data can be generated for production use or for training.

[0051] According to a preferred embodiment of the method, the discriminator can be a neural network with as many input nodes as the generator has output nodes and two output nodes. The two output nodes can classify the input of the discriminator into "true" or "false", thereby indicating "whether the input is artificially generated by the generator or the input is an original sample". If the discriminator can no longer distinguish the source of its input, the training of the generator can be considered completed.

[0052] According to a useful embodiment of the method, the discriminator can be a PatchGAN, i.e. a series of convolutional neural networks (CNNs) with batch normalization. Such a system can guarantee the best results with fast convergence during training.

[0053] According to an advantageous embodiment of the method, the joint loss function may be a weighted combination of loss functions. In this way, different aspects can be reflected during training and a good convergence during training according to the various aspects can be achieved.

[0054] According to another advantageous embodiment of the method, the loss function can be related to a content loss of the base data sample, wherein the content loss is determined using feature maps of a pre-trained neural network. The pre-trained neural network can in fact be of type VGG19 or ResNet-50.

[0055] According to an optional embodiment of the method, the modified data samples may include: solid lines instead of dashed lines; black and white modes instead of equivalent color modes; no text modes instead of modes with text, and no line images instead of mixed line / text images, compared to the related data samples. Thus, the generator can be trained to add or subtract information about the new input data provided during the inference phase.

[0056] According to a further developed embodiment of the method, providing data sample pairs may include providing a set of images having patterns, determining at least one pattern to be modified, using a random number generator to randomly modify at least one pattern of the image, and correlating an image in the set of images and a related image with at least one pattern defining one of the pairs of samples comprising a base data sample and a modified data sample. Thus, no labeled data is required at all. This can increase the speed at which the proposed system can be adapted to a variety of different application areas. Manual and labor-intensive tasks can be eliminated during the training process.

[0057] According to a useful embodiment of the method, when comparing the results of the current iteration with the results of the previous iteration, if the result of the joint loss function may be less than a relative threshold, the training of the generative adversarial network can be terminated. Among other options, this condition can also mark the adaptive limit of the training time of the generative adversarial network and the generator.

[0058] According to an optional embodiment of the method, the basic data sample and the modified data sample can be images. However, other data sample types (such as sound or text) or other so-called unstructured data can also be used as data samples. Other examples and results of these data samples will be described below.

[0059] A detailed description of the drawings will be given below. All illustrations in the figures are schematic. First, a block diagram of an embodiment of a computer-implemented method of the present invention for modifying patterns in a data set using a generative adversarial network is given. Thereafter, other embodiments and embodiments of a machine learning system for modifying patterns in a data set using a generative adversarial network will be described.

[0060] Figure 1 A block diagram of an embodiment of a computer-implemented method 100 for modifying patterns in a data set using a generative adversarial network is shown. The generative adversarial network includes a generator and a discriminator. The method includes providing 102 data sample pairs. An example of a data sample may be an image. However, other types of data samples may also be provided, such as sound data, text data, etc.

[0061] Each pair includes a base data sample having a pattern and a modified data sample having at least one corresponding modification pattern. The modification may involve, for example, a complete line, or involve, for example, not having a table pattern. Various modification patterns may add additional elements to the base data sample, or may reduce information from the base data sample. For example, the modification pattern may be determined by applying random modifications to the base data sample using a random number generator.

[0062] The method 100 also includes training 104 a generator using an adversarial training method in conjunction with the discriminator and using the data sample pairs as input to build a model of the generator. The discriminator receives a data set pair of the data set as input. Each data set pair includes a predicted output of the generator based on a base data sample and a corresponding modified data sample. Thus, a joint loss function for the generator and the discriminator can be optimized.

[0063] Last but not least, the method 100 includes predicting 106 an output data set for unknown data samples as input to a generator without a discriminator (i.e., removing the discriminator), and can be implemented for production use using only the generator network.

[0064] Figure 2The general setup of the system proposed herein used by the proposed method is shown. A generator network 202 with a deconvolutional neural network is trained with training data 204 including, for example, related image pairs 206, 208. The deconvolutional neural network 202 includes substantially the same number of nodes on the input side 210 (left) and the output side 212 (right). The number of nodes in the layers between the input layer 210 and the output layer 212 is relatively small.

[0065] The discriminator 214 (e.g., implemented as a convolutional neural network) includes substantially the same number of input nodes 216 as the generator network 202 has output nodes 212.

[0066] First, two datasets 206, 208 with paired related images must be synthetically generated for training. These datasets include modifications of the patterns of interest. Next, the GAN model is trained using these datasets until it is able to perform the desired modifications. Training continues until the requirements for the loss-based training stopping criteria are met. Once the GAN model is ready, the generator 202 is separated from the discriminator 214 for production use, and new pictures are sent to the generator 202 and modified to be output at the output layer 212.

[0067] It can be noted that the discriminator network 214 is shown as having two output nodes. It can be used to indicate whether the input data to the discriminator 214 is identified as artificially generated by the generator 202, or whether the input data to the discriminator 214 (which is the same as the output of the generator 202 during training) is the original unmodified data from the training data set. During training, the discriminator 214 receives the output data of the generator 202 and the original base data samples as input.

[0068] Figure 3 The main processing steps 300 are shown. During the generation 302 of the synthetic data set, the first step is to identify 304 and define the patterns that should be modified. This task should be the output of a data symbol study on the sample set that needs to be modified. In the case where the significance definition can be based on a threshold, this study can highlight the most important patterns in the sample. The first initialization of this process must be performed once at the beginning.

[0069] The pattern is then used to generate at least two corresponding synthetic data sets. The first synthetic data set includes the pattern and the second data set includes the modified pattern. The generation 306 of the synthetic data sets is based on the unique features of the initial sample set; that is, the features are changed by a pseudo-random number generator and added together to synthesize a temporary data set 308. The temporary data set is then discarded and an important feature is added (or removed) to one of the features (but not the second), the addition being a modification of the important feature.

[0070] It is useful to also include cases in the formation of the data set that do not necessarily need to be modified, so that the process learns to cope with features that do not necessarily need to be changed.

[0071] The training 310 of the GAN is performed in the following way: the input of the GAN is a set of images without modifications in the target features, and conversely, the data that the GAN learns to reproduce is from a set with modified features. It is worth noting that in this step, the input data is always coordinated in the data to be reproduced so that the only features that are changed are the ones that should be modified. Therefore, the remaining features have the same values ​​(controlled by random numbers) for all other features in both the input data and the version to be reproduced later.

[0072] Once training is complete, the generator portion of the GAN is used to reproduce 312 new samples having modifications from the input samples that include the portions of these features that were extracted during separate generation. Thus, training of the GAN does not require any labeled or annotated data, which represents a significant reduction in manual effort.

[0073] As mentioned above, depending on the application of the processing, some of the required corrections (i.e. modifications) may not be completed in one step. In this case, the processing can be continuously extended and the modifications can be done one by one, modifying one pattern at a time. To do this, a generator trained in a different way is needed.

[0074] Before moving on to examples of real training and result data, the adaptability of the used GAN architecture should be carefully studied.

[0075] The creation of the GAN architecture was inspired by several recent architectures, such as Pix2Pix, SuperResolution GAN, and in order to improve performance, improve training convergence, and separate the architectures of the generator and discriminator, the Wasserstein loss function was used instead of the adversarial function. This novel combination leads to better results than any individual component in the standard architecture. The following details outline the technical details of the foundation of the GAN network used:

[0076] The generator is similar to U-NET (a special form of fully convolutional network). The architecture is inspired by deep convolutional autoencoders and has the additional important advantage of skipping the connections between the corresponding symmetric convolutional layers in the encoding and decoding parts. This allows the relevant low-level information to be passed directly to the deeper layers, making training more efficient because the skipped layers facilitate gradient backpropagation and overcome the gradient vanishing problem of deeper networks. The absence of dense layers provides the flexibility to test on input shapes different from those used for training.

[0077] The network of the discriminator is a PatchGAN, which is a series of convolutional layers with batch normalization. The difference between this network and a regular discriminator is that instead of mapping the input to a single number (which corresponds to the probability that the input is true), the input is mapped to NxN blocks. Depending on the loss function used, each scalar value of the output block is able to classify whether the block of input corresponding to its receptive field is true (for adversarial loss) or how true it is (Wasserstein loss). The progress of this network is first of all the reduction in the number of required parameters (which leads to savings in memory and time), as well as the fact that it is possible to apply this architecture to arbitrary input shapes, a requirement needed to achieve a general and flexible approach in applications.

[0078] In the proposed architecture, a weighted combination of different loss functions is applied. First, knowing the fact that pixel-wise MSE (mean square error) and MAE (mean average error) losses tend to produce smoother and blurrier results, the content loss used is based on feature maps of pre-trained networks such as VGG19 (i.e., a CNN trained on more than a million images from the ImageNet database) or ResNet-50 (also a CNN with 50 layers enabled to classify images into 1000 object categories).

[0079] In addition, instead of using the traditional adversarial loss that is the typical cross entropy of the generator and the discriminator, this paper uses the Wasserstein loss function with gradient penalty, which corresponds to the bulldozer distance between the expected data distribution and the generated data distribution. There are two reasons for choosing to introduce this loss function into the architecture proposed in this paper: first, when the critic is trained to the best, this loss function enhances training stability and achieves faster convergence without the gradient vanishing problem. This is an important aspect considering the fact that GAN training is fragile and balanced training between the discriminator and the generator is usually difficult to achieve. The second reason lies in the following observation: traditional and latest GAN architectures (such as Pix2Pix and SRGAN) use adversarial losses and cannot map loss evolution to the quality of the generated samples. This is not the case in the implementation proposed in this paper. In fact, due to the nature of the Wasserstein loss, there is a correspondence between the loss and the output quality of the generator.

[0080] Figure 4An example 400 of a pattern 402 to be modified and ground truth data 404 of a modified output 1 and ground truth data 406 of a modified output 2 of a generator 408 is shown. The training set includes completely synthetic data generated using the above principles. More precisely, in the example shown in the accompanying drawings, it can be seen that the data pattern that needs to be modified as the input to the generator 408, and the desired modified pattern is used as the ground truth 404, 406 at the output of the generator. Here, the pattern 404 modified for output 1 with a single data set is: (i) converting dotted lines to solid lines once, (ii) filling in randomly missing parts in the lines, (iii) converting color image input (not shown) to black and white representation, and (iv) removing text (indirect detection). Output 2 406 includes all remaining parts of the input without any lines.

[0081] The figure below shows the performance achieved by the generator on a validation set that also includes images from the same synthetic dataset that have not been used for training.

[0082] Figure 5 The example 500 of converting a corrupted input image 502 into a modified output image 1 504 and a modified image output 2 506 by the generator 508 is clearly shown.

[0083] To test the generality and scalability of the proposed method, the generator was extracted after the GAN completed the training procedure as described above, and data modification was applied in three different datasets that have the exact regional correlation with the training center used for the synthetic dataset. However, the patterns under modification already exist in the training dataset.

[0084] Figure 6 is an example of a reconstruction of a phase map 600. In this case, the reconstruction of the map can be defined as successful when the dashed lines of the pattern to be modified 602 are converted to solid lines and the missing parts of the map are correctly filled, as shown in the reconstructed phase map 604. It can be seen that the generator 606 is able to perform the task almost perfectly.

[0085] Figure 7 Another example 700 involving a form is shown. Here, the task is to modify a form by "separating" a line from the text from the pattern 702 to be modified or "separating" a pattern 702 to be modified from a line from the text. Thus, depending on the desired output, the generator can be "perceived" as a form structure extractor (lines only) or a text extractor, as clearly seen in the form of form structure output 704 and form text output 706 produced by generator 708.

[0086] Figure 8 and 9Other examples 800, 900 of a very challenging dataset of very noisy scanned documents 802, 902 are shown. However, it turns out that the generator 804, 904 is unable to successfully perform the task because the input 802, 902 is different from the input of the training dataset due to the presence of high levels of noise.

[0087] Fig.10 and 11 Examples 1000, 1100 are shown, where by placing another generator 1004, 1104 in series with the generator 1006, 1106, which is trained to "separate" noise from the document, the text extraction performance is actually similar to the one of the examples discussed above. This demonstrates that the efficiency of the proposed method also applies to multiple steps with different generators in succession. Fig.10 , the pattern 1002 to be modified is filtered by a denoising generator 1004 and a text extraction generator 1006 to generate a form text output 1008.

[0088] exist Fig.11 , the pattern 1102 to be modified is first filtered by a denoising generator 1104 and then filtered by a text extraction generator 1106 to generate a form text output 1108.

[0089] It should also be noted that all content considered to be related to an image is not limited to the images described above by the examiner. In fact, other use cases can also be solved using the method proposed in this article (e.g., using acoustic data, using text data or other structured data).

[0090] For acoustic or voice data, the pattern to be modified may be, for example, language accent, so that the proposed method can be used to modify the accent in a recording.

[0091] For text data, the pattern to be modified may be, for example, the usage of specific expressions and sentences, so that the proposed method can change the specific expressions and sentences with other expressions, thereby maintaining the semantics in the sentences.

[0092] For unstructured data, the pattern to be modified could be, for example, the chronological transformation of the financial data of day x and day x + 1. The proposed method can then be applied at present (x = today) in order to predict tomorrow's trend.

[0093] For completeness reasons, Fig.12An embodiment of a machine learning system 1200 for modifying patterns in a data set using a generative adversarial network 1202 is shown, the system comprising a generator network system 1204 and a discriminator network system 1206. The machine learning system 1200 comprises a receiving unit 1202 adapted to provide pairs of data samples. Each pair comprises a base data sample having a pattern and a modified data sample having a corresponding modified pattern. Thus, the modified pattern is determined by applying at least one random modification to the base data sample.

[0094] The system further includes a training module 1208, which is adapted to control the use of an adversarial training method and use data sample pairs as input to train the generator network system to build a model of the generator network system. The discriminator network system 1206 receives a data set pair of data sets as input. Each data set pair includes a predicted output of the generator network system 1202 based on a base data sample and a corresponding modified data sample; thereby, the joint loss function of the generator and the discriminator can be optimized.

[0095] Last but not least, the system 1200 comprises a prediction unit 1210 adapted to predict an output data set for unknown data samples as input to the generator without a discriminator.

[0096] Embodiments of the invention can be implemented with nearly any type of computer, regardless of the platform being suitable for storing and / or executing program code. Fig.13 As an example, a computing system 1300 suitable for executing program code related to the proposed method is shown.

[0097] The computing system 1300 is only one example of a suitable computer system and is not intended to imply any limitation on the scope of use or functionality of the embodiments of the present invention described herein, and is not related to whether the computer system 1300 can be implemented and / or perform any of the functions described above. In the computer system 1300, there are components that can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system / server 1300 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems or devices, etc. The computer system / server 1300 can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system 1300. Generally, a program module can include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 1300 may be implemented in a distributed cloud computing environment where remote processing devices linked through a communications network perform tasks. In a distributed cloud computing environment, program modules may be located on both local and remote computer system storage media including memory storage devices.

[0098] As shown in the figure, computer system / server 1300 is shown in the form of a general computing device. The components of computer system / server 1300 may include, but are not limited to: one or more processors or processing units 1302, system memory 1304, and a bus 1306 that connects different system components including system memory 1304 to processor 1302. Bus 1306 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any bus structure in a variety of bus structures. For example, these architectures include, but are not limited to, an industrial standard architecture (ISA) bus, a microchannel architecture (MAC) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus. Computer system / server 1300 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer system / server 1300, including volatile and non-volatile media, removable and non-removable media.

[0099] System memory 1304 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1308 and / or cache memory 1310. Computer system / server 1300 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1312 may be used to read and write non-removable, non-volatile magnetic media (not shown in the figure, usually referred to as a "hard drive"). Although not shown in the figure, a disk drive for reading and writing removable non-volatile disks (such as "floppy disks") and an optical disk drive for reading and writing removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media) may be provided. In these cases, each drive may be connected to bus 1306 via one or more data media interfaces. Memory 1304 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of various embodiments of the present invention.

[0100] A program / utility having a set (at least one) of program modules 1316 may be stored, for example, in memory 1304, such program modules 1316 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 1316 generally perform the functions and / or methods of the embodiments described herein.

[0101] The computer system / server 1300 may also communicate with one or more external devices 1318 (e.g., keyboards, pointing devices, displays 1320, etc.), may communicate with one or more devices that enable a user to interact with the computer system / server 1300, and / or may communicate with any device that enables the computer system / server 1300 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interfaces 1314. Furthermore, the computer system / server 1300 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapters 1322. As shown, the network adapters 1322 communicate with other modules of the computer system / server 1300 via bus 1306. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer system / server 1300, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0102] Additionally, a machine learning system 1200 for modifying patterns in a data set using a generative adversarial network can be attached to the bus system 1306 .

[0103] The description of the embodiments of the present invention has been provided for illustrative purposes, and the above description is not exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to technologies in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0104] The present invention may be embodied as a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0105] The medium may be an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system or propagation medium. Examples of computer readable media may include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk and optical disk. Current examples of optical disks include compact disk - read only memory (CD-ROM), compact disk - read / write (CD-R / W), DVD and Blu-ray disk.

[0106] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0107] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0108] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0109] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or another device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or another device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] The flow chart and / or block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square frame in the block diagram and / or flow chart, and the combination of the square frames in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the specified function or action, or can be realized by a combination of dedicated hardware and computer instructions.

[0113] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of claimed features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0114] The corresponding structures, materials, operations and equivalent replacements of all functionally defined devices or steps in the following claims are intended to include any structure, material or operation for performing the function in combination with other units specifically indicated in the claims. The description of one or more aspects of the invention given is exemplary and descriptive, not exhaustive, and is not limited to the disclosed form of the invention. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The selection and description of the embodiments are intended to best explain the principles of the invention, the practical application, and to enable other technicians in the field to understand the various embodiments of the invention with various modifications when suitable for the specific use contemplated.

Claims

1. A computer-implemented method for modifying patterns in a data set, the method using a generative adversarial network comprising a generator and a discriminator, the method comprising: providing data sample pairs, each of the pairs comprising a base data sample and a modified data sample; The generator is trained using an adversarial training method to build a model of the generator, the adversarial training method inputting two samples of the data sample pair into the generator, wherein the base data sample has a plurality of patterns, wherein the modified data sample has a first pattern of the plurality of patterns, the first pattern is randomly modified compared to the base data sample, wherein the random modification includes randomly removing a pattern from the first pattern, wherein the modified data sample has at least one pattern of the plurality of patterns, the at least one pattern is unmodified compared to the base data sample, wherein the modified pattern is determined by applying at least one random modification to the base data sample, wherein the discriminator receives a dataset pair of datasets as input, each of the dataset pairs comprising a predicted output of the generator based on a base data sample and a corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, and In the absence of the discriminator, predicting the output data set for unknown data samples as input to the generator, The basic data samples and modified data samples are image data, or sound data, or text data.

2. The method according to claim 1, wherein: The joint loss function is the Wasserstein loss function.

3. The method according to claim 1, further comprising: The adversarial training method is used and different models of the generator are trained using the data sample pairs as input, wherein the modified data samples are modified according to different aspects.

4. The method according to claim 1, wherein: The generator is a neural network with the same number of output nodes as input nodes and fewer hidden layer nodes than the number of input nodes.

5. The method according to claim 1, wherein: The discriminator is a neural network having as many input nodes as the generator has output nodes and having two output nodes.

6. The method according to claim 1, wherein: The discriminator is PatchGAN.

7. The method according to claim 1, wherein: The joint loss function is a weighted combination of loss functions.

8. The method according to claim 1, wherein: The loss function is related to the content loss of the base data sample, and wherein the content loss is determined using a feature map of a pre-trained neural network.

9. The method according to claim 1, wherein: Compared to the related data samples, the modified data samples include solid lines instead of dashed lines, black and white modes instead of equivalent color modes, no text modes instead of modes with text, and no line images instead of mixed line / text images.

10. The method according to claim 1, wherein: The provided data sample pairs include: Provides an image set with a pattern, determining at least one mode to be modified, randomly modifying the at least one pattern of the image using a random number generator, and An image in the set of images and related images are correlated with the at least one pattern defining one of the pairs including the base data sample and the modified data sample.

11. The method according to claim 1, wherein: When comparing the results of the current iteration with the previous iteration, if the result of the joint loss function is less than a relative threshold, the training of the generative adversarial network is terminated.

12. A machine learning system for modifying patterns in a data set using a generative adversarial network, the machine learning system comprising a generator network system and a discriminator network system, the machine learning system comprising: a receiving unit adapted to provide pairs of data samples, each said pair comprising a base data sample and a modified data sample; a training module adapted to control the use of an adversarial training method to train the generator network system to construct a model of the generator network system, the adversarial training method inputting two samples of the data sample pair into the generator, wherein the base data sample has a plurality of patterns, wherein the modified data sample has a first pattern of the plurality of patterns, the first pattern is randomly modified compared to the base data sample, wherein the random modification includes randomly removing a pattern from the first pattern, wherein the modified data sample has at least one pattern of the plurality of patterns, the at least one pattern is unmodified compared to the base data sample, wherein the modified pattern is determined by applying at least one random modification to the base data sample, wherein the discriminator network system receives a dataset pair of the dataset as input, each of the dataset pairs comprising a predicted output of the generator based on a base data sample and a corresponding modified data sample, thereby optimizing a joint loss function for the generator and the discriminator, and a prediction unit adapted to predict an output data set for unknown data samples as input to the generator in the absence of the discriminator, The basic data samples and modified data samples are image data, or sound data, or text data.

13. The system according to claim 12, wherein: The joint loss function is a Wasserstein loss function, and / or The system uses the adversarial training method and uses the data sample pairs as input to train different models of the generator network system, wherein the modified data samples are modified according to different aspects.

14. The system according to claim 12, wherein: The generator network system is a neural network having the same number of output nodes as input nodes and fewer hidden layer nodes than the number of input nodes, or The discriminator network system is a neural network having the same number of input nodes as the output nodes of the generator and two output nodes.

15. The system of claim 12, wherein: The discriminator is PatchGAN.

16. The system of claim 12, wherein: The loss function is related to the content loss of the base data sample, and wherein the content loss is determined using a feature map of a pre-trained neural network.

17. The system of claim 12, wherein: The provided data sample pairs include: Provides an image set with a pattern, determining at least one mode to be modified, randomly modifying the at least one pattern of the image using a random number generator, and An image in the set of images and related images are correlated with the at least one pattern defining one of the pairs including the base data sample and the modified data sample.

18. A computer-readable storage medium having program instructions embodied thereon, wherein the program instructions can be executed by one or more computing systems or controllers to cause the one or more computing systems or controllers to perform the method according to any one of claims 1 to 11.

19. A computer system comprising: one or more processors; A computer-readable storage medium coupled to the one or more processors, the computer-readable storage medium comprising instructions that, when executed by the one or more processors, perform the method according to any one of claims 1 to 11.

20. A system for modifying a pattern in a data set, the system comprising modules respectively configured to perform each step of the method according to any one of claims 1 to 11.

Images