Method and device related to data generation framework

Abstract

The invention provides a method and a device related to a data generation framework. A first code is obtained from reference data by a style encoder. A second code is obtained from the potential code through the mapping network, the first code and the second code are style codes, and each style code corresponds to one or more style options; inputting the first source data to the generator, and outputting a first output corresponding to the first source data by referring to the style code through the generator; the first output is input to the discriminator, and a second output corresponding to the first output is output through the discriminator, where the second output indicates whether the first output corresponds to the style option. Thereby, the source content may be retained, and data corresponding to the style options may be generated.

Description

Technical Field

[0001] The present invention relates to a data generation technology, and in particular to a method and device related to a data generation framework. Background Art

[0002] Figure 1 This is a schematic diagram of the simulation situation. Figure 1 In the prior art, the simulator can generate corresponding images according to style conditions. For example, the style conditions are sunny noon, foggy noon and rainy night. Alternatively, the simulator can generate corresponding images according to spatial conditions. For example, the spatial conditions are semantic masks and depth maps that define corresponding categories of image areas. However, existing simulators can only generate fake images in virtual situations, and are even difficult to use in applications involving real scenes. For example, applications such as autonomous driving scene understanding and component defect detection. In addition, the design of three-dimensional objects is to design objects that are close to the real world in order to reduce the gap between the content generated by the simulator and the real world. However, designing diverse three-dimensional objects requires considerable costs and resources. Summary of the Invention

[0003] The present invention is directed to a method and apparatus for a data generation framework that can generate more realistic data based on customizable conditions.

[0004] According to an embodiment of the present invention, a method related to a data generation framework is implemented by a processor and includes (the following steps): obtaining a first code from reference data through a style encoder; obtaining a second code from a latent code through an image network, wherein the first code and the second code are both style codes, and each style code corresponds to one or more style options; inputting first source data to a generator, and outputting a first output corresponding to the first source data through the generator with reference to the style code; and inputting the first output to a discriminator, and outputting a second output corresponding to the first output through the discriminator, wherein the second output indicates whether the first output corresponds to a style option.

[0005] According to an embodiment of the present invention, an apparatus related to a data generation framework includes (but is not limited to) a memory and a processor. The memory is used to store program code. The processor is coupled to the memory. The processor is configured to load the program code to execute: obtaining a first code from reference data via a style encoder; obtaining a second code from a latent code via an image network, wherein both the first code and the second code are style codes, and each style code corresponds to one or more style options; inputting first source data to a generator, and outputting a first output corresponding to the first source data by the generator with reference to the style code; and inputting the first output to a discriminator, and outputting a second output corresponding to the first output by the discriminator, wherein the second output indicates whether the first output corresponds to a style option.

[0006] Based on the above, the method and apparatus for the data generation framework according to the embodiments of the present invention provide high-quality, realistic, and reliable data generation by enabling the generator to learn how to use style codes to transform input data into real data and using the discriminator to determine whether the input data is real data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present invention and together with the description serve to explain the principles of the present invention.

[0008] Figure 1 It is a schematic diagram of the simulation situation;

[0009] Figure 2 is a block diagram of components of an apparatus related to a data generation framework according to one embodiment of the present invention;

[0010] Figure 3 is a flowchart related to spatial learning of a data generation framework according to one embodiment of the present invention;

[0011] Figure 4 is a schematic diagram of spatial learning related to a data generation framework according to one embodiment of the present invention;

[0012] Figure 5 is a flow chart of spatial noise removal related to a data generation framework according to one embodiment of the present invention;

[0013] Figure 6 is a schematic diagram of spatial noise removal related to a data generation framework according to one embodiment of the present invention;

[0014] Figure 7 is a schematic diagram illustrating noise removal according to an embodiment of the present invention;

[0015] Figure 8is a flowchart of multiple domain conditional learning related to a data generation framework according to one embodiment of the present invention;

[0016] Figure 9 is a schematic diagram of multi-domain conditional learning related to a data generation framework according to one embodiment of the present invention;

[0017] Figure 10 is a schematic diagram of a fusion layer according to one embodiment of the present invention;

[0018] Figure 11 is a schematic diagram of a generator according to one embodiment of the present invention;

[0019] Figure 12A is a schematic diagram of style similarity error according to one embodiment of the present invention;

[0020] Figure 12B is a schematic diagram of a source preservation error according to an embodiment of the present invention;

[0021] Figure 12C is a schematic diagram of content comparison error according to one embodiment of the present invention;

[0022] Figure 13 is a schematic diagram of style options according to one embodiment of the present invention;

[0023] Figure 14 is a flowchart of inference of multi-domain conditions related to a data generation framework according to one embodiment of the present invention;

[0024] Figure 15 is a schematic diagram of inference of multi-domain conditions related to a data generation framework according to one embodiment of the present invention;

[0025] Figure 16A is a schematic diagram of scene image generation according to one embodiment of the present invention;

[0026] Figure 16B is a schematic diagram of facial image generation according to one embodiment of the present invention;

[0027] Figure 17A and Figure 17B is an experimental result of a spatially conditioned pipeline according to an embodiment of the present invention;

[0028] Figure 18 1 is an experimental result of a multi-domain conditional pipeline according to an embodiment of the present invention.

[0029] Explanation of Figure Numbers

[0030] 100: device;

[0031] 110: memory;

[0032] 120: processor;

[0033] 1000, S310, S320, S330, S340, S510, S520, S530, S810, S820, S830, S840, S1401, S1402: steps;

[0034] 401: training data;

[0035] 451: initial encoder;

[0036] 402: potential performance;

[0037] 411: Gaussian distribution;

[0038] 412: first noise data;

[0039] 421: Potential performance under noise;

[0040] 422: initial prediction;

[0041] 430: prediction model;

[0042] 431, D01, E01, M01: encoder;

[0043] 435,651,G02:decoder;

[0044] 432,G11: coding block;

[0045] 433: downsampling;

[0046] 436,G12: decoding block;

[0047] 403, 404: First semantic shielding data;

[0048] 453: Change size;

[0049] 437: upsampling;

[0050] 423: predicted mean;

[0051] 424: prediction variance;

[0052] 440: prediction error;

[0053] 441: first error;

[0054] 442: second error;

[0055] 611,1511:Gaussian distribution;

[0056] 601: second noise data;

[0057] 603,604: Second semantic shielding data;

[0058] 625,701-704: first generated data;

[0059] 621,622: prediction noise;

[0060] 623,624: Denoised data;

[0061] G: generator;

[0062] D: discriminator;

[0063] E: style encoder;

[0064] M: Image network;

[0065] 121,123,128,901: first source data;

[0066] 125,129,902: reference data;

[0067] 1631, E11~E13, M11~M13: style options;

[0068] 912,1512,9121,9123: style code;

[0069] 903: Latent coding;

[0070] G01: encoder;

[0071] 905,1505: content features;

[0072] 921: first output;

[0073] G03: fusion layer;

[0074] 1001: fusion block;

[0075] 1002, D11~D13: predictor;

[0076] 111: feature data;

[0077] 112: Input;

[0078] 941: style error;

[0079] LSS: style similarity error;

[0080] LSP: Source preservation error;

[0081] LCC: content comparison error;

[0082] 122: second source data;

[0083] 127: second generated data;

[0084] 124,1602,1603,1703,1706: generate data;

[0085] 9122: second style code;

[0086] 126: second reference data;

[0087] 9051: first characteristic manifestation;

[0088] 9052: Second characteristic manifestation;

[0089] PP1, PP2: positive pair;

[0090] MP: reverse pair;

[0091] 9053: The third characteristic manifestation;

[0092] 130: third generated data;

[0093] 1501: third source data;

[0094] 1502: second potential code;

[0095] 1503: fourth generated data;

[0096] 1601,1604: semantic masking data;

[0097] 1611: data generator;

[0098] 1612: Conditional Fusion;

[0099] 1605: Generate data;

[0100] 1632:Style options;

[0101] 1606: Generate data;

[0102] 1701,1704: semantic masking data;

[0103] 1702,1705: True labels. DETAILED DESCRIPTION

[0104] Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the description to refer to the same or like parts.

[0105] Figure 2FIG is a block diagram of components of an apparatus 100 related to a data generation framework according to an embodiment of the present invention. Figure 2 , device 100 includes (but is not limited to) a memory 110 and a processor 120. Device 100 can be a mobile phone, tablet computer, laptop computer, desktop computer, server, voice assistant device, smart home appliance, wearable device, vehicle system or other electronic device.

[0106] The memory 110 can be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or similar device. In one embodiment, the memory 110 is used to store program code, software modules, configurations, data (e.g., model parameters, data sets, samples, features, or predictions), or files, as will be described in detail in subsequent embodiments.

[0107] The processor 120 is coupled to the memory 110. The processor 120 may be a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), a programmable controller, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a neural network processor (NPU), a tensor processing unit (TPU), an artificial intelligence (AI) accelerator, a neural network engine, or other similar components or a combination of the above components. In one embodiment, the processor 120 is used to perform all or part of the operations of the device 100 and can load and execute various program codes, software modules, files, and data stored in the memory 110.

[0108] Hereinafter, the method according to the embodiment of the present invention will be described with reference to various devices, components, and modules in the device 100. Each process of the method can be adjusted according to the implementation situation, and is not limited thereto.

[0109] Figure 3 This is a flowchart of spatial learning related to a data generation framework according to one embodiment of the present invention. Figure 3 , the processor 120 obtains a latent representation from the training data through the initial encoder (step S310). Specifically, the training data can be images. For example, image samples obtained from a (training) dataset from various resource sources. For example, Figure 4 is a schematic diagram of spatial learning related to a data generation framework according to one embodiment of the present invention. Figure 4 , the training data 401 is an image of a street. In other embodiments, the training data may be sound, text, sensory intensity, angle, amplitude, position, or other forms of data.

[0110] In one embodiment, the initial encoder 451 is a vector quantized encoder (VQ encoder). The initial encoder 451 can divide the continuous high-dimensional vector space into several regions, and each region corresponds to a representative representation (for example, in the form of a (compressed) vector, code vector, or codeword). During the encoding process of the initial encoder 451, the processor 120 maps the input training data to the closest latent representation 402. The initial encoder 451 can map high-dimensional data to a low-dimensional space while retaining important information about the data. This low-dimensional representation is generally referred to as the latent representation 402.

[0111] In another embodiment, the initial encoder 451 is another encoder for dimensionality reduction or compression.

[0112] Please refer to Figure 3 , the processor 120 combines the potential performance and the first noise data to generate the noisy potential performance (step S320). Specifically, the first noise data can be a numerical value based on a statistical distribution such as Gaussian distribution, uniform distribution, or Poisson distribution, or data generated based on noise in a real environment. Figure 4 For example, the processor 120 samples the Gaussian distribution 411 and generates the first noise data 412 based on the sampled values. Then, the processor 120 adds the potential performance 402 and the first noise data 412 and uses the addition result as the noisy potential performance 421.

[0113] In one embodiment, the potential representation and the first noise data are in vector form. Processor 120 may perform element-wise addition of the potential representation and the first noise data, and use the result of the addition as the noisy potential representation. For example, the first element of the potential representation is added to the first element of the first noise data to form the first element of the noisy potential representation. The second element of the potential representation is added to the second element of the first noise data to form the second element of the noisy potential representation. The rest of the process is analogous and will not be further described herein.

[0114] Please refer to Figure 3 , the processor 120 inputs the noisy potential representation into the prediction model, and outputs the initial prediction corresponding to the noisy potential representation by referring to the semantic mask data through the prediction model (step S330). Specifically, in the training phase, the noisy potential representation and its ground truth (e.g., the first noise data) are used as training samples of the prediction model. The processor 120 can train the prediction model through a machine learning algorithm. The machine learning algorithm can analyze the labeled training samples (e.g., the noisy potential representation with the corresponding ground truth label) to establish the potential representation embedded / added with noise (i.e., the input of the model, such as Figure 4 The noisy potential representation 421 shown is compared with the embedded / added noise (i.e., the output of the model, such as Figure 4 The prediction model can be learned and used to infer the data to be evaluated (e.g., potential performance or other data to be evaluated) to output the initial prediction corresponding to the data to be evaluated (e.g., noise data in the potential performance or other data).

[0115] The type of machine learning algorithm can vary depending on the application scenario. The machine learning algorithm can be, but is not limited to, Semantic Latent Diffusion, Latent Diffusion, or Stochastic Diffusion.

[0116] Figure 4 The prediction model 430 is exemplified by the semantic latent diffusion network. The prediction model 430 includes an encoder 431 and a decoder 435. The encoder 431 is connected to the decoder 435. For example, the output of the encoder 431 (e.g., a feature map) serves as the input of the decoder 435. In one embodiment, for example, the encoder 431 and the decoder 435 are configured in a U-Net connection architecture.

[0117] Encoder 431 includes one or more encoding blocks 432. Encoding block 432 is, for example, a diffusion encoder residual block (DER). The semantic latent diffusion residual block is a core component of the encoder portion of the semantic latent diffusion model. Its primary function is to encode or extract features from noisy input data (e.g., the noisy latent representation 421). Encoding block 432 may include, but is not limited to, a convolutional layer, an activation function, a residual connection, and normalization processing.

[0118] In one embodiment, encoder 431 includes multiple encoding blocks 432. Processor 120 may downsample 433 the output of encoding block 432 to obtain data of smaller size or lower resolution. In other words, downsampling 433 is used to reduce the size or resolution. This smaller size or lower resolution data may be input to the next encoding block 432. In other words, these encoding blocks 432 correspond to data of different sizes or resolutions.

[0119] The decoder 435 includes one or more decoding blocks 436. The decoding block 436 is, for example, a diffusion decoding residual block (DDResblock). The semantic latent diffusion residual block is a core component of the encoder part in the semantic latent diffusion model. Its main function is to combine the features (or feature maps) obtained by the encoder 431 with semantic information (for example, the first semantic masking data 403, 404, which will be introduced later). The decoding block 436 includes, for example, a deconvolutional layer, a convolutional layer, and normalization processing, but is not limited to this.

[0120] In one embodiment, decoder 435 includes multiple decoding blocks 436. Processor 120 may up-sample 437 the output of decoding block 436 to obtain data of larger size or higher resolution. In other words, upsampling 437 is used to increase the size or resolution. This larger size or higher resolution data may be input to the next decoding block 436 or used as the output of decoder 435 or prediction model 430 (i.e., initial prediction 422). In other words, each decoding block 436 corresponds to data of different sizes or resolutions.

[0121] The first semantic masking data 403 and 404 define one or more first semantic categories for the training data 401. The first semantic masking data 403 is labeled data of the same size or resolution as the training data 401. The first semantic masking data 403 and 404 are composed of multiple blocks, elements, or pixels. Taking an image as an example, each pixel in the first semantic masking data 403 and 404 is assigned a predefined (first) semantic category (also known as a label or semantic information). For example, in a scene application, the semantic category can be lane, car, pedestrian, building, or sky.

[0122] In one embodiment, the processor 120 may generate the first semantic masking data 403 using a direct correspondence generator. The generator may generate the first semantic masking data 403 randomly or based on a rule. In another embodiment, the processor 120 may receive a user operation and define one or more first semantic categories in the first semantic masking data 403 and the first semantic categories corresponding to the blocks, elements, or pixels in the first semantic masking data 403 based on parameters corresponding to the user operation.

[0123] In one embodiment, the processor 120 may generate the first semantic mask data 404 at multiple sizes (or multiple resolutions) (i.e., by varying the size 453 or resolution of the first semantic mask data 403). The size or resolution of the first semantic mask data 404 is smaller or lower than the size or resolution of one or more of the first semantic mask data 403. The first semantic mask data 404 of a certain size is aligned with the size of the feature data (e.g., the output of the encoder 431 or the upsampled feature map of another decoding block 436) input to the corresponding decoding block 436, and both the first semantic mask data 404 of the same size and the feature data are input to the decoding block 436. In one embodiment, in response to the decoder 435 including multiple decoding blocks 436, the first semantic mask data 404 of multiple sizes or resolutions are respectively input to the same or corresponding decoding blocks 436, thereby increasing the sensitivity of the prediction model 430 to semantic information of multiple resolutions or sizes.

[0124] In one embodiment, the decoder 435 or the decoding block 436 may adaptively adjust the mean and variance (corresponding to the variation range of the mean) of the input feature map according to the first semantic masking data 403 .

[0125] In one embodiment, the initial prediction 422 includes a predicted mean 423 and a predicted variance 424 corresponding to the predicted mean 423. The initial prediction 422 or the predicted mean 423 is, for example, noise data predicted by the prediction model 430 for the noisy potential performance 421 (e.g., noise data predicted to be added or embedded in the noisy potential performance 421 or first noise data predicted).

[0126] Please refer to Figure 3 , the processor 120 updates the prediction model according to the prediction error between the initial prediction and the first noise data to generate a trained prediction model (step S340). Specifically, one of the multiple objectives of the training phase is to minimize the loss function (Loss function (related to the output of the prediction model (i.e., Figure 4 The error / loss between the initial prediction 422) and the true label in the training sample (for example, the first noise data 412). In one embodiment, during the training phase of the prediction model, the parameters of the prediction model are recursively updated by minimizing the loss function. For example, the back-propagation method. The parameters of the model are, for example, weights, number of layers, location or number of neurons, activation function or offset, but are not limited thereto. The method of updating the parameters is, for example, through gradient descent, adaptive moment estimation (Adam) optimizer, momentum method, adaptive gradient (Adagrad), or conjugate gradient method, but is not limited thereto. That is, one of the multiple goals of the training phase is to make the initial prediction output by the prediction model close to or the same as the corresponding true label. In one embodiment, a trained prediction model means that its loss function has converged, the prediction accuracy has reached the corresponding threshold, or the training has reached the standard for early stopping of training, but the standard for completion of training can still be adjusted according to other tasks or needs.

[0127] Please refer to Figure 4 The loss function is based on a prediction error 440 between the initial prediction 422 and the first noise data 412. In one embodiment, the prediction error 440 is related to a first error 441 and a second error 442. For example, the first error 441 and the second error 442 are the sum of the first error 441 and the second error 442. The processor 120 may calculate the difference between the prediction mean 423 and the mean of the first noise data 412 to determine the first error 441 (e.g., a first-order loss (L1 loss)). The mean of the first noise data 412 is, for example, the mean of the Gaussian distribution 411. The first error 441 is, for example, the difference between the prediction mean 423 and the mean of the Gaussian distribution 411. The processor 120 may calculate the difference between the prediction variance 424 and the variance of the first noise data 412 to determine the second error 442 (e.g., a KL divergence). The variance of the first noise data 412 is, for example, the variance of the Gaussian distribution 411. The second error 442 is, for example, a difference between the prediction variance 424 and the variance of the Gaussian distribution 411 .

[0128] Figure 5 This is a flowchart of spatial noise removal related to a data generation framework according to one embodiment of the present invention. Figure 5 The processor 120 inputs the second noise data into the trained prediction model, and outputs predicted noise corresponding to the second noise data by referring to the second semantic masking data through the trained prediction model (step S510). Specifically, the second noise data can be a numerical value based on a statistical distribution such as a Gaussian distribution, a uniform distribution, or a Pashon distribution, or data generated based on noise in a real environment. Figure 6 FIG. 1 is a schematic diagram of spatial noise removal related to a data generation framework according to an embodiment of the present invention. Figure 6 For example, the processor 120 samples the Gaussian distribution 611 and generates the second noise data 601 based on the sampled values. Then, the processor 120 uses the second noise data 601 as input to the trained prediction model 430. In other words, the second noise data 601 is input to the trained prediction model 430.

[0129] The second semantic masking data 603 defines one or more second semantic categories for the first generated data 625. The second semantic masking data 603 is labeled data of the same size or resolution as the first generated data 625. The second semantic masking data 603 and 604 are composed of multiple blocks, elements, or pixels. Taking image form as an example, each pixel in the second semantic masking data 603 and 604 is assigned a predefined (second) semantic category (or label, semantic information). Taking scene applications as an example, the semantic category can be lane, car, pedestrian, building, or sky. Taking face applications as an example, the semantic category can be black, nose, double eyelids, or curly hair. The first generated data 625 is the data to be generated. The first generated data 625 can be in image form. In other embodiments, the first generated data 625 can be sound, text, sensor intensity, angle, amplitude, position, or other forms of data.

[0130] In one embodiment, the processor 120 may generate the second semantic masking data 603 using a direct correspondence generator. This generator may generate the second semantic masking data 603 randomly or based on a rule. In another embodiment, the processor 120 may receive a user operation and define one or more second semantic categories in the second semantic masking data 603 and the second semantic categories corresponding to the blocks, elements, or pixels in the second semantic masking data 603 based on parameters corresponding to the user operation.

[0131] In one embodiment, the processor 120 may generate the second semantic mask data 604 at multiple sizes (or multiple resolutions) (i.e., by varying the size 453 or resolution of the second semantic mask data 603). The size or resolution of the second semantic mask data 604 is smaller or lower than the size or resolution of one or more of the second semantic mask data 603. The second semantic mask data 604 of a certain size is aligned with the size of the feature data (e.g., the output of the encoder 431 or the upsampled feature map of another decoding block 436) input to the corresponding decoding block 436, and both the second semantic mask data 604 of the same size and the feature data are input to the decoding block 436. In one embodiment, in response to the decoder 435 including multiple decoding blocks 436, the second semantic mask data 604 of multiple sizes or resolutions are respectively input to the same or corresponding decoding blocks 436, thereby increasing the sensitivity of the prediction model 430 to semantic information of multiple resolutions or sizes.

[0132] The decoder 435 or the trained prediction model 430 outputs a predicted noise 621 corresponding to the second noise data 601. The predicted noise 621 includes a mean and a variance corresponding to the mean. The processor 120 may generate a predicted noise 622 based on the predicted noise 621. For example, the predicted noise 622 is n+(e 0.5v *0.5), where n is the mean of the prediction noise 621 and v is the variance of the prediction noise 621.

[0133] Please refer to Figure 5 , the processor 120 generates denoised data according to the difference between the predicted noise and the second noise data (step S520). Assuming that the data is in vector form, the denoised data is the vector obtained by subtracting the predicted noise from the second noise data. Figure 6 For example, the processor 120 subtracts the predicted noise 622 from the second noise data 601 and generates denoised data 623 .

[0134] In one embodiment, the processor 120 may input the denoised data 623 into the trained prediction model 430 and, by reference to the second semantic masking data 604, output the predicted noise corresponding to the denoised data. Specifically, the processor 120 uses the denoised data 623 as input to the trained prediction model 430, or replaces the second noise data in step S510 with the denoised data 623. Next, the processor 120 proceeds to step S520. The processor 120 may repeat the above steps (i.e., inputting the denoised data 623 into the trained prediction model 430, outputting the predicted noise corresponding to the denoised data by reference to the second semantic masking data 604, and generating another denoised data 623 based on the difference between the predicted noise 622 and the second noise data 601) until a stopping condition is met. For example, the stopping condition may be repeating the above steps 1000 times, but this is not a limitation. The denoised data 624 is the denoised data generated when the stopping condition is met.

[0135] Please refer to Figure 5 , the processor 120 converts the denoised data into first generated data through the decoder corresponding to the initial encoder of step S310 (step S530). In one embodiment, the decoder corresponding to the initial encoder is a vector quantized decoder (VQ decoder). Figure 6 For example, the decoder 651 can convert the discrete potential representation / representation (e.g., in the form of a (compressed) vector, code vector, or codeword) generated by the vector quantization encoder back to the original data space to achieve data reconstruction or generation. During the decoding process of the decoder 651, the processor 120 takes the retrieved (compressed) vector, code vector, or codeword as input and converts it back to the original data space through a series of operations (e.g., deconvolution, convolution, etc.) to reconstruct the data. For example, the denoised data 624 is converted into the first generated data 625. The decoder 651 can map the low-dimensional data to a high-dimensional space and perform feature conversion.

[0136] In another embodiment, the decoder 651 is another decoder for dimension upscaling or decompression.

[0137] Figure 7 is a schematic diagram illustrating noise removal according to an embodiment of the present invention. Figure 7 , the first generated data 701, 702, 703, 704 are respectively steps repeated for different times (for example, Figure 6The denoised data 623 is input to the trained prediction model 430, and the trained prediction model 430 refers to the second semantic masking data 604 to output the predicted noise 622 corresponding to the denoised data 623, and generates another denoised data 623 based on the difference between the predicted noise 622 and the second noise data 601. The denoised data 624 generated is converted into generated data by the decoder 651. As the number of repetitions increases (for example, the number of repetitions corresponding to the first generated data 704 is greater than the number of repetitions corresponding to the first generated data 703, the number of repetitions corresponding to the first generated data 703 is greater than the number of repetitions corresponding to the first generated data 702, and the number of repetitions corresponding to the first generated data 702 is greater than the number of repetitions corresponding to the first generated data 701), the quality of the image gradually improves. In some application scenarios, by repeatedly denoising, the quality of the data can be effectively improved and also approach the real situation. For example, the first generated data 704 is a realistic street map.

[0138] Figure 8 This is a flowchart of the Multiple Domain Conditioned learning related to the data generation framework according to one embodiment of the present invention. Figure 8 , the processor 120 obtains a first code from the reference data through the style encoder (step S810). Specifically, Figure 9 Schematic diagram of multi-domain conditional learning related to the data generation framework according to one embodiment of the present invention. Figure 9 , the processor 120 can obtain a training pair (including the first source data 901 and the reference data 902) from the memory 110. The first source data 901 and the reference data 902 can come from different data sets. For example, taking scene generation as an example, the first source data 901 is a street map on a sunny day, and the reference data 902 is a street map on a rainy day. However, the content of the first source data 901 and the reference data 902 can still be changed according to actual needs. The first source data 901 and the reference data 902 can be in the form of images. In other embodiments, the first source data 901 and the reference data 902 can be sound, text, sensor intensity, angle, amplitude, position or other forms of data.

[0139] The style encoder E includes an encoder E01. The encoder E01 can be a shared encoder for multi-task learning. Multiple tasks share the same encoder E01, and the representation / performance learned by the encoder E01 can capture the common features between multiple tasks. These tasks correspond to different style options E11, E12, and E13. For example, the style option E11 with number 0 is sunny. The style option E12 with number y (y is a positive integer greater than 0) is rainy. The style option E13 is one of the style options E11 with number 0 to the style option E12 with number y. The first code is a style code 912 corresponding to the style option E13. That is, the encoder E01 obtains the feature representation / expression (for example, in vector or matrix form) generated for the style option E13 from the reference data 902.

[0140] In one embodiment, the style encoder E also corresponds to a task classifier or decoder, and this classifier or decoder is used to convert the output of encoder E01 into a prediction result for the corresponding task. For example, classifier 0 evaluates the effect of the sunny sky generated by encoder E01, which is converted into the corresponding style option E11. In other words, it assigns a score to the generated result for the corresponding style option.

[0141] Please refer to Figure 8 , the processor 120 obtains the second code from the latent code through the image network (step S820). Figure 9 , the image network M includes an encoder M01. The encoder M01 can be a shared encoder for multi-task learning. Multiple tasks share the same encoder M01, and the representation / performance learned by the encoder M01 can capture the common features between multiple tasks. These tasks correspond to different style options M11 and M13, for example. For example, the style option M11 of number 0 is daytime. The style option M13 is one of the style options M11 from the number 0 to the other number. For example, daytime is selected as the style option M13. The second code is a style code 912 corresponding to the style option M13. That is, the encoder M01 obtains the feature representation / expression (for example, in vector or matrix form) generated for the style option M13 from the latent code 903.

[0142] In one embodiment, the mapping network M also corresponds to a task classifier or decoder, and this classifier or decoder is used to convert the output of encoder M01 into a prediction result for the corresponding task. For example, classifier 0 converts the output of encoder M01 into an evaluation of the generation effect of the daytime scene corresponding to style option M11. In other words, it assigns a score to the generation result of the corresponding style option.

[0143] In one embodiment, the latent code 903 is noise data and can be a numerical value based on a statistical distribution such as a Gaussian distribution, a uniform distribution, or a Pashon distribution, or data generated based on noise in a real environment. For example, the processor 120 samples the Gaussian distribution and generates the latent code 903 based on the sampled numerical value.

[0144] Both the first and second codes are style codes 912. The second code is output from the same branch as the style encoder E. That is, it is fed / input into subsequent modules. Furthermore, each style code 912 corresponds to one or more style options, for example, style options E13 and M13.

[0145] It should be noted that the content and type of style options can still be changed according to actual needs and are not limited by the embodiment of the present invention.

[0146] Please refer to Figure 8 , the processor 120 inputs the first source data to the generator, and the generator outputs the first output corresponding to the first source data by referring to the style code (step S830). Figure 9 The generator is a key component of generative adversarial networks (GANs). It generates new data samples (e.g., images, text, music, etc.) from a latent space or random noise. In some applications, the generator is used to learn the distribution of real data and generate new samples similar to the real data.

[0147] Please refer to Figure 9 , the generator G includes an encoder G01 and a decoder G02. The encoder G01 is connected to the decoder G02. For example, the output of the encoder G01 (e.g., content features 905 / feature maps) serves as the input of the decoder G02. The encoder G01 can encode the input data (e.g., the first source data 901) into a latent vector (e.g., content features 905 or latent features) and obtain important information and variation factors in the input data. The decoder G02 can convert the latent vector into a high-dimensional data representation and convert the data representation into an output (e.g., a first output 921) in the same format as the real data.

[0148] In one embodiment, the generator G further includes a fusion layer G03 . The fusion layer G03 is connected between the encoder G01 and the decoder G02 . Figure 10 is a schematic diagram of the fusion layer G03 according to an embodiment of the present invention. Figure 10The fusion layer G03 includes a fusion block 1001 and a predictor 1002. The fusion block 1001 fuses the output (e.g., feature data 111 or other feature representations / expressions) from a decoding block of the decoder G01 (to be described in subsequent embodiments) with the style code 912 from the style encoder E and / or the mapping network M to generate fused data, which is then used to enhance the style information in the output of the decoding block of the encoder G01. The predictor 1002 is connected to the fusion block 1001. The predictor 1002 can convert the fused data output by the fusion block 1001 into style options corresponding to the style code 912. The output of the fusion layer G03 can serve as the input 112 of a decoding block of the decoder G02 (to be described in subsequent embodiments). In other words, the fusion processing of the fusion layer G03 implements the processing of the reference style code 912.

[0149] Figure 11 is a schematic diagram of a generator G according to an embodiment of the present invention. Figure 11 , the encoder G01 includes one or more encoding blocks G11 (for example, blocks numbered 1, blocks numbered 2 to blocks numbered N, and N is a positive integer). The encoding block G11 includes, for example, a convolution layer, a downsampling process, and a residual connection process, but is not limited thereto. The decoder G02 includes one or more decoding blocks G12 (for example, blocks numbered 1, blocks numbered 2 to blocks numbered N, and N is a positive integer). The decoding block G12 includes, for example, a deconvolution layer, a fully connected layer, an upsampling process, and an excitation function process, but is not limited thereto. A fusion layer G03 can be inserted into a U-net similar network. For example, the generator G01 shown in the figure is composed of an encoder G01 and a decoder G02. The encoder G01 can downsample the dimensions of the latent space, and the decoder G02 can upsample the dimensions of the latent space. The generator G may include one or more fusion layers G03. The input of each fusion layer G03 is the same as or corresponds to the size or resolution of its output. The processor 120 can input the fused data and another feature data to the decoding block G12, and this other feature data is the output of the encoding block G11 located adjacent to the front (such as the adjacent left side in the figure) (for example, the content feature 905) or the output of another decoding block G12 located adjacent to the front (such as the adjacent left side in the figure).

[0150] In addition, the user may insert one or more fusion layers G03 between the encoding block G11 and the decoding block G12 corresponding to a specific size or dimension according to actual needs, and the embodiment of the present invention is not limited thereto.

[0151] Since the decoder G02 also inputs the output of the fusion layer G03 (the style information of the entrained style code 912), the output of the decoder G02 (for example, Figure 9 The first output 921) can be regarded as style translation data. Figure 9 For example, if the first source data 901 is in image format, the first output 921 is a style-translated image. In addition, the first output 921 has the style options corresponding to the style code 912 (for example, rainy day), but still retains the same lane, vehicle, and scenery in the first source data 901.

[0152] Please refer to Figure 8 , the processor 120 inputs the first output to the discriminator, and outputs the second output corresponding to the first output through the discriminator (step S840). Specifically, the discriminator is another key component in the generative adversarial network (GAN). The generator can distinguish between real data (for example, images, text, music, etc.) and the first output output by the generator G. That is, determine whether the input data (for example, the first output of the generator) is real (true) or false (pseudo). In an embodiment of the present invention, the second output output by the discriminator includes true or false corresponding to the style option. That is, the second output indicates whether the first output corresponds to the style option. "Real" means corresponding to or the same as the style option, and "Fake" means not corresponding to or different from the style option. The discriminator includes, for example, a convolutional layer, a fully connected layer, an excitation function and a normalization layer, but is not limited thereto.

[0153] by Figure 9 For example, the discriminator D includes an encoder D01. The encoder D01 can be a shared encoder for multi-task learning. Multiple tasks share the same encoder D01, and the representation / performance learned by the encoder D01 can capture the common features between multiple tasks. These tasks correspond to different style options. For example, the style option numbered 0 is sunny. The style option numbered y (y is a positive integer greater than 0) is rainy. The style option is one of the style options from number 0 to number y. For example, rainy day is selected as the style option number The encoder D01 may obtain the feature representation / expression (eg, in vector or matrix form) generated for a specific style option from the first output 921 .

[0154] The discriminator D includes predictors D11 numbered 0 to D12 numbered y (y is a positive integer). Predictor D13 is one of Predictors D11 (number 0) through D12 (number y). Each numbered predictor is used to determine whether it corresponds to a specific style option. For example, Predictor D11 (number 0) determines whether the style option "Sunny" exists. "True" means it corresponds to or is the same as the "Sunny" style, while "Fake" means it does not correspond to or is different from the "Sunny" style.

[0155] In one embodiment, the style options corresponding to the predictors D11 to D13 of the discriminator D may correspond to the style options of the style encoder E and the mapping network M. In other embodiments, the content of the style options may still be changed according to actual needs.

[0156] In one embodiment, please refer to Figure 9 The processor 120 may update at least one of the style encoder E, the mapping network M, the generator G, and the discriminator D based on the style error 941. One of the multiple objectives of the training phase is to minimize a loss function (the error / loss between the output of the style encoder E, the mapping network M, the generator G, and / or the discriminator D and the corresponding data). In one embodiment, during the training phase of the prediction model, the parameters of the prediction model are recursively updated by minimizing the loss function. For example, backpropagation is used. Model parameters include, but are not limited to, weights, the number of layers, the location or number of neurons, activation functions, or biases. Parameter updating methods include, but are not limited to, gradient descent, adaptive estimation optimizer, momentum method, adaptive gradient, or conjugate gradient method. In other words, one of the multiple objectives of the training phase is to make it difficult for the discriminator D to identify the output of the generator G as "real" or "fake." In one embodiment, a trained prediction model means that its loss function has converged, the prediction accuracy has reached a corresponding threshold, or the training has reached the criterion for early stopping. However, the training completion criteria can be adjusted based on other tasks or requirements.

[0157] Figure 12A is the style similarity error L according to one embodiment of the present invention SS Please refer to the schematic diagram of Figure 12A , Figure 9 The style error 941 includes the style similarity error L SS The processor 120 may calculate the style encoder E from the first source data 121 (eg, Figure 9 The first code obtained by the first source data 901) and the second source data 122 (forming a positive pair (Positive Pair) PP1) to determine the style similarity error L SS The second source data 122 corresponds to the same style option as the first source data 121. For example, sunny day. In addition, the style similarity error LSS For example, the first code in vector form With the first code That is, one of the multiple objectives of the loss function is that the style codes generated by the style encoder E for multiple data of the same style option are the same or similar.

[0158] Figure 12B is the source preservation error L according to one embodiment of the present invention SP Please refer to the schematic diagram of Figure 12B , Figure 9 The style error 941 includes the source preservation error L SP The processor 120 may calculate the first source data 123 (eg, Figure 9 The difference between the first source data 901) and the second generated data 127 is used to determine the source preservation error L SP The generator G refers to the style code 9121 and outputs the third generated data 124 corresponding to the first source data 123. This style code 9121 is generated by the style encoder E from the reference data 125 (for example, Figure 9 The first code obtained by the reference data 902) Generator G refers to the second style code 9122 and outputs the second generated data 127 corresponding to the third generated data 124. The second style code 9122 is obtained by the style encoder E from the second reference data 126. The second reference data 126 corresponds to the same style option (e.g., sunny) as the first source data 123. The reference data 125 and the first source data 123 may correspond to the same or different style options. In addition, the source preservation error L SP For example, the difference between the first source data 123 and the second generated data 127 in matrix form is obtained by subtracting them. In other words, one of the multiple objectives of the loss function is to restore the second generated data 124 generated by the generator G to the first source data 123 or to be similar to the first source data 123 after the generator G refers to the style code of the same style option as the first source data 123.

[0159] Figure 12C is the content comparison error L according to one embodiment of the present invention CC Please refer to the schematic diagram of Figure 12C , Figure 9 The style error 941 includes the content contrast error L CC The processor 120 may calculate the difference between the first feature representation 9051 and the second feature representation 9052 (forming a positive pair PP2), and calculate the difference between the second feature representation 9052 and the third feature representation 9053 (forming a negative pair NP2) to determine the content comparison error L.CC The first feature representation 9051 is generated by the encoder G01 of the generator G from the first source data 128 (e.g., Figure 9 The generator G refers to the style code 9123 and outputs the third generated data 130 corresponding to the first source data 128. This style code 9123 is generated by the style encoder E from the reference data 129 (for example, Figure 9 The first code obtained by the reference data 902) The second feature expression 9052 is obtained by the encoder G01 from the third generated data 130. In addition, the third feature expression 9053 is obtained by the encoder G01 from the reference data 129. Content comparison error L CC For example, the difference (e.g., difference) between the second feature expression 9052 and the third feature expression 9053 is used as the denominator and the difference (e.g., difference) between the first feature expression 9051 and the second feature expression 9052 is used as the numerator. Since one of the multiple objectives of the loss function is to minimize the output value, the larger the difference (considered as content similarity) between the second feature expression 9052 and the third feature expression 9053 is, the better and / or the smaller the difference (considered as content similarity) between the first feature expression 9051 and the second feature expression 9052 is, the better. Since the encoder G01 is used to obtain content features, the more similar or close the content features of the first source data 128 and the third generated data 130 generated by the generator G are, the better, but the more dissimilar or dissimilar the content features of the reference data 129 and the third generated data 130 are, the better.

[0160] In one embodiment, Figure 9 The style error 941 can be Figure 12A The style similarity error L SS 、 Figure 12B The source of the preservation error L SP and Figure 12C Content comparison error L CC In another embodiment, according to different design requirements, the style error 941 can also be the style similarity error L SS , source storage error L SP and content comparison error L CC The result of the weighted operation. That is, the style similarity error L SS , source storage error L SP and content comparison error L CC Assign corresponding weights or priorities respectively.

[0161] Figure 13 is a schematic diagram of style options according to an embodiment of the present invention. Figure 13Different style options can be customized based on different design requirements. For example, for scenario simulation, style options for weather conditions include sunny, rainy, and cloudy; style options for day and night conditions include day and night; and style options for scene conditions include city, street, and freeway. Processor 120 can train separate module pairs (i.e., style encoder E and mapping network M corresponding to the same condition / type). Users can select the desired condition / type based on their needs.

[0162] Figure 14 This is a flowchart of the inference of multiple domain conditions related to the data generation framework according to one embodiment of the present invention. Figure 14 Processor 120 obtains a third code from the second latent code using the (trained) mapping network (step S1401). Specifically, the second latent code can be a numerical value based on a statistical distribution such as a Gaussian distribution, a uniform distribution, or a Pashon distribution, or data generated based on noise in a real environment. Figure 15 FIG is a schematic diagram of the inference of multi-domain conditions related to the data generation framework according to one embodiment of the present invention. Figure 15 For example, the processor 120 obtains Figure 5 The first generated data or the third source data generated in step S530. The introduction of the third source data 1501 can be referred to Figure 9 The first source data 901 is not described in detail here.

[0163] In addition, the processor 120 samples the Gaussian distribution 1511 and generates a second latent code 1502 based on the sampled value. Then, the processor 120 uses the second latent code 1502 as the input of the trained image network M. That is, the second latent code 1502 is input to the trained image network M. The representation / performance learned by the encoder M01 of the image network M can capture the common features between multiple tasks. These tasks correspond to different style options M11, M12, and M13, for example. For example, the style option M11 numbered 0 is cloudy, the style option M12 numbered y is sunny, and the style option M13 numbered y is sunny. The style option M13 is for rainy days. The style option M13 is one of the style options from the style option M11 numbered 0 to the style options numbered 1. As described above for the second code, the second code generated by the image network M is a style code corresponding to a certain style option. Similarly, the third code generated by the image network M is a style code 1512 corresponding to a certain style option. For example, Style option M13.

[0164] Please refer to Figure 14, the processor 120 inputs the first generated data or the third source data to the trained generator, and the trained generator refers to the third code to output the fourth generated data corresponding to the first generated data or the third source data (step S1402). Figure 15 , the encoder G01 of the generator G obtains the content feature 1505 from the first generated data or the third source data 1501. The fusion layer G03 fuses a certain coding block (such as Figure 11 The processor 120 combines the feature data and style code 1512 output by the encoding block G11 (of the source data) to generate fused data. The processor 120 inputs this fused data and feature data (e.g., content features 1505 or outputs of other decoding blocks) into the corresponding decoding block of the decoder G02. Ultimately, the decoder G02 outputs fourth generated data 1503. The fourth generated data 1503 is data having the same content as the third source data 1501 or the first generated data and the style options corresponding to the style code 1502.

[0165] Figure 16A This is a schematic diagram of scene image generation according to one embodiment of the present invention. Figure 16A , embodiments of the present invention provide a spatially conditioned generation pipeline and a multi-domain conditioned pipeline. For application scenarios of scene image generation (e.g., scenarios for self-driving car training), a simulator (e.g., a CARLA (Car Learning to Act) simulator or an annotated film sequence) may generate semantic masking data 1601. The semantic masking data 1601 defines semantic categories corresponding to multiple pixels of the generated data 1602. For example, pixels in a certain image area correspond to lanes, pixels in another image area correspond to vehicles, and pixels in another image area correspond to the sky. It should be noted that the semantic masking data 1601 is not limited to defining two-dimensional data, but can also be used to define three-dimensional, four-dimensional, or more dimensional data. For example, a three-dimensional semantic mask consists of three channels (i.e., height, width, and number of categories), and a four-dimensional semantic mask also includes a time dimension compared to a three-dimensional semantic mask. Data generator 1611 (for executing Figure 5 In steps S510 to S530, the semantic masking data 1601 may be referenced and predicted noise output. De-noised data is generated based on the difference between the predicted noise and the noise data. The de-noised data is then converted into generated data 1602 by a decoder corresponding to the initial encoder. The multiple image regions in the generated data 1602 may correspond to the corresponding semantic categories defined in the semantic masking data 1601.

[0166] Next, the condition fusion unit 1612 (for executing Figure 14In steps S1401 and S1402, the processor 120 obtains the style code corresponding to style option 1631 (e.g., snowy or rainy), and uses the trained generator to convert the input generated data 1602 into generated data 1603. For example, a street map on a rainy day. The processor 120 may store this generated data 1603 in a database corresponding to style option 1631. Multiple generated data 1603 may form a data set. Style option 1631 may be input content corresponding to a user operation, or may be generated based on specific conditions or random numbers, and may be changed according to the needs of the user.

[0167] Figure 16B FIG is a schematic diagram of facial image generation according to an embodiment of the present invention. Figure 16B For the application scenario of facial image generation, semantic masking data 1604 can be obtained from the dataset. The semantic masking data 1604 defines the semantic categories corresponding to the plurality of pixels of the generated data 1605. For example, pixels in a certain image region correspond to hair, pixels in another image region correspond to nose, and pixels in another image region correspond to mouth. Similarly, the data generator 1611 (for executing Figure 5 In steps S510 to S530 of the initial encoder, the semantic masking data 1604 may be referenced and predicted noise output. De-noised data is generated based on the difference between the predicted noise and the noise data. The de-noised data is then converted into generated data 1605 by a decoder corresponding to the initial encoder. The multiple image regions in the generated data 1605 may correspond to the corresponding semantic categories defined by the semantic masking data 1604.

[0168] Next, the condition fusion unit 1612 (for executing Figure 14 In steps S1401 and S1402 , the style code corresponding to the style option 1632 (e.g., male or female) is obtained, and the input generated data 1605 is converted into generated data 1606 through the trained generator. For example, a female face image.

[0169] It should be noted that the training datasets used for the spatial conditional pipeline and the multi-domain conditional pipeline may be different, and the training stages of the two pipelines may be performed separately.

[0170] Figure 17A and Figure 17B This is the experimental result of the spatially conditioned pipeline according to one embodiment of the present invention. Figure 17A Based on the semantic masking data 1701, generated data 1703 corresponding to the real label 1702 can be generated. The multiple image areas in the generated data 1703 can correspond to the corresponding semantic categories defined by the semantic masking data 1701. Figure 17BBased on the semantic masking data 1704 , generated data 1706 corresponding to the real label 1705 may be generated. The multiple image regions in the generated data 1706 may correspond to the corresponding semantic categories defined by the semantic masking data 1704 .

[0171] Figure 18 This is the experimental result of a multi-domain conditional pipeline according to an embodiment of the present invention. Figure 18 Assuming the input image is a street map of a sunny day, we can generate street maps with cloudy, rainy, and snowy styles. Aside from the weather, the image content of these street maps corresponds to the input image. Alternatively, assuming the input image is a street map of a snowy, rainy, and cloudy day, we can convert them into sunny street maps while retaining the original street content.

[0172] In summary, in the methods and apparatus related to the data generation framework of the embodiments of the present invention, a prediction model for identifying / predicting noisy data is trained, and the corresponding denoised data is converted into generated data. This allows for the generation of high-resolution data that approximates the real world. Furthermore, a style code corresponding to a style option is obtained, referenced by the generator, and a discriminator determines whether the generator's output is real or fake. Thus, the trained generator can be used to generate generated data that conforms to a specific style option, while retaining the source content. The generated data can be evaluated on a variety of datasets to verify the quality and robustness of the data generation.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method related to a data generation framework, implemented by a processor, comprising: Obtaining a first code from reference data by a style encoder; Obtaining a second code from the latent code through a mapping network, wherein the first code and the second code are both style codes, and each of the style codes corresponds to at least one style option; Inputting first source data to a generator, and outputting a first output corresponding to the first source data by the generator with reference to the style code; as well as The first output is input to a discriminator, and a second output corresponding to the first output is output by the discriminator, wherein the second output indicates whether the first output corresponds to the style option.

2. The method according to claim 1, wherein the generator comprises a first encoder and a first decoder, the first encoder being connected to the first decoder, the first encoder comprising at least one first encoding block, and the first decoder comprising at least one first decoding block, and the step of outputting the first output corresponding to the first source data by the generator with reference to the style code comprises: fusing the feature data output by the first encoding block with the style code to generate fused data; as well as The fused data and another feature data are input to the first decoding block, wherein the another feature data is an output of the first encoding block or an output of another first decoding block.

3. The method according to claim 1 , further comprising: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to a style error, wherein the style error includes a style similarity error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to the style error comprises: A difference between two first codes obtained by the style encoder from the first source data and the second source data is calculated to determine the style similarity error.

4. The method according to claim 1 , further comprising: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator based on a style error, wherein the style error includes a source preservation error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator based on the style error comprises: The difference between the first source data and the second generated data is calculated to determine the source preservation error, wherein the generator outputs third generated data corresponding to the first source data with reference to the style code, and the generator outputs the second generated data corresponding to the third generated data with reference to a second style code, the second style code is obtained by the style encoder from a second reference data, and the second reference data corresponds to the same style option as the first source data.

5. The method according to claim 1 , further comprising: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to a style error, wherein the style error comprises a content contrast error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to the style error comprises: Calculate the difference between a first feature expression and a second feature expression, and calculate the difference between the second feature expression and a third feature expression to determine the content comparison error, wherein the first feature expression is obtained from the first source data by the first encoder of the generator, the generator outputs third generated data corresponding to the first source data with reference to the style code, the second feature expression is obtained from the third generated data by the first encoder, and the third feature expression is obtained from the reference data by the first encoder.

6. The method according to claim 1 , further comprising: Obtaining a third code from the second latent code by the mapping network; as well as The first generated data or the third source data is input to the trained generator, and the trained generator outputs the fourth generated data corresponding to the first generated data or the third source data with reference to the third code.

7. The method according to claim 5, further comprising: Obtaining latent representation from training data via the initial encoder; combining the potential representation and the first noise data to generate a noisy potential representation; Inputting the noisy potential representation into a prediction model, and outputting an initial prediction corresponding to the noisy potential representation by the prediction model with reference to first semantic masking data, wherein the first semantic masking data defines at least one semantic category for the training data; as well as The prediction model is updated according to a prediction error between the initial prediction and the first noise data to generate a trained prediction model.

8. The method according to claim 7, wherein the prediction model comprises a second encoder and a second decoder, the second encoder being connected to the second decoder, the second encoder comprising at least one second encoding block, and the second decoder comprising at least one second decoding block, and the step of outputting the initial prediction corresponding to the noisy potential representation by the prediction model with reference to the first semantic masking data comprises: inputting the noisy potential representation into the second encoder; Inputting the first semantic masking data and feature data to the second decoding block, wherein the feature data is an output of the second encoding block or an output of another second decoding block; as well as The initial prediction is output by the second decoder.

9. The method according to claim 7, wherein the prediction error is the sum of a first error and a second error, the initial prediction includes a prediction mean and a prediction variance, and the step of updating the prediction model based on the prediction error between the initial prediction and the first noise data comprises: calculating a difference between the predicted mean and the mean of the first noise data to determine the first error; as well as The difference between the prediction variance and the variance of the first noise data is calculated to determine the second error.

10. The method according to claim 7, further comprising: inputting second noise data into the trained prediction model, and outputting predicted noise corresponding to the second noise data by the trained prediction model with reference to second semantic masking data, wherein the second semantic masking data defines at least a second semantic category for the first generated data; generating denoised data according to a difference between the predicted noise and the second noise data; as well as The denoised data is converted into the first generated data by a decoder corresponding to the initial encoder.

11. An apparatus related to a data generation framework, comprising: a memory for storing program codes; as well as A processor, coupled to the memory, and configured to load the program code to execute: Obtaining a first code from reference data by a style encoder; Obtaining a second code from the latent code through a mapping network, wherein the first code and the second code are both style codes, and each of the style codes corresponds to at least one style option; Inputting first source data to a generator, and outputting a first output corresponding to the first source data by the generator with reference to the style code; as well as The first output is input to a discriminator, and a second output corresponding to the first output is output by the discriminator, wherein the second output indicates whether the first output corresponds to the style option.

12. The apparatus according to claim 11, wherein the generator comprises a first encoder and a first decoder, the first encoder being connected to the first decoder, the first encoder comprising at least a first encoding block, the first decoder comprising at least a first decoding block, and the processor being further configured to: fusing the feature data output by the first encoding block with the style code to generate fused data; and The fused data and another feature data are input to the first decoding block, wherein the another feature data is an output of the first encoding block or an output of another first decoding block.

13. The apparatus according to claim 11, wherein the processor is further configured to: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to a style error, wherein the style error includes a style similarity error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to the style error comprises: A difference between two first codes obtained by the style encoder from the first source data and the second source data is calculated to determine the style similarity error.

14. The apparatus according to claim 11, wherein the processor is further configured to: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator based on a style error, wherein the style error includes a source preservation error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator based on the style error comprises: The difference between the first source data and the second generated data is calculated to determine the source preservation error, wherein the generator outputs third generated data corresponding to the first source data with reference to the style code, and the generator outputs the second generated data corresponding to the third generated data with reference to a second style code, the second style code is obtained by the style encoder from a second reference data, and the second reference data corresponds to the same style option as the first source data.

15. The apparatus according to claim 11, wherein the processor is further configured to: Updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to a style error, wherein the style error comprises a content contrast error, and updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to the style error comprises: Calculate the difference between a first feature expression and a second feature expression, and calculate the difference between the second feature expression and a third feature expression to determine the content comparison error, wherein the first feature expression is obtained from the first source data by the first encoder of the generator, the generator outputs third generated data corresponding to the first source data with reference to the style code, the second feature expression is obtained from the third generated data by the first encoder, and the third feature expression is obtained from the reference data by the first encoder.

16. The apparatus according to claim 11, wherein the processor is further configured to: Obtaining a third code from the second latent code by the mapping network; and The first generated data or the third source data is input to the trained generator, and the trained generator outputs the fourth generated data corresponding to the first generated data or the third source data with reference to the third code.

17. The apparatus according to claim 11, wherein the processor is further configured to: Obtaining latent representation from training data via the initial encoder; combining the potential representation and the first noise data to generate a noisy potential representation; inputting the noisy potential representation into a prediction model, and outputting an initial prediction corresponding to the noisy potential representation by the prediction model with reference to first semantic masking data, wherein the first semantic masking data defines at least one first semantic category for the training data; and The prediction model is updated according to a prediction error between the initial prediction and the first noise data to generate a trained prediction model.

18. The apparatus according to claim 17 , wherein the prediction model comprises a second encoder and a second decoder, the second encoder being connected to the second decoder, the second encoder comprising at least one second encoding block, the second decoder comprising at least one second decoding block, and the processor is further configured to: inputting the noisy potential representation into the second encoder; inputting the first semantic masking data and feature data to the second decoding block, wherein the feature data is an output of the second encoding block or an output of another second decoding block; and The initial prediction is output by the second decoder.

19. The apparatus according to claim 17 , wherein the prediction error is a sum of a first error and a second error, the initial prediction comprises a prediction mean and a prediction variance, and the processor is further configured to: calculating a difference between the predicted mean and the mean of the first noise data to determine the first error; and The difference between the prediction variance and the variance of the first noise data is calculated to determine the second error.

20. The apparatus according to claim 17, wherein the processor is further configured to: inputting second noise data into the trained prediction model, and outputting predicted noise corresponding to the second noise data by the trained prediction model with reference to second semantic masking data, wherein the second semantic masking data defines at least a second semantic category for the first generated data; generating denoised data based on a difference between the predicted noise and the second noise data; and The denoised data is converted into the first generated data by a decoder corresponding to the initial encoder.

Images