A continuous label filling method and system based on standardized flow and noise reduction auto-encoding fusion

Abstract

This invention discloses a continuous label imputation method and system based on the fusion of normalized flow and denoising autoencoder, belonging to the field of normalized flow and denoising autoencoder fusion technology, including the following steps: Step S1, constructing a SERDA model, wherein the SERDA model includes: an affine coupling layer, used to segment input variables based on a masking mechanism, and to implement the scaling function s(•) and translation function t(•) through a multilayer perceptron to complete the nonlinear transformation of features; a denoising autoencoder layer, used to introduce active information loss after feature transformation and to suppress noise interference through a multilayer fully connected network and LeakyReLU activation function. This invention outperforms other state-of-the-art models such as GAIN, GINN, and MissForest in imputation of continuous non-discrete data, and also solves the problem of low model accuracy due to insufficient data in small-sample learning and prediction tasks, exhibiting superior performance in multiple aspects.

Description

Technical Field

[0001] This invention relates to the field of fusion technology of standardized stream and noise reduction autoencoder, and in particular to a continuous label filling method and system based on the fusion of standardized stream and noise reduction autoencoder. Background Technology

[0002] Missing values ​​are a common problem in real-world datasets. Market data, due to factors such as differences in market liquidity, limitations in data transmission technology, and regulatory and confidentiality provisions, often affects the analysis of market microstructure, leading to calculation biases and impacting the accuracy of predictive models. In scientific survey data, factors such as survey fatigue and inappropriate experimental design can cause sample bias, resulting in analysis results that do not fully represent the target population. Therefore, effectively handling missing key values ​​has become a crucial issue that various industries urgently need to address before applying data.

[0003] Data imputation techniques have been extensively studied and applied to solve the aforementioned problems. Traditional imputation methods are typically based on statistical principles, using the mean (for continuous data), median (for skewed continuous data), or mode (for categorical data) to fill in missing values. K-Nearest Neighbors (KNN) imputation identifies neighboring samples to fill in missing values ​​through weighted or voting methods. This method performs well when the data has local correlations. However, KNN may fail when data is severely missing or the sample space is sparse. Chained Multiple Imputation (MICE) generates multiple imputation datasets by modeling the conditional distribution of missing data and introduces an uncertainty measure for iterative integration to improve imputation accuracy. The MissForest method uses a random forest model trained on non-missing features for predictive imputation. Such supervised methods rely on complete label information, so their performance may significantly degrade when labels are missing.

[0004] Semi-supervised learning is gaining attention as a promising alternative. Semi-supervised learning methods train on observed data and impute missing values ​​based on learned patterns, showing significant value, especially when labels and data distributions are incomplete. Generative Adversarial Networks (GANs) are models that apply deep learning to impute missing values ​​in tabular data. GAIN uses a masking mechanism, simultaneously inputting mask information into the generator and discriminator to define a random process for generating data, lacking structured distribution constraints. Another well-known adversarial training model is the Graph Inductive Neural Network (GINN), which introduces a graph convolutional network for imputation, treating the imputation task as a graph denoising autoencoder to uncover similarities between data patterns. However, GINN has limitations in mini-batch analysis and does not consider imbalanced distance relationships between variables.

[0005] Semi-supervised normalized flow models learn underlying patterns and estimate missing data points by integrating labeled and unlabeled data, effectively capturing the inherent structure of data. Flow models are trained using maximum likelihood estimation to learn invertible mappings in complex data distributions. Masked Autoregressive Flow (MAF), Real-Valued Non-Volume Preservation Flow (RealNVP), and Diffusion Normalized Flow are commonly used flow architectures for data imputation. Among them, RealNVP achieves fast convergence even with fewer iterations and has higher accuracy than MAF. Based on the idea of ​​RealNVP, research has proposed methods for discrete classification label imputation, which outperform models such as GAIN and GINN. In regression tasks, the target variable is modeled as a continuous label; existing flow models lack sufficient support for continuous label regression tasks, and iterative refinement mechanisms for continuous labels have not yet been addressed.

[0006] Overall, imputing large numbers of consecutive labels in datasets remains a significant challenge. There is an urgent need for a solution to overcome the dependence of supervised learning on complete labels, accommodate imbalanced distance relationships between data points, and construct an iterative refinement mechanism for consecutive labels. Semi-supervised flow models hold promise as a key direction for addressing this domain's pain points.

[0007] Based on the above, we designed a continuous label imputation method based on the fusion of normalized flow and denoising autoencoder to solve the above problems. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a continuous label filling method based on the fusion of standardized stream and noise reduction autoencoder.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A continuous label imputation method based on the fusion of normalized stream and denoising autoencoder includes the following steps:

[0011] Step S1: Construct the SERDA model, which includes:

[0012] The affine coupling layer is used to segment the input variables based on the masking mechanism, and to implement the scaling function s(•) and translation function t(•) through the multilayer perceptron to complete the nonlinear transformation of the features;

[0013] A noise reduction autoencoder layer is used to introduce active information loss after feature transformation and suppress noise interference through a multi-layer fully connected network and the LeakyReLU activation function;

[0014] The inverse transform layer is used to map the denoising results back to the original variable space, thereby reconstructing the data distribution.

[0015] Step S2: Train the SERDA model on the complete dataset, while minimizing the reconstruction loss function and the regression loss function, and balancing the supervised signal and the unsupervised signal through dynamic weights;

[0016] Step S3: Transfer the pre-trained model to the target dataset with missing labels, fine-tune the model by combining some observable labels, and generate continuous label imputation values ​​that conform to the original distribution.

[0017] Preferably, in step S1, the SERDA model, short for "semi-supervised enhanced regression & denoising autoencoder," includes a forward affine coupling layer that segments the input variables based on a masking mechanism. It performs feature transformation using a scaling function s(•) and a translation function t(•), and the output satisfies the following equation:

[0018]

[0019]

[0020]

[0021] The denoising autoencoder adds active information loss to the output of the affine coupling layer, which is then denoised by a fully connected layer and the LeakyReLU activation function. Finally, the affine coupling inverse transform layer reconstructs the original variable space.

[0022] Preferably, in step S1, the mask covers 70% of the input variable data, and the scaling function s(•) and translation function t(•) are implemented by a multilayer perceptron.

[0023] Preferably, in step S2, the backpropagation optimization process consists of two parts: reconstruction loss and regression loss, calculated as follows:

[0024]

[0025] in The numerical features after masking To reconstruct numerical features, For unobserved target labels, Predict labels for the model, The weights of reconstruction loss and regression loss are controlled and adjusted according to the gradient descent in each iteration until convergence.

[0026] Preferably, step S3 includes: selecting a complete dataset with a variable distribution similar to the target dataset, satisfying a difference in the total number of variables ≤10% and a mean squared error of non-missing variables ≤5%. During fine-tuning, the parameters of the affine coupling layer and the denoising autoencoder layer are frozen, and only the regression layer parameters are optimized.

[0027] Preferably, in step S3, a Gaussian mixture model is used to evaluate the consistency of data distribution before and after filling.

[0028] A system based on a continuous label imputation method fused with normalized stream and denoising autoencoder, comprising:

[0029] Data acquisition module: Matches complete datasets with distributions similar to the target dataset;

[0030] Preprocessing module: Performs ordinal encoding of categorical variables and standardization of numerical variables;

[0031] Model training module: Builds and pre-trains the SERDA model;

[0032] Migration Filling Module: Fine-tunes model parameters and outputs filling results.

[0033] Compared with existing technologies, the beneficial effects of this invention are as follows: The continuous label imputation method of this invention realizes data imputation based on a semi-supervised learning flow model framework. Through mechanisms such as masking, dimensionality-upgrading mapping, noise reduction, backpropagation, and transfer learning, it establishes the intrinsic relationships between variables in the data, thereby obtaining a complete dataset that highly conforms to the distribution pattern of the original data. This invention outperforms other state-of-the-art models, such as GAIN, GINN, and MissForest, in imputation of continuous non-discrete data. It also solves the problem of low model accuracy due to insufficient data volume in small-sample learning and prediction tasks, demonstrating superior performance in multiple aspects. Attached Figure Description

[0034] Figure 1 A schematic diagram of the continuous label imputation method based on semi-supervised learning provided in an embodiment of the present invention;

[0035] Figure 2 This is a block diagram of the SERDA model provided in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the architecture of a continuous label filling system based on semi-supervised learning, provided in an embodiment of the present invention. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0038] Reference Figure 1-3 A continuous label imputation method based on the fusion of normalized stream and denoising autoencoder includes the following steps:

[0039] Step S1: Construct the SERDA model, which includes:

[0040] The affine coupling layer is used to segment the input variables based on the masking mechanism, and to implement the scaling function s(•) and translation function t(•) through the multilayer perceptron to complete the nonlinear transformation of the features;

[0041] A noise reduction autoencoder layer is used to introduce active information loss after feature transformation and suppress noise interference through a multi-layer fully connected network and the LeakyReLU activation function;

[0042] The inverse transform layer is used to map the denoising results back to the original variable space, thereby reconstructing the data distribution.

[0043] Step S2: Train the SERDA model on the complete dataset, while minimizing the reconstruction loss function and the regression loss function, and balancing the supervised signal and the unsupervised signal through dynamic weights;

[0044] Step S3: Transfer the pre-trained model to the target dataset with missing labels, fine-tune the model by combining some observable labels, and generate continuous label imputation values ​​that conform to the original distribution.

[0045] More specifically, in step S1, the SERDA model, short for "semi-supervised enhanced regression & denoising autoencoder," includes a forward affine coupling layer that segments the input variables based on a masking mechanism. It performs feature transformation using a scaling function s(•) and a translation function t(•), and the output satisfies the following equation:

[0046]

[0047]

[0048]

[0049] The denoising autoencoder adds active information loss to the output of the affine coupling layer, which is then denoised by a fully connected layer and the LeakyReLU activation function. Finally, the affine coupling inverse transform layer reconstructs the original variable space.

[0050] In step S1, the mask covers 70% of the input variable data, and the scaling function s(•) and translation function t(•) are implemented by a multilayer perceptron.

[0051] More specifically, in step S2, the backpropagation optimization process consists of two parts: reconstruction loss and regression loss, calculated as follows:

[0052]

[0053] in The numerical features after masking To reconstruct numerical features, For unobserved target labels, Predict labels for the model, The weights of reconstruction loss and regression loss are controlled and adjusted according to the gradient descent in each iteration until convergence.

[0054] Further, step S3 includes filtering a complete dataset with a variable distribution similar to the target dataset, satisfying a difference in the total number of variables ≤10% and a mean squared error of non-missing variables ≤5%. During fine-tuning, the parameters of the affine coupling layer and the denoising autoencoder layer are frozen, and only the regression layer parameters are optimized.

[0055] In step S3, a Gaussian mixture model is used to evaluate the consistency of data distribution before and after the imputation.

[0056] More specifically, a system for a continuous label imputation method that integrates standardized streams and denoising autoencoders includes:

[0057] Data acquisition module: Matches complete datasets with distributions similar to the target dataset;

[0058] Preprocessing module: Performs ordinal encoding of categorical variables and standardization of numerical variables;

[0059] Model training module: Builds and pre-trains the SERDA model;

[0060] Migration Filling Module: Fine-tunes model parameters and outputs filling results.

[0061] Example: Continuous label imputation of a carbon dioxide emissions dataset from a steel plant

[0062] This example is based on statistics of carbon dioxide emissions from a small steel plant, totaling 3101 samples. Due to non-compliant personnel training, only 930 of these samples recorded carbon emission data. This example uses the UCI steel plant energy consumption dataset for validation (http: / / archive.ics.uci.edu / dataset / 851). First, outlier cleaning is performed. For the numerical variable "electricity consumption," its mean μ and standard deviation σ are calculated, and outlier samples exceeding the range of μ ± 3σ are removed.

[0063] The categorical variable "Workday" in the dataset is ordinal-encoded, and the numerical variables "Leading Current Reactive Power", "Lagging Current Reactive Power", "Leading Current Power Factor", "Lagging Current Power Factor", and "Electricity Consumption" are Z-score standardized. The preprocessed feature matrices are then merged into a unified input through a concatenation operation.

[0064]

[0065]

[0066] in This is a list of indices for categorical variables sorted by cardinality. For normalized numerical variables, This represents the process of juxtaposing categorical and numerical variables.

[0067] Based on the distribution similarity criteria mentioned above, a pre-training subset is divided from the UCI dataset to ensure that the total number of variables is the same, and the distribution consistency with the target dataset is verified using KL divergence. Missing data scenarios are simulated for the variables in the pre-training dataset using a random mask with a masking ratio of 70%, corresponding to the missing data scenarios encountered in the target dataset.

[0068] according to Figure 2 The SERDA model architecture shown feeds pre-trained data into an affine coupling layer, and uses a three-layer perceptron to implement the scaling function s(•) and translation function t(•). The hidden layer dimension is set to twice the input dimension. Dropout=0.2 is applied after the output of the affine coupling layer, and the data is processed through a five-layer fully connected network using the LeakyReLU activation function. The fully connected network structure is as follows:

[0069] h = dim * 2, nn.Linear(dim, dim), nn.Linear(dim, h), nn.Linear(h, h),nn.Linear(h, h), nn.Linear(h, dim)

[0070] The training process uses five-fold cross-validation to shard the data, evaluating the model simultaneously across all five shards. The Adam optimizer (learning rate 0.001, β1=0.9, β2=0.999) is used to minimize the joint loss function. Dynamic weights Adjustments are made in real time based on the proportion of observable labels. The training termination condition is that the loss decrease rate is less than 0.0001 for 10 consecutive epochs (actually converged on the 97th epoch).

[0071] In the pre-training phase, the model's performance is measured by the mean absolute error (MAE) and the coefficient of determination (R²). 2 This reflects the fact that pre-trained models are saved for fine-tuning on the target dataset.

[0072] Load the target dataset, using the saved preprocessor to load both observable and unobservable data. Load the pre-trained model and freeze all layers except the regression head. Fine-tune the regression head using the observable data. Use MSE loss as the loss function and Adam as the optimizer, optimizing until convergence.

[0073] The pre-trained model was transferred to a target dataset containing 70% missing labels. The parameters of the first three layers of the affine coupling layer and the denoising autoencoder layer were frozen, and only the last fully connected regression layer was optimized. A Gaussian mixture model was used to calculate the KL divergence of the distribution before and after imputation, which was 0.15, lower than the threshold requirement of 0.2.

[0074] As shown in Table 1, this method outperforms existing technologies in all key indicators:

[0075] Table 1. Comparison of the imputation performance of each model on the pre-training and target datasets.

[0076]

[0077] This invention introduces transfer learning strategies into semi-supervised data imputation tasks, successively incorporating flow models and denoising coding to effectively learn the dependencies and distribution patterns between data, thereby achieving high-precision data imputation results. In practical implementation, technicians can treat the above steps as independent entities and combine them arbitrarily according to the task, adding strategies or mechanisms to cope with various scenarios.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention, all of which fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A continuous label imputation method based on the fusion of normalized stream and denoising autoencoder, characterized in that, Includes the following steps: Step S1: Construct the SERDA model. SERDA stands for Semi-Supervised Augmented Regression and Denoising Autoencoder. The SERDA model includes: The affine coupling layer is used to segment the input variables based on the masking mechanism, and to implement the scaling function s(•) and translation function t(•) through the multilayer perceptron to complete the nonlinear transformation of the features; A noise reduction autoencoder layer is used to introduce active information loss after feature transformation and suppress noise interference through a multi-layer fully connected network and the LeakyReLU activation function; The inverse transform layer is used to map the denoising results back to the original variable space, thereby reconstructing the data distribution. Step S2: Train the SERDA model on the complete dataset, minimizing both the reconstruction and regression loss functions. Validate the model using the UCI steel plant energy consumption dataset by dynamically balancing the supervised and unsupervised signals with weights. Ordinal encoding is applied to the categorical variable "Workday" in the dataset, and Z-score standardization is performed on the numerical variables "Leading Current Reactive Power," "Lagging Current Reactive Power," "Leading Current Power Factor," and "Electricity Consumption." The preprocessed feature matrices are then merged into a unified input through a concatenation operation. ， in This is a list of indices for categorical variables sorted by cardinality. For normalized numerical variables, The process of juxtaposing categorical and numerical variables; The backpropagation optimization process consists of two parts: reconstruction loss and regression loss, which are calculated as follows: ， in The numerical features after masking To reconstruct numerical features, For unobserved target labels, Predict labels for the model, Control the weight balance between reconstruction loss and regression loss, and adjust it according to the gradient descent in each iteration until convergence; Step S3: Transfer the pre-trained model to the target dataset with missing labels, and fine-tune the model by combining some observable labels to generate continuous label imputation values ​​that conform to the original distribution. This includes selecting complete datasets with variable distributions similar to the target dataset, satisfying a total difference of ≤10% in the total number of variables and a mean square error of ≤5% for non-missing variables; and freezing the parameters of the affine coupling layer and the denoising autoencoder layer during fine-tuning, optimizing only the regression layer parameters.

2. The continuous label imputation method based on the fusion of normalized flow and denoising autoencoder as described in claim 1, characterized in that, In step S1, the included forward affine coupling layer segments the input variables based on a masking mechanism and uses a scaling function. s (•) and translation function t (•) Perform feature transformation, and the output satisfies the following equation: ， The denoising autoencoder adds active information loss to the output of the affine coupling layer, which is then denoised by a fully connected layer and the LeakyReLU activation function. Finally, the affine coupling inverse transform layer reconstructs the original variable space.

3. The continuous label imputation method based on the fusion of normalized flow and denoising autoencoder as described in claim 1, characterized in that, In step S1, the mask covers 70% of the input variable data, and the scaling function... s (•) and translation function t (•) Implemented by a multilayer perceptron.

4. The continuous label imputation method based on the fusion of normalized flow and denoising autoencoder as described in claim 1, characterized in that, In step S3, a Gaussian mixture model is used to evaluate the consistency of data distribution before and after the imputation.

5. A continuous label filling system based on the method of any one of claims 1-4, characterized in that, include: Data acquisition module: Matches complete datasets with distributions similar to the target dataset; Preprocessing module: Performs ordinal encoding of categorical variables and standardization of numerical variables; Model training module: Builds and pre-trains the SERDA model; Migration Filling Module: Fine-tunes model parameters and outputs filling results.

Images