Missing data filling method in table data processing system

Abstract

The invention provides a missing data filling method in a table data processing system, which comprises the following steps of: S1, dividing numeric data and classified data in table data to obtain preprocessed data; s2, performing data enhancement by adopting a K nearest neighbor algorithm, generating a three-dimensional data matrix, and obtaining a training set containing missing values; s3, generating a global feature and a local feature, and fusing the global feature and the local feature into a fused feature; s4, generating a plurality of candidate filling results, and performing weighted summation according to the confidence of each candidate filling result to obtain a final filling result; s5, optimizing model parameters by minimizing weighted combination items of the global loss and the local loss to obtain a trained model; and S6, outputting a filling result. According to the technical scheme, the problems of data sparsity, heterogeneity and uncertainty of an existing missing value filling method are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more particularly to a method for filling in missing data in a tabular data processing system. Background Technology

[0002] In recent years, tabular data has been widely used in various fields such as finance, healthcare, social science research, and industrial systems, making data quality a key factor in determining the reliability and accuracy of analytical results. However, the problem of missing values ​​is prevalent in tabular data, posing a significant challenge to the application of artificial intelligence models.

[0003] Existing rule-based or neural network-based missing value imputation techniques suffer from three main problems, especially when dealing with numerical and categorical data from diverse sources. 1) Sparsity: With the increasing prevalence of unstructured data (e.g., text, images, and audio), the size of structured datasets required for traditional machine learning tasks is gradually decreasing. This shift leads to the scarcity of tabular data, posing two main challenges to effective tabular imputation: (i) the need to extract rich latent structures from inherently “simple” tabular representations, and (ii) the limited generalization ability of models due to data finiteness. 2) Heterogeneity: Unlike image pixels or word tags, which exhibit relative homogeneity and semantic invariance under spatial transformations, tabular data is inherently heterogeneous. This heterogeneity manifests in three key aspects: (i) differences in feature representations, (ii) non-uniform formats and structures, and (iii) different data distributions among features. Therefore, assuming a uniform data distribution for missing value imputation often results in poor performance. This challenge underscores the need for a learning model capable of adapting to the heterogeneous distribution of tabular data. 3) Uncertainty Issues: Missing values ​​in tabular data introduce inherent uncertainty because their true values ​​are unobservable and cannot be uniquely determined from the existing data alone. However, mainstream existing methods typically employ single imputation, ignoring the uncertainty and ambiguity introduced by missing values. To address this limitation, multiple imputation is recommended. This method combines multiple different feasible datasets to derive the final imputation result, thus providing a more accurate and robust estimate than single imputation. However, due to the complexity of implementing multiple imputation and its computational requirements, related research remains relatively scarce. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose a method for filling in missing data in a tabular data processing system, which effectively solves the shortcomings of existing data filling methods in the face of these challenges.

[0005] The technical means employed in this invention are as follows: A method for imputing missing data in a tabular data processing system includes the following steps: S1. Divide the numerical data and categorical data in the table into numerical data and normalize the numerical data to obtain the preprocessed data. S2. Using the preprocessed data, the K-nearest neighbor algorithm is used to augment the data, generating a three-dimensional data matrix. Then, the missing data in the three-dimensional data matrix is ​​simulated by artificial masking technology to obtain a training set containing missing values. S3. Using a training set containing missing values, fine-grained feature modeling is performed by integrating a capsule network into a Gaussian mixture variational autoencoder to generate global and local features, and then the global and local features are fused into a fused feature. S4. Using fusion features, multiple global features are sampled and fused with local features to generate multiple candidate filling results. The results are then weighted and summed based on the confidence level of each candidate filling result to obtain the final filling result. S5. Using the final imputation results, optimize the model parameters by minimizing the weighted combination of global and local losses to obtain the trained model. S6. Using the trained model, impute missing values ​​in the target imputation data and output the imputation results.

[0006] Furthermore, S1 specifically includes the following steps: S11. Based on the data dictionary, divide the features into numerical and categorical types; let the number of numerical features be... The number of categorical features is , ; S12. Standardize the numerical data using the Min-Max normalization method.

[0007] Furthermore, S2 specifically includes the following steps: S21. For numerical features Cosine similarity is used to measure samples With any complete sample The similarity between them is denoted as ; S22. Regarding classification features Using Hamming distance to measure samples and The similarity is denoted as ; S23, Sample and The overall similarity is obtained by adding the numerical feature similarity and the categorical feature similarity:

[0008] S24, for each sample Select the K nearest neighbors with the highest similarity, denoted as:

[0009] S25, Based on each sample K-nearest neighbor set By adding a nearest neighbor dimension, the original two-dimensional table data is transformed. Expanded into a three-dimensional data matrix ; S26. Randomly mask a portion of the observed values ​​in the training set according to a given artificial missing rate: first generate and augment the training set. All elements have the same shape, and each element is defined as follows:

[0010] in, It is a missing mask matrix One of the elements; S27, using the missing mask matrix With augmented training set Perform element-wise multiplication to obtain the training set with missing values. .

[0011] Furthermore, S3 specifically includes the following steps: S31. Global Feature Modeling: The GMVAE model learns the mean and variance of latent variables through an encoder network, and generates the distribution of latent variables by combining convolutional layers and linear transformations; it then uses reparameterization techniques to generate the corresponding global features. ; S32. Local feature modeling, given inputs that have undergone data augmentation and corruption processing. Capsule encoder generation eigenvectors Attention weights corresponding to feature vectors Each feature vector is processed through a template. Perform an affine transformation; from The transformed features of each capsule are concatenated along the feature dimension to obtain the final local features. :

[0012] S33. Feature fusion: merging the global feature vector... and local feature vectors By splicing together, a fusion feature is formed. .

[0013] Furthermore, S4 specifically includes the following steps: S41. Sample T global features from the posterior distribution of the global feature modeling. , each global feature With local features Connect the components to obtain the fused features. Based on each fusion feature The corresponding filling results are obtained. ; S42. Generate the confidence weight for each candidate imputation result:

[0014] in, For learnable parameters, It is an activation function; S43. The candidate filling results are weighted and summed according to their confidence weights to obtain the final filling result.

[0015] Furthermore, S5 specifically includes the following steps: S51. Calculate the global loss. The global loss is used to measure the consistency between the reconstructed data and the global latent representation, and the posterior distribution is regularized by the KL regularization term. S52. Calculate the local loss, which measures the degree of alignment between the augmented training set and local features in the local feature space. S53. Optimize the model interpolation quality by minimizing the weighted combination of global and local losses.

[0016] The present invention also provides a storage medium comprising a stored program, wherein, when the program is executed, the missing data filling method in any of the above-described lattice data processing systems is performed.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the missing data filling method in any of the above tabular data processing systems through the computer program.

[0018] Compared with the prior art, the present invention has the following advantages: This invention provides a missing data imputation method for tabular data processing systems, addressing the problem that existing missing value imputation algorithms cannot effectively handle complex data patterns in tabular data. Considering the diversity of tabular data and fully taking into account its sparsity, heterogeneity, and uncertainty, a high-quality imputation model method is proposed. This method enhances the model's generalization ability by utilizing neighboring samples to enrich observed information. Capsule networks are used to model fine-grained feature dependencies, while multi-sample inference further improves the accuracy and robustness of the imputation results by introducing multiple latent variables. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the method of the present invention.

[0021] Figure 2 This is a diagram illustrating the missing data filling process in the missing data filling method of the present invention.

[0022] Figure 3 This is used to fill in the model diagram for the data of this invention.

[0023] Figure 4 This is a comparison image of the MINST filling process before and after the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] like Figure 1 As shown, the present invention provides a method for imputing missing data in a tabular data processing system, comprising the following steps: S1. Divide the numerical data and categorical data in the table into numerical data and normalize the numerical data to obtain the preprocessed data. S11. Based on the data dictionary, divide the features into numerical and categorical types; let the number of numerical features be... The number of categorical features is Used for subsequent similarity calculations and data augmentation.

[0027] S12. Numerical data is standardized using the Min-Max normalization method.

[0028] S2. Using the preprocessed data, the K-nearest neighbor algorithm is used to augment the data, generating a three-dimensional data matrix. Then, the missing data in the three-dimensional data matrix is ​​simulated by artificial masking technology to obtain a training set containing missing values. S21. For numerical features Cosine similarity is used to measure the similarity of samples. With any complete sample The similarity between them is denoted as .

[0029] S22. Regarding classification features Hamming distance is used to obtain and The similarity is denoted as .

[0030] S23, Sample and The overall similarity is obtained by adding the numerical feature similarity and the categorical feature similarity:

[0031] S24. After completing the similarity calculation, for each sample Select the K nearest neighbors with the highest similarity, denoted as:

[0032] S25. Nearest neighbor-based data augmentation, based on each sample K-nearest neighbor set By adding a nearest neighbor dimension, the original two-dimensional table data... Expanded into a three-dimensional data matrix .

[0033] S26. After data augmentation, randomly mask a portion of the observed values ​​in the training set according to a given artificial missing value rate: first generate and augment the training set. A consistent missing mask matrix:

[0034] S27, using the missing mask matrix With augmented training set Perform element-wise multiplication to obtain the training set with missing values. .

[0035] S3. Using a training set containing missing values, fine-grained feature modeling is performed by integrating a capsule network into a Gaussian mixture variational autoencoder to generate global and local features, and then the global and local features are fused into a fused feature. S31. Global Feature Modeling: The GMVAE model learns the mean and variance of latent variables through an encoder network, and generates the distribution of latent variables by combining convolutional layers and linear transformations. Subsequently, the corresponding global features are generated using reparameterization techniques. .

[0036] S32. Local feature modeling, given inputs that have undergone data augmentation and corruption processing. The capsule encoder first generates eigenvectors and their corresponding attention weights Then, each feature vector is processed through a template. Perform an affine transformation. Finally, the result from... The transformed features of each capsule are concatenated along the feature dimension to obtain the final local features. .

[0037]

[0038] S33. Feature fusion: In order to fully utilize global semantic information and local structural features, the global feature vector is fused... and local feature vectors By splicing together, a fusion feature is formed. .

[0039] S4. Using fusion features, multiple global features are sampled and fused with local features to generate multiple candidate filling results. The results are then weighted and summed based on the confidence level of each candidate filling result to obtain the final filling result. S41. Sample T global features from the posterior distribution introduced in global feature modeling. Then, each global feature With local features Connect the components to obtain the fused features. Based on each fusion feature The corresponding filling results are obtained. .

[0040] S42. Generate the confidence weight for each candidate imputation result.

[0041]

[0042] in, It is a learnable parameter. It is an activation function.

[0043] S43. Each candidate filling result is weighted and summed according to its confidence weight to obtain the final filling result.

[0044] S5. Using the final imputation results, optimize the model parameters by minimizing the weighted combination of global and local losses to obtain the trained model. The global loss measures the consistency between the reconstructed data and the global latent representation it depends on, while regularizing the posterior distribution through variational inference. Its initial form is defined as the expectation of the likelihood ratio: that is, under the distribution of the latent variables, it compares the likelihoods of the reconstructed data with those of the true data, thereby constraining the model to maintain consistency at the global statistical level, and using a KL regularization term to encourage the posterior to approach the prior, avoiding overfitting.

[0045] Local loss is used to measure the degree of alignment between the augmented training set and the local features generated by the capsule network in the local feature space.

[0046] S6. Using the trained model, impute missing values ​​in the target imputation data and output the imputation results.

[0047] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for imputing missing data in a tabular data processing system, characterized in that, Includes the following steps: S1. Divide the numerical data and categorical data in the table into numerical data and normalize the numerical data to obtain the preprocessed data. S2. Using the preprocessed data, the K-nearest neighbor algorithm is used to augment the data, generating a three-dimensional data matrix. Then, the missing data in the three-dimensional data matrix is ​​simulated by artificial masking technology to obtain a training set containing missing values. S3. Using a training set containing missing values, fine-grained feature modeling is performed by integrating a capsule network into a Gaussian mixture variational autoencoder to generate global and local features, and then the global and local features are fused into a fused feature. S4. Using fusion features, multiple global features are sampled and fused with local features to generate multiple candidate filling results. The results are then weighted and summed based on the confidence level of each candidate filling result to obtain the final filling result. S5. Using the final imputation results, optimize the model parameters by minimizing the weighted combination of global and local losses to obtain the trained model. S6. Using the trained model, impute missing values ​​in the target imputation data and output the imputation results.

2. The method for filling in missing data in a tabular data processing system according to claim 1, characterized in that, S1 specifically includes the following steps: S11. Based on the data dictionary, divide the features into numerical and categorical types; let the number of numerical features be... The number of categorical features is , ; S12. Standardize the numerical data using the Min-Max normalization method.

3. The method for filling in missing data in a tabular data processing system according to claim 1, characterized in that, S21. For numerical features Cosine similarity is used to measure samples With any complete sample The similarity between them is denoted as ; S22. Regarding classification features Using Hamming distance to measure samples and The similarity is denoted as ; S23, Sample and The overall similarity is obtained by adding the numerical feature similarity and the categorical feature similarity: S24, for each sample Select the K nearest neighbors with the highest similarity, denoted as: S25, Based on each sample K-nearest neighbor set By adding a nearest neighbor dimension, the original two-dimensional table data is transformed. Expanded into a three-dimensional data matrix ; S26. Randomly mask a portion of the observed values ​​in the training set according to a given artificial missing rate: first generate and augment the training set. All elements have the same shape, and each element is defined as follows: in, It is a missing mask matrix One of the elements; S27, using the missing mask matrix With augmented training set Perform element-wise multiplication to obtain the training set with missing values. .

4. The method for filling in missing data in a tabular data processing system according to claim 1, characterized in that, S3 specifically includes the following steps: S31. Global Feature Modeling: The GMVAE model learns the mean and variance of latent variables through an encoder network, and generates the distribution of latent variables by combining convolutional layers and linear transformations; it then uses reparameterization techniques to generate the corresponding global features. ; S32. Local feature modeling, given inputs that have undergone data augmentation and corruption processing. Capsule encoder generation eigenvectors Attention weights corresponding to feature vectors ; Each feature vector is passed through a template Perform an affine transformation; from The transformed features of each capsule are concatenated along the feature dimension to obtain the final local features. : S33. Feature fusion: merging the global feature vector... and local feature vectors By splicing together, a fusion feature is formed. .

5. The method for filling in missing data in a tabular data processing system according to claim 1, characterized in that, S4 specifically includes the following steps: S41. Sample T global features from the posterior distribution of the global feature modeling. , each global feature With local features Connect the components to obtain the fused features. Based on each fusion feature The corresponding filling results are obtained. ; S42. Generate the confidence weight for each candidate imputation result: in, For learnable parameters, It is an activation function; S43. The candidate filling results are weighted and summed according to their confidence weights to obtain the final filling result.

6. The method for filling in missing data in a tabular data processing system according to claim 1, characterized in that, S5 specifically includes the following steps: S51. Calculate the global loss. The global loss is used to measure the consistency between the reconstructed data and the global latent representation, and the posterior distribution is regularized by the KL regularization term. S52. Calculate the local loss, which measures the degree of alignment between the augmented training set and local features in the local feature space. S53. Optimize the model interpolation quality by minimizing the weighted combination of global and local losses.

7. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it performs the missing data filling method in the tabular data processing system according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the missing data filling method in the tabular data processing system according to any one of claims 1 to 6 through the computer program.

Images