A data missing processing and predicting system and method based on diffusion model

Abstract

This invention discloses a data missing data processing and prediction system and method based on a diffusion model. The system includes a data governance module and a model inference module. The data governance module calls different diffusion models for targeted data completion based on the data missing pattern. The model inference module predicts the completed data and uses the prediction performance as feedback to evaluate the effectiveness of the data completion, forming a governance-verification closed loop. This invention effectively solves the problem of the disconnect between data governance and prediction applications under high missing rate and mixed missing patterns by deeply coupling a bimodal completion mechanism with the prediction task, significantly improving the robustness and accuracy of prediction in scenarios with incomplete data.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data science, and in particular to a data missing processing and prediction system and method based on a diffusion model. Background Technology

[0002] In data-driven decision-making systems, such as disease risk prediction in the healthcare field, high-quality and complete data are the foundation for building high-precision predictive models. However, real-world data often contains a large number of missing values ​​due to difficulties in collection, incomplete records, or privacy protection, which severely restricts the performance and application of predictive models.

[0003] Existing technology CN114997313A discloses an anomaly detection method for marine online monitoring data. This scheme labels historical monitoring data with point and sequence anomalies and comprehensively utilizes multiple methods, including threshold detection, mean square error detection, abrupt change detection, weighted moving average fitting detection, Holt-Winter fitting detection, and isolated forest detection, to effectively identify different types of anomalies in marine online monitoring data. This method focuses on the "detection" stage of data quality, i.e., identifying and marking anomalous data points or sequences. Its core is to improve the accuracy and recall of anomaly detection to ensure data usability. However, this scheme mainly focuses on the "anomaly discovery" stage and does not provide further recovery or filling mechanisms for data that has been marked as anomalous or missing. In practical applications, especially for downstream prediction tasks, simple data anomaly labeling is insufficient to support the complete operation of the model; if the removed or marked missing data cannot be reasonably filled in, it will directly affect the continuity of subsequent analysis and the accuracy of prediction.

[0004] In summary, the existing technical solutions have the following shortcomings: (1) the data missing processing method is isolated from the downstream prediction task, and there is a lack of post-event verification and feedback optimization mechanism for the completion effect; (2) the anomaly detection scheme only stops at discovering problems and fails to form a closed loop linkage with data completion and prediction; (3) under high missing rate and complex missing mode, a single model is difficult to meet the dual requirements of completion quality and prediction performance. Therefore, there is an urgent need for a system and method that can realize an integrated closed loop of data governance-verification-prediction. Summary of the Invention

[0005] The main objective of this invention is to provide a data missing processing and prediction system and method based on a diffusion model, which aims to solve the problems of existing data missing processing methods being singular and the data completion process being independent of the subsequent prediction and analysis process.

[0006] To achieve the above objectives, the present invention provides a data missing information processing and prediction system based on a diffusion model, the system comprising:

[0007] The data governance module is used to receive structured tabular data containing missing values, identify the missing data patterns, and adaptively select the corresponding model from at least two preset diffusion models to complete the missing values ​​based on the identified missing patterns, and output the completed data.

[0008] The model inference module, connected to the data governance module, is used to receive the completed data and perform analysis and prediction using a combination of discriminative and generative prediction to obtain prediction results.

[0009] The result output module is connected to the model inference module and is used to format and output the prediction results.

[0010] The prediction performance index output by the model inference module is configured as a feedback signal to evaluate the effectiveness of the data governance module in completing the data, thus forming a closed loop from data completion to prediction verification.

[0011] Optionally, the missing patterns include random missing of some features and complete missing of the entire column of features;

[0012] The data governance module is configured to: when the missing pattern is that some features are randomly missing, call the self-supervised diffusion model to complete the missing data; when the missing pattern is that the entire column of features is completely missing, call the conditional table diffusion model to perform generative completion based on the observed features.

[0013] Optionally, the conditional table diffusion model employs a classifier-free guidance mechanism during the generation process.

[0014] Optionally, the model inference module includes a collaborative prediction unit, which includes:

[0015] The discriminative prediction subunit outputs the first prediction result based on a machine learning classifier.

[0016] The generative prediction subunit, based on the conditional table diffusion model, uses input features as conditions and outputs a second prediction result through generative modeling.

[0017] The result fusion subunit is used to fuse the first prediction result and the second prediction result to generate the prediction result.

[0018] Optionally, the generative prediction subunit reuses the conditional table diffusion model used in the data governance module to handle cases where all features are completely missing.

[0019] Optionally, the result output module further includes an interpretation unit, used to transform the prediction results into domain-specific insights or suggestions based on application domain rules.

[0020] Optionally, the data governance module further includes an adaptive inference unit, which receives instructions from the model inference module regarding the key missing features on which the current prediction task depends, and drives the corresponding completion model to perform targeted completion of the key missing features.

[0021] Optionally, the model inference module further includes a model update and maintenance unit, which monitors the prediction performance of the model inference module and automatically uses newly added data to iteratively optimize the prediction model inside the model inference module when a preset trigger condition is met.

[0022] Furthermore, to achieve the above objectives, the present invention also provides a method for data missing information processing and prediction based on a diffusion model. This method is applied to any of the systems described above and includes the following steps:

[0023] Step 1: Receive structured tabular data containing missing values;

[0024] Step 2: Identify the missing data patterns, and based on the identified missing patterns, adaptively select the corresponding model from at least two preset diffusion models to complete the missing values, thus obtaining the completed data.

[0025] Step 3: Based on the completed data, analyze and predict using a combination of discriminative and generative prediction methods to obtain the prediction results;

[0026] Step 4: Format and output the prediction results;

[0027] The prediction performance metrics output in the model inference step are used as feedback signals to evaluate the effectiveness of data completion in the data governance step.

[0028] Optionally, step 2 specifically includes:

[0029] The first prediction result is obtained through a discriminative prediction model;

[0030] The second prediction result is obtained by using a generative prediction model conditioned on the input features;

[0031] The first prediction result and the second prediction result are fused together to obtain the prediction result.

[0032] The beneficial effects of this invention are as follows:

[0033] This invention effectively addresses the problem of the disconnect between data governance and predictive applications under high missing rate and complex missing modes by constructing a system that integrates bimodal adaptive completion, governance-verification closed loop, and collaborative prediction. Specifically, it is reflected in:

[0034] (1) High accuracy and strong targeting: By distinguishing the data missing mode into "random missing part of the feature" and "complete missing of the entire column of features", and using self-supervised diffusion model and conditional table diffusion model respectively for targeted treatment, the shortcomings of traditional single imputation method are poor adaptability and insufficient accuracy in complex mixed missing scenarios.

[0035] (2) An integrated governance-verification closed loop was constructed, enhancing practical value: This invention establishes a governance-verification closed loop starting from the end goal. The system uses the performance indicators of downstream prediction tasks (such as AUC and accuracy) as standards and feedback signals to evaluate the effectiveness of data completion. This ensures that the completed data is not only statistically reasonable, but also effectively improves the efficiency of the final business application, effectively alleviating the problem of data preprocessing being disconnected from business objectives.

[0036] (3) By integrating the two different prediction perspectives of discriminative prediction and generative prediction through the collaborative prediction module, the system's stability in the face of data uncertainty, noise, and distribution changes is improved by leveraging the complementary advantages of the former's high accuracy and the latter's probabilistic nature and deep understanding of data distribution. In addition, the adaptive inference unit can dynamically identify and request the completion of key missing features during prediction, further ensuring the reliability of decision-making in scenarios with incomplete data.

[0037] (4) Independent privacy-preserving data generation units can generate synthetic data that is highly consistent with the distribution of the original data but is untraceable. This can be used to securely expand training samples and improve the generalization ability of models, and also provides a privacy-preserving solution for cross-institutional data collaboration, expanding the boundaries of data value utilization.

[0038] (5) The model update and maintenance module of the system design can be automatically retrained through performance monitoring, enabling the system to adapt to changes in data distribution and achieve continuous optimization of long-term performance. At the same time, its core architecture solves the universality problem of governance of missing tabular data and predictive collaboration, and can be easily extended to multiple fields such as medical, financial, and industrial fields that require intelligent analysis of incomplete structured data, with broad application prospects. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0040] Figure 1 This is a block diagram of a data missing processing and prediction system based on a diffusion model according to the present invention.

[0041] Figure 2 This is a flowchart of the SimpDM model.

[0042] Figure 3 The graph shows a comparison of the feature completion effects of systolic blood pressure (BP_HIGH), aspartate aminotransferase (AST), low-density lipoprotein cholesterol (LDL_C), and serum creatinine (SCR) in the 10% missing MCAR scenario in Group 2 of Table 2. The horizontal axis units are: BP_HIGH in mmHg (millimeters of mercury), AST in U / L (units per liter), LDL_C in mmol / L (millimoles per liter), and SCR in μmol / L (micromoles per liter).

[0043] Figure 4 The comparison chart shows the effect of feature completion for 10% missing features of alanine aminotransferase (ALT), age (AGE), serum albumin (ALB), and hematocrit (PCV) in the MCAR scene of Group 2 in Table 2; the horizontal axis units are: ALT in U / L (unit / liter), AGE in years, ALB in g / L (gram / liter), and PCV in L / L (red blood cell volume / whole blood volume).

[0044] Figure 5 The table shows the t-SNE scatter plot and density contour plot of the MCAR scene after 10% missing data filling in Group 2 of Table 2; the dimensions 1 and 2 of the t-SNE are dimensionless projected coordinates used to show the clustering structure of the data.

[0045] Figure 6 A comparison of the kernel density estimation (KDE) curves of the synthesized data of glycated hemoglobin (HBA1C) generated for the TabDiff model with the original data; the unit of glycated hemoglobin (HBA1C) in the figure is %.

[0046] Figure 7 The figure shows a comparison of kernel density estimation (KDE) curves of the synthetic and raw data for systolic blood pressure BP_HIGH; the unit of systolic blood pressure BP_HIGH in the figure is mmHg (millimeters of mercury).

[0047] Figure 8 This is a comparison of the kernel density estimation (KDE) curves of synthesized and raw data of aspartate aminotransferase (AST); the AST unit in the figure is U / L (unit / liter).

[0048] Figure 9 This is a comparison of the kernel density estimation (KDE) curves of the synthetic and raw data of blood urea BU; the unit of blood urea BU is mmol / L (millimoles per liter).

[0049] Figure 10This is a comparison of kernel density estimation (KDE) curves between the synthetic and raw data of low-density lipoprotein cholesterol (LDL_C); the unit for LDL_C is mmol / L.

[0050] Figure 11 This is a comparison of the kernel density estimation (KDE) curves of the synthesized and original serum creatinine SCR data; the unit of serum creatinine SCR is μmol / L.

[0051] Figure 12 A comparison of kernel density estimation (KDE) curves of body mass index (BMI) generated for the TabDiff model with those of the original data; BMI is measured in kg / m². 2 .

[0052] Figure 13 This is a comparison of the nuclear density estimation (KDE) curves of synthetic data and raw data for alanine aminotransferase (ALT); the unit for ALT is U / L (units per liter).

[0053] Figure 14 This is a comparison of kernel density estimation (KDE) curves for synthetic and raw data of age (AGE); the unit for age (AGE) is years.

[0054] Figure 15 This is a comparison of the kernel density estimation (KDE) curves of the synthesized serum albumin (ALB) data and the original data; the unit for serum albumin (ALB) is g / L.

[0055] Figure 16 A comparison of the nuclear density estimation (KDE) curves of synthetic and raw PCV data; PCV is expressed in L / L (red blood cell volume / whole blood volume).

[0056] Figure 17 This is a comparison of the nuclear density estimation (KDE) curves between the synthesis data and the original data of glutamine transferase (GGT); the unit for GGT is U / L (units per liter).

[0057] Figure 18 A bar chart comparing the distribution of synthetic and original data for kidney disease generated by the TabDiff model;

[0058] Figure 19 A bar chart comparing the distribution of synthetic and raw data for lower extremity arterial lesions;

[0059] Figure 20 A bar chart comparing the distribution of synthetic and raw data for coronary heart disease;

[0060] Figure 21A bar chart comparing the distribution of synthetic and original data for tag frequencies.

[0061] Explanation of icon numbers:

[0062] 100 - Data Governance Module, 101 - Data Collection and Preprocessing Unit, 102 - Dual-Modal Data Completion Unit, 103 - Privacy Data Generation Unit;

[0063] 200-Model inference module, 201-Feature selection unit, 202-Collaborative prediction unit, 203-Model update and maintenance unit;

[0064] 300 - Result Output Module.

[0065] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0066] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0067] Explanation of the name:

[0068] t-SNE (t-distributed stochastic neighbor embedding) is a commonly used high-dimensional data visualization method. Its output dimensions 1 and 2 are dimensionless projected coordinates used to display the relative position and clustering structure of data in a reduced-dimensional space, and do not have physical units. This invention uses t-SNE plots to visually compare the similarity between generated data and real data in global distribution, thereby verifying the effectiveness of the data completion method.

[0069] See Figure 1 The present invention provides an architecture block diagram of a data missing processing and prediction system based on a diffusion model. The system includes a data governance module 100, a model inference module 200, and a result output module 300.

[0070] The data governance module 100 is the front end of the system, used to optimize and process the input raw tabular data (structured tabular data) to ensure the accuracy and consistency of the data and provide a reliable data foundation for subsequent analysis. Furthermore, the data governance module 100 includes a data collection and preprocessing unit 101 and a bimodal data completion unit 102.

[0071] The data collection and preprocessing unit 101 is used to receive raw tabular data from multiple data sources and perform data cleaning, format unification, and standardization operations. For example, the Min-Max standardization formula is:

[0072] ;

[0073] X is the original feature value, X min and X max X represents the minimum and maximum values ​​of this feature, respectively. norm This is the standardized value.

[0074] And a dual-modal data completion unit 102, used to call the corresponding diffusion model to complete the missing values ​​according to the missing pattern of the preprocessed structured table data. The missing pattern of the data is identified in advance, that is, the missing distribution of the data matrix is ​​analyzed. If the missing values ​​are scattered across multiple features of multiple samples, it is determined as "random missing of some features"; if the entire column of features is missing in all samples, it is determined as "complete missing of the entire column of features".

[0075] Then, the completion strategy is automatically selected based on the missing pattern. Specifically, if the missing pattern is that some features are randomly missing, the first diffusion model is called to refine the missing values; if the missing pattern is that the entire column of features is completely missing, the second diffusion model is called to perform generative completion of the entire column of missing features based on the observed features.

[0076] Preferably, the first diffusion model is a self-supervised diffusion model (SimpDM), which is trained through a self-supervised alignment mechanism and a state-dependent data augmentation strategy to ensure that the distribution of the imputed values ​​is consistent with that of the original data.

[0077] Specifically, such as Figure 2 As shown, it comprises three core stages: sampling masking and mean imputation, state-related data augmentation, and a self-supervised alignment mechanism, ultimately serving the closed-loop validation of downstream predictions. This model constructs a self-supervised task by masking partial observations and learns the data distribution through a diffusion process, enabling refined and context-aware imputation of missing data.

[0078] In the sampling mask and mean imputation stage, the system takes the original data x as input, first generates a missing mask m (0, 1, 0 in the example in the figure), and then generates a pseudo mask m after sampling. p (Examples in the diagram are 0, 0, and 1). This is then combined with the conditional mask m. c (The example in the figure is 0, 1, 1) Perform the calculation and process it by mean filling to obtain the initial filled data x0.

[0079] In the state-related data augmentation stage, based on the initial padding data x0, time step parameters t1 and t2 (whose values ​​follow a uniform discrete distribution Uniform([0,1,...,T]) from 0 to T) are introduced, along with noise. and (Its values ​​follow a normal distribution N(0, I) with a mean of 0 and a covariance equal to the identity matrix). This generates state data x corresponding to different time steps. t1 and x t2 Then, by combining the state information including GT (Ground Truth, referring to the original true data state), MS (Mean State, referring to the data state obtained based on mean filling), and PM (Pseudo Mask State, referring to the data state after processing with pseudomasks), the enhanced data is generated. and This provides enhanced samples for subsequent model training.

[0080] In the self-supervised alignment mechanism stage, the enhanced data are input into the diffusion model, and the model outputs the corresponding completion results. and Subsequently, the two completion results are constrained and aligned using self-supervised alignment loss (Lsa, used to constrain the consistency of the two completion results) to improve the consistency of the completed data.

[0081] Preferably, the second diffusion model is a conditional tabular diffusion model (TabDiff), which uses observed features as conditions to reconstruct the entire column of missing data through a classifier-free generation process. Its core logic is expressed as follows:

[0082] ;

[0083] in, For the diffusion model at time step state, For known observations, This is the conditional probability distribution after adjustment without classifier guidance. Indicates given known information In this case, the denoising direction predicted by the model. By randomly assigning conditions during training Setting it to empty indicates that the data is distributed globally. The guiding coefficient. When At that time, the model will enhance the conditions. The impact on the generation process makes the imputed results more consistent with the constraints of known features, thereby improving the accuracy of the imputation. Specifically, the model uses other observed feature columns in the data as conditions, and synthesizes the missing feature columns through a generation process of forward diffusion and reverse denoising. The generated data maintains consistency and correlation with the observation conditions.

[0084] In addition, for refined imputation of partial missing features, imputation models based on generative adversarial networks (GANs) or variational autoencoders (VAEs) can be used; for generative imputation of missing entire columns, conditional generative adversarial networks (CGANs) or conditional VAEs can also be used.

[0085] Furthermore, the bimodal data completion unit 102 also includes an adaptive inference subunit, which is used to receive instructions from an external prediction task. When the prediction task determines that the input data is missing key features, it initiates a high-precision completion request for the key features to the bimodal data completion unit 102.

[0086] The workflow of the adaptive inference unit includes:

[0087] Receive instructions from the model inference module 200, the instructions including the current prediction task identifier and the feature identifier that is determined to be a key missing feature;

[0088] Based on the missing pattern of the key missing features, the corresponding diffusion model in the dual-modal data completion unit 102 is invoked for high-precision filling;

[0089] The completed data, now filled in, is returned to the model inference module 200 to perform the final prediction. Furthermore, the adaptive inference unit transforms the data governance process and downstream prediction tasks from a "pipeline" relationship to a "closed-loop collaborative" relationship, establishing a governance-verification closed loop oriented towards practical application effectiveness. This ensures that the supplemented data is not merely statistically reasonable, but truly useful for the final task.

[0090] Furthermore, the data governance module 100 also includes a privacy-preserving data generation unit 103, independent of the bimodal data completion unit. This unit uses a tab diff model to generate synthetic data with a distribution highly consistent with the original data, used to securely expand training samples or facilitate data collaboration. Although also based on a conditional tab diff model (TabDiff), this unit's purpose is not to complete specific missing values, but rather to generate unconditionally to create entirely new synthetic samples. This synthetic data does not contain any traceable information about real individuals, but retains the key statistical regularities and feature associations of the original dataset. Therefore, it can be securely used to expand local training datasets, improve model generalization capabilities, and, in cross-institutional collaborations, replace sensitive real data for exchange or joint modeling, fundamentally mitigating the risk of privacy breaches.

[0091] Furthermore, the model inference module 200 receives the high-dimensional feature data supplemented by the data governance module 100 and performs analysis and prediction. It includes a feature filtering unit 201, a collaborative prediction unit 202, and a model update and maintenance unit 203.

[0092] The feature selection unit 201 is used to select a subset of core features that contribute significantly to the prediction task from the input high-dimensional feature data. Specifically, the feature selection unit 201 adopts a fusion selection strategy, which is implemented as follows:

[0093] In the first screening stage, based on statistical testing methods, features with statistically significant correlation to the prediction target are initially screened from the original high-dimensional feature data, forming a preliminary feature set. Preferably, the Mann-Whitney U test is used for continuous features and categorical targets, and the chi-square test is used for categorical features and categorical targets. This stage can efficiently narrow down the range of candidate features. Therefore, this stage uses non-parametric statistical testing methods to quickly filter out features that are clearly irrelevant to the prediction target, constituting a statistical preliminary screening.

[0094] In the second screening stage, based on multiple ensemble learning models (such as Random Forest and AdaBoost), importance scores are calculated for features in the initial screening feature set, and the features are ranked according to their comprehensive scores to determine the final core feature subset. These learning models can capture complex nonlinear relationships. By calculating and aggregating the feature importance scores from different models, a comprehensive and robust importance ranking is obtained. Finally, the top-ranked features are selected to form the core feature subset. This method effectively avoids the bias of a single model and ensures that the selected features have a broad and solid contribution to the prediction task. This belongs to model fine screening.

[0095] A collaborative prediction unit 202, connected to the feature filtering unit 201, is used to output the final prediction result based on the core feature subset by fusing the results of discriminative prediction and generative prediction. Specifically, the collaborative prediction unit 202 includes a discriminative prediction subunit, a generative prediction subunit, and a result fusion subunit.

[0096] The discriminative prediction subunit, based on a machine learning classifier, utilizes its inherent nonlinear mapping capability to map a subset of input core features to target labels, outputting a first prediction result. The specific form of this mapping function is automatically determined by the training process of the selected classifier (such as a gradient boosting decision tree, support vector machine, or deep neural network), which is a mature technology in this field. This invention does not improve its internal mapping mechanism. Preferably, the machine learning classifier used in the discriminative prediction subunit includes at least one of gradient boosting decision tree, support vector machine, or deep neural network. If a discriminative model such as XGBoost is used, the decision boundary between input features and output labels can be directly learned, achieving high prediction accuracy and efficiency, and providing an explanation of feature importance. In a preferred embodiment, the effectiveness of the subunit was verified using the XGBoost model.

[0097] The generative prediction subunit, based on a conditional table diffusion model, uses input features as conditions to generate target labels through generative modeling and outputs a second prediction result. Preferably, the generative prediction unit reuses the conditional table diffusion model from the data governance module used to handle cases where entire columns of features are completely missing. It generates prediction targets using the aforementioned observed features as conditions, and the model directly "generates" or infers the most likely target label based on the learned joint distribution of the data. This provides a probabilistic, generative prediction perspective based on the overall data distribution.

[0098] The result fusion subunit is used to fuse the first prediction result and the second prediction result to generate the final prediction result. Specifically, this fusion strategy can be a simple weighted average, voting, or a more complex meta-learner. The fundamental advantage of this collaboration lies in improving the robustness and stability of the system. When the data distribution changes slightly, the discriminative subunit may be sensitive, while the generative subunit may remain stable due to its understanding of the inherent distribution of the data; conversely, when the data is clear, the discriminative subunit can provide a high-precision benchmark. The two complement each other, making the entire collaborative prediction unit more reliable in the face of uncertainty and noise, and achieving a prediction performance of "1+1>2".

[0099] A model update and maintenance unit 203 is also included, used to iteratively optimize the model in the collaborative prediction unit 202 based on new data and prediction feedback. Specifically, this can be achieved by establishing performance monitoring and automatically triggering retraining processes, thereby ensuring the continuous evolution of module performance. For example, key performance indicators of the collaborative prediction unit 202 on the latest data or the retained validation set are continuously tracked. When the performance indicators are continuously lower than a preset threshold, or when a module update instruction is received periodically, the module automatically triggers a retraining process. The accumulated new data (or combined with historical data) is then used to retrain or fine-tune the discriminative and / or generative subunits in the collaborative prediction unit 202, and the performance of the new model is evaluated using the validation set. After confirming the improvement, the online collaborative prediction unit 202 is seamlessly updated.

[0100] The result output module 300, connected to the model inference module 200, is responsible for converting the prediction results into user-understandable and executable output. This module embodies the transformation of the system's core capabilities into application value, including:

[0101] Standardized output: Receives prediction results and interpretable data, and encapsulates them into structured data objects for external systems to call or display.

[0102] Interpretation Unit: Based on the application domain, load the corresponding rules or templates to transform general prediction results into domain-specific insights and suggestions (such as clinical early warning and risk control points).

[0103] Furthermore, to better illustrate the effects of the present invention, specific embodiments are described below.

[0104] Taking the specific data analysis task of predicting the risk of diabetic retinopathy (DR) as an example, this paper systematically demonstrates the complete workflow and verification process of the system from data input to result output.

[0105] System initialization and data preparation

[0106] Data Source: National Population Health Science Data Center Data Warehouse PHDA. A General Hospital. Diabetes Complications Early Warning Dataset. This dataset contains data on diabetic retinopathy, consisting of 3,000 patients with diabetes and diabetic retinopathy (DR). The dataset includes detailed patient information, including demographic characteristics such as patient ID, disease type, age, gender, and marital status, as well as laboratory test results such as blood glucose, blood lipids, and kidney function, totaling 87 features.

[0107] Data cleaning. Eighteen features with a missing value rate ≥20% and all samples containing missing values ​​were removed, ultimately retaining 2329 valid samples, ensuring the initial quality of the data. Subsequently, numerical features were Min-Max standardized and divided into training and test sets in a 7:3 ratio, laying the foundation for subsequent model training and evaluation.

[0108] Automated Feature Filtering

[0109] After obtaining a well-structured dataset, a multi-level strategy combining statistical tests (Mann-Whitney U test, chi-square test) and various ensemble learning models (random forest, AdaBoost, Extra Trees, and GBDT) was employed to precisely select 15 core feature sets most valuable for DR prediction from 87 initial features, as shown in Table 1. Table 1 explains the sample size: This study included 2329 patients, of whom 1121 were in the diabetes group and 1208 were in the diabetic retinopathy group.

[0110]

[0111] Data completion performance verification

[0112] To verify the system's robustness in environments with incomplete data, various missing feature scenarios (MCAR, MAR, MNAR) were systematically simulated on the test set, targeting the most common case of "partial feature loss". Five independent tests were conducted for each missing rate, with eight features randomly selected in each test to simulate missing features, and the self-supervised diffusion model (SimpDM) in the data governance module was used to fill in the missing features.

[0113] In the completely random missing features (MCAR) scenario with missing rates of 10%, 30%, and 70%, this model exhibits excellent imputation performance, with average RMSE values ​​of 0.1086, 0.1083, and 0.1184, respectively. Table 2 shows the MCAR scenario with 10% missing features and the corresponding RMSE data.

[0114]

[0115] Among them, HBA1C is glycated hemoglobin, AST is aspartate aminotransferase, BU is blood urea, LDL_C is low-density lipoprotein cholesterol, ALT is alanine aminotransferase, AGE is age, PCV is hematocrit, GGT is gamma-glutamyl transferase, BP_HIGH is systolic blood pressure, ALB is serum albumin, SCR is serum creatinine, and BMI is body mass index.

[0116] Taking Group 2 of Table 2 as an example, draw a comparison chart of the feature completion effects, such as... Figure 3-4 As shown, the distributions of missing features LDL_C, SCR, ALT, and AGE are generally the same as the original data. The distribution patterns of the remaining features are similar in the two sets of data, with only slight differences in the higher value ranges. These differences may be due to the influence of certain extreme values ​​on missing value imputation. Figure 5 The scatter plot in the t-SNE shows that the points in the generated data overlap somewhat with the points in the original data in terms of overall distribution trends, indicating that the model is effective in capturing the approximate distribution characteristics of the original data; and Figure 5 The density contour plots in the t-SNE show that the density contours of the original data and the generated data have similar shapes in some major high-density areas, indicating that the generated data has a certain degree of matching with the original data in terms of overall density distribution pattern.

[0117] With a 10% missing rate as a fixed condition, the generalization ability was analyzed for two missing scenarios: massively random missing (MAR) and non-massive missing (MNAR). The average RMSE for MAR was 0.1187 (as shown in Table 3), and for MNAR it was 0.1064 (as shown in Table 4), both within the low error range. This indicates that the model can stably complete the imputation task under different missing mechanisms, and the imputation results deviate little from the real data, meeting the accuracy requirements of conventional applications.

[0118]

[0119]

[0120] Among them, HBA1C is glycated hemoglobin, AST is aspartate aminotransferase, BU is blood urea, LDL_C is low-density lipoprotein cholesterol, ALT is alanine aminotransferase, AGE is age, PCV is hematocrit, GGT is gamma-glutamyl transferase, BP_HIGH is systolic blood pressure, ALB is serum albumin, SCR is serum creatinine, and BMI is body mass index.

[0121] To establish a closed-loop validation mechanism, after completing the missing data, the system inputs the completed dataset into a pre-defined prediction module. This module employs an XGBoost classifier, using the completed features (such as glycated hemoglobin HBA1C, systolic blood pressure BP_HIGH, etc.) as input and the presence or absence of retinal disease as the binary classification label for model training and prediction. The prediction process follows a standard machine learning workflow: the dataset is divided into training and testing sets in a 7:3 ratio, and hyperparameters are tuned using five-fold cross-validation. Accuracy, recall, precision, F1 score, and AUC are used as evaluation metrics. First, a baseline model is trained using the original complete data (without missing data) to obtain benchmark performance on the test set (see the first row of Table 5). Subsequently, the data completed using the method of this invention for each missing scenario is input into the same discriminative prediction unit (XGBoost) to evaluate its predictive performance. The results (Table 5) show that even with a high missing rate of 70%, the model prediction AUC can still be maintained at 0.7751. This result directly confirms that the data processed by the dual-modal completion mechanism of this system can effectively support downstream prediction tasks, and achieves the design goal of using prediction performance (AUC) as a quantitative indicator to conduct closed-loop verification of the effectiveness of data completion.

[0122]

[0123] Generative prediction verification

[0124] By using an independent privacy-preserving data generation branch (TabDiff), 19 sets of synthetic data with a distribution highly consistent with the original training set were successfully generated. In terms of synthetic data quality, feature distribution visualization was achieved. Figures 6 to 21 The results show that the distributions of multiple core features in the generated data are highly similar to those in the original data, and the single-feature distribution fit is good, indicating that TabDiff can accurately capture the single-feature probability distributions of the original data. For example, Figure 6 The probability density curve of the synthetic data of glycated hemoglobin (HBA1C) almost completely overlaps with the original data, and the peak position and distribution width are highly consistent, indicating that the model accurately captures the central tendency and dispersion of this key indicator. Figure 7 The curve of systolic blood pressure (BP_HIGH) also closely matches the original data, with the main peak position and skewness characteristics being well reproduced; Figure 11As an indicator with a wide distribution range and long tail characteristics, serum creatinine (SCR) maintained a high degree of fit in the main density region of the synthetic data, with only a slight deviation in the extreme tail of high value, which is consistent with the generative model's restoration law of rare extreme values. Figure 18-21 The category frequency distributions of the classification features shown (such as kidney disease, lower extremity arterial disease, etc.) are also basically consistent, and the differences in the positive rates between real and synthetic samples are within acceptable ranges. At the same time, the correlations between multiple features are effectively preserved, providing a reliable data foundation for subsequent prediction tasks based on synthetic data.

[0125] After training the XGBoost model on each set of samples, predictions were made on the same test set. As shown in Table 6, the model trained on synthetic data achieved an average AUC of 0.786, demonstrating the system's ability to effectively alleviate data scarcity while protecting privacy. Furthermore, the average C2ST metric of 95.698 reflects a high degree of consistency in distribution between the synthetic and original data, further confirming the quality of the synthetic data.

[0126]

[0127] Meanwhile, for generative prediction, using the aforementioned 15 core features as conditions to directly generate DR labels, the AUC reached 0.7842. Although this result is slightly lower than that of the discriminative model, it provides agents with a new and potentially promising reasoning perspective, enriching their problem-solving toolset.

[0128] Conditional reasoning verification

[0129] To verify the functionality of the adaptive inference unit in the model inference module, a typical application was simulated: in a primary healthcare scenario, the input data was assumed to contain only a subset of easily accessible basic features (5 indicators), while a key feature (HBA1C) crucial to the prediction task was partially missing.

[0130] Following a pre-defined task flow, this unit first performs an initial prediction based on available basic features (AUC=0.671). Then, it calls a self-supervised diffusion model (SimpDM) to impute the missing key feature HBA1C. The imputed HBA1C is then combined with the basic features for the final prediction, resulting in a significant performance improvement (Table 7, AUC=0.7286).

[0131] This process fully demonstrates how the system can ensure the reliability of decisions and improve prediction accuracy by adaptively coordinating data completion and multi-stage inference when data is incomplete.

[0132]

[0133] Implementation of Column Feature Completion

[0134] As an extended application of the bimodal data completion unit, the system can be configured to handle scenarios where an entire column of features is completely missing. Specifically, when the system detects that data in a certain feature column is completely missing, a second completion mode can be triggered, invoking the conditional table diffusion model (TabDiff). This model uses all other observed feature columns as conditions and, through its inherent classifier-free guided generation process, restores the missing data in the entire column. Although this function was not fully verified in this experiment, based on the system's successfully implemented TabDiff data generation capability, those skilled in the art can understand and implement this application.

[0135] Continuous learning mechanism

[0136] To ensure system performance during long-term deployment, a model update and maintenance mechanism can be introduced. This mechanism continuously monitors the performance metrics of the prediction model (such as accuracy and AUC); when performance degradation exceeds a preset threshold, or when the system accumulates a sufficient amount of new data, the model update process is triggered. Updates can employ various techniques, such as full retraining, incremental learning, or fine-tuning with synthetic data augmentation, to ensure the model can adapt to potential changes in data distribution and maintain the system's advancement and reliability.

[0137] Multi-domain extended applications

[0138] The core of this system addresses the universal technical challenges of "managing missing data in tabular tables" and "collaborative optimization of prediction tasks." Therefore, it can be easily extended to numerous fields requiring missing data handling and prediction in structured tabular tables, not just the healthcare scenario described in the examples.

[0139] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A data missing information processing and prediction system based on a diffusion model, characterized in that, The system includes: The data governance module is used to receive structured tabular data containing missing values, identify the missing data patterns, and adaptively select the corresponding model from at least two preset diffusion models to complete the missing values ​​based on the identified missing patterns, and output the completed data. The diffusion models include self-supervised diffusion model and conditional table diffusion model. The model inference module, connected to the data governance module, is used to receive the completed data and perform analysis and prediction using a combination of discriminative and generative prediction to obtain prediction results. The result output module is connected to the model inference module and is used to format and output the prediction results. The prediction performance index output by the model inference module is configured as a feedback signal to evaluate the effectiveness of the data governance module in completing the data, thus forming a closed loop from data completion to prediction verification.

2. The system according to claim 1, characterized in that, The missing patterns include random missing parts of features and complete missing of an entire column of features; The data governance module is configured to: when the missing pattern is that some features are randomly missing, call the self-supervised diffusion model SimpDM to complete the missing data; when the missing pattern is that the entire column of features is completely missing, call the conditional table diffusion model TabDiff to perform generative completion based on the observed features.

3. The system according to claim 2, characterized in that, The conditional table diffusion model employs a classifier-free guidance mechanism during its generation process.

4. The system according to claim 1, characterized in that, The model inference module includes a collaborative prediction unit, which includes: The discriminative prediction subunit outputs the first prediction result based on a machine learning classifier. The generative prediction subunit, based on the conditional table diffusion model, uses input features as conditions and outputs a second prediction result through generative modeling. The result fusion subunit is used to fuse the first prediction result and the second prediction result to generate the prediction result.

5. The system according to claim 4, characterized in that, The generative prediction subunit reuses the conditional table diffusion model used in the data governance module to handle cases where all features are completely missing.

6. The system according to claim 1, characterized in that, The result output module also includes an interpretation unit, which is used to transform the prediction results into domain-specific insights or suggestions based on application domain rules.

7. The system according to claim 1, characterized in that, The data governance module also includes an adaptive inference unit, which receives instructions from the model inference module regarding the key missing features on which the current prediction task depends, and drives the corresponding completion model to perform targeted completion of the key missing features.

8. The system according to claim 1, characterized in that, The model inference module also includes a model update and maintenance unit, which monitors the prediction performance of the model inference module and automatically uses new data to iteratively optimize the prediction model inside the model inference module when a preset trigger condition is met.

9. A method for handling and predicting missing data based on a diffusion model, wherein the method is applied to the system described in any one of claims 1 to 8, characterized in that, The method includes the following steps: Step 1: Receive structured tabular data containing missing values; Step 2: Identify the missing data patterns, and based on the identified missing patterns, adaptively select the corresponding model from at least two preset diffusion models to complete the missing values, thus obtaining the completed data. Step 3: Based on the completed data, analyze and predict using a combination of discriminative and generative prediction methods to obtain the prediction results; Step 4: Format and output the prediction results; The prediction performance metrics output in the model inference step are used as feedback signals to evaluate the effectiveness of data completion in the data governance step.

10. The method according to claim 9, characterized in that, The model inference steps include: The first prediction result is obtained through a discriminative prediction model; The second prediction result is obtained by using a generative prediction model conditioned on the input features; The first prediction result and the second prediction result are fused together to obtain the prediction result.

Images