Method and device for supplementing missing values of customer data

Abstract

The application discloses a kind of client data missing value supplement method and device, for financial field and data processing technical field, the method includes: identifying that missing data record exists missing value in client data set;Mark out missing feature in each missing data record;For each missing feature, extract the data record of the missing value of the missing feature as sampling sample;Determine a plurality of related features related to the missing feature;The multidimensional feature combination obtained by the combination of the missing feature and the related feature is discretized to obtain the joint empirical distribution function of dimension;Generate multidimensional simulation data associated with the missing feature by Markov Chain Monte Carlo algorithm;According to the feature similarity of multidimensional simulation data and target missing data record, select the supplementary data corresponding to the missing feature from multidimensional simulation data.The application is to improve the accuracy and reliability of the supplementary data of client data missing value.

Description

Technical Field

[0001] The method and apparatus for supplementing missing customer data values ​​of the present invention can be used in the financial field and the field of data processing technology, and can also be used in any field other than the financial field. The application field of the method and apparatus for supplementing missing customer data values ​​of the present invention is not limited. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Data missing is a feature engineering problem that is difficult to handle in the modeling process. The quality of a model depends to a large extent not on how sophisticated the model design is, but on the quality of the sample data.

[0004] For a long time, feature engineering has lacked good methods for handling missing data. Traditional methods are often too simplistic, such as assigning 0 to all missing values; or performing distribution analysis on the features of missing values ​​to find the overall distribution function of the samples, and then randomly generating random samples under the distribution based on the distribution function to assign missing values. This approach of making rough predictions based on the statistical distribution of features will lose a lot of effective information from the data, and the reliability and accuracy of the supplemented data are extremely low. Summary of the Invention

[0005] This invention provides a method for supplementing missing values ​​in customer data, thereby improving the accuracy and reliability of the supplemented data. The method includes:

[0006] Identify missing data records with missing values ​​in the customer dataset; mark the missing features in each missing data record;

[0007] For each missing feature in the target missing data record, extract the data records with missing values ​​of the missing feature from the customer dataset and use them as sampling samples; perform feature analysis on the sampling samples to determine multiple related features associated with the missing feature;

[0008] The multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain the joint empirical distribution function corresponding to the missing feature.

[0009] Using the Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function;

[0010] Based on the feature similarity between the multidimensional simulation data and the target missing data record, supplementary data corresponding to the missing feature is selected from the multidimensional simulation data.

[0011] This invention also provides a device for supplementing missing customer data values, used to improve the accuracy and reliability of supplemented data for missing customer data values. The device includes:

[0012] The missing feature determination module is used to identify missing data records with missing values ​​in the customer dataset; and to mark the missing features in each missing data record.

[0013] The relevant feature determination module is used to extract data records containing missing values ​​of the missing feature from the customer dataset for each missing feature in the target missing data records, and use them as sampling samples; perform feature analysis on the sampling samples to determine multiple relevant features related to the missing feature;

[0014] The joint empirical distribution function determination module is used to discretize the feature dimensions of the multi-dimensional feature combination obtained by combining the missing feature and the related features, so as to obtain the joint empirical distribution function corresponding to the missing feature.

[0015] The multidimensional simulation data determination module is used to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function using the Markov chain Monte Carlo algorithm.

[0016] The supplementary data determination module is used to select supplementary data corresponding to the missing feature from the multidimensional simulation data based on the feature similarity between the multidimensional simulation data and the target missing data record.

[0017] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for supplementing missing customer data values.

[0018] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for supplementing missing customer data values.

[0019] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned method for supplementing missing customer data values.

[0020] In this embodiment of the invention, missing data records with missing values ​​in a customer dataset are identified; missing features in each missing data record are marked; for each missing feature in a target missing data record, data records with missing values ​​containing that missing feature are extracted from the customer dataset as sampling samples; feature analysis is performed on the sampling samples to determine multiple related features associated with the missing feature; the multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain a joint empirical distribution function corresponding to the missing feature; using a Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function; supplementary data corresponding to the missing feature is selected from the multidimensional simulated data based on the feature similarity between the multidimensional simulated data and the target missing data record. Compared with existing technologies that use fixed values ​​or features to roughly predict missing values ​​through statistical distribution, this invention first identifies and marks the specific missing features in each missing data record. By analyzing the correlation between each missing feature and other features in the target missing data record, multiple related features are selected. Then, for the selected related features and the target missing feature, a multi-dimensional feature combination is introduced and discretized to obtain a joint empirical distribution function. This overcomes the limitation of existing technologies that ignore the correlation between features, utilizing the inherent correlation of data to improve the accuracy and rationality of supplementary data. By employing the Markov chain Monte Carlo algorithm, multi-dimensional simulated data is generated based on the joint empirical distribution function, which can efficiently simulate complex distributions and help generate multi-dimensional simulated data that is closer to the actual data distribution, thereby improving the credibility of the supplementary data. Finally, supplementary data is selected based on the feature similarity between the multi-dimensional simulated data and the target missing data record, avoiding blind filling and ensuring that the supplementary data is highly similar to the original data in the feature space, thus improving the accuracy of data supplementation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0022] Figure 1 This is a flowchart illustrating a method for supplementing missing values ​​in customer data according to an embodiment of the present invention.

[0023] Figure 2 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0024] Figure 3This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0025] Figure 4 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0026] Figure 5 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0027] Figure 6 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0028] Figure 7 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0029] Figure 8 This is a specific example diagram of a method for supplementing missing values ​​in customer data according to an embodiment of the present invention;

[0030] Figure 9 This is a schematic diagram of a device for supplementing missing customer data values ​​in an embodiment of the present invention;

[0031] Figure 10 This is a schematic diagram of a computer device used to supplement missing values ​​in customer data in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0033] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] In the description of this specification, the terms "comprising," "including," "having," and "containing" are open-ended terms, meaning that they include but are not limited to. The terms "an embodiment," "a specific embodiment," "some embodiments," and "for example," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. The order of steps involved in the various embodiments is used to illustrate the implementation of this application, and the order of steps is not limited and can be adjusted appropriately as needed.

[0035] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. The information collected in this application is authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant national and regional laws, regulations, and standards, and necessary confidentiality measures have been taken. This application does not violate public order and good morals, and provides corresponding access points for users to choose to authorize or refuse. In addition, this application provides users with corresponding access points to choose to agree to or refuse automated decision-making results. If the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0036] It should be noted that in the embodiments of this application, certain existing solutions in the industry, such as software, components, and models, may be mentioned. For example, some existing software tools, components, algorithm models, or solutions well-known in other technical fields may be cited. These should be considered exemplary, and their purpose is only to illustrate the feasibility of implementing the technical solution of this application. These mentions should be understood as typical examples, and their core purpose is to illustrate and verify the rationality and feasibility of implementing the technical solution proposed in this application. However, this does not mean that the applicant has already used or necessarily used the solution. Such citations do not imply that the applicant has actually adopted these existing solutions, or that it will necessarily adopt these methods in its technical implementation process in the future. In other words, these mentions are only illustrative in nature, helping to understand the connection and transcendence of the innovation points of this application with the prior art, and do not constitute an endorsement or reliance statement on a specific prior art product.

[0037] Data missing is one of the most troublesome feature engineering problems in modeling. The quality of a model largely depends not on how sophisticated its design is, but on the quality of the sample data. Just like a steel mill can only produce high-end steel if it provides refined ore of a certain purity.

[0038] For a long time, feature engineering has lacked good methods for handling missing data. A simple and crude solution is to assign 0 to all missing values. A more advanced approach is to perform distribution analysis on the features of missing values, find the overall distribution function of the sample, and then randomly generate random samples under that distribution to assign missing values. However, this approach loses a lot of effective information from the data, and the reliability of the supplemented data is extremely low. Therefore, how to use sampling techniques to solve the impact of insufficient data on model accuracy has become a very valuable task.

[0039] Traditional processing methods are often oversimplified, such as assigning 0 to all missing values; or performing distribution analysis on the characteristics of missing values ​​to find their overall sample distribution function, and then randomly generating random samples under that distribution based on the distribution function and assigning them missing values. This approach of making rough predictions based on the statistical distribution of features will lose a lot of effective information from the data, and the reliability and accuracy of the supplemented data are extremely low.

[0040] To address the aforementioned problems, embodiments of the present invention provide a method for supplementing missing customer data values, thereby improving the accuracy and reliability of the supplemented data. See [link to relevant documentation]. Figure 1 The method may include:

[0041] Step 101: Identify missing data records with missing values ​​in the customer dataset; mark the missing features in each missing data record;

[0042] Step 102: For each missing feature in the target missing data record, extract the data records with missing values ​​containing that missing feature from the customer dataset as a sampling sample; perform feature analysis on the sampling sample to determine multiple related features associated with the missing feature;

[0043] Step 103: Discretize the feature dimensions of the multidimensional feature combination obtained by combining the missing feature and the related features to obtain the joint empirical distribution function corresponding to the missing feature;

[0044] Step 104: Using the Markov chain Monte Carlo algorithm, generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function;

[0045] Step 105: Based on the feature similarity between the multidimensional simulation data and the target missing data record, select supplementary data corresponding to the missing feature from the multidimensional simulation data.

[0046] In this embodiment of the invention, missing data records with missing values ​​in a customer dataset are identified; missing features in each missing data record are marked; for each missing feature in a target missing data record, data records with missing values ​​containing that missing feature are extracted from the customer dataset as sampling samples; feature analysis is performed on the sampling samples to determine multiple related features associated with the missing feature; the multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain a joint empirical distribution function corresponding to the missing feature; using a Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function; supplementary data corresponding to the missing feature is selected from the multidimensional simulated data based on the feature similarity between the multidimensional simulated data and the target missing data record. Compared with existing technologies that use fixed values ​​or features to roughly predict missing values ​​through statistical distribution, this invention first identifies and marks the specific missing features in each missing data record. By analyzing the correlation between each missing feature and other features in the target missing data record, multiple related features are selected. Then, for the selected related features and the target missing feature, a multi-dimensional feature combination is introduced and discretized to obtain a joint empirical distribution function. This overcomes the limitation of existing technologies that ignore the correlation between features, utilizing the inherent correlation of data to improve the accuracy and rationality of supplementary data. By employing the Markov chain Monte Carlo algorithm, multi-dimensional simulated data is generated based on the joint empirical distribution function, which can efficiently simulate complex distributions and help generate multi-dimensional simulated data that is closer to the actual data distribution, thereby improving the credibility of the supplementary data. Finally, supplementary data is selected based on the feature similarity between the multi-dimensional simulated data and the target missing data record, avoiding blind filling and ensuring that the supplementary data is highly similar to the original data in the feature space, thus improving the accuracy of data supplementation.

[0047] In practice, the first step is to identify missing data records with missing values ​​in the customer dataset and mark the missing features in each missing data record.

[0048] In this embodiment, the first step involves the accurate identification and feature labeling of missing values ​​in the customer dataset, which is the foundation for building an efficient data supplementation mechanism. The specific steps are as follows:

[0049] First, a comprehensive review of the collected customer dataset is conducted. Advanced data cleaning and preprocessing techniques are employed to automatically identify and label all data records containing missing values. This process, implemented through programming logic, efficiently traverses every record in the dataset, identifying any unfilled or invalid fields to ensure no omissions.

[0050] After identifying records with missing values, this invention further refines the operation, precisely marking the specific missing features in each record. This means that for each data record, the system not only identifies the presence of missing values ​​but also clearly indicates which feature value(s) are missing. This approach provides a clear objective for subsequent feature engineering processing, enabling targeted data supplementation.

[0051] Through the above steps, this invention not only ensures a comprehensive understanding of the data missing situation but also lays a solid foundation for subsequent intelligent data supplementation algorithms based on Markov chain Monte Carlo statistical sampling methods. This series of operations not only improves the efficiency of data processing but also ensures the accuracy and relevance of data supplementation, effectively overcoming the shortcomings of existing technologies in handling missing data—namely, their coarseness and lack of detailed feature identification.

[0052] In specific implementation, after step 101: identifying missing data records with missing values ​​in the customer dataset; and marking the missing features in each missing data record, step 102: for each missing feature in the target missing data record, extracting the data records with missing values ​​containing the missing feature from the customer dataset as sampling samples; and performing feature analysis on the sampling samples to determine multiple related features associated with the missing feature.

[0053] In this embodiment, for each identified missing feature in the target missing data records, the aim is to accurately extract records with the same missing feature from the large customer dataset as sampling samples. This strategy aims to construct a targeted sample set specifically for analyzing and handling the problem of supplementing specific missing features. The specific operations are as follows:

[0054] First, data mining algorithms are used to systematically filter out all data records containing specific missing features from the customer dataset. This process requires not only comprehensive control over the dataset but also a high degree of screening accuracy to ensure that the extracted sample set focuses on the missing features being studied and eliminates interference from other factors.

[0055] Next, in-depth feature analysis is performed on the extracted samples. Statistical methods and machine learning techniques are used to analyze the performance of these samples on features other than the missing feature, in order to identify multiple features highly correlated with the target missing feature. In this step, statistical measures such as Pearson correlation coefficient and mutual information are used to quantify the correlation strength between features, thereby ensuring that the selected features are not only closely related to the missing feature, but also provide rich contextual information and reliable supplementary basis for subsequent missing value prediction.

[0056] Through the above two steps, this invention not only ensures the relevance and effectiveness of the sampling samples, but also deeply explores the intrinsic relationships between data through feature analysis strategies, providing a foundation for subsequent multi-dimensional linked sampling and data supplementation using the Markov chain Monte Carlo algorithm. This innovative method significantly surpasses the simple and crude processing methods in existing technologies, greatly improving the accuracy of missing data supplementation and the predictive performance of the model by fully utilizing the complex correlations between data.

[0057] In one embodiment, for each missing feature in the target missing data records, data records containing missing values ​​of that missing feature are extracted from the customer dataset and used as a sampling sample, such as... Figure 2 As shown, it includes:

[0058] Step 201: For each missing feature in the target missing data record, determine the set of no missing features for that target missing data record; the set of no missing features consists of data records excluding the missing features present in the missing data record.

[0059] Step 202: Extract the data records of missing values ​​with the missing feature and each feature value in the set of no missing features from the customer dataset, and use them as sampling samples.

[0060] In this embodiment, a systematic data processing workflow was employed for each missing feature to be supplemented, to ensure the accuracy and effectiveness of the data supplementation. The specific operation steps are as follows:

[0061] First, for each missing feature in the target missing data record, an algorithm identifies and determines the set of all non-missing features in that record. This set of non-missing features represents all known and valid data features in the target record, excluding the specific missing feature. This step is crucial for constructing the selection criteria for subsequent sampling, as it forms the basis for analyzing correlations and building similarity models.

[0062] Based on the definition of a set of features without missing features, data records with the following characteristics are carefully selected from the entire customer dataset as sampling samples: these records not only need to have missing values ​​for the specific missing features, but also their feature values ​​should match the corresponding feature values ​​in the target missing data records in the remaining set of features without missing features. In this way, a highly relevant and targeted sampling sample set is constructed, providing high-quality input data for subsequent multi-dimensional linked sampling and data supplementation using the Markov chain Monte Carlo algorithm.

[0063] This innovative process, by precisely selecting records with similar missing features to the target data, not only effectively utilizes the correlation between data points but also minimizes prediction errors caused by feature mismatches, providing a more intelligent and refined solution to the data missing problem. This method overcomes the limitations of existing technologies that simply replace or ignore missing values; through in-depth analysis of the internal structure of the data, it significantly improves the accuracy of data supplementation and the overall performance of the model.

[0064] In one embodiment, feature analysis is performed on the sampled data to identify multiple related features associated with the missing feature, such as... Figure 3 As shown, it includes:

[0065] Step 301: Select features from the sampled sample that belong to the same feature category as the missing feature as the selected features;

[0066] Step 302: Based on the feature values ​​of the sampled sample, calculate the correlation coefficient between the selected feature and each other feature in the sampled sample;

[0067] Step 303: Identify a preset number of other features whose correlation coefficient with the selected feature exceeds a preset value as related features to the missing feature.

[0068] In implementing this invention, the extracted sample aims to discover multiple features closely related to the target missing feature. This step ensures the high relevance and accuracy of the supplementary data. The specific steps are as follows:

[0069] First, features belonging to the same feature category as the target missing feature are identified from the sampled data. These features, due to their similarity in data structure and business logic, are considered potential related feature selection targets. This operation helps narrow the scope of analysis, focusing on the data dimensions most directly related to the missing feature.

[0070] Subsequently, statistical and machine learning methods are used to analyze each feature value in the sampled sample, calculating the correlation coefficient between the selected feature and all other features in the sampled sample. This step encompasses various correlation measures, including but not limited to Pearson correlation coefficient and Spearman rank correlation coefficient, to comprehensively capture both linear and nonlinear relationships between features.

[0071] Based on the calculated correlation coefficients, a preset numerical threshold is set, retaining only those features whose correlation coefficients with the selected features exceed this threshold. Furthermore, a preset number is defined, selecting the most strongly correlated features (a preset number) from those meeting the criteria, confirming them as closely related to the target missing feature. This strategy ensures the relevance of the supplementary data while controlling the complexity of the analysis and preventing overfitting.

[0072] By employing the above method, this invention not only accurately identifies the feature set highly correlated with the missing features, but also ensures the reliability of the selected features and the effectiveness of the supplementary data through a scientifically sound screening mechanism, providing a solid foundation for subsequent data imputation based on the Markov chain Monte Carlo algorithm. This strategy is significantly superior to traditional methods, which often overlook the complex inherent relationships within the data, resulting in limited accuracy and practicality of the data supplementation.

[0073] In the above embodiments, based on the feature values ​​of the sampled sample, the correlation coefficient between the selected feature and each other feature in the sampled sample is calculated, such as... Figure 4 As shown, it includes:

[0074] Step 401: Based on the absolute value of the feature value of the sampled sample, calculate the Pearson correlation coefficient, Spearman correlation coefficient, and rank correlation coefficient between the selected feature and each other feature in the sampled sample;

[0075] Step 402: Based on the preset weighting coefficients, perform a weighted average of the Pearson correlation coefficient, Spearman correlation coefficient, and rank correlation coefficient to obtain the correlation coefficient between the selected feature and each other feature in the sampled sample.

[0076] In one embodiment, during the implementation of this invention, the feature values ​​of the sampled samples are analyzed in depth to ensure the accuracy and efficiency of data supplementation. Specifically, a mathematical calculation process is used to evaluate the correlation between features, with the following steps:

[0077] First, for each feature value in the sample, its absolute value is standardized to eliminate the influence of dimensions and scale. Then, for each selected feature, the Pearson correlation coefficient, Spearman correlation coefficient, and rank correlation coefficient are calculated between it and all other features in the sample. These coefficients reflect the linear, monotonic, and ordinal relationships between features, respectively, and together constitute a comprehensive correlation assessment system.

[0078] Considering the limitations of a single correlation coefficient, this invention further introduces a weighted average method to integrate different types of correlation information. Specifically, based on pre-set weight coefficients (a, b, and c corresponding to the weights of Pearson, Spearman, and rank correlation coefficients, respectively), the three calculated correlation coefficients are weighted and summed to obtain a comprehensive correlation coefficient that considers linear, nonlinear, and ranking relationships. This strategy aims to more comprehensively and accurately measure the complex relationships between features, ensuring the accuracy and applicability of the supplementary data.

[0079] Through the above steps, this invention not only achieves a multi-dimensional assessment of the correlation between data features, but also optimizes the calculation of the correlation coefficient through a weighted average method, making it more closely reflect the complex characteristics of actual data. This innovative method, when dealing with data missing features, can more effectively identify other features closely related to the target missing feature, providing strong theoretical and technical support for subsequent intelligent data supplementation based on these features.

[0080] In specific implementation, in step 102: for each missing feature in the target missing data record, extract the data records with missing values ​​of the missing feature from the customer dataset as sampling samples; perform feature analysis on the sampling samples to determine multiple related features associated with the missing feature, and then proceed to step 103: discretize the feature dimensions of the multidimensional feature combination obtained by combining the missing feature and the related features to obtain the joint empirical distribution function corresponding to the missing feature.

[0081] In this embodiment, a highly systematic discretization method was used to construct a joint empirical distribution function for the identified missing features and their related features. The specific operation steps are as follows:

[0082] First, the missing target feature is combined with several relevant features identified in previous steps to form a multi-dimensional feature combination. This combination strategy aims to capture the complex correlations inherent in the data, providing a comprehensive and accurate framework for subsequent discretization and distribution estimation.

[0083] To effectively manage and analyze multidimensional data, this invention employs a discretization technique. For each feature dimension, an equidistant or equal-frequency binning strategy is implemented, dividing the continuous feature value range into a predetermined number of discrete intervals, i.e., bins. For example, for a certain feature, its value range may be divided into 10 equally spaced bins, each bin representing a specific value range. This process not only reduces the complexity of data processing but also ensures that the discretized data can better preserve the characteristics of the original feature distribution.

[0084] After discretizing the feature dimensions, the frequency of each bin is calculated based on the discretized data; that is, the proportion of data points in each bin to the total number of data points. Using these frequency values, a joint empirical distribution function is constructed for the missing features. This function integrates the discrete probabilities of each feature in the multi-dimensional feature combination, forming a model that describes the joint probability of these features. This joint distribution function is derived from practical data experience, laying the foundation for subsequent generation of simulated data conforming to a specific distribution using the Markov chain Monte Carlo algorithm.

[0085] Through the above discretization process and the construction of the joint distribution function, this invention not only effectively manages the complexity of high-dimensional data, but also improves the accuracy and efficiency of the data completion algorithm, ensuring that the inherent relationship between data can be fully utilized when processing missing data, thereby improving the accuracy and reliability of model prediction and analysis.

[0086] In one embodiment, the multidimensional feature combination obtained by combining the missing feature and the related features is discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature, such as... Figure 5 As shown, it includes:

[0087] Step 501: Combine the missing feature and the relevant feature to obtain a multidimensional feature combination;

[0088] Step 502: Obtain the feature values ​​of the multidimensional feature combination from the sampled sample;

[0089] Step 503: Based on the binning processing rules of the multidimensional discretized sample space, the feature values ​​of the multidimensional feature combination are discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature.

[0090] In the above embodiments, when facing the problem of missing features in a dataset, this invention proposes an innovative method aimed at effectively generating high-quality supplementary data through a rigorous mathematical processing procedure. Specifically, the missing features and their related features are integrated and then discretized to construct a joint empirical distribution function for the missing features. The entire process strictly follows statistical principles and machine learning theory.

[0091] First, for a missing feature in a customer data record, this invention employs a systematic method to identify other features highly correlated with it. The selection of these correlated features is based on a comprehensive consideration of multiple correlation coefficients (including but not limited to Pearson, Spearman, and rank correlation coefficients), and a weighted combination method ensures accurate capture of key relationships between features even in nonlinear relationships. Several features with the highest correlation to the missing feature are selected, typically not exceeding three dimensions to maintain computational feasibility, to form a high-dimensional feature combination. This combination strategy not only considers the direct correlation between features but also their potential interactions in higher-dimensional space, providing a richer information foundation for subsequent discretization and distribution estimation.

[0092] After constructing the multidimensional feature combinations, this invention utilizes existing sampled data as the data source. These samples are screened to ensure that only observations with complete data records for both the missing feature and its related features are included. Subsequently, the feature values ​​of these multidimensional feature combinations are discretized according to predefined binning rules. Each feature dimension is divided at equal intervals; for example, the value range of each feature is uniformly divided into 10 bins. This simplifies the complexity of subsequent analysis while relatively accurately preserving the distribution pattern of the original data.

[0093] After discretization, the frequency of each bin is used to estimate the joint empirical distribution of multidimensional feature combinations. This distribution function reflects the probability of occurrence of each discretized feature value combination, essentially an empirical approximation of the joint distribution of the original dataset. The joint empirical distribution function obtained in this way not only considers the distribution characteristics of individual features, but more importantly, it captures the complex interaction effects between features, providing a solid statistical basis for the reasonable imputation of missing data.

[0094] In summary, this invention provides a novel and effective approach to solving the data missing problem by constructing a multi-dimensional feature combination, obtaining and discretizing feature values, and finally obtaining a joint empirical distribution function. This method not only overcomes the limitations of traditional single feature imputation techniques but also significantly improves the accuracy and robustness of model predictions while maintaining data integrity.

[0095] In specific implementation, after performing step 103: discretizing the feature dimensions of the multidimensional feature combination obtained by combining the missing feature and the related features to obtain the joint empirical distribution function corresponding to the missing feature, step 104: generating multidimensional simulation data associated with the missing feature based on the joint empirical distribution function using the Markov chain Monte Carlo algorithm.

[0096] In this embodiment, a Markov Chain Monte Carlo (MCMC) algorithm is employed to efficiently generate multidimensional simulated data closely associated with missing features, based on the constructed multidimensional feature combination and the corresponding joint empirical distribution function. The MCMC algorithm, with its powerful sampling capabilities, can simulate complex distributions and converge to the target distribution. It is particularly suitable for handling high-dimensional and non-standard distribution data, providing a powerful tool for the accurate imputation of missing data.

[0097] First, based on the joint empirical distribution function obtained in the preceding steps, a suitable Markov chain model is defined, which reflects the transition probabilities between various features. The state space of the Markov chain corresponds to the discrete combinations of values ​​in the joint empirical distribution function, with each state representing a specific combination of multidimensional feature values. By constructing a probability transition matrix to describe the probability of transitioning from the current state to the next state, this ensures that the algorithm can gradually approximate the steady-state probability distribution described by the joint empirical distribution function after multiple iterations.

[0098] A starting state, i.e., a combination of multidimensional feature values, is randomly selected from the discretized sample space of the joint empirical distribution as the initial point of the MCMC algorithm. Subsequently, a warm-up phase is performed, allowing the Markov chain to run for a sufficiently long time to overcome the influence of the initial state and ensure that subsequent sampling reflects the target distribution.

[0099] In each iteration, a new candidate state (i.e., a new combination of multidimensional eigenvalues) is proposed based on the current state and the probability transition matrix. Whether to accept this transition depends on an acceptance probability calculated based on the Metropolis-Hastings criterion or other relevant criteria to ensure that the final samples of the algorithm conform to the joint empirical distribution of the target.

[0100] The algorithm is iterated until it reaches a predetermined convergence criterion, such as by monitoring intra-chain correlations or using convergence diagnostics of multiple independent chains. Once the Markov chain is determined to have reached a steady state, samples are extracted from the chain as simulation data. These samples are considered reasonable supplements to missing features because they are statistically consistent with the patterns of the original dataset.

[0101] Through the steps described above, this invention achieves efficient and accurate generation of simulated data for missing features. This data not only maintains consistency with the original dataset but also fully utilizes the correlations between multidimensional features, providing more complete and high-quality information support for subsequent data analysis and model building. This method not only enhances the accuracy and robustness of model predictions but also provides an innovative and practical strategy for handling data missing issues in large datasets.

[0102] In one embodiment, a Markov chain Monte Carlo algorithm is used to generate multidimensional simulated data associated with the missing feature based on the joint empirical distribution function, including:

[0103] Using the multivariate Gibbs sampling algorithm in the Markov chain Monte Carlo algorithm, based on the pre-set transition matrix, the joint empirical distribution function is taken as the steady-state distribution π, and simulated samples are extracted from the joint empirical distribution function to generate multidimensional simulated data under the same distribution, which serves as multidimensional simulated data associated with the missing feature.

[0104] In one embodiment, for accurate data imputation, the Markov Chain Monte Carlo (MCMC) algorithm framework, especially its core component—the multivariate Gibbs sampling algorithm—is used to generate multidimensional simulated data closely related to the missing features in a scientific and systematic manner. This process is detailed below:

[0105] First, a subset is extracted from customer data with complete feature records, specifically designed for customer data lacking certain features. This data is then discretized into a predetermined number of intervals for each feature dimension (e.g., 10 equally spaced intervals per feature dimension), thereby constructing a joint empirical distribution function. This function captures the joint probability distribution among the features, providing a foundation for subsequent sampling.

[0106] Based on the joint empirical distribution among features, a suitable transition matrix is ​​defined, which reflects the probability of transitioning from one eigenvalue combination state to another during the MCMC process. This transition matrix must ensure that after a sufficiently long iteration sequence, the Markov chain converges to a stable probability distribution, the so-called steady-state distribution π, which closely matches the joint empirical distribution function.

[0107] Within the MCMC framework, a multivariate Gibbs sampling algorithm is employed to iteratively update the feature values ​​in each dimension while keeping the feature values ​​in other dimensions fixed, gradually approximating the steady-state distribution π. In each iteration, for a given missing feature, simulated samples are extracted based on the current feature values ​​in other dimensions and the joint empirical distribution function, ensuring that the newly generated feature values ​​maintain statistical consistency with the existing feature value set. This process is repeated until a predetermined convergence criterion is met, such as good chain mixing or the convergence diagnostic index is satisfied.

[0108] As Gibbs sampling proceeds, simulated samples drawn from the joint empirical distribution function gradually accumulate, forming a series of multidimensional simulated data highly correlated with the missing features. These data not only match the original dataset in statistical properties but also fully consider the complex relationships between features. Therefore, as a supplement to the missing data, they can significantly improve the model's predictive ability and stability.

[0109] In summary, this invention utilizes the multivariate Gibbs sampling technique in the Markov chain Monte Carlo algorithm, based on a joint empirical distribution function, to systematically generate multidimensional simulated data closely related to missing features. This effectively solves the problem of missing data in big data analysis and improves the accuracy and practicality of data analysis.

[0110] In specific implementation, after step 104: generating multidimensional simulated data associated with the missing feature using the Markov chain Monte Carlo algorithm based on the joint empirical distribution function, step 105: selecting supplementary data corresponding to the missing feature from the multidimensional simulated data based on the feature similarity between the multidimensional simulated data and the target missing data record.

[0111] In this embodiment, to accurately supplement the features of the target missing data records, the present invention proposes an intelligent screening mechanism based on the feature similarity comparison between multidimensional simulated data and the target missing data records. The specific operation process is as follows, aiming to ensure the accuracy and logical consistency of the supplemented data:

[0112] First, a comprehensive feature similarity calculation model is constructed. This model not only considers traditional similarity measurement methods, such as Euclidean distance and cosine similarity, but also incorporates a weighted strategy based on feature relevance. Specifically, for each missing target data record, the corresponding multi-dimensional feature vector from the related simulated dataset is selected. By calculating the comprehensive correlation coefficient between the target record and the target record across the selected three feature dimensions (combining Pearson correlation coefficient, Spearman rank correlation coefficient, and Kendall rank correlation coefficient, with different weights assigned according to actual needs), a comprehensive similarity score reflecting global similarity is obtained.

[0113] To efficiently filter high-quality supplementary candidate data from massive amounts of multidimensional simulation data, a strict similarity threshold, such as 90%, is set. This means that only simulation data that shows extremely high similarity to the target missing data record on selected features (i.e., similarity scores higher than the set threshold) will be further considered as supplementary data sources.

[0114] Based on the aforementioned similarity calculation model, each data point in the multidimensional simulated dataset is compared one by one with the target missing data record. Data points that meet the threshold requirements are selected according to the similarity score, and the indices of these data points are labeled as a set. The data in this set are not only numerically similar to the target record, but more importantly, they maintain a high degree of consistency in the correlation patterns between features, thus providing a solid foundation for the effective supplementation of missing features.

[0115] For each missing feature in the target missing data records, the simulated data point with the highest similarity is selected from the set, and its corresponding feature value is used as a candidate value for the supplementary data. If multiple data points in the simulated dataset have the same highest similarity score, the average or median of these data points on that feature can be used as the final supplementary value to increase the robustness of the prediction. This method ensures that the supplementary data is consistent with the original dataset in terms of statistical characteristics, while also preserving the inherent structure and correlation of the data to the greatest extent.

[0116] In summary, this invention, through a similarity calculation model and a rigorous data screening process, enables the accurate selection of supplementary data corresponding to the missing features from a multidimensional simulated dataset, effectively solving the problem of missing data and improving the accuracy and completeness of data analysis.

[0117] In one embodiment, supplementary data corresponding to the missing feature is selected from the multidimensional simulation data based on the feature similarity between the multidimensional simulation data and the target missing data record, such as... Figure 6 As shown, it includes:

[0118] Step 601: Obtain the first value of the relevant feature in the target missing data record;

[0119] Step 602: Calculate the feature similarity between the value of the relevant feature in each multidimensional simulation data and the first value;

[0120] Step 603: Use the multidimensional simulation data whose feature similarity exceeds a preset value as the target multidimensional simulation data;

[0121] Step 604: Calculate supplementary data corresponding to the missing feature based on the feature value selected from the target multidimensional simulation data corresponding to the missing feature.

[0122] In the implementation of this invention, a sophisticated screening and calculation strategy is employed to supplement the features of the target missing data records, ensuring the accuracy and reliability of the supplemented data. The specific operation steps are as follows:

[0123] First, other features related to the feature to be supplemented (i.e., relevant features) are extracted from the target missing data records, and the specific values ​​of these relevant features are recorded, referred to as the "first values". These values ​​serve as a reference benchmark for evaluating the similarity of the simulated data.

[0124] Subsequently, for each multidimensional simulated data point, the similarity between its corresponding relevant feature values ​​and the "first value" determined in the first step is calculated. Various statistical measures can be used for similarity calculation, such as Euclidean distance and cosine similarity; the specific choice depends on the actual data characteristics and requirements. This process aims to identify the simulated data that is most similar to the target record in terms of feature distribution.

[0125] Based on a pre-set threshold, simulated data whose relevant feature values ​​are more similar to the target record than this threshold are selected from all multidimensional simulation data. This data is considered "target multidimensional simulation data." This selection step ensures that only highly matching simulation data is considered, thereby improving the quality of the supplementary data.

[0126] Finally, based on the selected target multidimensional simulation data, corresponding feature values ​​are selected from these data for the missing features, and one or a set of supplementary data values ​​are calculated using statistical methods (such as the mean, median, or other statistics selected according to the specific context). These calculated values ​​are considered reasonable supplements to the missing features, aiming to restore the true situation of the missing data to the greatest extent possible, and to ensure the accuracy and completeness of model prediction and analysis.

[0127] Through the above steps, this invention systematically utilizes the feature similarity between multidimensional simulated data and target missing data records to achieve accurate supplementation of missing features, effectively solving the challenges brought about by missing data and improving the scientificity and practicality of data processing.

[0128] In one embodiment, such as Figure 7 As shown, it also includes:

[0129] Step 701: Receive the actual data of the missing feature;

[0130] Step 702: Based on the real data and the supplementary data, correct the similarity parameter used in calculating the supplementary data corresponding to the missing feature.

[0131] During the continuous optimization phase of this invention, a dynamic feedback and self-adjustment mechanism was designed to ensure the long-term accuracy of data supplementation and the continuous improvement of the model. This mechanism specifically includes the following steps:

[0132] First, the system can receive and integrate real data on previously missing features subsequently entered by users or the system. This step is based on the importance of continuous data updates and improvements, ensuring that the model can absorb new information in a timely manner and improve the timeliness and accuracy of predictions.

[0133] For the missing feature data that has been supplemented, the system uses the newly acquired real data for comparative analysis. This process involves reviewing the similarity calculation parameters used when supplementing the data, including but not limited to the comprehensive correlation coefficient of other features related to the missing feature.

[0134] Based on the differences between real and supplementary data, the system designs a correction algorithm to adjust the similarity parameter in the calculation formula. In practice, if the evaluation reveals that the difference between the real and predicted values ​​indicates that the original similarity parameter underestimated or overestimated the importance of certain features, the system will adjust these parameters appropriately. For example, it may fine-tune the similarity coefficient as a percentage based on the direction and magnitude of the difference, aiming to optimize the performance of the prediction model. If the adjusted prediction result shows an improved match with the real data, the correction is confirmed; otherwise, a conservative strategy is adopted to avoid model instability caused by over-correction.

[0135] Through the steps described above, the model can not only respond instantly to data updates, but also continuously learn and iterate to optimize the algorithm logic for supplementing missing data, thereby improving its ability to predict and supplement future data gaps. This dynamic correction mechanism imbues the model with the ability to self-evolve, ensuring that it can continuously adapt to data changes and maintain prediction accuracy and model performance in long-term applications.

[0136] In summary, this invention provides an intelligent and dynamically adjustable solution for data missing supplementation by integrating a real data feedback mechanism with a dynamic correction strategy for similarity parameters. This effectively overcomes the limitation of traditional methods that lack continuous optimization after missing data supplementation, and significantly improves the accuracy of model prediction and the level of intelligence in data processing.

[0137] In specific implementation, the method for supplementing missing customer data values ​​provided in this embodiment of the invention may further include:

[0138] If supplementary data corresponding to the missing feature cannot be selected from the multidimensional simulation data, then the Markov chain Monte Carlo algorithm is used again to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function; the above steps are repeated until supplementary data corresponding to the missing feature is selected.

[0139] In this embodiment, under certain circumstances, if the initially generated multidimensional simulation dataset fails to identify effective data suitable for filling in specific missing features, this invention proposes a cyclical feedback and optimization strategy to ensure the accuracy and completeness of data supplementation. The specific implementation steps are as follows:

[0140] First, a detailed analysis is performed on the multidimensional simulated dataset generated by the Markov Chain Monte Carlo (MCMC) algorithm to identify whether there are data records with sufficient matching degree to the feature to be completed. If, after rigorous screening, it is found that there are no data pairs in the current simulated dataset that are highly correlated with the missing feature, the restart process will be automatically triggered.

[0141] This step does not simply repeat the previous MCMC process, but rather, based on the experience gained from the initial attempt, involves a detailed review and possible adjustments to the joint empirical distribution function. This means using existing data feedback to deepen and optimize the understanding of the previously constructed joint distribution among features, thereby generating a more accurate transition matrix or adjusting sampling parameters to better approximate the intrinsic structure and patterns of the actual data.

[0142] Subsequently, based on the updated joint empirical distribution function, the MCMC algorithm will be used again to generate a new multidimensional simulation dataset. This process involves adjusting algorithm parameters and potential dimensionality reduction or increase strategies to improve the similarity and matching degree between simulated and real data while maintaining data generation efficiency.

[0143] The newly generated simulated dataset will undergo another rigorous similarity evaluation process, continuously monitoring until supplementary data that meets the preset similarity threshold is found. This process may require multiple iterations until data points that highly match the missing features are successfully identified and adopted as supplementary data.

[0144] Each iteration not only aims to fill in a single missing feature but also serves as a learning opportunity to fine-tune and optimize the entire data imputation algorithm. Through continuous feedback and adjustments, the system gradually refines its strategies for handling complex data gaps, ensuring the long-term stability and accuracy of the model.

[0145] Through the aforementioned iterative process, this invention ensures that even if the initial attempt fails, it can still accurately find suitable supplementary data for the missing features through continuous learning and optimization strategies. This demonstrates the system's flexibility and intelligence in data processing, effectively solves the problem of missing data, and improves the overall quality and reliability of data analysis.

[0146] For example, the intelligent supplementation and correction mechanism for model prediction results is a method for subsequent optimization of missing data that has already been supplemented. This mechanism is initiated after data supplementation is completed, aiming to continuously adjust and optimize the accuracy of this supplemented data. The specific operation process includes: First, for feature data that was originally missing but was later supplemented with true values ​​by the customer, the system will identify this part of the dataset and record the corresponding customer number.

[0147] Next, using the defined similarity calculation formula, which considers the comprehensive correlation coefficient between features to enhance the rationality of the similarity calculation, the previously predicted missing values ​​are compared and analyzed with the newly provided true values ​​by the client. By using the difference between the similarity of the predicted results and the true values ​​as the objective function, the system gradually fine-tunes the similarity parameters in the calculation formula in order to find the similarity configuration that minimizes the prediction error.

[0148] If the accuracy of the adjusted prediction improves, meaning it matches the true value better, the system accepts the correction. Otherwise, it adopts a conservative strategy, appropriately reducing the adjustment magnitude (e.g., increasing or decreasing the similarity coefficient by 1%) to ensure the stability of the prediction model is not significantly affected. This mechanism ensures the dynamism and self-optimization capability of data supplementation, continuously improving the model's performance and the accuracy of data supplementation over time with the addition of new data.

[0149] The following is a specific embodiment to illustrate the application of the method of the present invention. This specific embodiment illustrates the writing approach and specific application scenario of the present invention.

[0150] Data gaps are a common problem in real-world data analysis, and related processing methods are among the most cutting-edge technologies focused on by feature engineering. For a single customer data record, the absence of one or two key pieces of information often renders the entire record invalid and forcing its abandonment. Such situations of discarding entire data entries due to missing information are numerous, and discarding all incomplete data represents a significant waste of the dataset's information.

[0151] The main problems with existing technologies lie in two aspects: the inaccuracy of handling missing data and the lack of dynamic optimization mechanisms.

[0152] 1. Data augmentation methods are crude and lack intelligence: Existing feature engineering methods for handling missing data often employ simplistic and imprecise approaches, such as uniformly assigning all missing values ​​to 0 or making rough predictions based on the statistical distribution of features. These methods neglect the inherent relationships between data and the effective utilization of correlated information, resulting in inaccurate augmented data that fails to accurately reflect the true situation.

[0153] 2. Lack of effective secondary correction and feedback mechanisms: Once the data has been initially supplemented, existing technologies often lack further mechanisms to verify and correct the accuracy of this supplemented data. Even if truly missing data is later collected, this new information cannot be fed back into the model for correction, thus missing opportunities to improve the model's predictive accuracy. This results in the data supplementation effect not continuously optimizing with time and the addition of new data, limiting the model's predictive power and adaptability.

[0154] Therefore, the proposed solution in this embodiment aims to address the aforementioned problem by constructing an intelligent data missing value imputation algorithm based on Markov chain Monte Carlo (MCMC) statistical sampling. This algorithm not only generates sample data that more accurately conforms to the actual distribution and imputs missing values, but also incorporates a dynamic data self-correction mechanism to ensure that the data imputation results are continuously iterated and optimized with newly acquired real data, thereby improving the accuracy and long-term effectiveness of model predictions.

[0155] The specific implementation process of this plan can be summarized in the following steps:

[0156] 1. Data preprocessing and feature selection:

[0157] Identify records in the dataset that have missing values ​​and mark the missing features in each record.

[0158] Extract data records with complete features from the dataset to serve as the basic sampling population sample.

[0159] Perform feature analysis on the basic sampling population sample, and select features that are highly correlated with the missing features (using the comprehensive correlation coefficient method, considering Pearson, Spearman and rank correlation coefficients). Generally, select the three most correlated features.

[0160] 2. Feature discretization and sample space construction:

[0161] Discretize the selected features, for example, by dividing them into 10 bins, to reduce computational complexity while ensuring that the main distribution characteristics of the data are captured.

[0162] A four-dimensional feature combination is constructed, consisting of one missing feature and three highly correlated non-missing features, forming a joint discretized sample space.

[0163] 3. MCMC sampling generates simulation data:

[0164] Multivariate MCMC sampling methods, such as Gibbs sampling, are applied to generate a large amount of simulated data based on the joint empirical distribution function of four-dimensional feature combinations. This data is used to fill in missing values.

[0165] Ensure that the size of the simulated dataset is much larger than the original dataset to cover all possible combinations of values, thereby improving the accuracy and coverage of data supplementation.

[0166] 4. Similarity matching and data supplementation:

[0167] Design a similarity calculation method based on the comprehensive correlation coefficient to compare the similarity of each customer with the data in the simulated dataset.

[0168] Simulated data with a similarity to the target customer data higher than a certain threshold (such as 90%) are selected as predicted values ​​for missing values ​​to fill in the gaps.

[0169] If there is not enough similar data in the simulated dataset, MCMC sampling needs to be re-executed until enough high-similarity samples are obtained.

[0170] 5. Intelligent correction of model prediction results:

[0171] When the client provides actual missing data afterward, this information is used to correct the model.

[0172] By comparing the difference between the predicted value and the true value, the similarity weights between features are adjusted, and the data supplementation algorithm is fine-tuned to make the prediction result closer to the true value.

[0173] Implement a conservative correction strategy: if the correction causes the predicted result to deviate more from the true value, then appropriately reduce the correction magnitude; otherwise, increase it moderately to maintain the stability of the prediction.

[0174] The entire process revolves around intelligent data gap supplementation and dynamic optimization, aiming to overcome the shortcomings of traditional methods and improve the accuracy of model predictions and the effective utilization of data.

[0175] The goal of this embodiment is to explore a novel missing data completion algorithm based on a sampling method. For ease of understanding, the following example of a bank customer is used for illustration, but the applicable range of values ​​for this embodiment is by no means limited to this one scenario.

[0176] Assume the existing bank customer dataset is as shown in Table 1 below:

[0177] Table 1

[0178]

[0179] As shown in the table above, the characteristic data for customer k is labeled as follows:

[0180] X k =(x k1 ,x k2 ,x k3 ,……x kM), k = 1, 2, ..., N

[0181] N is the maximum number of customer IDs, which is also the number of customers, and M is the number of features.

[0182] {X k In the dataset {k = 1, 2, ..., N}, records with missing data are extracted and marked as follows: Where n is the dataset containing all missing data. The total amount, and simultaneously marking each The missing feature data is That is, for each missing data record Its missing feature data is marked as That is, the values ​​of these features are null, p t That is Total number of missing features.

[0183] To facilitate understanding, here is an example. Suppose that the characteristic data of customer 1 is known as X1 = (x 11 ,x 12 ,x 13 ,……x 1M Given that features 1, 3, and 4 are missing, the data record is as follows: Assuming customer 1 is also The first record with missing data in the list, then its location The mark in is Furthermore, it lacks three features, i.e., p1 = 3, which means... Assuming there are a total of 100 customers with missing data records, like Customer 1, then n = 100, meaning the dataset of customers with missing data is...

[0184] Having introduced the basic labeling, the next step is to construct a data supplementation algorithm based on the intelligent multidimensional linkage sampling method.

[0185] I. Markov Chain Monte Carlo Algorithm (MCMC)

[0186] The MCMC algorithm is a sampling technique built on the property that sampling of Markov chain transition matrices will eventually converge to a specific probability distribution. Based on this sampling method, data samples that follow a specific distribution can be stably generated, making it one of the most cutting-edge and promising application areas of sampling technology.

[0187] Suppose variable X has D states, namely states 1, 2, 3, ..., D. The probability transition matrix between these D states is as follows:

[0188]

[0189] The matrix finally yields element P. ij This represents the probability that variable X will change from its current state i to the next state j. Probability theory research on Markov chains has proven that, under certain conditions, the probabilities described by the transition matrix above can be derived. As variable X changes infinitely, it will eventually converge to a stable probability distribution, called the steady-state probability distribution. This steady-state probability distribution is independent of the initial value of X. An example can be used to illustrate this:

[0190] Suppose we have 1000 variables that are independent and identically distributed with respect to X, denoted as Y1, Y2, ..., Yn. 1000 They follow the same probability transition matrix P as X, and their steady-state probability distribution after multiple changes is denoted as π = (π1, π2, ..., π). D It can be proven that when the number of changes exceeds a certain amount, Y1, Y2, ..., Y 1000 In the state 1, 2, 3, ..., the number of D will remain basically stable. No matter how many times the number of changes increases, regardless of Y1, Y2, ..., Y 1000 What is the initial state? The proportion of this final practical state remains stable. It means rounding down, for example And it can be proven that Pπ = π.

[0191] Based on this characteristic, modelers can approximate and discretize the model according to the specific distribution F they need, that is, by taking the empirical distribution F of their samples. * , will F * Given a steady-state distribution π, and a suitable patent probability matrix P, a composite specific distribution F can be obtained by continuously generating new random number samples. * The new sample values ​​are obtained, which is the core idea of ​​the Markov chain Monte Carlo algorithm. Gibbs sampling and MH sampling, which are constructed based on this idea, are the culmination of this field.

[0192] Having introduced the basic sampling techniques, we can now focus on the core of this embodiment: a new intelligent algorithm based on the MCMC algorithm and big data analysis methods to achieve accurate evaluation of missing data.

[0193] II. Intelligent sampling data supplementation algorithm based on MCMC algorithm.

[0194] Given the known characteristic that the MCMC algorithm can produce specific distributions, we will now use this characteristic to achieve precise sampling of the target distribution function.

[0195] Using the same notation as before, suppose we have a customer Q, 1≤Q≤N, whose feature data is recorded as X. Q =(x Q1 ,x Q2 ,x Q3 ,……x QM ), and the number of its missing features is p. Q One, in If a value belongs to the q-th element, then the missing feature data of customer Q can be denoted as... Therefore, its set of features without missing data is:

[0196] We need to fill in the missing data one by one. First of all, it is The following steps are required to perform data supplementation:

[0197] (1) As can be seen from the markings The column to which it belongs is a feature Features in all customer data With H Q The sample records with no missing values ​​in the H feature are extracted to form the basic sampling population sample required for sampling. For ease of representation, H is... Q Marked as Therefore, the basic sampling population sample can be labeled as...

[0198] Where D Q In features With H Q The set of customers with no missing values ​​for the middle feature.

[0199] (2) Based on the basic sampling population sample B Q As a sample basis, discretizing it can yield features. With H Q The joint empirical distribution function of the features in the middle.

[0200] It should be noted that the complexity of the joint empirical distribution function increases exponentially with the increase of feature dimension. Discretizing the function using 10 bins per feature dimension results in 10 different bins in a 3-dimensional feature sample space. 3 To ensure that the observed samples can represent the overall probability distribution, statistically speaking, each feature dimension needs to be assigned at least 30 different observations to accurately represent a certain statistical trend or large sample property. That is, 30*30*30 = 2.7*10^30 observations are required. 4 30 different observation samples; and so on. If the sample consists of 4 features, the required observation samples are 30*30*30*30 = 8.1*10. 5Each additional feature dimension increases the sample complexity by a factor of 30.

[0201] To address the high complexity of real-world data sampling, the technical disclosure proposes using four-dimensional features to construct the sample space to balance accuracy and feasibility. This strategy aims to ensure the accuracy of data supplementation while avoiding excessive computational complexity. Figure 8 As shown, the specific steps include:

[0202] Step 801: Feature Selection: Select three features that are highly correlated with the missing target feature using the correlation coefficient method. This step utilizes a weighted combination of Pearson, Spearman, and rank correlation coefficients to comprehensively evaluate the correlation, with a greater emphasis on non-linear correlation (Spearman and rank correlation coefficients). The overall correlation coefficient is required to be at least 0.5 to ensure that the selected features are significantly correlated.

[0203] Step 802: Data preprocessing: Discretize the four selected features (including missing features and three highly correlated features) into 10 bins to form 10,000 possible combinations, thereby simplifying the multidimensional space and facilitating subsequent sampling operations.

[0204] Step 803: Feature Discretization: Convert the relevant features in the basic sampling population into box numbers to facilitate sampling and subsequent data supplementation.

[0205] Step 804: Multidimensional MCMC Sampling: A multidimensional MCMC algorithm, such as Gibbs sampling, is used to generate a large amount of simulated data based on the constructed four-dimensional discretized sample space. The size of the generated sample is much larger than the original data to ensure that there is enough data to accurately fill in missing values.

[0206] In summary, this technical solution provides an efficient and accurate method for missing data supplementation by carefully selecting highly relevant features, performing effective data preprocessing and discretization, and combining multidimensional MCMC sampling technology. It solves the problem of excessive complexity in high-dimensional data sampling while retaining sufficient information to improve the accuracy of data supplementation.

[0207] For example, this level of complexity is simply unacceptable for real-world data sampling. However, if the joint sample space dimension is too low, the lost joint distribution correlation information will be too large, which will seriously affect the accuracy of subsequent data supplementation algorithms based on the sampling results. Therefore, under a comprehensive balance, a dimension of 3-4 is generally acceptable.

[0208] Therefore, this embodiment uses 4-dimensional features to construct the sample space, that is, one dimension is the missing data feature. and H Q Three features in H QHow to select the three features? Here, we construct a correlation coefficient method for selection, and select the features. Three data points with high correlation are selected to improve the accuracy of the data supplementation algorithm. The specific processing steps are as follows:

[0209] 1. The basic sampling population sample B Q Chinese characteristics The corresponding data is extracted and matched one by one with H. Q The correlation coefficients of the various features in the data are calculated. Here, the correlation coefficient is not limited to the traditional Pearson correlation coefficient (denoted as β), because the Pearson correlation coefficient is a coefficient that reflects linear correlation and has significant limitations. To fully demonstrate the correlation, it is recommended to use the Spearman correlation coefficient (denoted as β). The correlation coefficient (τ) is calculated together with the rank correlation coefficient, and the composite correlation coefficient is obtained by weighting the three correlation coefficients to determine the overall correlation coefficient. Here, a, b, and c are the weighting coefficients, respectively.

[0210] The weighting coefficients can be determined independently, but it is generally recommended that the rank correlation coefficient have the highest weight, followed by the Spearman correlation coefficient, and then the Pearson correlation coefficient have the lowest weight. Note that the formula for ξ uses the absolute values ​​of the three correlation coefficients for calculation, because this embodiment requires the correlation with the feature... Other highly related H Q Features, whether positively or negatively correlated, if a certain feature is related to feature If the absolute value of the correlation between two features is close to 1, it means that the two features are almost uniquely mutually definite, which is of great significance for subsequent supplementary data. Conversely, if the absolute value of the correlation is close to 0, it means that the features are not mutually definite. Features with weak correlation, where the numerical relationship between the two is almost completely random, must be discarded. In practice, the overall correlation coefficient ξ of the selected features should generally be at least greater than 0.5. If all features are related to the feature... If the correlation coefficient ξ is less than 0.5, it indicates that the correlation with the characteristic... Highly correlated features do not exist; features If it is difficult to obtain supplementary information through other highly similar related features, then the focus on those features should be abandoned. Supplement the data.

[0211] 2. If it is related to the characteristics Related H Q If multiple correlation coefficients ξ of the features are greater than 0.5, then further data supplementation can be carried out, and H can be added. Q Zhongyu The three features with the highest correlation ξ were extracted and labeled as h. q1 h q2 hq3 Next, we will discuss these three features and... These features are combined to form a four-dimensional feature set. As mentioned above, each dimension of the feature is binned into 10 parts, which is essentially discretization. The constructed four-dimensional discretized sample space was divided into 10 4 There are 10 discrete value combinations (each feature dimension can take 10 different values, and the four dimensions combined mean that the sample in this sample space can have 10*10*10*10 = 10,000 different values).

[0212] The basic sampling population sample B Q Corresponding features The data is extracted and labeled as According to the binning rules of the four-dimensional discretized sample space, that is, the rule of dividing each dimension into 10 equal parts, the following will be performed: middle The values ​​of the four characteristics are converted into corresponding box numbers, which achieves... Discretization of the four features.

[0213] To facilitate understanding, the following example is given: Suppose Middle feature h q1 The data range is 0-1000. Dividing it into 10 equal bins results in the following bins: [0,100), [100,200), [200,300), [300,400), [400,500), [500,600), [600,700), [700,800), [800,900), [900,1000]. These correspond to bin numbers 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, respectively. Chinese h q1 The numerical values ​​can be converted into corresponding container numbers from 1 to 10 based on this correspondence and the range they belong to. When supplementing data based on these container numbers, the container numbers predicted by the supplementation algorithm can be used to inversely deduce the range of values ​​corresponding to the predicted numbers. Generally, the median of this range is taken as the final prediction result for the supplemented data. The bin discretization rules for other features follow the same principle.

[0214] 3. After feature discretization, identically distributed samples can be generated using the previously mentioned MCMC method. It's important to note that the MCMC algorithm described earlier generates data from a specific distribution in the case of one-dimensional data. This embodiment deals with multi-dimensional data; therefore, slightly differently, it uses a multivariate MCMC algorithm capable of simulating multi-dimensional data generation, such as the classic multivariate Gibbs sampling method. In the previous step... The four-dimensional feature empirical distribution obtained after discretization of the four features is taken as the steady-state distribution π, and large-scale generation of four-dimensional simulated sample data under the same distribution is started, labeled as... The data volume is generally higher than One to two orders of magnitude, that is, if There are 10,000 data points. The simulated data should contain 1 million samples. This is sufficient for subsequent supplementation of missing samples.

[0215] (3) Simulated dataset of four-dimensional features Once generated, it can be based on Characteristics of customer Q Missing data The following information has been added:

[0216] Customer Q in h q1 ,h q2 ,h q3 The values ​​of the three features are remember The dataset is labeled as D Q That is, the feature mentioned in (1) With H Q The set of customers with no missing values ​​for the middle feature.

[0217] Next, we will design a missing value prediction algorithm based on For the missing The missing data prediction algorithm is based on a similarity comparison design and performs predictions and data supplementation.

[0218] Specifically, it is by and Compare them one by one and select those with high similarity. corresponding Numerical pairs are filled with predictions for each group. Its and The formula for calculating similarity is:

[0219]

[0220] Among them, w1, w2, and w3 are based on h q1 ,h q2 ,h q3 Three characteristics and features The comprehensive correlation coefficient ξ is constructed, denoted as h. q1 ,h q2 ,h q3The combined correlation coefficients of the three features are ξ1, ξ2, and ξ3, respectively.

[0221] This is a kind of h q1 ,h q2 ,h q3 A similarity calculation method that uses vectorized three feature values ​​and then fine-tunes the similarity by combining the magnitude of the correlation coefficient.

[0222] Unlike traditional similarity calculations, this approach not only considers the similarity between components to uniformly calculate the overall similarity, but also considers the relationship between each feature component and the feature... The correlation magnitude is used to further refine different feature components, in h q1 ,h q2 ,h q3 Three characteristics and features When the correlation coefficients differ significantly, this difference will be highlighted in the formula calculation, making the similarity calculation in the formula more reasonable.

[0223] After the calculations were completed, a simulated dataset was obtained. Various data and similarity But from Selecting s with high similarity L Marked as a set The selection criterion here is similarity score (s). L It should be above 90%. That is D Q Medium similarity s L The set of subscripts for over 90% of the simulated data. If the generated simulated dataset... If there are no samples with a similarity exceeding 90%, or only a very small number (e.g., a single-digit number) of samples with a similarity exceeding 90%, then the simulated dataset is... The data quality is poor, and the simulated dataset should be regenerated using the MCMC method described above. This continues until a large amount of highly similar data appears.

[0224] High similarity data generated by simulation Once sufficient data is prepared, it can be supplemented. The supplementation algorithm is as follows:

[0225]

[0226] in yes The number of samples included.

[0227] Thus, the characteristics of customer Q are... The missing data has now been filled in. Following a similar method, the missing data will be... As other missing feature data are completed one by one, the missing feature data for customer Q will be... That completes the process.

[0228] All customers with valid data can supplement their data by following the same method described above for customer Q to fill in the missing data.

[0229] III. Intelligent Supplementation and Correction Mechanism for Model Prediction Results

[0230] After the data is supplemented, the results need to be continuously optimized and adjusted. In fact, for some customers, the missing data is not permanent; some missing data is simply due to the customer forgetting to fill it in. At some point in the future, the customer may fill in the missing data themselves, which gives us the opportunity to correct and adjust the model algorithm.

[0231] Continuing with the previous notation, let's assume we're targeting a certain feature. The missing dataset that was partially supplemented was labeled as G is the set of customer IDs for the supplemented real data, and the supplemented real data for the customers is...

[0232] As can be seen from the previous formula design

[0233]

[0234] That is, missing values The prediction results are mainly affected by their corresponding similarity s L The weighting effect changes s L The value can be changed The prediction results are shown here in the formula above. Replace with The elements in the comparison Compared with the true value To correct the differences The corresponding similarity s in the calculation formula L This will improve the accuracy of data supplementation.

[0235] The specific steps are as follows: The objective function is to change... Similarity in the calculation formula To change The value, until it makes When the similarity score reaches its minimum, the combination of similarity values ​​at that point is taken as the minimum. contrast and If it makes Reaching the minimum Compared to s L If it becomes larger, then the formula The corresponding s in L Revised to That is, the s in the original calculation formula L Expand by 1% to prevent if Compared to s L If it becomes smaller, then s L Revised to That is, the s in the original calculation formula L Reducing the forecast by 1% is a relatively conservative adjustment method to fine-tune the prediction results in order to ensure the stability of the prediction effect.

[0236] The steps of the data missing information imputation method based on Markov chain Monte Carlo (MCMC) sampling in this embodiment can be summarized as follows:

[0237] The dataset containing missing values ​​is preliminarily processed to identify and separate records with missing data, while determining the specific feature dimensions involved in the missing data.

[0238] The identified feature data without missing values ​​are discretized. Binning technology is used to divide the continuous features into multiple intervals, each interval representing a discretized value range. The joint distribution of the features is characterized based on an empirical distribution function.

[0239] Using the MCMC method, especially the multivariate Gibbs sampling technique, a large-scale simulated sample dataset with the same distribution as the original data is generated based on the discretized empirical distribution of the above features. This ensures that the amount of simulated data is much larger than the amount of original data, so as to cover more potential sample space.

[0240] A similarity calculation method based on comprehensive correlation coefficient is designed. For each sample with missing features, data records with high similarity to other non-missing features are selected from the simulated dataset as candidates for predicting missing values. The similarity calculation formula combines the comprehensive correlation coefficient of multiple features to achieve weighted adjustment of the importance of different features.

[0241] Based on the similarity matching results, the corresponding feature values ​​of the simulated data with a similarity higher than a preset threshold (e.g., 90%) are selected to fill the missing data, forming a supplemented dataset.

[0242] For the real-world data that may be obtained in the future, the missing values ​​that have already been filled are corrected. By comparing the difference between the predicted and actual values, the weights in the similarity calculation are dynamically adjusted, and the prediction algorithm is fine-tuned to improve the accuracy of data completion. Specifically, the prediction error is minimized by setting an objective function, and the similarity calculation parameters are adjusted according to the error to ensure the continuous optimization and feedback learning capabilities of the model.

[0243] Based on new real data feedback, the data supplementation model is continuously adjusted and optimized to achieve self-iteration and long-term performance improvement, ensuring the long-term effectiveness of data supplementation and the stability of model prediction.

[0244] This invention aims to overcome the shortcomings of existing technologies that simply fill in data directly or lack dynamic optimization mechanisms. By comprehensively utilizing MCMC sampling, multi-dimensional feature similarity matching, and intelligent correction strategies, it achieves effective and accurate data supplementation for missing data, thereby improving the accuracy of data analysis and the reliability of model prediction.

[0245] In this embodiment, a novel data missing completion algorithm based on the Markov chain Monte Carlo statistical sampling method is constructed to help improve the sample generation accuracy of sampling techniques by designing novel similarity matching algorithms based on existing data in the case of missing data.

[0246] For the sampled data results, a brand-new dynamic data self-correction mechanism was designed to ensure that the data can continue to be compared with new real data as time goes by after the data sampling, and the accuracy of the model prediction can be continuously optimized and updated. A growth-type self-iteration mechanism was added to the model.

[0247] Among them, the "Intelligent Sampling Data Supplementation Algorithm Based on MCMC Algorithm" is an innovative data processing technology designed to accurately fill in missing feature values ​​in datasets. This algorithm first utilizes the principle of Markov Chain Monte Carlo (MCMC) method to generate high-quality simulated data by constructing a joint empirical distribution function of multi-dimensional features. Specifically, it involves: extracting subsets related to missing features from the complete feature data, performing multi-dimensional discretization to reduce computational complexity, and using a weighted correlation coefficient method to select the most relevant feature combinations. Subsequently, it employs multivariate MCMC sampling techniques, such as Gibbs sampling, to generate multi-dimensional simulated samples that closely approximate the actual data distribution. Next, through a similarity comparison mechanism based on a comprehensive correlation coefficient, it selects samples from these simulated samples that are most similar to the missing data records, using these samples to fill in the missing features. Furthermore, the algorithm includes a feedback correction mechanism that continuously optimizes and adjusts the previously supplemented missing values ​​based on subsequently collected real data, ensuring the accuracy and stability of the model's predictions. This method overcomes the limitations of traditional missing data processing techniques, such as the inaccuracy of simple assignment or prediction based on a single distribution, and achieves intelligent data supplementation and improved accuracy.

[0248] Of course, it is understood that there may be other variations of the above detailed process, and all such variations should fall within the protection scope of this invention.

[0249] In this embodiment of the invention, missing data records with missing values ​​in a customer dataset are identified; missing features in each missing data record are marked; for each missing feature in a target missing data record, data records with missing values ​​containing that missing feature are extracted from the customer dataset as sampling samples; feature analysis is performed on the sampling samples to determine multiple related features associated with the missing feature; the multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain a joint empirical distribution function corresponding to the missing feature; using a Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function; supplementary data corresponding to the missing feature is selected from the multidimensional simulated data based on the feature similarity between the multidimensional simulated data and the target missing data record. Compared with existing technologies that use fixed values ​​or features to roughly predict missing values ​​through statistical distribution, this invention first identifies and marks the specific missing features in each missing data record. By analyzing the correlation between each missing feature and other features in the target missing data record, multiple related features are selected. Then, for the selected related features and the target missing feature, a multi-dimensional feature combination is introduced and discretized to obtain a joint empirical distribution function. This overcomes the limitation of existing technologies that ignore the correlation between features, utilizing the inherent correlation of data to improve the accuracy and rationality of supplementary data. By employing the Markov chain Monte Carlo algorithm, multi-dimensional simulated data is generated based on the joint empirical distribution function, which can efficiently simulate complex distributions and help generate multi-dimensional simulated data that is closer to the actual data distribution, thereby improving the credibility of the supplementary data. Finally, supplementary data is selected based on the feature similarity between the multi-dimensional simulated data and the target missing data record, avoiding blind filling and ensuring that the supplementary data is highly similar to the original data in the feature space, thus improving the accuracy of data supplementation.

[0250] This invention also provides a device for supplementing missing customer data values, as described in the following embodiments. Since the principle behind this device is similar to the method for supplementing missing customer data values, its implementation can be found in the implementation of the method for supplementing missing customer data values; repeated details will not be elaborated further.

[0251] This invention also provides a device for supplementing missing customer data values, to improve the accuracy and reliability of the supplemented data for missing customer data values, such as... Figure 9 As shown, the device includes:

[0252] The missing feature determination module 901 is used to identify missing data records with missing values ​​in the customer dataset and to mark the missing features in each missing data record.

[0253] The relevant feature determination module 902 is used to extract data records with missing values ​​of the missing feature from the customer dataset for each missing feature in the target missing data record, and use them as sampling samples; perform feature analysis on the sampling samples to determine multiple relevant features related to the missing feature;

[0254] The joint empirical distribution function determination module 903 is used to discretize the feature dimensions of the multi-dimensional feature combination obtained by combining the missing feature and the related features to obtain the joint empirical distribution function corresponding to the missing feature.

[0255] The multidimensional simulation data determination module 904 is used to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function using the Markov chain Monte Carlo algorithm.

[0256] The supplementary data determination module 905 is used to select supplementary data corresponding to the missing feature from the multidimensional simulation data based on the feature similarity between the multidimensional simulation data and the target missing data record.

[0257] In one embodiment, for each missing feature in the target missing data records, data records containing missing values ​​of that missing feature are extracted from the customer dataset as a sampling sample, including:

[0258] For each missing feature in the target missing data record, determine the set of no missing features for that target missing data record; the set of no missing features consists of data records excluding the missing features present in the missing data record.

[0259] Extract the missing values ​​of the missing feature from the customer dataset and the data records of each feature value in the set of no missing features, and use them as sampling samples.

[0260] In one embodiment, feature analysis is performed on the sampled sample to determine multiple related features associated with the missing feature, including:

[0261] Features in the sample that belong to the same feature category as the missing feature are selected features;

[0262] Based on the feature values ​​of the sampled sample, calculate the correlation coefficient between the selected feature and each other feature in the sampled sample;

[0263] A preset number of other features whose correlation coefficient with the selected feature exceeds a preset value are identified as related features to the missing feature.

[0264] In one embodiment, based on the feature values ​​of the sampled sample, calculating the correlation coefficient between the selected feature and each other feature in the sampled sample includes:

[0265] Based on the absolute value of the feature value of the sampled sample, calculate the Pearson correlation coefficient, Spearman correlation coefficient and rank correlation coefficient between the selected feature and each other feature in the sampled sample;

[0266] Based on preset weighting coefficients, the Pearson correlation coefficient, Spearman correlation coefficient, and rank correlation coefficient are weighted and averaged to obtain the correlation coefficient between the selected feature and each other feature in the sampled sample.

[0267] In one embodiment, the multidimensional feature combination obtained by combining the missing feature and the related features is discretized in terms of feature dimensions to obtain a joint empirical distribution function corresponding to the missing feature, including:

[0268] The missing feature and the relevant feature are combined to obtain a multidimensional feature combination;

[0269] The feature values ​​of the multidimensional feature combination are obtained from the sampled sample;

[0270] Based on the binning processing rules of the multidimensional discretized sample space, the feature values ​​of the multidimensional feature combination are discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature.

[0271] In one embodiment, a Markov chain Monte Carlo algorithm is used to generate multidimensional simulated data associated with the missing feature based on the joint empirical distribution function, including:

[0272] Using the multivariate Gibbs sampling algorithm in the Markov chain Monte Carlo algorithm, based on the pre-set transition matrix, the joint empirical distribution function is taken as the steady-state distribution π, and simulated samples are extracted from the joint empirical distribution function to generate multidimensional simulated data under the same distribution, which serves as multidimensional simulated data associated with the missing feature.

[0273] In one embodiment, supplementary data corresponding to the missing feature is selected from the multidimensional simulation data based on the feature similarity between the multidimensional simulation data and the target missing data record, including:

[0274] Obtain the first value of the relevant feature in the target missing data record;

[0275] Calculate the feature similarity between the value of the relevant feature in each multidimensional simulation data and the first value;

[0276] Multidimensional simulation data with feature similarity exceeding a preset value are used as target multidimensional simulation data;

[0277] Based on the feature values ​​corresponding to the missing feature selected from the target multidimensional simulation data, supplementary data corresponding to the missing feature is calculated.

[0278] In one embodiment, it also includes:

[0279] Receive the actual data of the missing feature as input;

[0280] Based on the real data and the supplementary data, the similarity parameters applied in the process of calculating the supplementary data corresponding to the missing feature are corrected.

[0281] In one embodiment, it also includes:

[0282] If supplementary data corresponding to the missing feature cannot be selected from the multidimensional simulation data, then the Markov chain Monte Carlo algorithm is used again to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function; the above steps are repeated until supplementary data corresponding to the missing feature is selected.

[0283] This invention provides an embodiment of a computer device for implementing all or part of the above-described method for supplementing missing customer data values. The computer device specifically includes the following components:

[0284] The computer device comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between related devices; the computer device can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the computer device can be implemented with reference to the embodiments of the method for supplementing missing customer data values ​​and the embodiments of the device for supplementing missing customer data values, the contents of which are incorporated herein by reference, and repeated details will not be described again.

[0285] Figure 10 This is a schematic block diagram illustrating the system configuration of the computer device 1000 according to an embodiment of this application. Figure 10 As shown, the computer device 1000 may include a central processing unit 1001 and a memory 1002; the memory 1002 is coupled to the central processing unit 1001. It is worth noting that... Figure 10 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0286] In one embodiment, the function of supplementing missing customer data values ​​can be integrated into the central processing unit 1001. The central processing unit 1001 can be configured to perform the following control:

[0287] Identify missing data records with missing values ​​in the customer dataset; mark the missing features in each missing data record;

[0288] For each missing feature in the target missing data record, extract the data records with missing values ​​of the missing feature from the customer dataset and use them as sampling samples; perform feature analysis on the sampling samples to determine multiple related features associated with the missing feature;

[0289] The multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain the joint empirical distribution function corresponding to the missing feature.

[0290] Using the Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function;

[0291] Based on the feature similarity between the multidimensional simulation data and the target missing data record, supplementary data corresponding to the missing feature is selected from the multidimensional simulation data.

[0292] In another embodiment, the device for supplementing missing customer data values ​​can be configured separately from the central processing unit 1001. For example, the device for supplementing missing customer data values ​​can be configured as a chip connected to the central processing unit 1001, and the function of supplementing missing customer data values ​​can be realized through the control of the central processing unit.

[0293] like Figure 10 As shown, the computer device 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, and a power supply 1007. It is worth noting that the computer device 1000 does not necessarily need to include... Figure 10 All components shown; in addition, the computer device 1000 may also include Figure 10 For components not shown, please refer to existing technologies.

[0294] like Figure 10 As shown, the central processing unit 1001, sometimes also referred to as a controller or operation control, may include a microprocessor or other processor device and / or logic device. The central processing unit 1001 receives input and controls the operation of various components of the computer device 1000.

[0295] The memory 1002 may be, for example, one or more of a cache, flash memory, hard drive, removable medium, volatile memory, non-volatile memory, or other suitable device. It can store the aforementioned device-related information, and may also store programs for executing that information. The central processing unit 1001 can execute the program stored in the memory 1002 to perform information storage or processing, etc.

[0296] Input unit 1004 provides input to central processing unit 1001. This input unit 1004 may be, for example, a keypad or touch input device. Power supply 1007 provides power to computer device 1000. Display 1006 displays images, text, and other display objects. This display may be, for example, an LCD display, but is not limited to this.

[0297] The memory 1002 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs, etc. The memory 1002 can also be some other type of device. The memory 1002 includes a buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application / function storage unit 1022 for storing application programs and function programs or processes for executing operations of the computer device 1000 via the central processing unit 1001.

[0298] The memory 1002 may also include a data storage unit 1023 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the computer device. The driver storage unit 1024 of the memory 1002 may include various drivers for the computer device for communication functions and / or for performing other functions of the computer device (such as messaging applications, address book applications, etc.).

[0299] The communication module 1003 is a transmitter / receiver 1003 that transmits and receives signals via the antenna 1008. The communication module (transmitter / receiver) 1003 is coupled to the central processing unit 1001 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0300] Based on different communication technologies, multiple communication modules 1003 can be configured in the same computer device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and receive audio input from the microphone 1010, thereby realizing typical telecommunications functions. The audio processor 1005 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 1005 is also coupled to a central processing unit 1001, enabling on-device recording via the microphone 1010 and on-device playback of stored sound via the speaker 1009.

[0301] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for supplementing missing customer data values.

[0302] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-mentioned method for supplementing missing customer data values.

[0303] In this embodiment of the invention, missing data records with missing values ​​in a customer dataset are identified; missing features in each missing data record are marked; for each missing feature in a target missing data record, data records with missing values ​​containing that missing feature are extracted from the customer dataset as sampling samples; feature analysis is performed on the sampling samples to determine multiple related features associated with the missing feature; the multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain a joint empirical distribution function corresponding to the missing feature; using a Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function; supplementary data corresponding to the missing feature is selected from the multidimensional simulated data based on the feature similarity between the multidimensional simulated data and the target missing data record. Compared with existing technologies that use fixed values ​​or features to roughly predict missing values ​​through statistical distribution, this invention first identifies and marks the specific missing features in each missing data record. By analyzing the correlation between each missing feature and other features in the target missing data record, multiple related features are selected. Then, for the selected related features and the target missing feature, a multi-dimensional feature combination is introduced and discretized to obtain a joint empirical distribution function. This overcomes the limitation of existing technologies that ignore the correlation between features, utilizing the inherent correlation of data to improve the accuracy and rationality of supplementary data. By employing the Markov chain Monte Carlo algorithm, multi-dimensional simulated data is generated based on the joint empirical distribution function, which can efficiently simulate complex distributions and help generate multi-dimensional simulated data that is closer to the actual data distribution, thereby improving the credibility of the supplementary data. Finally, supplementary data is selected based on the feature similarity between the multi-dimensional simulated data and the target missing data record, avoiding blind filling and ensuring that the supplementary data is highly similar to the original data in the feature space, thus improving the accuracy of data supplementation.

[0304] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0305] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0306] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0307] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0308] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for supplementing missing values ​​in customer data, characterized in that, include: Identify missing data records in the customer dataset that have missing values; Mark the missing features in each missing data record; For each missing feature in the target missing data record, extract the data records with missing values ​​of that missing feature from the customer dataset and use them as sampling samples; Feature analysis is performed on the sampled data to identify multiple related features associated with the missing feature; The multidimensional feature combination obtained by combining the missing feature and the related features is discretized to obtain the joint empirical distribution function corresponding to the missing feature. Using the Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function; Based on the feature similarity between the multidimensional simulation data and the target missing data record, supplementary data corresponding to the missing feature is selected from the multidimensional simulation data; For each missing feature in the target missing data records, extract the data records containing missing values ​​of that missing feature from the customer dataset as a sampling sample, including: For each missing feature in the target missing data record, determine the set of no missing features for that target missing data record; the set of no missing features consists of data records excluding the missing features present in the missing data record. Extract the data records of each feature value in the set of no missing features and the missing feature set from the customer dataset, and use them as sampling samples; The multidimensional feature combination obtained by combining the missing feature and the related features is discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature, including: The missing feature and the relevant feature are combined to obtain a multidimensional feature combination; The feature values ​​of the multidimensional feature combination are obtained from the sampled sample; Based on the binning processing rules of the multidimensional discretized sample space, the feature values ​​of the multidimensional feature combination are discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature.

2. The method as described in claim 1, characterized in that, Feature analysis is performed on the sampled data to identify multiple related features associated with the missing feature, including: Features in the sample that belong to the same feature category as the missing feature are selected features; Based on the feature values ​​of the sampled sample, calculate the correlation coefficient between the selected feature and each other feature in the sampled sample; A preset number of other features whose correlation coefficient with the selected feature exceeds a preset value are identified as related features to the missing feature.

3. The method as described in claim 2, characterized in that, Based on the feature values ​​of the sampled sample, the correlation coefficient between the selected feature and each other feature in the sampled sample is calculated, including: Based on the absolute value of the feature value of the sampled sample, calculate the Pearson correlation coefficient, Spearman correlation coefficient and rank correlation coefficient between the selected feature and each other feature in the sampled sample; Based on preset weighting coefficients, the Pearson correlation coefficient, Spearman correlation coefficient, and rank correlation coefficient are weighted and averaged to obtain the correlation coefficient between the selected feature and each other feature in the sampled sample.

4. The method as described in claim 1, characterized in that, Using the Markov chain Monte Carlo algorithm, multidimensional simulated data associated with the missing feature is generated based on the joint empirical distribution function, including: Using the multivariate Gibbs sampling algorithm in the Markov chain Monte Carlo algorithm, based on the pre-set transition matrix, the joint empirical distribution function is taken as the steady-state distribution π, and simulated samples are extracted from the joint empirical distribution function to generate multidimensional simulated data under the same distribution, which serves as multidimensional simulated data associated with the missing feature.

5. The method as described in claim 1, characterized in that, Based on the feature similarity between the multidimensional simulation data and the target missing data records, supplementary data corresponding to the missing features is selected from the multidimensional simulation data, including: Obtain the first value of the relevant feature in the target missing data record; Calculate the feature similarity between the value of the relevant feature in each multidimensional simulation data and the first value; Multidimensional simulation data with feature similarity exceeding a preset value are used as target multidimensional simulation data; Based on the feature values ​​corresponding to the missing feature selected from the target multidimensional simulation data, supplementary data corresponding to the missing feature is calculated.

6. The method as described in claim 5, characterized in that, Also includes: Receive the actual data of the missing feature as input; Based on the real data and the supplementary data, the similarity parameters applied in the process of calculating the supplementary data corresponding to the missing feature are corrected.

7. The method as described in claim 1, characterized in that, Also includes: If supplementary data corresponding to the missing feature cannot be selected from the multidimensional simulation data, then the Markov chain Monte Carlo algorithm is used again to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function; the above steps are repeated until supplementary data corresponding to the missing feature is selected.

8. A device for supplementing missing values ​​in customer data, characterized in that, include: The missing feature determination module is used to identify missing data records with missing values ​​in the customer dataset; Mark the missing features in each missing data record; The relevant feature determination module is used to extract data records containing missing values ​​of the missing feature from the customer dataset for each missing feature in the target missing data record, and use them as sampling samples. Feature analysis is performed on the sampled data to identify multiple related features associated with the missing feature; The joint empirical distribution function determination module is used to discretize the feature dimensions of the multi-dimensional feature combination obtained by combining the missing feature and the related features, so as to obtain the joint empirical distribution function corresponding to the missing feature. The multidimensional simulation data determination module is used to generate multidimensional simulation data associated with the missing feature based on the joint empirical distribution function using the Markov chain Monte Carlo algorithm. The supplementary data determination module is used to select supplementary data corresponding to the missing feature from the multidimensional simulation data based on the feature similarity between the multidimensional simulation data and the target missing data record. The relevant feature determination module is specifically used for: For each missing feature in the target missing data record, determine the set of no missing features for that target missing data record; the set of no missing features consists of data records excluding the missing features present in the missing data record. Extract the data records of each feature value in the set of no missing features and the missing feature set from the customer dataset, and use them as sampling samples; The joint empirical distribution function determination module is specifically used for: The missing feature and the relevant feature are combined to obtain a multidimensional feature combination; The feature values ​​of the multidimensional feature combination are obtained from the sampled sample; Based on the binning processing rules of the multidimensional discretized sample space, the feature values ​​of the multidimensional feature combination are discretized in terms of feature dimensions to obtain the joint empirical distribution function corresponding to the missing feature.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images