A relationship graph-based intelligent missing data imputation method and system

Abstract

The present application relates to the technical field of information processing, in particular to a missing data intelligent interpolation method and system based on a relationship graph, which is generally based on the regression interpolation idea, introduces the relationship graph between data as the input control strategy of the missing value prediction model, uses the improved neural network model, so that the missing value prediction of multiple variables can use the same model, and for the scene of "large range, large proportion" missing data, a set of high credibility interpolation sequence control strategy and secondary interpolation strategy are constructed. In general, the present application reduces the complexity of the interpolation system, and improves the calculation efficiency of the interpolation process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, in particular to a missing data intelligent imputation method and system based on a relationship graph. BACKGROUND

[0002] With the wide application of machine learning and digital twin technology, the dependence of software systems on data has greatly increased, and higher requirements have been put forward for the integrity and reliability of data input. However, due to defects in the collection and storage process, raw data often has missing values, and the imputation of these missing data is a problem that the engineering field has to face.

[0003] The existing technology mainly includes the following three kinds: hot card imputation method, regression imputation method and multiple imputation method, wherein,

[0004] The hot card imputation method finds an object most similar to it in the complete data, sometimes more than one similar object, and randomly selects one from all matching objects as the filling value. This method is conceptually simple and uses the relationship between data to evaluate null values, but the disadvantage is that the "similarity standard" is difficult to accurately define and is greatly influenced by subjective factors.

[0005] Multiple imputation method considers that the missing values are randomly distributed. Multiple imputation algorithms such as MICE algorithm will first estimate the value to be imputed using regression imputation, then simulate noise to form multiple sets of imputation values, and finally compare the generated multiple data sets with the original data set to select the set with the smallest distribution offset as the final result. Multiple imputation can only handle random missing, cannot handle non-random missing, and also requires a lot of calculations.

[0006] The regression imputation method uses supervised machine learning methods such as regression, nearest neighbor, random forest, support vector machine, etc. to establish a prediction model based on the complete data set, and predicts the missing attribute by substituting the known attribute into the model. Its advantage is high prediction accuracy, and its disadvantage is that it requires a lot of calculations, which greatly reduces the processing speed of missing values.

[0007] Specifically, the regression imputation method will establish a missing value prediction model for each variable. In the big data scenario of "many tables and many fields", modeling for each variable will consume a lot of resources and greatly increase the complexity of the system. In addition, during model training and actual prediction, the regression imputation method takes all variables except the target variable as input, which requires a lot of computing power and calculation time, and also forms a dependence on these variables. When there are multiple missing data in the same row, the model input will also have missing values, making the model difficult to calculate, and therefore cannot be applied to "large-scale, large-scale" data missing scenarios. SUMMARY

[0008] The purpose of the present application is to provide a one-stop data imputation technology, based on the idea of regression imputation, introducing the relationship graph between data as input for training of the missing value prediction model, so that the same model can realize missing value prediction of multiple variables; through the constructed imputation sequence control strategy considering the missing range and missing correlation in the same row, it can be applied to the "large range and large proportion" data missing scenario, realize high credibility and high computational efficiency of data imputation, to solve the problems pointed out in the background technology.

[0009] The embodiments of the present application are implemented by the following technical solutions: a missing data intelligent imputation method based on a relationship graph, comprising the following steps:

[0010] Generate a variable data set and perform feature numericalization and numerical normalization preprocessing;

[0011] Based on the correlation coefficient between variables, a variable relationship graph is established;

[0012] The adjacent variables of each variable in the relationship graph are taken as input to train a neural network model to obtain a missing value prediction model, the adjacent variables being variables directly connected to the target variable in the relationship graph;

[0013] Based on the imputation sequence control strategy considering the missing range and missing correlation in the same row, the missing data intelligent imputation is realized using the missing value prediction model;

[0014] The variables are decoded and restored.

[0015] According to a preferred embodiment, the variable relationship graph is established based on the correlation coefficient between variables, comprising:

[0016] Calculate the correlation matrix between all variables and perform binary processing on the correlation matrix;

[0017] Set the diagonal elements of the binary processed correlation matrix to 0 to obtain an adjacency matrix, and construct a relationship graph based on the adjacency matrix.

[0018] According to a preferred embodiment, the variable relationship graph is established based on the correlation coefficient between variables, further comprising:

[0019] Based on the obtained adjacency matrix, the expert experience data is used to optimize and adjust it.

[0020] According to a preferred embodiment, the adjacent variables of each variable in the relationship graph are taken as input to train a neural network model to obtain a missing value prediction model, comprising:

[0021] Taking the adjacency vector of each variable, a Hadamard product is performed with the input tensor row by row to generate N intermediate tensors of the same dimension, N being the number of variables in the input tensor;

[0022] The intermediate tensors are input into the model for N rounds of forward propagation, generating N output tensors, wherein no parameter update is performed after each round of forward propagation;

[0023] The input tensor is input into the model for one round of forward propagation, and the process parameters are updated;

[0024] The elements of the output tensors except the jth column are set to zero, and the N output tensors are summed to obtain a final output tensor, j being the number of forward propagation rounds of the output tensor;

[0025] Backpropagation is performed based on the deviation of the final output tensor and the input tensor, and this is repeated until the network converges or the number of training times reaches a set value, completing the training of the missing value prediction model.

[0026] According to a preferred embodiment, the interpolation sequence control strategy considering the missing range in the same row is:

[0027] The sufficiency of all null values in the current data row is verified, and the null values meeting the sufficiency verification requirement are filled;

[0028] The loop is iterated until there is no null value meeting the sufficiency verification requirement.

[0029] According to a preferred embodiment, the intelligent interpolation of missing data using the missing value prediction model comprises:

[0030] The Hadamard product of the adjacency vector of the current data row and the target null value is performed to obtain a vector after shielding processing;

[0031] The above vector is input into the missing value prediction model to calculate the result;

[0032] The column corresponding to the target null value in the calculation result is extracted as the predicted value, and the target null value is replaced.

[0033] According to a preferred embodiment, the interpolation sequence control strategy considering the missing correlation degree in the same row is:

[0034] The null values are sorted according to the missing correlation degree, which is the correlation proportion of the missing value in all adjacent variables of the current null value, expressed as follows:

[0035]

[0036] In the above formula, r ijdenotes the element in the i-th row and j-th column of the correlation matrix, l j is the j-th element of the adjacency vector L, z j is the j-th element of the missing state vector Z, z j = 1 if the data in the j-th bit of the data row is empty, otherwise z j = 0.

[0037] The missing values in the adjacency variables are replaced by default values as input, and the filling is performed in order of low to high missing correlation.

[0038] According to a preferred embodiment, after the filling is performed, further comprising:

[0039] Calculate the credibility of the interpolated data to form a credibility table, wherein the interpolated data is divided into original data and interpolated values, and the interpolated values are calculated according to the following expression:

[0040]

[0041] In the above formula, ε is a harmonic coefficient, η is a model loss and gain coefficient, represents the credibility loss caused by using the model for prediction, λ j represents the credibility of the j-th variable in the current data row.

[0042] According to a preferred embodiment, the adjacency variables of each variable in the relationship graph are used as input to train a neural network model to obtain a missing value prediction model, further comprising:

[0043] The missing value prediction model is used as a pre-trained model to realize intelligent interpolation of missing data using the pre-trained model;

[0044] Calculate the average credibility of each row of interpolated data, and use the data row with an average credibility higher than a preset threshold as new input to perform secondary training on the pre-trained model to obtain a final missing value prediction model.

[0045] The present application also provides a missing data intelligent interpolation system based on a relationship graph, which is applied to the method described above, comprising:

[0046] A processing module for generating a variable data set and performing feature numericalization and numerical normalization preprocessing;

[0047] A relationship graph construction module for establishing a variable relationship graph based on correlation coefficients between variables;

[0048] A training module for training a neural network model using adjacency variables of each variable in the relationship graph to obtain a missing value prediction model, wherein the adjacency variables are variables directly connected to the target variable in the relationship graph;

[0049] An interpolation module is configured to use the missing value prediction model to realize intelligent interpolation of missing data based on an interpolation sequence control strategy considering the missing range and missing correlation in the same row.

[0050] A decoding module is configured to decode and restore the variables.

[0051] The technical scheme of the embodiment of the present application has at least the following advantages and beneficial effects: the present application comprises a variable relationship graph construction strategy and a control strategy for adjusting the input and output of a relationship graph model, which greatly reduces the number of input variables and the dependence on other variables in the data set based on the traditional interpolation method, and has stronger compatibility for the case of "large range and large proportion" of data; the present application comprises a unified missing value prediction model training strategy, in the big data scene of "many tables and many fields", the prediction of all missing variables will use the same model, which greatly reduces the modeling time and system complexity; the present application comprises an interpolation sequence control strategy considering the missing range and missing correlation in the same row, which can maximize the retention of data authenticity by adjusting the interpolation sequence, and provide an important reliability reference for subsequent work. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of the intelligent interpolation method for missing data based on the relationship graph provided by the embodiment 1 of the present application is shown.

[0053] Figure 2 A relationship graph provided by the embodiment 1 of the present application is shown.

[0054] Figure 3 A forward propagation and loss calculation process provided by the embodiment 1 of the present application is shown.

[0055] Figure 4 A flowchart of the intelligent interpolation provided by the embodiment 1 of the present application is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be described clearly and completely in combination with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiments of the present application, not all the embodiments. The components of the embodiment of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0057] Embodiment 1

[0058] The intelligent interpolation method for missing data based on the relationship graph comprises the following steps: Figure 1

[0059] ​1) Data preprocessing step, since the input variables may exist in numerical, character, time and other formats, therefore, the variable data set needs to be preprocessed before modeling, the specific steps are as follows:

[0060] 1.1) Numerical processing of character information, the numerical processing includes but is not limited to label encoding, one-hot encoding, serial number encoding, frequency encoding, relative time, etc.

[0061] The following takes one-hot encoding as an example for a simple description:

[0062] For example: characters include southern airlines, China Airlines, Eastern Airlines, Hainan Airlines, Xiamen Airlines, Sichuan Airlines, Shenzhen Airlines, Shandong Airlines, Jixiang Airlines, Spring Airlines, etc. These characters are not continuous, but discrete and unordered.

[0063] Numerical processing of the above character examples, according to the principle of N-bit status register to encode N states, after processing, (here there are only nine features, so N = 9) are obtained:

[0064] Southern Airlines→100000000

[0065] China Airlines→010000000

[0066] Eastern Airlines→001000000

[0067] Hainan Airlines→000100000

[0068] Xiamen Airlines→000010000

[0069] Sichuan Airlines→000001000

[0070] Shenzhen Airlines→000000100

[0071] Jixiang Airlines→000000010

[0072] Spring Airlines→000000001

[0073] 1.2) Normalization processing of numerical information, the normalization includes but is not limited to deviation standardization, logarithmic normalization, zero-mean normalization, etc., which will not be described in detail here.

[0074] After preprocessing, store the processing method and parameters for subsequent decoding.

[0075] 2) According to statistical correlation and / or business correlation, build a relationship map between all variables in the data set, the specific steps are as follows:

[0076] 2.1) The specific steps of building a relationship map based on statistical correlation are as follows:

[0077] 2.1.1), calculate the correlation matrix between all variables, and the correlation matrix calculation means includes but is not limited to Pearson correlation coefficient, Spearman correlation coefficient, Kendall correlation coefficient and other methods, wherein the calculation expression of Pearson correlation coefficient is as follows:

[0078]

[0079] In the above formula, X represents the mean value of variable X i Y represents the mean value of variable Y i It should be noted that the range of Pearson correlation coefficient is-1 to 1, the coefficient value r is 1, which indicates that X i and Y i have a linear relationship, and the coefficient value r is 0, which indicates that X i and Y i have no linear relationship. Specifically, if and only if X i and Y i fall on the same side of the respective mean value, the correlation coefficient is positive; if X i and Y i tend to fall on the opposite side of the respective mean value, the correlation coefficient is negative.

[0080] 2.1.2), the correlation matrix is binarized, and the threshold of binarization can be a preset value or a default value, and specific details are not described too much.

[0081] 2.1.3), the diagonal elements of the correlation matrix are set to 0, so that the variable itself is not considered as adjacent, and the finally obtained matrix is used as the adjacency matrix of the relationship graph to describe the association relationship between variables, as shown in Table 1, Table 1 is an example of the adjacency matrix between variables provided by the embodiment of the application:

[0082] Table 1. Adjacency matrix between variables

[0083] Passenger count Bag count Door close time Task timeout rate … Passenger count 0 1 1 0 … Bag count 1 0 1 0 … Door close time 1 1 0 1 … Task timeout rate 0 0 1 0 … … … … … … …

[0084] It should be noted that the above adjacency matrix is a symmetric matrix, and the adjacency vector is the matrix row corresponding to the target variable, and the element of 1 indicates that the two variables corresponding to the row and column are associated, and the element of 0 indicates that they are not associated.

[0085] 2.2), the relationship graph is optimized and adjusted based on business correlation. Specifically, considering that the employees are more familiar with business data, they can capture the causal relationship other than statistical correlation, and can also eliminate the homogenized association relationship in the relationship graph (taking the relationship between the two variables as an example). Figure 2 ​Taking civil aviation scenario data as an example, passenger numbers and baggage numbers are highly correlated. Removing either one when predicting closing time can reduce computational load and data dependency (the benefits outweigh the loss in accuracy). Therefore, in this embodiment, based on the generated adjacency matrix, the corresponding elements of the relationships to be removed are set to 0, and the corresponding elements of the relationships to be added are set to 1, based on expert experience data. The final relationship graph is shown below. Figure 2 As shown.

[0086] In summary, through step 2), this invention significantly reduces the number of input variables and the dependence on other variables in the dataset compared to traditional interpolation methods, and has stronger compatibility with situations where the data is "large-scale and large-proportion".

[0087] 3) Use the adjacent variables of each variable in the relation graph as input to train the neural network model to obtain the missing value prediction model. The specific steps are as follows:

[0088] 3.1) Model initialization: Set the common parameters for training the neural network model, such as the number of neural network layers, number of neurons, activation function, learning rate, loss function, and optimizer. It should be noted that in this embodiment, the model requires that the input and output dimensions be the same, both being N.

[0089] In addition, before the forward propagation begins, the neuron weights need to be initialized. This process is the same as that of traditional feedforward neural networks, and will not be elaborated on here.

[0090] It should be noted that the embodiments of the present invention use an improved neural network model. This improved neural network model refers to improvements made to the timing and number of forward and backward propagation, as well as the organization of input and output, based on the deep feedforward network. It does not impose restrictions on general neural network parameters such as the number of network layers, the number of neurons, and activation functions.

[0091] 3.2) Forward propagation, in this embodiment the input tensor is P M×N The P M×N It is an M×N matrix, where M is the batch size, representing the number of rows of data in the batch input, and N represents the number of variables in the dataset; it should be noted that the input tensor used in the training process is the complete data rows in the complete dataset.

[0092] Furthermore, take P M×N The adjacency vectors of each variable in the vectors are intersected row by row with P. M×N Perform the Hadamard product to generate N intermediate tensors of dimension M×N. The goal is to generate an intermediate tensor that does not contain non-adjacent variables and a target variable, where the adjacent variables are those directly connected to the target variable in the relation graph.

[0093] will be described below. N forward propagations as input to generate N output tensors Wherein, each forward propagation is not immediately updated by backpropagation, only record the output.

[0094] Finally, the P M×N A round of forward propagation as input, the purpose is to update the process parameters for subsequent gradient calculation.

[0095] 3.3), calculate the loss, the specific steps are as follows:

[0096] 3.3.1), the P Zero the other elements in the N M×N , the purpose is to extract the effective column group in N The final output tensor O The number of forward propagation rounds.

[0097] 3.3.2), based on the loss function to calculate the O M×N The deviation of P M×N , it should be noted that P M×N That is, the correct value of the output, so the deviation of O M×N The deviation of P M×N The loss of the current model; the loss function includes but is not limited to L1 norm loss, mean square error loss, cross entropy loss and KL divergence loss, etc. General machine learning loss function, not described here.

[0098] 4), based on the deviation of O M×N The deviation of P M×N Backpropagation will calculate the contribution of each neuron to the loss, and update the weight according to the gradient calculated by the backpropagation algorithm. The process is the same as the traditional feedforward neural network, which will not be described here. It should be noted that the backpropagation process takes a much longer time than the forward propagation process in the entire deep neural network training process, so multiple forward propagations will not significantly increase the training time. The specific forward propagation and loss calculation process is shown in Figure 3 .

[0099] Repeat steps 2)-4) above until the network converges or the number of training times reaches the set value, complete the missing value prediction model training.

[0100] In summary, the model provided by step 4) is used for interpolation. In the big data scene of "table multiple, field multiple", the prediction of all missing variables will use the same model, which greatly reduces the modeling time and system complexity.

[0101] 5), based on considering the same row in the range of missing, missing correlation interpolation sequence control strategy, using the missing value prediction model to achieve intelligent interpolation of missing data, the specific steps are as follows:

[0102] 5.1), initial row number m = 1, null screening, see Figure 4 As shown in the figure, the specific steps are as follows:

[0103] 5.1.1), the sufficiency of all null values in the current data row is verified; it should be noted that the sufficiency verification means whether the adjacent variables corresponding to the current null value are all non-null, if all non-null, it meets the sufficiency verification.

[0104] 5.1.2), the null value meeting the sufficiency verification requirement is filled, and the specific steps are as follows:

[0105] 5.1.2.1), the hadamard product of the adjacent vector of the current data row and the target null value is performed, and the vector after shielding processing is obtained;

[0106] 5.1.2.2), the above vector is taken as input, and the result of the missing value prediction model is calculated;

[0107] 5.1.2.3), the column corresponding to the target null value in the calculation result is extracted as the prediction value, and the target null value is replaced.

[0108] 5.1.3), loop iteration until there is no null value meeting the sufficiency verification requirement.

[0109] 5.2), null value sorting, it should be noted that when there is no null value meeting the sufficiency verification, it does not mean that all null values have been filled, there may still be multiple null values dependent on each other and cannot be filled; the specific steps of the null value sorting are as follows:

[0110] 5.2.1), sorting according to the missing correlation R of the current null value, the expression of the missing correlation is as follows:

[0111]

[0112] In the above formula, r ij represents the element of the i-th row and the j-th column of the correlation matrix, l j represents the j-th element of the adjacent vector, z j represents the j-th element of the missing state vector, if the data in the j-th position of the data row is null, z j = 1, otherwise z j = 0.

[0113] 5.2.2), assignment filling, filling in the order of missing correlation R from low to high according to the filling procedure provided in step 5.1.2), until the row number m is greater than the total row M, otherwise m is added by 1 and returns to step 5.1.1).

[0114] It should be noted that the default value is used instead of the null value in the adjacent variable as the model input before filling, and the default value includes but is not limited to the median, mode or mean of the variables in the data set, etc., which will not be described in detail here.

[0115] It should be noted that there are more default values in the input variables in the initial stage of the assignment filling of the same row, but the influence is small because the missing correlation R is lower; the closer the assignment filling of the same row is to the later stage, the fewer the default values in the input variables, and the higher the missing correlation, which maximizes the credibility of the imputed data as a whole.

[0116] Further, the credibility of the imputed data is calculated after each null value is imputed to form a credibility table, wherein the imputed data is divided into original data and imputed values, and the imputed value calculation expression is as follows:

[0117]

[0118] In the above formula, ε is a harmonic coefficient, η is a model loss and gain coefficient, indicating the credibility loss caused by using the model for prediction, λ j represents the credibility of the jth variable of the current data row (if it is original data, then λ j = 1; if it is a default value, then λ j = μ, μ represents a default loss and gain coefficient, representing the credibility loss caused by using the default value as the input; if it is a value generated by the previous step, then λ j is the calculation value of the formula in the previous step), and ε, η and μ are constants that can be pre-set or use default values.

[0119] As described above, the present application can maximize the data authenticity by adjusting the imputation order through step 5), providing an important reliability reference for subsequent work.

[0120] 6), since the imputed data and the original data are in an encoded state, the processing method and parameters stored according to step 1.2) are also needed to decode and restore the variables, and finally form a new data set after the implementation of the imputation.

[0121] Example 2

[0122] Different from example 1, in order to further improve the prediction accuracy of the model, the missing value prediction model obtained in step 3) is used as a pre-trained model for secondary training, and the pre-trained model is used to realize intelligent imputation of missing data.

[0123] Calculate the average reliability of each row of interpolated data, and the data row with an average reliability higher than a preset threshold is regarded as a new input to perform secondary training on the pre-trained model to obtain a final missing value prediction model, and the prediction interpolation is performed again.

[0124] The scheme provided in the embodiment further improves data utilization by combining pre-training and secondary training, has stronger adaptability in the case of large-scale missing data and few complete data rows, thereby improving the compatibility of the model to large-scale missing conditions, and can further improve the prediction accuracy compared with the scheme of embodiment 1.

[0125] Embodiment 3

[0126] The embodiment of the application provides a missing data intelligent interpolation system based on a relationship graph, and the application to the method described in embodiment 1 or embodiment 2 comprises:

[0127] The processing module is configured to generate a variable data set and perform feature numericalization and numerical normalization preprocessing.

[0128] The relationship graph construction module is configured to establish a variable relationship graph based on correlation coefficients between variables.

[0129] The training module is configured to train a neural network model by taking adjacent variables of each variable in the relationship graph as input to obtain a missing value prediction model, wherein the adjacent variables are variables directly connected to the target variable in the relationship graph.

[0130] The interpolation module is configured to use the missing value prediction model to realize intelligent interpolation of missing data based on an interpolation order control strategy considering missing ranges and missing correlation degrees in the same row.

[0131] The decoding module is configured to decode and restore the variables.

[0132] The above is only a preferred embodiment of the application and is not used to limit the application. For those skilled in the art, the application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A missing data intelligent imputation method based on a relationship graph, characterized in that, The method comprises the following steps: generating a variable dataset and performing feature numericalization and numerical normalization preprocessing; establishing a variable relationship graph based on correlation coefficients between variables; training a neural network model by taking adjacent variables of each variable in the relationship graph as input to obtain a missing value prediction model, the adjacent variables being variables directly connected to the target variable in the relationship graph; implementing intelligent imputation of missing data using the missing value prediction model based on an imputation sequence control strategy considering missing ranges and missing correlation degrees within the same row; decoding and restoring the variables; the training of the neural network model by taking the adjacent variables of each variable in the relationship graph as input to obtain the missing value prediction model comprises: taking the adjacent vectors of each variable, performing hadamard product with the input tensor row by row to generate N intermediate tensors of the same dimension, N being the number of variables in the input tensor; performing N rounds of forward propagation by taking the intermediate tensors as model input, generating N output tensors, wherein no parameter update is performed after each round of forward propagation; performing one round of forward propagation by taking the input tensor as input and updating the process parameters; setting the elements other than the jth column in the output tensors to zero and summing the N output tensors to obtain a final output tensor, j being the number of forward propagation rounds of the output tensor; performing back propagation based on the deviation between the final output tensor and the input tensor, and repeating until the network converges or the number of training times reaches a set value, completing the training of the missing value prediction model. 2.The relationship graph-based intelligent missing data imputation method of claim 1, wherein, The establishment of the variable relationship graph based on the correlation coefficients between the variables comprises: calculating the correlation matrix between all variables and performing binary processing on the correlation matrix; setting the diagonal elements of the binary-processed correlation matrix to 0 to obtain an adjacency matrix, and constructing a relationship graph based on the adjacency matrix. 3.The relationship graph-based intelligent missing data imputation method of claim 2, wherein, The establishment of the variable relationship graph based on the correlation coefficients between the variables further comprises: optimizing and adjusting the obtained adjacency matrix based on expert experience data. 4.The relationship graph-based intelligent missing data imputation method of claim 1, wherein, The imputation sequence control strategy considering the missing range within the same row comprises: performing sufficiency verification on all null values of the current data row, and performing filling on the null values meeting the sufficiency verification requirements; iterating until there is no null value meeting the sufficiency verification requirements. 5.The relationship graph-based intelligent missing data imputation method of claim 1, wherein, The intelligent imputation of missing data using the missing value prediction model comprises: performing hadamard product on the adjacent vectors of the current data row and the target null value to obtain a screened vector; taking the above vector as input and calculating the result through the missing value prediction model; extracting the column corresponding to the target null value in the calculation result as the prediction value to replace the target null value. 6.The relationship map-based intelligent missing data imputation method of claim 1, wherein, The imputation sequence control strategy considering the missing correlation degree within the same row comprises: sorting the null values according to the missing correlation degree, the missing correlation degree being the correlation proportion of the missing value in all adjacent variables of the current null value, and the expression being as follows: In the above formula, represents the element of the correlation matrix in the i-th row and the j-th column, is the j-th element of the adjacency vector is the j-th element of the missing state vector is the j-th element of the missing state vector if the data in the j-th position of the data row is empty, then , otherwise ; using the default value to replace the null values in the adjacent variables as input, and performing filling in the order from low to high according to the missing correlation degree. 7.The relationship graph-based intelligent missing data imputation method of claim 6, wherein, After performing filling, it further comprises: The reliability of the interpolation data is calculated to form a reliability table, wherein the interpolation data is divided into original data and interpolation values, and the interpolation values are calculated according to the following expression: In the above formula, is a reconciliation coefficient, is a model loss coefficient, indicating the loss of credibility caused by using the model for prediction, represents the credibility of the jth variable of the current data row. 8.The relationship map-based intelligent missing data imputation method of claim 1, wherein, The method further includes: The missing value prediction model is used as a pre-training model to realize intelligent interpolation of missing data; The average reliability of each row of interpolation data is calculated, and a data row with an average reliability higher than a preset threshold is used as new input to perform secondary training on the pre-training model to obtain a final missing value prediction model.

9. A system for intelligent imputation of missing data based on a relational graph, applied to the method according to any one of claims 1 to 8, characterized in that, The method includes: A processing module is configured to generate a variable data set and perform feature numericalization and numerical normalization preprocessing; A relationship graph construction module is configured to establish a variable relationship graph based on correlation coefficients between variables; A training module is configured to use adjacent variables of each variable in the relationship graph as input to train a neural network model to obtain a missing value prediction model, wherein the adjacent variables are variables directly connected to a target variable in the relationship graph; An interpolation module is configured to use the missing value prediction model to realize intelligent interpolation of missing data based on an interpolation sequence control strategy considering a missing range and missing correlation in the same row; A decoding module is configured to decode and restore variables.

Images