Sample data integration method

Abstract

The invention discloses an integration method for sample data, which relates to the field of data processing, and comprises the following steps: S1, summarizing and screening out sample data with a certain characteristic variable according to stock data information; s2, integrating and screening out sample data which can be incorporated into the model, and carrying out preliminary processing and arrangement on the original sample data to obtain processed sample data; s3, performing data conversion on the sample data processed in the step S2, and screening feature variables helpful for predicting future feature variable information from the stock data information with the step S1; s4, further extracting important features by constructing a gradient elevator algorithm model based on the various feature variables screened in the S3, and evaluating the advantages and disadvantages of the model by using a test set so as to obtain feature variables which can be helpful to predict variables finally; according to the method, the feature model is constructed, so that the subsequent data analysis work can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method for integrating sample data. Background Art

[0002] In many fields such as scientific research, product quality testing, and medical diagnosis, a large amount of sample data often needs to be analyzed. However, the sample data currently obtained comes from a wide range of sources, and there are great differences in their formats, storage methods, etc. Due to the dispersion and inconsistency of such data, researchers or testers often need to spend a lot of time collecting, organizing, and converting data when conducting data analysis, which is not only inefficient but also prone to errors.

[0003] Some existing data integration tools are often targeted at specific types of data or specific industries, lack versatility, and cannot meet the needs of integrating multiple types of sample data. There is an urgent need for a system that can effectively integrate sample data from different sources and in different formats. Summary of the invention

[0004] The purpose of the present invention is to provide a method for integrating sample data, which solves the technical problems that existing data integration tools have poor versatility, resulting in a large amount of manpower required for integration, high cost and low efficiency, and realizes the technical effect of automatically identifying data formats, centrally storing scattered data, and performing preliminary cleaning and preprocessing, so that subsequent data analysis work can be carried out efficiently and accurately.

[0005] To achieve the above object, the present invention provides the following solution: The present invention provides a method for integrating sample data.

[0006] Preferably, a method for integrating sample data is characterized by comprising:

[0007] S1. Based on the stock data information, summarize and select the sample data that clearly has a certain characteristic variable;

[0008] S2. Integrate and screen out sample data that can be included in the model, and preliminarily process and organize the original sample data to obtain processed sample data;

[0009] S3, perform data conversion on the sample data processed by S2, and select characteristic variables that are helpful for predicting future characteristic variable information from the stock data information with S1;

[0010] S4. Based on the various feature variables that have been screened in S3, further important features are extracted by building a gradient boosting algorithm model, and the model is evaluated using a test set to obtain feature variables that can ultimately help predict variables.

[0011] Preferably, S2 specifically includes the following steps:

[0012] S21, sample data fusion;

[0013] S22. Sample data cleaning: remove obvious errors and duplicate data from the stock data;

[0014] S23, sample data quality analysis, conduct detailed statistical analysis on the quality of the data in the data table processed by S22. The statistical analysis includes regression analysis and variance analysis to finally obtain the processed sample data.

[0015] Preferably, S3 specifically includes the following steps:

[0016] S31. Using the processed sample data as the original feature variable pool, the feature correlation and feature predictability are used to complete the screening of the better feature fields in the feature variable pool. For feature correlation analysis, the Kendall rank correlation coefficient is used to implement it. For two variables with high correlation, the one with higher overall correlation with other variables is deleted to optimize the feature processing process.

[0017] S32. Use the group stability index to analyze other characteristic variables that have the greatest contribution to the S1 characteristic variable.

[0018] Preferably, S4 specifically includes the following steps:

[0019] S41, using mixed sampling to balance the number of sample data, so that the number of data samples with S1 characteristic variables and the number of data samples with the opposite characteristic variables of S1 tend to be consistent;

[0020] S42, splitting the data into a training set and a test set according to the data after equalization processing in S41, using the gradient boosting algorithm to perform model fitting on the training data in the training set, randomly sampling in the original data set to form multiple different sample data sets, and then building multiple different decision models based on these data sets, and finally obtaining the optimal parameters and the final decision model according to the average value or voting results of these decision models;

[0021] S43. Use the final decision model to predict the sample test set and obtain the prediction result.

[0022] Preferably, in S4, using the test set to evaluate the quality of the model includes: using the test set to evaluate the model, evaluating the quality of the model through evaluation indicators and providing an evaluation value of the model.

[0023] The present invention further provides a storage medium, which includes a stored program, wherein when the program is run, any of the above-mentioned methods for integrating sample data is executed.

[0024] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any one of the above-mentioned methods for integrating sample data through the operation of the computer program.

[0025] Compared with the prior art, the advantages of the present invention are:

[0026] The present invention performs correlation processing before screening the population stability index, and can optimize the feature processing process for two variables with high correlation.

[0027] The present invention performs feature extraction based on the gradient boosting algorithm, can perform multi-dimensional feature fusion on data from different scenes, and then perform feature extraction. The gradient boosting algorithm is used to train multiple decision models using multiple selected sample sets and feature sets, which can ensure that the adopted model will not rely solely on a certain batch of features and cause prediction deviation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1 As shown, a method for integrating sample data, characterized by comprising:

[0032] S1. Based on the stock data information, summarize and select the sample data that clearly has a certain characteristic variable;

[0033] S2. Integrate and screen out sample data that can be included in the model, and preliminarily process and organize the original sample data to obtain processed sample data;

[0034] S3, perform data conversion on the sample data processed by S2, and select characteristic variables that are helpful for predicting future characteristic variable information from the stock data information with S1;

[0035] S4. Based on the various feature variables that have been screened in S3, further important features are extracted by building a gradient boosting algorithm model, and the model is evaluated using a test set to obtain feature variables that can ultimately help predict variables.

[0036] As an implementable example, S2 specifically includes the following steps:

[0037] S21, sample data fusion;

[0038] S22. Sample data cleaning: remove obvious errors and duplicate data from the stock data;

[0039] S23, sample data quality analysis, conduct detailed statistical analysis on the quality of the data in the data table processed by S22. The statistical analysis includes regression analysis and variance analysis to finally obtain the processed sample data.

[0040] As an implementable example, S3 specifically includes the following steps:

[0041] S31. Using the processed sample data as the original feature variable pool, the feature correlation and feature predictability are used to complete the screening of the better feature fields in the feature variable pool. For feature correlation analysis, the Kendall rank correlation coefficient is used to implement it. For two variables with high correlation, the one with higher overall correlation with other variables is deleted to optimize the feature processing process.

[0042] S32. Use the group stability index to analyze other characteristic variables that have the greatest contribution to the S1 characteristic variable.

[0043] As an implementable example, S4 specifically includes the following steps:

[0044] S41, using mixed sampling to balance the number of sample data, so that the number of data samples with S1 characteristic variables and the number of data samples with the opposite characteristic variables of S1 tend to be consistent;

[0045] S42, splitting the data into a training set and a test set according to the data after equalization processing in S41, using the gradient boosting algorithm to perform model fitting on the training data in the training set, randomly sampling in the original data set to form multiple different sample data sets, and then building multiple different decision models based on these data sets, and finally obtaining the optimal parameters and the final decision model according to the average value or voting results of these decision models;

[0046] S43. Use the final decision model to predict the sample test set and obtain the prediction result.

[0047] As an implementable example, in S4, using the test set to evaluate the quality of the model includes: using the test set to evaluate the model, evaluating the quality of the model through evaluation indicators and providing an evaluation value of the model.

[0048] As an implementable example, the present invention further includes a storage medium, characterized in that the storage medium includes a stored program, wherein when the program is run, any of the above-mentioned methods for integrating sample data is executed.

[0049] As an implementable example, the present invention also includes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any one of the above-mentioned methods for integrating sample data through the computer program.

[0050] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-restrictive from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.

[0051] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for integrating sample data, characterized in that: include: S1. Based on the stock data information, summarize and select the sample data that clearly has a certain characteristic variable; S2. Integrate and screen out sample data that can be included in the model, and preliminarily process and organize the original sample data to obtain processed sample data; S3, perform data conversion on the sample data processed by S2, and select characteristic variables that are helpful for predicting future characteristic variable information from the stock data information with S1; S4. Based on the various feature variables that have been screened in S3, further important features are extracted by building a gradient boosting algorithm model, and the model is evaluated using a test set to obtain feature variables that can ultimately help predict variables.

2. A method for integrating sample data according to claim 1, characterized in that: S2 specifically includes the following steps: S21, sample data fusion; S22. Sample data cleaning: remove obvious errors and duplicate data from the stock data; S23, sample data quality analysis, conduct detailed statistical analysis on the quality of the data in the data table processed by S22. The statistical analysis includes regression analysis and variance analysis to finally obtain the processed sample data.

3. A method for integrating sample data according to claim 1, characterized in that: S3 specifically includes the following steps: S31. Using the processed sample data as the original feature variable pool, the feature correlation and feature predictability are used to complete the screening of the better feature fields in the feature variable pool. For feature correlation analysis, the Kendall rank correlation coefficient is used to implement it. For two variables with high correlation, the one with higher overall correlation with other variables is deleted to optimize the feature processing process. S32. Use the group stability index to analyze other characteristic variables that have the greatest contribution to the S1 characteristic variable.

4. A method for integrating sample data according to claim 1, characterized in that: S4 specifically includes the following steps: S41, using mixed sampling to balance the number of sample data, so that the number of data samples with S1 characteristic variables and the number of data samples with the opposite characteristic variables of S1 tend to be consistent; S42, splitting the data into a training set and a test set according to the data after equalization processing in S41, using the gradient boosting algorithm to perform model fitting on the training data in the training set, randomly sampling in the original data set to form multiple different sample data sets, and then building multiple different decision models based on these data sets, and finally obtaining the optimal parameters and the final decision model according to the average value or voting results of these decision models; S43. Use the final decision model to predict the sample test set and obtain the prediction result.

5. A method for integrating sample data according to claim 4, characterized in that: In S4, using the test set to evaluate the quality of the model includes: using the test set to evaluate the model, evaluating the quality of the model through evaluation indicators and giving an evaluation value of the model.

6. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the method for integrating sample data described in any one of claims 1 to 5 is executed.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method for integrating sample data according to any one of claims 1 to 5 through the operation of the computer program.

Images