Unified social credit code multi-data completion fusion method and system

Abstract

The invention discloses a unified social credit code multi-data completion fusion method and system, relates to the field of data quality improvement, and is used for improving the accuracy and reliability of data completion. According to the method, a plurality of data completion models are respectively trained to calculate the completion value of the economic industry in each missing data sample, and the economic industry is randomly deleted from the complete data samples for completion so as to respectively calculate the credibility of each data completion model; and further performing weighted fusion on the complementation values of the data complementation models based on the credibility of the data complementation models to obtain a final prediction value, and finally performing credibility quantification on the final prediction value. According to the method, the advantages of various data completion models are fused, the robustness of the final completion result is improved, so that the data quality is improved, credibility quantification is carried out on the final predicted value, and the reliability of the final completion result is improved.

Description

Technical Field

[0001] The present invention relates to the field of data quality improvement, and in particular to a method and system for completing and fusing multiple data of a unified social credit code. Background Art

[0002] The rapid development of digitalization and informatization is driving the development of the digital economy. Among them, big data technology has played a vital supporting role in promoting the efficient implementation of digital application scenarios.

[0003] As one of the most common application scenarios of big data technology, artificial intelligence has been increasingly used in natural language processing, image recognition, data mining, etc. As a foundation, a large number of data samples need to be provided for feature learning and model training to discover the potential correlation between data.

[0004] Unified social credit code data (also known as institutional information) refers to a basic data set formed by integrating and verifying various types of legal person and other organizational information sent back by various registration management departments based on the unified social credit code as the unique identifier and relying on the unified social credit code database. It uses the unified social credit code of the institution as the unique identifier, and associates multiple data attributes such as the name of the institution, registered address, economic industry, registered capital, and number of employees. Due to abnormal conditions that may be encountered in the collection, transfer, cleaning and other links, it is inevitable that some data items (or data attributes) of some data samples will be missing. Missing data samples cannot be directly used to train machine models, otherwise it is easy to cause overfitting of the machine model, resulting in poor accuracy and reliability of the predicted data. For these missing data samples, the traditional approach is to delete these data samples; however, there are still valuable data items in the missing data samples. Directly deleting such data samples as a whole will cause a waste of data resources and is not allowed in some scenarios. In order to make up for the defects of deleting data samples as a whole, some scholars have proposed to use interpolation, mean / median method, nearest neighbor method, regression analysis and rule-based methods to complete the missing data items; for example, the "Satellite Power System Missing Data Completion Method Based on Random Forest Network" proposed in the Chinese patent document with publication number CN115454988A, this method completes the missing segments of telemetry data by training the neural random forest NRF model. However, traditional data completion algorithms rely on a single method, lack the comprehensive use and credibility evaluation of multiple completion strategies, and are limited in dealing with different missing modes, making it difficult to ensure data quality. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for integrating multiple data completion of a unified social credit code to improve the accuracy and reliability of data completion in order to address all or part of the above-mentioned problems.

[0006] The technical solution adopted by the present invention is as follows: A method for completing and fusing multiple data of a unified social credit code, comprising: S1. Collect data samples of unified social credit code data, where the data attributes contained in the data samples include at least: institution name, registered address, economic industry, registered capital and number of employees; S2, preprocessing the data sample; S3. Taking the economic industry as the completion target, use the complete data samples to train multiple data completion models respectively; use the trained data completion models to calculate the completion value of the economic industry in each missing data sample; S4. Randomly delete economic sectors from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic sectors, and use the accuracy as the credibility of the corresponding data completion model; S5. Based on the credibility of each data completion model, the completion values ​​of each data completion model are weighted and integrated to obtain the final prediction value of the economic industry in the missing data sample; S6. quantify the credibility of the final prediction value.

[0007] In order to address the above problems, the present invention also provides a unified social credit code multiple data completion and fusion system, which includes: A data collection module, used to collect data samples of unified social credit code data, wherein the data attributes contained in the data samples include at least: institution name, registered address, economic industry, registered capital and number of employees; A data preprocessing module, used for preprocessing the data samples; The data completion module is used to take the economic industry as the completion target and use the complete data samples to train multiple data completion models respectively; the trained data completion models are used to calculate the completion value of the economic industry in each missing data sample; The credibility calculation module is used to randomly delete economic sectors from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic sectors, and use the accuracy as the credibility of the corresponding data completion model; The completion value fusion module is used to perform weighted fusion on the completion values ​​of each data completion model based on the credibility of each data completion model to obtain the final prediction value of the economic industry in the missing data sample; The credibility quantification module is used to quantify the credibility of the final prediction value.

[0008] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: The unified social credit code multiple data completion fusion scheme of the present invention fuses the completion values ​​of multiple different data completion models, integrates the characteristics of different learning models, improves the robustness of the final completion result, and thus improves the data quality. The unified social credit code multiple data completion fusion method and system proposed by the present invention are used to complete the economic industry of the missing unified social credit code data (data attributes of similar nature can also be completed in the same way) in the scenario of data application to the unified social credit code database for analysis, data mining, user modeling, etc., so as to complete the missing unified social credit code data to be consistent with the data distribution law of the complete unified social credit code data in the database, so that the missing unified social credit code data can be analyzed and utilized without introducing too many errors in the completed data attributes. The present invention also quantifies the credibility of the final prediction value, which can improve the reliability of the final completion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 It is a flow chart of the method for completing and integrating multiple data of a unified social credit code provided in an embodiment of the present application.

[0010] Figure 2 It is a structural diagram of the unified social credit code multiple data completion and fusion system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0011] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0012] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other alternative features that are equivalent or have similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0013] In view of the problem that traditional data completion methods rely on a single completion mode and have poor data quality when dealing with different missing modes, the embodiments of the present application provide a unified social credit code multiple data completion and integration method and system, aiming to make the completed data more consistent with the distribution pattern of the original data and improve the reliability of the completed data.

[0014] Example 1 In some embodiments, Figure 1 As shown in the figure, the unified social credit code multiple data completion and integration method includes the following processes: S1. Collect data samples of unified social credit code data, which contain multiple data attributes, including at least: institution name, registered address, economic industry, registered capital and number of employees.

[0015] According to the format designed for the data sample, it contains multiple data attributes, that is, each data sample contains multiple data attributes. Among the large number of data samples collected, there are complete data samples, that is, there are no missing data attributes, and there are inevitably missing data samples, that is, there are missing data attributes (missing attribute values).

[0016] In some embodiments, for a specific scenario, step S1 includes the following sub-steps: S1.1. Collect multiple data samples in the unified social credit code database, each of which contains multiple data attributes. These data attributes include data attributes such as institution name, registered address, economic industry, registered capital, and number of employees. Table 1 shows an example of some data attributes of the collected data samples.

[0017] Table 1. Partial data attribute table of unified social credit code data

[0018] S1.2. For the collected data samples, each data attribute is divided into a discrete data attribute or a continuous data attribute.

[0019] Discrete data attributes have text characteristics and cannot be directly used for data calculation; continuous data attributes have numerical characteristics and data calculation can be performed directly based on their values.

[0020] For the data attributes of the unified social credit code data, "organization name", "registered address" and "economic industry" are discrete data attributes, while "registered capital" and "number of employees" (or in some embodiments, "latitude and longitude") are continuous data attributes. Table 2 shows the division results of some data attributes of the unified social credit code data.

[0021] Table 2 Classification of some data attributes of unified social credit code data

[0022] S2. Preprocess the data samples.

[0023] The collected data samples may not be accurate, especially for massive data sources, there are inevitably outliers. In addition, the various data attributes contained in the data samples also need to consider how to facilitate computer calculation and processing.

[0024] In some embodiments, step S2 includes the following sub-steps: S2.1. Eliminate outliers in the data sample.

[0025] Data samples are collected one by one. When storing each data sample, each data attribute will be traversed in turn. For each data attribute, there is a reasonable range of values ​​(which can be set in advance). For example, some data attributes (such as registered capital, number of employees, etc.) cannot be negative values; when a data attribute exceeds its reasonable value, it is judged as an outlier and is eliminated.

[0026] S2.2. Label encode the discrete data attributes in the data sample and normalize the continuous data attributes in the data sample.

[0027] In the unified social credit code data, the "registration address" is labeled and coded step by step according to the province-city-district-street-road level, and the "economic industry" is mapped according to the numerical association table to complete the label coding. For example, after sorting all the "economic industry" attribute values, continuous label coding is performed in sequence starting from 1. To express, Indicates the first The label encoding of the attribute value is , as shown in Table 3; the “registered capital” is normalized.

[0028] Table 3 Economic industry coding table

[0029] Assuming that there are 9,000 attribute values ​​in the economic industry, the label encoding results of each attribute value are: .

[0030] In some feasible implementations, the normalization method for continuous data attributes is: , in, Indicates the first attribute values, represents the number of collected data samples, Express The normalized processing result.

[0031] S2.3. Convert all data attributes into floating point data to facilitate computer processing.

[0032] S3. Taking the economic industry as the completion target, use the complete data samples to train multiple data completion models respectively; use the trained data completion models to calculate the completion value of the economic industry in each missing data sample.

[0033] It should be noted that for a data sample, each of its data attributes belongs to a feature of the data sample, reflecting its characteristics in a certain aspect. Under a large number of data samples, the distribution characteristics of each data attribute should tend to be the same. In some embodiments of the present application, the economic industry here can be replaced by other data attributes.

[0034] In some embodiments, step S3 includes the following sub-steps: S3.1. Randomly divide the complete data sample into training set and test set according to a predetermined ratio.

[0035] Generally speaking, the ratio of training set to test set is 7:3. Of course, it can also be other ratios, such as 8:2, but the proportion of training set needs to be larger than that of test set.

[0036] S3.2. Taking all data attributes in the training set and the test set as input features (with the economic industry as the true value) and the economic industry as the target feature, multiple learning models are used to train the corresponding data completion models on the training set and the test set respectively.

[0037] As an optional mode, the multiple learning models include at least one of a mean filling model, a regression analysis model, a K-nearest neighbor model, a random forest model, an XGBOOST model, and an NGBoost model. Assume that the selected learning model is a K-nearest neighbor model, a random forest model, an XGBOOST model, and an NGBoost model.

[0038] The pre-processed "organization name", "economic industry", "registered capital", "registered address", "number of employees" and other data samples in the training set are input into each learning model in batches for training, where "economic industry" is used as the true value to guide the adjustment of model parameters. After the training of the training set is completed, the trained learning model is tested by the test set, and the corresponding data completion model is obtained after completion. The same applies if other data attributes such as "registered capital" need to be completed in some embodiments.

[0039] S3.3. Use each data completion model to complete the economic industry in each missing data sample and obtain the corresponding completion value.

[0040] The missing data samples of "economic industry" are completed using four data completion models trained by K-nearest neighbor model, random forest model, XGBOOST model and NGBoost model (only learning models selected for one embodiment), and each missing data sample will get four completion values.

[0041] S4. Randomly delete economic sectors from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic sectors, and use the accuracy as the credibility of the corresponding data completion model.

[0042] In some embodiments, step S4 includes the following sub-steps: S4.1. Randomly delete economic industries with multiple complete data samples.

[0043] As a feasible approach, in the scenario of completing economic industries, data samples can be randomly selected from all complete data samples, and the economic industries of the selected data samples can be deleted.

[0044] S4.2. For each data completion model, the accuracy rate of completing each deleted economic industry is calculated as the credibility of the data completion model.

[0045] For each data completion model, based on the data attributes retained in each sample data, the completion value of the missing economic industry is predicted, and the deleted economic industry is used to evaluate whether the completion value is correct. The number of sample data with correct completion values ​​is counted, and the accuracy of the data completion model is calculated based on the total number of completed sample data. The accuracy is used as the credibility of the data completion model. The credibility calculation expression is: , In the formula, the total number of complete data samples is the number of data samples with economic industries deleted, and the number of correctly completed data samples is the number of data samples of economic industries correctly predicted by the data completion model. Indicates The credibility of the data completion model, Represents the total number of data completion models. Taking the four learning models selected in the previous article, namely, the K-nearest neighbor model, the random forest model, the XGBOOST model, and the NGBoost model, as examples, here .

[0046] For example, in a certain application scenario, the calculated credibility of four data completion models is shown in Table 4.

[0047] Table 4 Data completion model credibility table

[0048] S5. Based on the credibility of each data completion model, the completion values ​​of each data completion model are weighted and integrated to obtain the final prediction value of the economic industry in the missing data sample.

[0049] Each data completion model has its own characteristics. The embodiment of the present application combines the characteristics of different data completion models to complete the missing data attributes. In different application scenarios, the completion effects reflected by various data completion models show different credibility due to different scenarios. Therefore, for different application scenarios, the embodiment of the present application uses the credibility shown by different data completion models in the application scenario as the contribution of its completion value, and weights its completion value based on the credibility of different data completion models, and finally integrates the completion values ​​of all data completion models as the final prediction value.

[0050] In some embodiments, step S5 includes the following sub-steps: S5.1. Based on the DS (Dempster-Shafer) evidence theory, a recognition framework covering all completion values ​​is constructed.

[0051] Taking the four learning models selected in the previous article as an example, the completion values ​​of the four data completion models are expressed as , constructing the identification framework based on DS evidence theory ,in, Propositional Representation is the correct completion value, is the complete set of all possible hypotheses.

[0052] S5.2. The credibility of each data completion model is used as the basic probability distribution of its completion value.

[0053] The four data completion models are regarded as an independent source of evidence and expressed as , the basic probability distribution of each completion value can be expressed as: .in The value of indicates the source of evidence Topic (Right now to complete the value correctly), Indicates that the completion value is In the case of The credibility of the data completion model for the completion value, Indicates source of evidence Will The basic probability of is assigned to the uncertain hypothesis (i.e. the full set). The basic probability distribution of each data completion model is shown in Table 5.

[0054] Table 5 Basic probability distribution table

[0055] S5.3. Use the DS fusion rule to obtain the basic trust function value of each completion value.

[0056] The DS fusion rule is used to combine multiple credibility into a basic trust function value, which requires calculating the conflict coefficient first. K , the calculation method is as follows: .

[0057] The basic trust function value of the complement value is calculated based on the conflict coefficient K, and finally the basic trust function value of each proposition is obtained. , the calculation method is as follows: .

[0058] The basic trust function values ​​of the final completion values ​​are shown in Table 6.

[0059] Table 6 Basic trust function value table of completion value

[0060] S5.4. Perform weighted averaging of the completed values ​​using the basic trust function values ​​to obtain the final predicted value.

[0061] The method of using each basic trust function value to calculate the weighted average of the completion value is: , In the formula, Represents the final predicted value.

[0062] S6. quantify the credibility of the final prediction value.

[0063] The multi-data completion and fusion method of the embodiment of the present application also performs credibility evaluation on the completed data attributes. In some embodiments, step S6 includes: quantifying the information entropy using the basic trust function value of each completion value, and obtaining the credibility of the final prediction value based on the quantized information entropy.

[0064] Information Entropy H The calculation method is: , Credibility Defined as the inverse of information entropy: .

[0065] Information entropy can be used to measure the distribution of basic trust function values. The more dispersed the distribution means the greater the uncertainty of the fusion value. The higher the information entropy, the lower the credibility.

[0066] Example 2 like Figure 2 As shown, in some embodiments, the unified social credit code multiple data completion and fusion system includes a data acquisition module, a data preprocessing module, a data completion module, a credibility calculation module, a completion value fusion module and a credibility quantification module.

[0067] The data collection module is used to collect data samples of unified social credit code data, and the data samples contain multiple data attributes, including institution name, registered address, economic industry, registered capital and number of employees, etc.

[0068] As a feasible approach, the data collection module is configured to: collect multiple data samples in the unified social credit code database; for the collected data samples, divide each data attribute into a discrete data attribute or a continuous data attribute.

[0069] The data preprocessing module is used to preprocess the data samples.

[0070] As a feasible approach, the data preprocessing module is configured to: eliminate outliers in data samples; perform label encoding on discrete data attributes in data samples, and normalize continuous data attributes in data samples; and convert all data attributes into floating-point data for computer processing.

[0071] The data completion module is used to take the economic industry as the completion target and use complete data samples to train multiple data completion models; the trained data completion models are used to calculate the completion value of the economic industry in each missing data sample.

[0072] As a feasible approach, the data completion module is configured as follows: the complete data sample is randomly divided into a training set and a test set according to a predetermined ratio; all data attributes in the training set and the test set are used as input features, and the economic industry is used as the target feature, and a variety of learning models are used to train corresponding data completion models on the training set and the test set respectively; each data completion model is used to complete the economic industry in each missing data sample to obtain the corresponding completion value.

[0073] The credibility calculation module is used to randomly delete economic industries from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic industries, and use the accuracy as the credibility of the corresponding data completion model.

[0074] As a feasible approach, the credibility calculation module is configured to: randomly delete economic industries of multiple complete data samples; for each data completion model, calculate the accuracy rate of completing each deleted economic industry as the credibility of the data completion model.

[0075] The completion value fusion module is used to perform weighted fusion on the completion values ​​of each data completion model based on the credibility of each data completion model to obtain the final prediction value of the economic industry in the missing data sample.

[0076] As a feasible approach, the completion value fusion module obtains the final prediction value according to the following configuration: According to DS evidence theory, a recognition framework covering all complement values ​​is constructed; The credibility of each data completion model is used as the basic probability distribution of its completion value; The basic trust function value of each completion value is obtained by using the DS fusion rule; The final prediction value is obtained by performing weighted averaging on the completed values ​​using the basic trust function values.

[0077] The credibility quantification module is used to quantify the credibility of the final prediction value.

[0078] As a feasible approach, the credibility quantification module performs credibility quantification on the final prediction value according to the following configuration: The information entropy is quantified using the basic trust function value of each complement value, and the credibility of the final prediction value is obtained based on the quantified information entropy.

[0079] For the specific implementation of each module, the system embodiment can refer to the relevant features in the previous method embodiment, so it will not be repeated in this embodiment.

[0080] The unified social credit code multiple data completion and integration method and system provided in the embodiments of the present application are used to complete the economic industry (or other data attributes that are missing in some embodiments) of the unified social credit code data that is missing in the unified social credit code database.

[0081] An embodiment of the present application also provides another unified social credit code multiple data completion and integration system, which includes a processor and a storage medium, the storage medium stores a computer program, and the processor runs the computer program to execute the unified social credit code multiple data completion and integration method of the above embodiment.

[0082] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A method for completing and fusing multiple data of a unified social credit code, characterized in that: include: S1. Collect data samples of unified social credit code data, where the data attributes contained in the data samples include at least: institution name, registered address, economic industry, registered capital and number of employees; S2, preprocessing the data sample; S3. Taking the economic industry as the completion target, use the complete data samples to train multiple data completion models respectively; use the trained data completion models to calculate the completion value of the economic industry in each missing data sample; S4. Randomly delete economic sectors from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic sectors, and use the accuracy as the credibility of the corresponding data completion model; S5. Based on the credibility of each data completion model, the completion values ​​of each data completion model are weighted and integrated to obtain the final prediction value of the economic industry in the missing data sample; S6. quantify the credibility of the final prediction value.

2. The method for completing and fusing multiple data of a unified social credit code according to claim 1, characterized in that: The step S2 comprises: S2.1, eliminating outliers in the data sample; S2.2, label encoding the discrete data attributes in the data sample, and normalizing the continuous data attributes in the data sample; S2.

3. Convert all data attributes into floating point data.

3. The method for completing and fusing multiple data of a unified social credit code according to claim 1, characterized in that: The step S3 comprises: S3.

1. Randomly divide the complete data sample into training set and test set according to a predetermined ratio; S3.2, using all data attributes in the training set and the test set as input features, and the economic industry as the target feature, respectively using multiple learning models to train corresponding data completion models on the training set and the test set; S3.

3. Use each of the data completion models to complete the economic industry in each missing data sample to obtain the corresponding completion value.

4. The method for completing and fusing multiple data of a unified social credit code as claimed in claim 3, characterized in that: The multiple learning models include at least one of a mean filling model, a regression analysis model, a K-nearest neighbor model, a random forest model, an XGBOOST model, and an NGBoost model.

5. The method for completing and fusing multiple data of a unified social credit code according to claim 1, characterized in that: The step S4 comprises: S4.

1. Randomly delete economic industries with multiple complete data samples; S4.

2. For each data completion model, the accuracy rate of completing each deleted economic industry is calculated as the credibility of the data completion model.

6. The method for completing and fusing multiple data of a unified social credit code according to claim 1, characterized in that: The step S5 comprises: S5.

1. Construct a recognition framework covering all complement values ​​based on DS evidence theory; S5.2, respectively using the credibility of each data completion model as the basic probability distribution of its completion value; S5.3, using the DS fusion rule to obtain the basic trust function value of each completion value; S5.

4. Perform weighted averaging of the completed values ​​using the basic trust function values ​​to obtain the final predicted value.

7. The method for completing and fusing multiple data of a unified social credit code according to claim 6, characterized in that: The step S6 comprises: The information entropy is quantified using the basic trust function value of each complement value, and the credibility of the final prediction value is obtained based on the quantified information entropy.

8. A unified social credit code multiple data completion and fusion system, characterized by: include: A data collection module, used to collect data samples of unified social credit code data, wherein the data attributes contained in the data samples include at least: institution name, registered address, economic industry, registered capital and number of employees; A data preprocessing module, used for preprocessing the data samples; The data completion module is used to take the economic industry as the completion target and use the complete data samples to train multiple data completion models respectively; the trained data completion models are used to calculate the completion value of the economic industry in each missing data sample; The credibility calculation module is used to randomly delete economic sectors from the complete data sample, calculate the accuracy of each data completion model in completing the deleted economic sectors, and use the accuracy as the credibility of the corresponding data completion model; The completion value fusion module is used to perform weighted fusion on the completion values ​​of each data completion model based on the credibility of each data completion model to obtain the final prediction value of the economic industry in the missing data sample; The credibility quantification module is used to quantify the credibility of the final prediction value.

Images