Customer basic archive data checking method based on machine learning and storage medium
Through a machine learning-based method, a data verification model is built using unified social credit code and text information similarity judgment, which solves the problem of low data matching accuracy in the existing technology, and achieves efficient and accurate data verification and management.
Patent Information
- Application Number
- CN202410991838.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to adapt to changing business needs due to registration errors when the internal user data of the enterprise matches external industrial and commercial data.
Using a basic customer archive data verification method based on machine learning, we use the enterprise internal and external data to judge the existence of unified social credit code and the similarity of text information, and build a data verification model to improve matching accuracy and efficiency.
It effectively improves the accuracy and efficiency of data matching, ensures the accuracy and consistency of internal and external data in the enterprise, and adapts to complex business needs.
Smart Images

Figure CN118964597A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of basic archive data verification, and in particular to a verification method and storage medium for customer basic archive data based on machine learning. Background Art
[0002] With the rapid development of information technology, the amount of enterprise data has increased dramatically. How to efficiently and accurately manage and use these basic archival data has become an important issue. In existing technical solutions, the matching of internal enterprise user data with external industrial and commercial data often becomes complicated and difficult due to various registration errors, which not only affects the accuracy of the data, but also reduces the efficiency of data management. In the power grid system, if there are problems with the basic archival data of the enterprise, it will have a serious impact on related businesses.
[0003] At present, the technologies for data matching and verification mainly include string matching algorithms, rule-based expert systems, and some simple natural language processing technologies. When processing data matching, these technologies often rely on preset rules or simple text similarity calculations, lack sufficient flexibility and accuracy, and are difficult to adapt to changing business needs, resulting in low matching accuracy.
[0004] The inventors of the present application discovered during the process of implementing the present invention that the above-mentioned solution in the prior art has the defect of low matching accuracy. Summary of the invention
[0005] The purpose of the embodiments of the present invention is to provide a method and storage medium for verifying customer basic file data based on machine learning, which has the function of matching internal user data of an enterprise with external industrial and commercial data with high accuracy.
[0006] In order to achieve the above object, an embodiment of the present invention provides a method for verifying customer basic file data based on machine learning, comprising:
[0007] Access to the company's internal data and external databases;
[0008] Determine whether the unified social credit code in the internal data exists in the external database;
[0009] In the case where it is determined that the unified social credit code in the internal data exists in the external database, obtaining external data in the external database that matches the internal data of the enterprise;
[0010] Determining whether the text information in the internal data and the text information in the external data are the same;
[0011] In the case where it is determined that the text information in the internal data and the external data is the same, it is determined that the internal and external data of the enterprise are consistent;
[0012] In the case where it is determined that the text information in the internal data and the text information in the external data are different, it is determined that the internal and external data of the enterprise are inconsistent;
[0013] When it is determined that the unified social credit code in the internal data does not exist in the external database, it is determined that the internal data is incorrect.
[0014] Optionally, determining whether the unified social credit code in the internal data exists in the external database includes:
[0015] Obtaining a unified social credit code from the internal data;
[0016] Determining whether the first digit of the unified social credit code in the internal data is 9;
[0017] When it is determined that the first digit of the unified social credit code in the internal data is 9, sequentially matching the unified social credit code in the internal data with the unified social credit code in the external database;
[0018] When it is determined that the first digit of the unified social credit code in the internal data is not 9, it is determined that the unified social credit code in the internal data is incorrect.
[0019] Optionally, determining whether the text information in the internal data and the text information in the external data are the same includes:
[0020] Preprocessing the text information in the internal data and the external data, and performing feature extraction;
[0021] Acquire the similarity between the features of the internal data and the external data;
[0022] A data verification model is obtained according to the similarity between the features of the internal data and the external data.
[0023] Optionally, preprocessing the text information in the internal data and the external data includes:
[0024] Filling in missing parts and removing symbols from the text information;
[0025] Segmenting the text information to obtain an address portion, a subjective portion, a category portion, and a unit portion of the text information;
[0026] Preset stop words in the address part, the subjective part, the category part and the unit part of the text information are removed.
[0027] Optionally, segmenting the text information includes:
[0028] Acquire first historical data of the enterprise, wherein the first historical data includes first word data and a first word label;
[0029] Preprocessing the first historical data;
[0030] Build a data segmentation model;
[0031] The data segmentation model is trained using the preprocessed first historical data.
[0032] Optionally, segmenting the text information further includes:
[0033] Acquire second historical data of the enterprise, wherein the second historical data includes second word data and a second word label;
[0034] Preprocessing the second historical data;
[0035] Build data NER model;
[0036] The NER model is trained using the preprocessed second historical data.
[0037] Optionally, obtaining the similarity of the features of the internal data and the external data includes:
[0038] Vectorize the preprocessed text features;
[0039] The similarity between the internal data and the text features corresponding to the external data is obtained.
[0040] Optionally, obtaining the similarity between the internal data and the text features corresponding to the external data includes:
[0041] The similarity between the text features corresponding to the internal data and the external data is calculated according to formula (1).
[0042]
[0043] Among them, cosθ is the similarity between the text features corresponding to the internal data and the external data, A is the text feature vector of the internal data, B is the text feature vector of the external data, θ is the angle between the text feature vector of the internal data and the text feature vector of the external data, and ‖‖ is a modulo operation.
[0044] Optionally, acquiring a data verification model according to similarities between features of the internal data and the external data includes:
[0045] Dividing the internal data, the external data and the similarity of the text features of the two into a training set, a validation set and a test set;
[0046] Constructing a data verification model, wherein the data verification model includes a random forest model;
[0047] Using the training set to train the data verification model;
[0048] Using the validation set to perform hyperparameter training and optimization on the data verification model;
[0049] The test set is used to test and evaluate the data verification model.
[0050] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions are used to be read by a machine so that the machine executes any of the above-mentioned verification methods.
[0051] Through the above technical scheme, the verification method and storage medium of customer basic file data based on machine learning provided by the present invention determine whether there is an external database with the unified social credit code of the internal data of the enterprise, and if the unified social credit code exists in the external database, the external data corresponding to the unified social credit code in the external database and the internal data of the enterprise are further verified by text information. If the verification is consistent, it means that the internal and external data of the enterprise are consistent, otherwise they are inconsistent. The method of first matching the internal and external data with the unified social credit code and then further verifying the text information can effectively improve the accuracy of the verification on the one hand, and on the other hand, it can also effectively improve the efficiency of the verification.
[0052] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:
[0054] Figure 1 is a flow chart of a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention;
[0055] Figure 2 is a flow chart of internal data diagnosis in a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention;
[0056] Figure 3 It is a flowchart of preprocessing internal data and external data in a method for verifying customer basic archive data based on machine learning according to an embodiment of the present invention;
[0057] Figure 4 It is a flowchart of preprocessing internal data and external data in a method for verifying customer basic archive data based on machine learning according to an embodiment of the present invention;
[0058] Figure 5 It is a flow chart of word segmentation of text information in a method for verifying customer basic archive data based on machine learning according to an embodiment of the present invention;
[0059] Figure 6 It is a flow chart of calculating the similarity between internal data and external data in a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention;
[0060] Figure 7 It is a flow chart of training a data verification model in a verification method for customer basic profile data based on machine learning according to an embodiment of the present invention;
[0061] Figure 8 is a block diagram of a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention;
[0062] Fig. 9 The present invention provides an example of word segmentation using a word segmentation model and a NER model in a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0063] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.
[0064] Figure 1 1 is a flow chart of a method for verifying customer basic profile data based on machine learning according to an embodiment of the present invention. Figure 1 In the present invention, the verification method may include:
[0065] In step 10, the internal data and external database of the enterprise are obtained. The internal data of the enterprise may include the enterprise name, enterprise address, etc., and the external database may include the business data registered by the enterprise. Specifically, the internal data of the enterprise also includes the data set to be trained.
[0066] In step 11, it is determined whether the unified social credit code in the internal data exists in the external database. Taking the internal data of a certain enterprise as an example, in order to determine whether the enterprise has a record in the external industrial and commercial database (i.e., the external database), the unified social credit code of the internal data is first searched in the external database corresponding to the industrial and commercial data.
[0067] In step 12, when it is determined that the unified social credit code in the internal data exists in the external database, the external data in the external database that matches the internal data of the enterprise is obtained. If the external data contains the unified social credit code of the enterprise, it means that the enterprise has registered its enterprise data with the industrial and commercial administration department, and the registered data is used as the external data of the enterprise.
[0068] In step 13, it is determined whether the text information in the internal data and the text information in the external data are the same.
[0069] In step 14, when it is determined that the text information in the internal data and the external data is the same, it is determined that the internal and external data of the enterprise are consistent. If the text information in the internal data and the external data of the enterprise is the same, it means that the internal data and the external data can correspond, and the internal and external data of the enterprise are consistent. Specifically, the same here means substantially the same, and the content of the similarity calculation described below can be referred to.
[0070] In step 15, when it is determined that the text information in the internal data and the external data are different, it is determined that the internal and external data of the enterprise are inconsistent. If the text information in the internal data and the external data of the enterprise is different, it means that there is a difference between the internal data and the external data of the enterprise, and the internal and external data of the enterprise are inconsistent.
[0071] In step 16, if it is determined that the unified social credit code in the internal data does not exist in the external database, it is determined that the internal data is incorrect. If the unified social credit code of the internal data does not exist in the external database, it means that the internal data / unified social credit code of the enterprise is incorrect.
[0072] In step S10 to step S16, the internal data and external industrial and commercial database of the enterprise are first obtained, and the unified social credit code in the internal data is searched in the external industrial and commercial database. If the unified social credit code of the internal data exists in the external industrial and commercial database, the external data corresponding to the unified social credit code in the external industrial and commercial database is used as the data to be matched. Further, the text information of the internal data and the text information of the external data are matched. If the text information of the two is substantially the same, it is determined that the internal and external data of the enterprise are consistent in verification, otherwise the verification is inconsistent. In addition, if the unified social credit code of the internal data does not exist in the external industrial and commercial database, it means that the internal data of the enterprise is incorrect.
[0073] Traditional data matching and verification technologies mainly include string matching algorithms, rule-based expert systems, and some simple natural language processing technologies. These technologies often rely on preset rules or simple text similarity calculations when processing data matching, lack sufficient flexibility and accuracy, and are difficult to adapt to changing business needs, resulting in low matching accuracy. In this embodiment of the present invention, the unified social credit code is used to first match internal and external data, and then the text information is further verified. On the one hand, it can effectively improve the accuracy of the verification, and on the other hand, it can also effectively improve the efficiency of the verification.
[0074] In this embodiment of the present invention, before searching for the unified social credit code of the internal data, it is necessary to diagnose itself first. The specific diagnosis process can be as follows: Figure 2 Specifically, Figure 2 In the above, the verification method may further include:
[0075] In step S110, a unified social credit code in internal data is obtained, wherein the unified social credit code of the enterprise is unique and identifiable.
[0076] In step S111, it is determined whether the first digit of the unified social credit code in the internal data is 9. The first digit of the unified social credit code represents the registration management department code, and the first digit of the registration of the industrial and commercial administration department is 9.
[0077] In step S112, when it is determined that the first digit of the unified social credit code in the internal data is 9, the unified social credit code in the internal data is sequentially matched with the unified social credit code in the external database. If the first digit of the unified social credit code is 9, it means that the unified social credit code has been registered with the industrial and commercial administration department, and the other digits of the unified social credit code are further searched and matched.
[0078] In step S113, if the first digit of the unified social credit code in the internal data is not 9, it is determined that the unified social credit code in the internal data is incorrect. If the first digit of the unified social credit code is not 9, it means that the unified social credit code is incorrect.
[0079] In step S110 to step S113, the first digit of the unified social credit code of the internal data is diagnosed to determine whether it is 9. If the first digit of the unified social credit code is 9, the external industrial and commercial database is further searched for the remaining digits that match the unified social credit code. Otherwise, the unified social credit code is determined to be incorrect. Specifically, the method of performing a preliminary diagnosis on the unified social credit code of the internal data can effectively improve the efficiency and effect of subsequent text information verification. If the unified social credit codes are not the same, the subsequent text verification will be meaningless.
[0080] In this embodiment of the present invention, when the external data corresponding to the internal data is searched in the external industrial and commercial database, it is necessary to further verify the text information of the two. The specific verification steps can be as follows: Figure 3 and Figure 8 Specifically, Figure 3 and Figure 8 In the above, the verification method may further include:
[0081] In step S130, the text information in the internal data and the external data is preprocessed and features are extracted. The text information in the internal data and the external data may include company names, address information, etc. Specifically, the preprocessing of the text information may include deletion symbols, word segmentation, etc., and the preprocessed multiple words are used as text features of the internal data or the external data.
[0082] In step S131, the similarity of the features of the internal data and the external data is obtained. After the text features of the internal data and the external data are obtained, the similarity of each word is calculated respectively, and finally summarized to obtain the total similarity.
[0083] In step S132, a data verification model is obtained according to the similarity of the features of the internal data and the external data. After the similarity of the internal data and the external data is obtained, a training set of the data verification model is constructed and trained.
[0084] In step S130 to step S132, the text information of the internal data and the external data is preprocessed to obtain text features, and then the similarity between the two is calculated respectively, a training set is constructed according to the similarity, and the verification model is trained.
[0085] In this embodiment of the present invention, in the process of preprocessing the text information of the internal data and the external data, it is necessary to segment them according to the categories. The specific segmentation step can be as follows: Figure 4 Specifically, Figure 4 In the above, the verification method may further include:
[0086] In step S1300, the text information is filled with missing parts and symbols are removed. Taking the company name as an example, the text information can be divided into four parts: address part, subjective part, category part and unit part. Specifically, such as "Baidu Online Network Technology (Beijing) Co., Ltd.", it can be divided into address part (Beijing), subjective part (Baidu), category part (Online Network Technology), unit part (Co., Ltd.). Therefore, it is not difficult to see that the address part describes the geographical location information in the entity name, the subjective part has a special meaning for the entity or can represent the information of the entity, the category part describes the information of the business category, and the unit part describes the information of the entity's business unit.
[0087] According to the above four divisions, the internal and external names of the same entity may appear in the following different ways:
[0088] 1. The four parts are the same, but some symbols are added to one of them, or the Chinese and English formats of the symbols are different. For example, "State Grid Electric Vehicle Service (Shaanxi) Co., Ltd." and "State Grid Electric Vehicle Service (Shaanxi) Co., Ltd." have different Chinese and English formats of symbols, and "State Grid Electric Vehicle Service Shaanxi Co., Ltd." and "State Grid Electric Vehicle Service (Shaanxi) Co., Ltd." have a symbol added to one of them.
[0089] 2. The address parts are different, such as "Anhua Cement Products Factory" and "Xianyang Anhua Cement Products Co., Ltd." are missing addresses, "Shaanxi Wanzhou Electrical Equipment Co., Ltd." and "Xi'an Wanzhou Electrical Equipment Co., Ltd." are different address levels, and "Zhen'an County Shengshi Fengda Automobile Service Co., Ltd." and "Zhen'an Shengshi Fengda Automobile Service Co., Ltd." are missing geographical units.
[0090] 3. The categories are different, such as "Zhejiang Yunda Energy Technology Co., Ltd." and "Zhejiang Yunda Wind Power Co., Ltd." Specifically, we know that the subjective part of "Yunda" is the same, and the similar meanings of "energy technology" and "wind power" are the same entity, but the SQL query string commonly used in the prior art can only be queried by whether the text is the same, which leads to errors in matching.
[0091] 4. The unit part is different, such as "Anhua Cement Products Factory" and "Xianyang Anhua Cement Products Co., Ltd."
[0092] In addition, the subjective part is generally the same, and typos or similar words may appear in rare cases.
[0093] Therefore, some missing parts can be filled, data types can be converted, and special symbols can be removed to reduce the interference in the similarity calculation between the two.
[0094] In step S1301, the text information is segmented to obtain the address part, subjective part, category part and unit part of the text information. The above is the segmentation method for the enterprise name, and the keyword extraction method can be used for the enterprise address.
[0095] In step S1302, the preset stop words in the address part, subjective part, category part and unit part of the text information are removed. Among them, the preset stop words may include the first and second level addresses such as provinces and cities in the address part. Specifically, in most cases, the first and second level address divisions of provinces and cities in the address part of the two data associated with the unified social credit code are the same. After removing them, the similarity of the address part calculated later can be more accurate. In addition, considering the accuracy of word segmentation, the address part and the unit part can also be removed. Since the subjective part and the category part in the enterprise name can play a distinguishing role in entity discrimination, the similarity of the name processed in this way will also be more accurate.
[0096] In step S1300 to step S1302, the missing parts in the text information are first filled and the symbols are removed. Then the text information is segmented to obtain the address part, the subjective part, the analogy part and the unit part. Finally, the preset stop words in the four parts are removed to obtain the word features of the company name. In traditional data matching, the importance of pre-processing such as data filling and symbol removal is often ignored, resulting in lower similarity calculation and poor verification accuracy. The present application adopts methods such as missing value filling and symbol removal to effectively improve the accuracy and efficiency of internal and external data matching.
[0097] In this embodiment of the present invention, in order to further improve the accuracy of calculating the similarity of internal and external data, it is necessary to segment the text information and then calculate the similarity of each word. The specific segmentation step can be as follows: Figure 5 Specifically, Figure 5 In the above, the verification method may further include:
[0098] In step S13010, the first historical data of the enterprise is obtained, wherein the first historical data includes first word data and first word label. Specifically, the first historical data may include the enterprise name, enterprise address, etc. The word segmentation of the enterprise name and enterprise address may be divided according to the constituent words of the common enterprise name or enterprise address, specifically, including but not limited to the TF-IDF model and the TextRank model, etc.
[0099] In step S13011, the first historical data is preprocessed, wherein the preprocessing of the first historical data may include data cleaning, removal of stop words, etc. known to those skilled in the art.
[0100] In step S13012, a data segmentation model is constructed.
[0101] In step S13013, the data segmentation model is trained using the preprocessed first historical data.
[0102] In step S13014, the second historical data of the enterprise is obtained, wherein the second historical data includes second word data and a second word tag. Specifically, the second historical data may include the enterprise name, and the first word tag may include a subjective part and a category part.
[0103] In step S13015, the second historical data is preprocessed, wherein the preprocessing of the second historical data may include data cleaning, removal of stop words, etc. known to those skilled in the art.
[0104] In step S13016, a data NER model is constructed. The CRF model used in the data NER model is the best traditional machine learning algorithm. The model is trained with a large amount of name annotation data, and its classification accuracy can reach 95%. The model can obtain the subjective part and the category part of the enterprise name.
[0105] In step S13017, the preprocessed second historical data is used to train the NER model.
[0106] In step S13010 to step S13017, before the text information needs to be segmented, a data segmentation model and a data NER model are respectively constructed. The first historical data and the second historical data are used to train the data segmentation model and the data NER model respectively. Specifically, the data segmentation model can segment the company name and the company address, and the data NER model can segment the company name. Specifically, the data NER model can separate the subjective part and the category part of the company name. Two different segmentation models are used to consider the segmentation of multiple dimensions to obtain the similarity calculation of multiple dimensions, so as to further improve the calculation accuracy of the similarity of internal and external data. Specifically, the data segmentation model and the data NER model can be the segmentation model and the NER model known to those skilled in the art. Specifically, the feature extraction of the data segmentation model and the data NER model can be as follows. Fig. 9 The example given.
[0107] In this embodiment of the present invention, if the data segmentation model alone cannot separate the core content subjective part and the category part of the name, the data NER model can effectively extract both, thereby providing more valuable feature information for the subsequent classification model, thereby improving the accuracy of the final classification.
[0108] In this embodiment of the present invention, after the text information of the internal data and the external data is segmented, multiple word features of the internal data and the external data can be obtained. It is also necessary to simultaneously perform similarity calculation on the multiple word features corresponding to the internal data and the external data to obtain the similarity of the multiple word features. The specific calculation steps can be as follows: Figure 6 and Figure 8 Specifically, Figure 6 and Figure 8 In the above, the verification method may further include:
[0109] In step S1310, the preprocessed text features are vectorized. The text features are multiple word features, and the word vector model Word2Vec can be used to vectorize the word features to obtain word vectors. Specifically, the role of the word vector model is to convert natural language text into a numerical vector of a certain dimension, and the converted "distance of numerical vectors" can express the "similarity and distance between natural language texts.
[0110] In step S1311, the similarity between the text features of the internal data and the external data is obtained. The similarity between the text features of the internal data and the external data can be calculated according to the following formula. Specifically, the similarity between the text features of the internal data and the external data is calculated according to formula (1):
[0111]
[0112] Among them, γ is the similarity between the text features corresponding to the internal data and the external data, A is the text feature vector of the internal data, B is the text feature vector of the external data, and ‖‖ is the modulo operation. Specifically, according to formula (1), the similarity between the internal data and the external data of each word vector and text feature can be obtained. Specifically, after the word vector output by the data segmentation model is calculated by similarity, the similarity between the internal and external address word vectors and text features, and the similarity between the internal and external name word vectors and text features can be obtained; after the word vector output by the data NER model is calculated by similarity, the similarity of the subjective part of the name and the similarity of the category part of the name can be obtained.
[0113] In this embodiment of the present invention, after obtaining the similarity of the word vectors, it is necessary to train the data verification model according to the similarity. The specific training steps can be as follows: Figure 7 Specifically, Figure 7 In the above, the verification method may further include:
[0114] In step S1320, the similarity of internal data, external data and the text features of the two are divided into a training set, a validation set and a test set. The parameters for each sample in the training set may include internal and external address similarity, internal and external name similarity, internal and external name subjective category similarity, internal and external name category similarity and comprehensive similarity. Specifically, the internal and external address similarity and the internal and external name similarity can be calculated based on the similarity of each word vector in the address text and the name text and the mean of the text features; the comprehensive similarity can be calculated based on formula (2),
[0115] γ z =τ1·γ1+τ2·γ2+τ3·γ3+τ4·γ4, (2)
[0116] Among them, γ z is the comprehensive similarity, τ1 is the address weight, γ1 is the internal and external address similarity, τ2 is the name similarity, γ2 is the internal and external name similarity, τ3 is the subjective weight, γ3 is the subjective category similarity of the internal and external names, τ4 is the category weight, and γ4 is the category similarity of the internal and external names. Specifically, considering that the difference between corporate names mainly lies in the subjective part and the category part, the above weights can be determined according to the conditions shown in formula (3),
[0117]
[0118] According to formula (3), the weights of subjective category similarity of internal and external names and category similarity of internal and external names are relatively large, which can improve the accuracy of comprehensive similarity.
[0119] In addition, the target in the training set can be set to 1 or -1, that is, 1 represents that the internal and external data are consistent, and -1 represents that the internal and external data are inconsistent.
[0120] In step S1321, a data verification model is constructed, wherein the data verification model includes a random forest model. Specifically, the decision conditions of the decision trees in the random forest may include comparing the above features with corresponding thresholds respectively, and each decision tree randomly selects a preset number of decision conditions.
[0121] In step S1322, the data verification model is trained using the training set.
[0122] In step S1323, the validation set is used to train and tune the hyperparameters of the data verification model, wherein the hyperparameters may include the number of decision trees, the maximum depth of the decision tree, the number of small samples required for splitting the decision tree nodes, and the like.
[0123] In step S1324, the data verification model is tested and evaluated using a test set. For binary classification problems, models such as logistic regression, support vector machine, naive Bayesian classification, random forest, and XGBoost can also be used, but after evaluation on the test set, random forest has the best effect.
[0124] In step S1320 to step S1324, the calculated multiple similarity values are converted into multiple features as samples. According to the multiple features and objectives of the samples, a training set, a validation set, and a test set are constructed, and the data verification model is trained, hyperparameters are tuned, and evaluated, so as to obtain a complete data verification model. By using this data verification model, the internal and external data of the enterprise that needs to be queried in real time is input into the trained data verification model, and real-time and accurate verification of the internal and external data of the enterprise can be obtained.
[0125] On the other hand, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and the instructions are used to be read by a machine so that the machine executes any of the above verification methods.
[0126] Through the above technical scheme, the verification method and storage medium of customer basic file data based on machine learning provided by the present invention determine whether there is an external database with the unified social credit code of the internal data of the enterprise, and if the unified social credit code exists in the external database, the external data corresponding to the unified social credit code in the external database and the internal data of the enterprise are further verified by text information. If the verification is consistent, it means that the internal and external data of the enterprise are consistent, otherwise they are inconsistent. The method of first matching the internal and external data with the unified social credit code and then further verifying the text information can effectively improve the accuracy of the verification on the one hand, and on the other hand, it can also effectively improve the efficiency of the verification.
[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0128] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0132] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0135] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.
Claims
1. A method for verifying customer basic file data based on machine learning, characterized in that: include: Access to the company's internal data and external databases; Determine whether the unified social credit code in the internal data exists in the external database; In the case where it is determined that the unified social credit code in the internal data exists in the external database, obtaining external data in the external database that matches the internal data of the enterprise; Determining whether the text information in the internal data and the text information in the external data are the same; In the case where it is determined that the text information in the internal data and the external data is the same, it is determined that the internal and external data of the enterprise are consistent; In the case where it is determined that the text information in the internal data and the text information in the external data are different, it is determined that the internal and external data of the enterprise are inconsistent; When it is determined that the unified social credit code in the internal data does not exist in the external database, it is determined that the internal data is incorrect.
2. The verification method according to claim 1, characterized in that: Determining whether the unified social credit code in the internal data exists in the external database includes: Obtaining a unified social credit code from the internal data; Determining whether the first digit of the unified social credit code in the internal data is 9; When it is determined that the first digit of the unified social credit code in the internal data is 9, sequentially matching the unified social credit code in the internal data with the unified social credit code in the external database; When it is determined that the first digit of the unified social credit code in the internal data is not 9, it is determined that the unified social credit code in the internal data is incorrect.
3. The verification method according to claim 1, characterized in that: Determining whether the text information in the internal data and the text information in the external data are the same includes: Preprocessing the text information in the internal data and the external data, and performing feature extraction; Acquire the similarity between the features of the internal data and the external data; A data verification model is obtained according to the similarity between the features of the internal data and the external data.
4. The verification method according to claim 3, characterized in that: Preprocessing the text information in the internal data and the external data includes: Filling in missing parts and removing symbols from the text information; Segmenting the text information to obtain an address portion, a subjective portion, a category portion, and a unit portion of the text information; Preset stop words in the address part, the subjective part, the category part and the unit part of the text information are removed.
5. The verification method according to claim 4, characterized in that: Segmenting the text information includes: Acquire first historical data of the enterprise, wherein the first historical data includes first word data and a first word label; Preprocessing the first historical data; Build a data segmentation model; The data segmentation model is trained using the preprocessed first historical data.
6. The verification method according to claim 5, characterized in that: Segmenting the text information further includes: Acquire second historical data of the enterprise, wherein the second historical data includes second word data and a second word label; Preprocessing the second historical data; Build data NER model; The NER model is trained using the preprocessed second historical data.
7. The verification method according to claim 3, characterized in that: Acquiring the similarity of the features of the internal data and the external data includes: Vectorize the preprocessed text features; The similarity between the internal data and the text features corresponding to the external data is obtained.
8. The verification method according to claim 3, characterized in that: Obtaining the similarity between the internal data and the text features corresponding to the external data includes: The similarity between the text features corresponding to the internal data and the external data is calculated according to formula (1). Among them, cosθ is the similarity between the text features corresponding to the internal data and the external data, A is the text feature vector of the internal data, B is the text feature vector of the external data, θ is the angle between the text feature vector of the internal data and the text feature vector of the external data, and ‖‖ is a modulo operation.
9. The verification method according to claim 3, characterized in that: Acquiring a data verification model according to the similarity of the features of the internal data and the external data includes: Dividing the internal data, the external data and the similarity of the text features of the two into a training set, a validation set and a test set; Constructing a data verification model, wherein the data verification model includes a random forest model; Using the training set to train the data verification model; Using the validation set to perform hyperparameter training and optimization on the data verification model; The test set is used to test and evaluate the data verification model.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and the instructions are used to be read by a machine so that the machine executes the verification method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Data processing method and device, server and storage medium
CN110502529A
Certificate content error correction method, device and system
CN112364630A
Model training method, model training device and intelligent equipment
CN112528021A
User archive information improvement method and device, equipment and storage medium
CN113672702A
Enterprise industry recognition system and recognition method based on text similarity
CN114090736A
Cited By
Data checking management method and system based on artificial intelligence technology and medium
CN120030202A
Data Verification Management Method, System and Medium Based on Artificial Intelligence Technology
CN120030202B
