Method for automatically associating original data items with data elements and mapping original data items and data elements into new data items

Abstract

The invention discloses a method for automatically associating an original data item with a data element and mapping the original data item and the data element into a new data item. The method comprises the following steps: acquiring a data list needing to be processed from a service system; performing keyword matching on the original data items in the data list with a data element library and a data element association library by adopting a keyword wide matching algorithm to obtain data elements with the highest matching degree with the original data items; adopting a customizable feature extraction algorithm based on a regular expression to extract all features in the original data items and the customizable feature library, and performing rationality verification on all the features to obtain available features; and splicing the data element with the highest matching degree with the original data item and the available feature to generate a new data item, and checking the uniqueness of the new data item to finally form a mapped data item.

Description

Technical Field

[0001] This invention relates to the field of digitization, and more particularly to a method for automatically associating original data items with data elements and mapping them to new data items. Background Technology

[0002] To adapt to the development trend of big data and obtain the potential value of data assets, data analysis and data mining have become important means. However, because the data naming in the original information system is relatively independent and non-standard, there are significant differences in the names of data items on different devices and they do not have specific actual business meanings. Therefore, data governance has become the primary task.

[0003] Data governance requires associating raw data items with data elements in business systems, extracting usable features from the raw data items, and mapping them to new data items, which can then be used for data modeling. Currently, this data mapping process relies on manual configuration, that is, manually associating business system data with data elements and extracting usable features from the raw data items to map them to new data items. However, this process suffers from problems such as a huge workload and a high error rate, resulting in low efficiency.

[0004] To address the aforementioned technical issues, patent document No. 202011109507.2 discloses an automatic table relationship association method based on multiple hash functions, comprising the following steps: Step 1: Obtain the existing source database and, according to the connection configuration of the source database, obtain all data tables for which relationships to be discovered; Step 2: For each data table, first obtain the primary key of the data table, then initialize a binary object for the primary key data, and use the constructed hash function to map the original data of the source database, with the calculated binary vector corresponding one-to-one with the primary key sequence; Step 3: Based on the reference rules between primary and foreign keys, and based on the binary vector of the primary key, check the fields that may be related to the primary key, and record the relationship if the check passes.

[0005] The above technical solution automatically identifies the relationships between tables based on detailed data in the data table rather than table structure or field information, which solves the problems of incomplete relationships in existing database tables and the time-consuming and laborious manual verification.

[0006] However, the above technical solutions can only automatically associate table relationships used for data standardization, and cannot be applied to data in non-standard information systems.

[0007] Therefore, it is necessary to improve the existing technology to overcome the aforementioned defects. Summary of the Invention

[0008] The purpose of this invention is to provide a method for automatically associating original data items with data elements and mapping them to new data items, so as to improve the speed and quality of data mapping in the data governance process.

[0009] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0010] A method for automatically associating original data items with data elements and mapping them to new data items includes the following steps:

[0011] 1) Obtain the list of data to be processed from the business system;

[0012] 2) Using a broad keyword matching algorithm, the original data items in the data list are matched with the data element library and the data element association library respectively to obtain the data element with the highest matching degree with the original data item;

[0013] 3) A customizable feature extraction algorithm based on regular expressions is used to extract all features from the original data items and the custom feature library. After verifying the rationality of all features, usable features are obtained.

[0014] 4) The data element with the highest matching degree to the original data item and the available features are concatenated to generate a new data item. After verifying the uniqueness of the new data item, the mapped data item is finally formed.

[0015] Furthermore, the original data items in the data list include Chinese and English names, and the data list is in Excel format. If the Chinese and English names of the original data items are missing, the process will skip to the next original data item.

[0016] Furthermore, the data element library stores the Chinese and English names of data elements; the data element association library stores the Chinese and English names of the original data items, as well as the Chinese and English names of the data elements associated with the original data items.

[0017] Furthermore, the data element association library can be customized to add original data items and data element association data that do not exist in the current data element association library, so as to accurately associate the original data with the data elements in the future.

[0018] Furthermore, the correlation between the original data items and data elements will be statistically analyzed, and a prompt will be given for correlations that are too low, making it easier for manual verification of the accuracy of the correlation; if the correlation is incorrect, the correlation between this original data item and the correct data element will be added to the data element correlation library.

[0019] Furthermore, the specific method for step 3) is as follows:

[0020] 3.1) Extract all features of non-Chinese characters contained in the Chinese name of the original data item, including device number, device height, and serial number information;

[0021] 3.2) Examine the original data items according to the custom feature library, identify the custom features they contain, and convert them into corresponding feature values; the custom library stores the custom feature content and feature values.

[0022] 3.3) Verify the validity of all features extracted from the original data items and associated from the custom feature library one by one. The verification standard is to remove the special characters contained therein and ensure that the features do not contain any other characters except for English letters, numbers and underscores.

[0023] 3.4) Compare the above features with the English names of the associated data elements. If the data element already contains the feature, then ignore the feature.

[0024] Furthermore, the specific method for step 4) is as follows:

[0025] 4.1) Concatenate the available features as suffixes after the data element that best matches the original data item to generate the English and Chinese names of the data item, where different features are connected by "_".

[0026] 4.2) Verify the uniqueness of the English name of the new data item, distinguish duplicate new data items by random enumeration, and finally map the original data item to the new data item.

[0027] In summary, the present invention has the following beneficial effects:

[0028] The system obtains raw data items and a data element library from the business system. It then associates the raw data items and data elements using a keyword broad association algorithm. Finally, it extracts all features from the raw data items and the custom feature library using regular expressions. After verification, the features are concatenated with the associated data elements and duplicates are removed. This process effectively maps the raw data items to new data items, thus improving the speed and quality of data mapping in the data governance process. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method for automatically associating original data items with data elements and mapping them to new data items, as described in this invention.

[0030] Figure 2 This is a schematic diagram illustrating the process of associating original data items and data elements as described in this invention.

[0031] Figure 3 This is a schematic diagram of the feature extraction process described in this invention. Detailed Implementation

[0032] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to the figures and specific embodiments.

[0033] like Figures 1 to 3 As shown, the present invention proposes a method for automatically associating original data items with data elements and mapping them to new data items, comprising the following steps:

[0034] 1) Obtain the list of data to be processed from the business system;

[0035] 2) Using a broad keyword matching algorithm, the original data items in the data list are matched with the data element library and the data element association library respectively to obtain the data element with the highest matching degree with the original data item;

[0036] 3) A customizable feature extraction algorithm based on regular expressions is used to extract all features from the original data items and the custom feature library. After verifying the rationality of all features, usable features are obtained.

[0037] 4) The data element with the highest matching degree to the original data item and the available features are concatenated to generate a new data item. After verifying the uniqueness of the new data item, the mapped data item is finally formed.

[0038] Example

[0039] Step S10: Obtain the raw data items that need to be processed from the business system.

[0040] In this embodiment of the invention, a list of data to be mapped is obtained from the business system. The data list should include the Chinese and English names of the original data items, and the data list should be in Excel format. If the Chinese and English names of the original data items are missing, the process is skipped and the next item is continued.

[0041] Step S20: Based on the keyword broad matching algorithm, associate the original data items and data elements.

[0042] In this embodiment of the invention, a data element library and a pre-associated original data item and data element association library are obtained. A broad keyword matching algorithm is used to associate the original data items and data elements, obtaining the data element with the highest association degree. The data element library stores the Chinese and English names of the data elements, while the association library stores the Chinese and English names of the original data items and the Chinese and English names of the associated data elements.

[0043] In this step, the original data items and data element association libraries can be customized and added simultaneously. That is, original data items and data element association data that do not exist in the current association library can be added for accurate association of original data and data elements in the future.

[0044] In this step, the correlation between the original data items and the data source is statistically analyzed. Low correlations are flagged to facilitate manual verification of the accuracy of the correlations. If an incorrect correlation is found, the correlation between the original data item and the correct data element can be added to the correlation database.

[0045] Step S30: Extract the original data items and all features from the custom feature library based on regular expressions.

[0046] In this embodiment of the invention, all features of non-Chinese characters contained in the original data item's Chinese name are extracted, including but not limited to information such as device number, device height, and serial number.

[0047] In this step, the original data items are examined according to the custom feature library, the included custom features are identified, and they are converted into corresponding feature values. The custom library stores the feature content and feature values.

[0048] In this step, the legality of all features extracted from the original data and associated from the custom feature library is verified one by one. The verification standard is to remove special characters including but not limited to "*", "#", "+", "-", ".", "()", "~", "、", "\", " / ", "|", "[]", etc., to ensure that the features do not contain any other characters except English letters, numbers and underscores.

[0049] In this step, the verified features are compared with the English names of their associated data elements. If a feature is already contained in the data element, it is ignored.

[0050] Step S40: Map new data items based on the associated data elements and available features.

[0051] In this embodiment of the invention, available features are appended to data elements in the form of suffixes to generate the English and Chinese names of data items, wherein different features are connected by "_".

[0052] In this step, the uniqueness of the English name of the new data item is verified, and duplicate new data items are distinguished by random enumeration, such as adcdef_1, adcdef_2, adcdef_3, etc. Finally, the original data items in the business system are mapped to the new data items.

[0053] In this document, the terms "upper," "lower," "front," "back," "left," "right," "top," "bottom," "inner," "outer," "vertical," and "horizontal," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only used for the clarity of expressing the technical solution and for the convenience of description, and therefore should not be construed as limiting the present invention.

[0054] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0055] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatically associating and mapping original data items with data elements into new data items, characterized by, The method comprises the following steps: 1) obtaining a data list to be processed from a business system; 2) using a keyword broad match algorithm to perform keyword matching between original data items in the data list and a data element library and a data element association library respectively, to obtain data elements with the highest matching degree with the original data items; 3) using a regular expression-based customizable feature extraction algorithm to extract all features of the original data items and a customizable feature library, and after reasonable verification of all features, obtaining available features; 4) splicing the data elements with the highest matching degree with the original data items and the available features to generate new data items, and after verifying the uniqueness of the new data items, finally forming mapped data items.

2. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The original data items of the data list contain Chinese names and English names, and the data list is in excel format. If the Chinese name and English name of the original data item are missing, go to the next original data item.

3. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The data element library stores the Chinese name and English name of the data element; the data element association library stores the Chinese name, English name of the original data item, and the Chinese name and English name of the data element associated with the original data item.

4. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The data element association library can be customized to add original data items and data element association data that do not exist in the current data element association library for subsequent accurate association of original data and data elements.

5. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The association degree of the original data items and the data elements will be counted, and a prompt will be given for the association relationship with too low association degree, to facilitate manual confirmation of the accuracy of the association; if the association is incorrect, the association relationship between the original data item and the correct data element will be added to the data element association library.

6. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The specific method of step 3) is as follows: 3.1) Extract all features of non-Chinese characters contained in the Chinese name of the original data item, including device number, device height, and serial number information; 3.2) Check the original data item according to the customizable feature library, identify the customizable features contained therein and convert them to corresponding feature values; wherein the customizable library stores customizable feature content and feature values; 3.3) Check the legality of all features extracted from the original data item and associated from the customizable feature library one by one, and the standard for verification is to remove special characters contained, to ensure that the features do not contain other characters except English letters, numbers, and underscores; 3.4) Compare the above features with the English name of the associated data element, and if the data element already contains it, ignore this feature.

7. The method for automatically associating and mapping original data items with data elements as new data items of claim 1, wherein, The specific method of step 4) is as follows: 4.1) Splice the available features in the form of a suffix after the data element with the highest matching degree with the original data item to generate the English name of the data item, wherein different features are connected by "_""; 4.2) Verify the uniqueness of the English name of the new data item, distinguish the repeated new data items according to the random enumeration method, and finally map the original data items to new data items.

Images