Data identification method and device, equipment, storage medium and program product

Abstract

The invention discloses a data identification method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the data preprocessing of main data of different data sources collected in advance, and obtaining a plurality of data sets; wherein each data set comprises data items of a column corresponding to one data element; performing similarity matching on the data items in each data set, and determining a plurality of candidate main data items; verifying and evaluating the plurality of candidate main data items according to a pre-constructed identification and evaluation matrix, and determining a target main data item; according to the method, the problems of high cost, low efficiency, low accuracy and low universality of existing manual main data identification can be solved, the identification cost of the main data is effectively reduced, and the efficiency and the universality of the main data identification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data identification method, device, equipment, storage medium and program product. BACKGROUND

[0002] The master data is the data that is shared by multiple departments or systems, core, high value and relatively static, has the characteristics of uniqueness, sharing, stability, periodicity, is the data standard for enterprise basic data, business operation and decision analysis. With the continuous development of business and the continuous accumulation of data, the complexity of data is increasing, involving cross-business, cross-department, cross-process, cross-system and cross-technology. The identification and management of master data is the only way for massive data assets to play their due value, and is the key to the success of the digital transformation of enterprises. It is very important for enterprises to explore data value and improve core competitiveness.

[0003] At present, the existing identification of master data is mainly based on relevant standards and specifications and business personnel experience, and the master data is found and identified by manual analysis of data analysts. However, due to the different industries and fields of data, the business scope involved is also different, resulting in differences in the definition and rules of master data in different industries and fields. Therefore, the existing manual master data identification method has the problems of high cost, low efficiency, low accuracy and low universality in the process of master data identification. SUMMARY

[0004] In view of the problems in the prior art, the embodiments of the present application provide a data identification method, device, equipment, storage medium and program product, which can solve the problems of high cost, low efficiency, low accuracy and low universality of the existing manual master data identification, effectively reduce the identification cost of master data, and improve the efficiency and universality of master data identification.

[0005] In a first aspect, the embodiments of the present application provide a data identification method, comprising: data preprocessing of the master data of different data sources collected in advance to obtain a plurality of data sets; wherein each data set includes a data item corresponding to a column of data elements; Similarity matching is performed on the data items in each data set to determine a plurality of candidate master data items; According to the pre-constructed identification evaluation matrix, the plurality of candidate master data items are verified and evaluated to determine the target master data item.

[0006] As an improvement of the above-mentioned scheme, the data preprocessing of the master data of different data sources collected in advance to obtain a plurality of data sets comprises: Preprocess the main data to obtain a plurality of data items to be identified; wherein the preprocessing includes data cleaning and arrangement processing, data standardization processing; Divide a plurality of the data items according to data elements to obtain a plurality of data sets; wherein one data element corresponds to the division of column data items into one data set.

[0007] As an improvement of the above scheme, the similarity matching of the data items in each data set determines a plurality of candidate main data items, comprising: For each data set, length matching is performed on all data items in the data set to determine the first similarity of the data set; Content matching is performed on all data items in the data set to determine the second similarity of the data set; Rule matching is performed on all data items in the data set to determine the third similarity of the data set; The first similarity, the second similarity, and the third similarity are weighted and summed to obtain the data similarity of the data set; Select the data items in the data set with a data similarity greater than or equal to a preset similarity threshold as candidate main data items.

[0008] As an improvement of the above scheme, the length matching of all data items in the data set to determine the first similarity of the data set comprises: Length comparison is performed on the strings corresponding to all data items in the data set to obtain a first number; wherein the first number is the number of data items corresponding to strings of the same length; According to the proportion of the first number and the total number of data items in the data set, the first similarity is determined.

[0009] As an improvement of the above scheme, the content matching of all data items in the data set to determine the second similarity of the data set comprises: Content editing processing is performed on the strings corresponding to any two data items in the data set to calculate the edit distance between the strings corresponding to any two data items; wherein the content editing processing includes character deletion, character insertion, and character replacement; According to the edit distance of the strings corresponding to any two data items in the data set, the second similarity is determined.

[0010] As an improvement of the above scheme, the rule matching of all data items in the data set to determine the third similarity of the data set comprises: According to a preset regular expression, a rule matching is performed on strings corresponding to all data items in the data set, and a second quantity is determined; wherein the second quantity is the quantity of data items corresponding to strings conforming to the regular expression; According to the proportion of the second quantity and the total quantity of data items in the data set, the third similarity is determined.

[0011] As an improvement of the above-mentioned scheme, the candidate master data items are checked and evaluated according to the pre-constructed identification evaluation matrix, and the target master data item is determined, comprising: According to the first description of the table and the second description of the metadata of each candidate master data item, the data characteristics of each candidate master data item are scored according to the data bloodline link and the identification evaluation matrix, and the initial score of each candidate master data item on each data characteristic is determined; the data characteristics include: business value characteristics, data uniqueness characteristics, data sharing characteristics, data stability characteristics and data periodicity characteristics; For each candidate master data item, the initial score of the candidate master data item on each data characteristic is weighted and summed to obtain the check evaluation score of the candidate master data item; The candidate master data item with a check evaluation score greater than or equal to a preset evaluation score threshold is selected as the target master data item.

[0012] In a second aspect, an embodiment of the present application provides a data recognition device, comprising: A data preprocessing module is configured to perform data preprocessing on the master data of different data sources collected in advance to obtain a plurality of data sets; wherein each data set includes data items in a data element corresponding column; A similarity matching module is configured to perform similarity matching on the data items in each data set to determine a plurality of candidate master data items; A check evaluation module is configured to perform check evaluation on the plurality of candidate master data items according to a pre-constructed identification evaluation matrix to determine a target master data item.

[0013] In a third aspect, an embodiment of the present application provides a data recognition device, comprising: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the data recognition method of any one of the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein when the computer program runs, the computer readable storage medium controls the device where the computer readable storage medium is located to execute the data recognition method of any one of the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the data identification method according to any one of the first aspect.

[0016] Compared with the prior art, the data identification method, device, equipment, storage medium and program product provided by the embodiment of the present application can obtain a plurality of data sets by performing data preprocessing on the main data of different data sources collected in advance, wherein each data set comprises data items corresponding to a column of data elements; the data items in each data set are subjected to similarity matching to determine a plurality of candidate main data items; and the plurality of candidate main data items are subjected to verification and evaluation according to a pre-constructed identification evaluation matrix to determine a target main data item. The embodiment of the present application can solve the problems of high cost, low efficiency, low accuracy and low universality of manual main data identification, effectively reduce the identification cost of main data, and improve the efficiency and universality of main data identification. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the present application, the drawings used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a flowchart of a data identification method provided by an embodiment of the present application; Figure 2 is a flowchart of another data identification method provided by an embodiment of the present application; Figure 3 is a service value evaluation flowchart provided by an embodiment of the present application; Figure 4 is a data sharing evaluation flowchart provided by an embodiment of the present application; Figure 5 is a data uniqueness evaluation flowchart provided by an embodiment of the present application; Figure 6 is a data stability evaluation flowchart provided by an embodiment of the present application; Figure 7 is a data periodicity evaluation flowchart provided by an embodiment of the present application; Figure 8 is a structural block diagram of a data identification device provided by an embodiment of the present application; Figure 9This is a structural block diagram of a data recognition device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It is understood that the various numerical designations used in the embodiments of this invention are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0021] In embodiments of the invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element. The term "a plurality or several" refers to two or more.

[0022] Please see Figure 1 , Figure 1 This is a flowchart of a data recognition method provided by an embodiment of the present invention. The data recognition method specifically includes: S11: Perform data preprocessing on the master data collected from different data sources in advance to obtain multiple datasets; wherein each dataset includes a data item corresponding to a column of data elements; The master data includes data from different data sources, such as basic data from different sources within an enterprise's various business systems (i.e., different data sources), including customer data, product data, supplier data, and employee data. These data may have the same or different formats. In this embodiment of the invention, master data from different data sources can be collected using pre-selected data collection tools. For example, Extract Transform Load (ETL) tools can be used to uniformly collect, aggregate, and store master data from different sources within the enterprise's various business systems. It should be noted that this embodiment of the invention does not specifically limit the content, format, or data collection tools used for the master data.

[0023] It's understandable that a data item is an instantiation of a data element in a specific business scenario (i.e., the concrete use of a standard template). A master data item is a special subset of a data item; it refers to core data that remains unchanged within an enterprise and is shared across systems during business processes, and must adhere to unified standards and rules, such as product codes, names, and ID numbers. A data element is a general unit (i.e., a standardized template) used to describe the common attributes of data, providing a standardized basis for data items and master data items. The data items in the column corresponding to a data element can be understood as master data possessing the common attributes described by that data element.

[0024] In this embodiment of the invention, after collecting master data from different data sources, the data is first preprocessed, and then the data is organized according to data elements. Master data with the same general attributes are organized as data items and placed under the corresponding data element column. The data items of a column corresponding to a data element are divided into datasets, and master data items are identified at the dataset level.

[0025] S12: Perform similarity matching on each data item in the dataset to determine multiple candidate master data items; Among them, candidate master data items refer to multiple data items in the corresponding dataset that are matched by similarity.

[0026] S13: Based on the pre-constructed identification and evaluation matrix, verify and evaluate the multiple candidate master data items to determine the target master data item.

[0027] The identification and evaluation matrix uses the business value, sharing, uniqueness, stability, and periodicity of master data as evaluation indicators. It includes rules for comprehensively evaluating the five dimensions of business value, sharing, uniqueness, stability, and periodicity of master data items (such as the aforementioned candidate master data items).

[0028] In this embodiment of the invention, multiple candidate master data items are verified and evaluated by a pre-constructed identification and evaluation matrix to determine the verification and evaluation score of each candidate master data item, and the candidate master data item whose verification and evaluation score exceeds the preset evaluation score threshold is selected as the target master data item.

[0029] This invention employs a method combining similarity matching with an evaluation matrix based on multi-dimensional evaluation metrics for automatic identification of master data items. The overall process is as follows: Figure 2 As shown, the embodiments of the present invention can solve the problems of high cost, low efficiency, low accuracy and low universality of existing manual master data identification, and can effectively reduce the cost of master data identification and improve the efficiency and universality of master data identification.

[0030] In one optional embodiment, the pre-collected master data from different data sources is preprocessed to obtain multiple datasets, including: The master data is preprocessed to obtain multiple data items to be identified; wherein, the preprocessing includes data cleaning and sorting, and data standardization. Multiple data items are divided according to data elements to obtain multiple datasets; wherein, the data items in a column corresponding to one data element are divided into one dataset.

[0031] The preprocessing of master data includes, but is not limited to, data cleaning, data organization, and data standardization. Cleaning and organizing the collected master data to remove redundancy and missing values ​​ensures data integrity and accuracy, preventing omissions or errors. Before identifying master data items, data elements need to be standardized: this includes unifying naming rules, format specifications, and data types. Then, master data with the same common attributes are organized into the corresponding data element columns. The data items in these columns are then standardized according to the naming rules, format specifications, and data types of the corresponding data elements. This ensures the consistency and comparability of master data items, facilitating subsequent identification and verification.

[0032] After completing the preprocessing of the master data, the data items in the column corresponding to a data element are divided into datasets, and master data item identification is performed at the dataset level. This embodiment of the invention improves data reliability by preprocessing the master data, and improves the accuracy and efficiency of master data item identification by performing master data item identification at the data element level.

[0033] Furthermore, data elements can be pre-cleaned to remove redundant data elements and extract valid ones. For example, data elements related to technical and management metadata, such as creation date, update date, auto-incrementing primary keys, and operators, can be deleted. This can remove redundant fields in the master data and improve the identification efficiency of master data items.

[0034] Furthermore, fuzzy matching can be performed on data elements in advance, and the data items in the corresponding columns of the fuzzy-matched data elements can be grouped under the same column to form a dataset. For example, based on the metadata information of the data elements, fuzzy matching can be performed on the name and description (i.e., attribute description) of the data elements to find data elements with similar names and descriptions, which are then used as fuzzy-matched data elements. For example, data elements with names such as id_card, cardNo, etc., and descriptions such as ID card number, document number, etc., can be considered as fuzzy-matched data elements, all of which describe ID card numbers. Through fuzzy matching of data elements, the accuracy of master data item identification can be effectively improved.

[0035] It should be noted that the embodiments of the present invention do not specifically limit the fuzzy matching method of data elements. For example, if the character similarity or keyword overlap (i.e. the proportion of overlapping words) of the names exceeds the corresponding threshold, it is determined that the names are similar; if the semantic similarity of the description is calculated based on TF-IDF, and the semantic similarity exceeds the corresponding threshold, it is determined that the descriptions are similar.

[0036] In an optional embodiment, S12: performing similarity matching on data items in each of the datasets to determine multiple candidate master data items, including: For each dataset, length matching is performed on all data items in the dataset to determine the first similarity of the dataset; Specifically, the lengths of the strings corresponding to all data items in the dataset are compared to obtain a first quantity; wherein, the first quantity is the number of data items corresponding to strings of the same length; The first similarity is determined based on the ratio of the first quantity to the total number of data items in the dataset.

[0037] For example, the length of the strings of all data items in the dataset is identified. Since the length of the main data is generally fixed, such as the device serial number, ID card number, mobile phone number, and unified social credit code number, let's assume that the lengths of the strings corresponding to two data items are m and n respectively. If m=n, then the lengths of the strings corresponding to the two data items are the same. If m is not equal to n, then the lengths of the strings corresponding to the two data items are different. Then, we move on to the next data item for comparison. We traverse all data items for length comparison and finally calculate the proportion of data items with the same string length in the dataset. This proportion is called the uniform length ratio and is used as the first similarity.

[0038] For example, for all data items in the column with the data item name SN, a set T = {"FTXBB231", "FTWOI557", "FTCEE908", "FTCMM467", "FCVXX769"} is formed; the lengths of the strings in set T are compared using len(1, L). max ), where 1 represents the minimum length, L max This indicates the maximum length, meaning the length of any string cannot be less than 1 and cannot be greater than L. max By comparing the lengths, it was found that all strings in set T are 8 characters long, and the uniform length accounts for 100%. Therefore, the first similarity of the length of set T is 100%.

[0039] Perform content matching on all data items in the dataset to determine the second similarity of the dataset; Specifically, the content editing process is performed on the strings corresponding to any two data items in the dataset, and the edit distance between the strings corresponding to any two data items is calculated; wherein, the content editing process includes character deletion, character insertion, and character replacement; The second similarity is determined based on the edit distance between the strings corresponding to any two data items in the dataset.

[0040] For example, the Levenshtein similarity algorithm is used to perform replacement matching for all data items in the dataset, and the edit distance is calculated. The Levenshtein similarity algorithm evaluates the degree of difference between strings by calculating the minimum number of operations required to transform string A into string B. It is used to determine whether the values ​​in a column have a pattern. A two-dimensional matrix is ​​constructed to record the differences between strings, and the edit distance is calculated step by step. The second similarity is obtained based on the proportion of the edit distance of data items in the dataset within a preset distance range.

[0041] The preset distance range is determined based on the maximum length of the strings in the data items in the dataset. For example, the lower limit of the preset distance range is 0, which means that the two strings are the same and do not need to be edited, while the upper limit is the maximum length of the strings in the data items in the dataset, which means the minimum number of edits required to completely replace one string with another, or to insert all characters from an empty string.

[0042] Given that the values ​​in the master data are generated according to certain rules and are unique, a high matching degree after replacement indicates a high probability. Generally, assuming there are two strings A and B with lengths m and n respectively, a matrix D of size m+1 x n+1 is used to record intermediate calculation results. Initially, the first row and first column of matrix D represent the index positions of the empty string and the other string, respectively. For the i-th row and j-th column of matrix D, D(i, 0) and D(0, j) represent the edit distance between the first i characters of string A and the first j characters of string B, respectively. The other cells of matrix D are calculated step-by-step according to the following formula: If A[i]==B[j]:D(i,j)=D(i-1,j-1); Else: D(i,j)=min(D(i-1,j)+1, D(i,j-1)+1, D(i-1,j-1)+1).

[0043] If the i-th character of string A is the same as the j-th character of string B, then the edit distance depends on the previous edit distance D(i-1, j-1); otherwise, the edit distance depends on three possible operations: deletion D(i-1, j)+1, insertion D(i, j-1)+1, and replacement D(i-1, j-1)+1. The last cell D(m, n) of matrix D represents the edit distance between strings A and B. The shorter the edit distance, the more similar the strings are, indicating a relatively good match and a higher probability of successful matching.

[0044] For example, the intermediate calculation results can be recorded by multiplying the lengths of two strings in set T to obtain matrix D. For instance, if A = FTXBB231 and B = FTWOI557, the length of string A (lenA) is 8 and the length of string B (lenB) is 8. This yields the edit distance of matrix D (8, 8). The smaller the edit distance, the greater the similarity. Similarly, if all strings in set T fall within the range of (0, 8) in matrix D, the second similarity of the data items in set T is 100%.

[0045] Perform rule matching on all data items in the dataset to determine the third similarity of the dataset; Based on a preset regular expression, the strings corresponding to all data items in the dataset are matched according to the rules to determine a second quantity; wherein, the second quantity is the number of data items corresponding to strings that conform to the regular expression; The third similarity is determined based on the ratio of the second quantity to the total number of data items in the dataset.

[0046] For example, for all data items in the dataset, a general regular expression is used for rule matching to find if there are any common patterns, i.e., whether the general regular expression can match all strings. If the general regular expression can satisfy the composition verification methods of most strings, then the strings are considered to have a pattern, and the similarity probability is higher.

[0047] For example, if we use a general regular expression to match all strings in set T, and all strings in set T have the characteristic that the first 5 characters are letters and the last 3 are positive integers, we can find a pattern through the general regular expression. The strings in set T conform to the data value pattern, that is, all strings in set T conform to the pattern defined by the general regular expression, and the third similarity of set T on the rule is 100%.

[0048] The data similarity of the dataset is obtained by weighted summation of the first similarity, the second similarity, and the third similarity. Data items in the dataset whose data similarity is greater than or equal to a preset similarity threshold are selected as candidate master data items.

[0049] For example, after matching the length, content, and rules of all strings in the column containing the data item, the calculated first similarity, first similarity, and third similarity are weighted and summed using pre-set length weight, content weight, and rule weight (e.g., length weight = 30%, content weight = 30%, rule weight = 40%) to obtain the data similarity of the dataset. The result is calculated as: Data Similarity = First Similarity × 30% + First Similarity × 30% + Third Similarity × 40%. A higher data similarity increases the probability of a data item being identified as a primary data item. If the data similarity is below a preset similarity threshold, such as 70, the data item in the dataset is considered unlikely to be a primary data item; otherwise, it is considered a candidate primary data item.

[0050] For example, if the data similarity of set T is 100×30%+100×30%+100×40%=100>70, then the data item SN is determined to be the main data item.

[0051] In this embodiment of the invention, the main data item is identified by using a similarity algorithm to comprehensively match the strings corresponding to the data items in the dataset (i.e., all data items in the column corresponding to the data element) according to length, content, and format rules. This eliminates the need to re-specify the main data identification logic for specific business logic and avoids redeveloping the model due to changes in business logic, thus saving development costs and improving the coverage of main data item identification.

[0052] In an optional embodiment, S13: Based on a pre-constructed identification and evaluation matrix, the multiple candidate master data items are verified and evaluated to determine the target master data item, including: Based on the first description of the table containing each candidate master data item and the second description of the metadata, the data characteristics of each candidate master data item are scored according to the data lineage and the identification evaluation matrix to determine the initial score of each candidate master data item on each data characteristic; the data characteristics include: business value characteristics, data uniqueness characteristics, data sharing characteristics, data stability characteristics, and data periodicity characteristics; For each candidate master data item, the initial scores of the candidate master data item on each data feature are weighted and summed to obtain the verification evaluation score of the candidate master data item; Candidate master data items whose verification evaluation scores are greater than or equal to a preset evaluation score threshold are selected as target master data items.

[0053] Based on the evaluation matrix, this invention combines metadata information (including the first description of the table where the candidate master data item is located and the second description of the metadata) and data lineage links to verify and score the evaluated candidate master data items, and finally determines whether the candidate master data item is a core master data item (i.e., the target master data item).

[0054] Data lineage verification: Each candidate master data item's strings are deduplicated. Each candidate master data item contains multiple strings. The deduplicated strings in the candidate master data item are compared with the reference data indicated by data elements in the reference data dictionary corresponding to that candidate master data item, found based on the data lineage. If a match is found, the candidate master data item corresponding to the matching string is determined as the reference data, and the matching string is deleted, resulting in the verified master data item. In essence, data lineage refers to the visual representation and structured description of the relationships between data nodes throughout the entire data flow from the original data source to the final data product (such as reports, models, and applications).

[0055] For example, based on the identified candidate master data items, validation is performed according to the data lineage (field-level or table-level). Because master data is unique, stable, and shareable, it remains unchanged across different business systems throughout the entire business process. Simultaneously, since reference data also possesses similar properties, to distinguish between master data and reference data, candidate master data items are randomly selected and matched against the reference data dictionary during validation. Before matching against the reference data dictionary table, duplicate strings in the identified candidate master data items are removed, and then strings from the identified master data items are randomly selected for matching against the reference data dictionary. To improve matching efficiency, the corresponding data element in the reference data dictionary can be found based on the candidate master data item's metadata information or data lineage, and the string is matched against it. If the extracted string completely matches a data element in the reference data dictionary table, it is considered reference data and deleted.

[0056] In this embodiment of the invention, since master data is unique, stable, and shareable, it remains unchanged across different systems throughout the entire business process. Meanwhile, since reference data also possesses similar properties, to distinguish between master data and reference data, data is randomly sampled and matched against a reference data dictionary during verification. Before matching, the strings in the identified master data items are deduplicated, resulting in significantly fewer duplicate types, thus improving the accuracy of identifying reference data (due to the fixed and limited types of reference data). Then, strings from the identified master data items are randomly sampled and matched against the reference data dictionary. If the sampled string completely matches the reference data dictionary, it is considered reference data. After deleting all reference data from the candidate master data items, the verified master data items are obtained.

[0057] The evaluation matrix indicates the scoring range and weight of several evaluation indicators, including business value, data uniqueness, data sharing, data stability, and data periodicity, as well as the scoring logic. For example, considering that master data is the basic information reflecting the status of core business entities and meeting the needs of cross-departmental and cross-business collaboration, an evaluation matrix is ​​designed based on the characteristics of master data's business value, data uniqueness, data sharing, data stability, and data periodicity. The evaluation matrix is ​​shown in the table below:

[0058] The scoring logic for each evaluation indicator is described in the scoring process of the evaluation matrix verification below.

[0059] Assessment Matrix Verification: In the telecommunications industry, master data is critical data spanning customers, products, resources, channels, materials, and other business operations. It is used across departments, processes, and systems, possessing characteristics of uniqueness, sharing, stability, and periodicity. For example, a key master data item in a product's data is the SN code, which is data spanning supply chain systems, CRM systems, installation and maintenance systems, etc. In the supply chain system, the SN code is a unique identifier for products and equipment, consistent with the actual items in the inventory. In the CRM system, it is bound to the specific products and equipment ordered by the customer, i.e., bound to the SN code. In the installation and maintenance system, the specific SN code must be confirmed before installing products and equipment for the customer. Through this SN code master data item, control is implemented throughout the entire process from equipment receipt to installation and consumption, achieving product management and ensuring consistency between inventory and records. The specific verification process combining the assessment matrix and data lineage is as follows: Business Value: Data describes a company's core business and is its most valuable asset. For example... Figure 3 As shown, the business value scoring process is as follows: Preliminary preparation: Define the key business rule dictionary (i.e., reference data dictionary), including standard field names and standardized keywords (i.e., reference data) used to describe the business. Then, evaluate the identified candidate master data items as follows: Step a: Determine whether the name of the candidate master data item matches, that is, whether it contains the keywords indicated by the key business rule dictionary (found based on data lineage); Step b: If yes, perform business value scoring, increment the initial score of the candidate master data item in business value by 1 point, and proceed to step e; Step c: If not, determine whether the candidate master data item belongs to the core business based on the first description of the table where the candidate master data item is located and the second description of the metadata. The first description refers to the basic information of the data table to which the master data item belongs, including the name, purpose, and types of fields contained in the data table, which is used to clarify the position of the master data item at the data storage level and the overall situation of the related tables. The second description refers to the description of the metadata of the master data itself. Metadata is data that describes data, including the definition, data type, length, value range, business rules, and other information of the master data item.

[0060] For example, it can be determined whether the first description matches the keywords in the key business rules dictionary, and whether the second description conforms to the relevant description of the master data item defined by the core business (such as the definition, data type, length, value range, and business rules of the master data item). If the first description matches the keywords in the key business rules dictionary, and the second description conforms to the relevant description of the master data item defined by the core business, then the candidate master data item belongs to the core business; otherwise, the candidate master data item belongs to the non-core business.

[0061] Step d: If the candidate master data item belongs to the core business, perform business value scoring, increase the initial score of the candidate master data item in business value by 1 point, and jump to step e; if the candidate master data item belongs to the non-core business, end the business value scoring, and at this time the initial score of the candidate master data item in business value = 1 point.

[0062] Step e: Based on data lineage, trace the source of the candidate master data item to determine whether the source of the candidate master data item can be traced, i.e., the data source from which the candidate master data item comes; Step f: If the source can be identified, perform business value scoring. The initial score of the candidate master data item in business value is increased by 2 points, and the process jumps to step h. If the source cannot be identified, end the business value scoring. At this time, the initial score of the candidate master data item in business value is 1 point.

[0063] Step h: Based on the data lineage, perform verification at the source end, such as verifying whether the candidate master data item is consistent with the data recorded at the source end; if so, perform business value scoring, and the initial score of the candidate master data item in business value is increased by 3 points, at which point the initial score of the candidate master data item in business value = 1 + 2 + 3 = 6 points; if not, perform business value scoring, and the initial score of the candidate master data item in business value is increased by 2 points, at which point the initial score of the candidate master data item in business value = 1 + 2 + 2 = 5 points.

[0064] Data sharing: This typically refers to a data entity shared by multiple departments, businesses, and platforms within a group or enterprise, and currently or in the future, it has multiple data users. For example... Figure 4 As shown, the specific scoring process for data sharing is as follows: Step a: Perform a link determination on the identified candidate master data item to determine whether the candidate master data item can be completely found in the data lineage link; if not, end the scoring, and the initial score of the candidate master data item in data sharing = 0 points; if yes, proceed to step b. Step b: The candidate master data item can be found completely in the data lineage, that is, the candidate master data item can be found in multiple data tables based on the data lineage. The business value score is calculated, and the initial score of the candidate master data item in data sharing is increased by 1 point. Step c; Step c: Verify the metadata information of the table containing the candidate master data item to determine whether the metadata information fully reflects all business systems; if not, end the scoring, and at this time, the initial score of the candidate master data item in data sharing = 1 point; if yes, proceed to step d. Step d: Based on the number of business systems involved in the candidate master data item, score the master data sharing of the candidate master data item; for example, if the number of business systems is greater than the first value (a natural number greater than 1), the initial score of the candidate master data item in data sharing is increased by 2 points, and the initial score of the candidate master data item in data sharing is 1+2=3 points; if the number of business systems is greater than the second value (a natural number greater than the first value and greater than 1), the initial score of the candidate master data item in data sharing is increased by 3 points, and the initial score of the candidate master data item in data sharing is 1+3=4 points.

[0065] Data uniqueness: Within an enterprise, the same master data must have a unique identifier. This only checks the uniqueness of master data across different tables for the same master data item; the uniqueness of master data values ​​within the same column is already determined when identifying the master data item. For example... Figure 5 As shown, the specific scoring process for data uniqueness is as follows: Step a: For the identified candidate master data items, randomly select the data value (i.e., the string mentioned above) of the candidate master data item and perform link judgment. That is, find the data table where the candidate master data item is located based on the data lineage link, and match the data value of the same candidate master data item in different data tables. If there is no match, it means that the data value of the candidate master data item was not found in the same candidate master data item in other data tables, and the scoring ends. At this time, the initial score of the candidate master data item in data uniqueness is 0 points. If there is a match, the initial score of the candidate master data item in data uniqueness is increased by 1 point, and the process jumps to step b. Step b: Verify the uniqueness of the data value of the candidate master data item by performing similarity matching with the data values ​​of the same candidate master data items found in multiple data tables. For example, the average similarity between the data values ​​of the same candidate master data items is used as the matching degree. If the matching degree is greater than the first similarity threshold, the initial score of the candidate master data item in data uniqueness is increased by 2 points. At this time, the initial score of the candidate master data item in data uniqueness is 1 + 2 = 3 points. If the matching degree is greater than the second similarity threshold (greater than the first similarity threshold), the initial score of the candidate master data item in data uniqueness is increased by 3 points. At this time, the initial score of the candidate master data item in data uniqueness is 1 + 3 = 4 points.

[0066] Data stability: Master data is stable, generally only requiring additions or deactivations. Compared to transaction data, master data is relatively stable with a lower frequency of change. For example... Figure 6 As shown, the specific scoring process for data stability is as follows: Step a: For the identified candidate master data items, randomly select the data value (i.e., the string mentioned above) of the candidate master data item and perform link judgment. That is, find the data table where the candidate master data item is located based on the data lineage link, and match the data value of the same candidate master data item in different data tables. If there is no match, it means that the data value of the candidate master data item was not found in the same candidate master data item in other data tables, and the scoring ends. At this time, the initial score of the candidate master data item in data stability = 0 points. If there is a match, the initial score of the candidate master data item in data stability is increased by 1 point, and the process jumps to step b. Step b: Confirm the stability of the data value of the candidate master data item, that is, match the link lengths corresponding to multiple data tables where the candidate master data item is located. For example, the proportion of the number of data tables where the candidate master data item is located with the same link length is used as the matching degree. If the matching degree is greater than the first proportion threshold, the initial score of the candidate master data item in data stability is increased by 2 points. At this time, the initial score of the candidate master data item in data stability = 1 + 2 = 3 points. If the matching degree is greater than the second proportion threshold (greater than the first proportion threshold), the initial score of the candidate master data item in data stability is increased by 3 points. At this time, the initial score of the candidate master data item in data stability = 1 + 3 = 4 points.

[0067] Data periodicity: Master data is typically used over a long period of time, exhibiting a long lifespan. For example... Figure 7 As shown, the specific evaluation process is as follows: Step a: For the identified candidate master data items, randomly select the data value (i.e., the string mentioned above) of the candidate master data item and perform link judgment. That is, find the data table where the candidate master data item is located based on the data lineage link, and match the data value of the same candidate master data item in different data tables. If there is no match, it means that the data value of the candidate master data item was not found in the same candidate master data item in other data tables, and the scoring ends. At this time, the initial score of the candidate master data item in data periodicity = 0 points. If there is a match, the initial score of the candidate master data item in data periodicity is increased by 1 point, and the process jumps to step b. Step b: Perform periodic verification on the data value of the candidate master data item, that is, determine whether there is periodicity in the data values ​​of the same candidate master data item in multiple data tables containing the candidate master data item. Periodicity refers to the periodic change of data values ​​over time. For example, if the proportion of the data values ​​of the same candidate master data item in multiple data tables containing the candidate master data item that exhibit periodicity is greater than the first proportion threshold, the initial score of the candidate master data item on data periodicity is increased by 2 points. At this time, the initial score of the candidate master data item on data periodicity is 1 + 2 = 3 points. If the proportion of the data values ​​of the same candidate master data item in multiple data tables containing the candidate master data item that exhibit periodicity is greater than the second proportion threshold (greater than the first proportion threshold), the initial score of the candidate master data item on data periodicity is increased by 3 points. At this time, the initial score of the candidate master data item on data periodicity is 1 + 3 = 4 points.

[0068] The initial scores of each candidate master data item in terms of business value, data uniqueness, data sharing, data stability, and data periodicity are weighted and summed to obtain the verification evaluation score of the candidate master data item; finally, the candidate master data items whose verification evaluation scores are greater than or equal to the preset evaluation score threshold are selected as the target master data items.

[0069] Furthermore, the identified target master data items can be stored in a temporary master data item pool, including the master data item name, master data item type, master data item reference length, master data reference rules, whether it is a primary or foreign key, and other related attributes. Then, the results are output: the confirmed target master data items are moved from the temporary master data item pool to the formal master data items, and each master data item is displayed with the support of similar master data management platform software, and the relevant master data item attributes are modified according to the actual business requirements.

[0070] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: (1) A similarity algorithm is adopted, and an identification and evaluation matrix is ​​constructed by combining the business value, data sharing, data uniqueness, data stability, and data periodicity of master data to achieve the purpose of automatic identification of master data items. This eliminates the need to re-specify master data identification logic for specific business logic, thereby reducing manual costs. Through automatic identification of master data items, master data items are efficiently identified, the format of master data items is standardized, and the consistency of information in heterogeneous systems is ensured, reducing the cost of information verification. At the same time, the use of a similarity algorithm to achieve automatic matching and identification, combined with the verification and evaluation of multi-dimensional features of master data, improves the efficiency, accuracy, and universality of master data item identification. It can be applied to matching based on standard data elements in various fields of the entire industry, enhancing standardized management.

[0071] (2) The master data item identification in this embodiment of the invention breaks the limitations of the traditional method that only judges master data and that master data judgment requires sorting out business processes, system divisions, and subject domain divisions. The master data item identification in this embodiment of the invention does not focus on the division of business processes, business systems, and subject domains. It uses a similarity algorithm to identify master data items. It does not need to specify master data identification logic for specific business logic, nor does it need to redevelop the model for changes in business logic, thus saving development costs and improving the coverage of master data item identification. In addition, when the business processes, system divisions, and subject domain divisions are unclear, master data item identification uses a general evaluation matrix to verify master data items, thus comprehensively improving the efficiency and accuracy of master data item identification.

[0072] See Figure 8 , Figure 8 This is a structural block diagram of a data recognition device provided in an embodiment of the present invention. The data recognition device includes: Data preprocessing module 11 is used to preprocess master data from different pre-collected data sources to obtain multiple datasets; wherein each dataset includes a data item corresponding to a column of data elements; Similarity matching module 12 is used to perform similarity matching on each data item in the dataset to determine multiple candidate master data items; The verification and evaluation module 13 is used to verify and evaluate multiple candidate master data items according to a pre-constructed identification and evaluation matrix to determine the target master data item.

[0073] In an optional embodiment, the data preprocessing module 11 includes: A preprocessing unit is used to preprocess the master data to obtain multiple data items to be identified; wherein, the preprocessing includes data cleaning and sorting, and data standardization. A data partitioning unit is used to partition multiple data items according to data elements to obtain multiple datasets; wherein, a data item corresponding to a column of a data element is partitioned into one dataset.

[0074] In an optional embodiment, the similarity matching module 12 includes: The first similarity calculation unit is used to perform length matching on all data items in each dataset to determine the first similarity of the dataset. The second similarity calculation unit is used to perform content matching on all data items in the dataset and determine the second similarity of the dataset. The third similarity calculation unit is used to perform rule matching on all data items in the dataset to determine the third similarity of the dataset. The first weighted summation unit is used to perform a weighted summation of the first similarity, the second similarity, and the third similarity to obtain the data similarity of the dataset; The first master data item determination unit is used to select data items in the dataset whose data similarity is greater than or equal to a preset similarity threshold as candidate master data items.

[0075] In an optional embodiment, the first similarity calculation unit includes: The length comparison subunit is used to compare the lengths of the strings corresponding to all data items in the dataset to obtain a first quantity; wherein, the first quantity is the number of data items corresponding to strings of the same length; The first similarity determination subunit is used to determine the first similarity based on the ratio of the first quantity to the total number of data items in the dataset.

[0076] In one optional embodiment, the second similarity calculation unit includes: The content editing subunit is used to perform content editing processing on the strings corresponding to any two data items in the dataset, and to calculate the edit distance between the strings corresponding to any two data items; wherein, the content editing processing includes character deletion, character insertion and character replacement; The second similarity determination subunit is used to determine the second similarity based on the edit distance between the strings corresponding to any two data items in the dataset.

[0077] In one optional embodiment, the third similarity calculation unit includes: The rule matching subunit is used to perform rule matching on the strings corresponding to all data items in the dataset according to a preset regular expression to determine a second quantity; wherein, the second quantity is the number of data items corresponding to strings that conform to the regular expression; The third similarity determination subunit is used to determine the third similarity based on the ratio of the second quantity to the total number of data items in the dataset.

[0078] In an optional embodiment, the verification and evaluation module 13 includes: The scoring unit is used to score the data features of each candidate master data item according to the data lineage link and the identification evaluation matrix, based on the first description of the table where each candidate master data item is located and the second description of the metadata, and to determine the initial score of each candidate master data item on each data feature; the data features include: business value features, data uniqueness features, data sharing features, data stability features, and data periodicity features; The second weighted summation unit is used to perform a weighted summation of the initial scores of each candidate master data item on each data feature for each candidate master data item, so as to obtain the verification evaluation score of the candidate master data item. The second master data item determination unit is used to select candidate master data items whose verification evaluation scores are greater than or equal to a preset evaluation score threshold as target master data items.

[0079] It should be noted that the working process of each module in the data recognition device described in the embodiments of the present invention can refer to the working process of the data recognition method described in the above embodiments, and the technical effect achieved is the same as that of the data recognition method described in the above embodiments, so it will not be repeated here.

[0080] See Figure 9 , Figure 9 This is a structural block diagram of a data recognition device provided in an embodiment of the present invention. The data recognition device includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described data recognition method embodiments, such as steps S11 to S13.

[0081] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the data identification device.

[0082] The data identification device may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a data identification device and does not constitute a limitation on the data identification device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the data identification device may also include input / output devices, network access devices, buses, etc.

[0083] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the data recognition device, connecting all parts of the data recognition device via various interfaces and lines.

[0084] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the data recognition device by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0085] If the modules / units integrated into the data identification device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0086] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0087] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data recognition method, characterized in that, include: Data preprocessing is performed on master data collected from different data sources to obtain multiple datasets; each dataset includes a data item corresponding to a column of data elements; For each data item in the dataset, perform similarity matching to determine multiple candidate master data items; Based on the pre-constructed identification and evaluation matrix, multiple candidate master data items are verified and evaluated to determine the target master data item.

2. The data recognition method as described in claim 1, characterized in that, The process involves preprocessing master data from different pre-collected data sources to obtain multiple datasets, including: The master data is preprocessed to obtain multiple data items to be identified; wherein, the preprocessing includes data cleaning and sorting, and data standardization. Multiple data items are divided according to data elements to obtain multiple datasets; wherein, the data items in a column corresponding to one data element are divided into one dataset.

3. The data recognition method as described in claim 1, characterized in that, The step of performing similarity matching on data items in each dataset to determine multiple candidate master data items includes: For each dataset, length matching is performed on all data items in the dataset to determine the first similarity of the dataset; Perform content matching on all data items in the dataset to determine the second similarity of the dataset; Perform rule matching on all data items in the dataset to determine the third similarity of the dataset; The data similarity of the dataset is obtained by weighted summation of the first similarity, the second similarity, and the third similarity. Data items in the dataset whose data similarity is greater than or equal to a preset similarity threshold are selected as candidate master data items.

4. The data recognition method as described in claim 3, characterized in that, The step of performing length matching on all data items in the dataset to determine the first similarity of the dataset includes: The lengths of the strings corresponding to all data items in the dataset are compared to obtain a first quantity; wherein, the first quantity is the number of data items corresponding to strings of the same length; The first similarity is determined based on the ratio of the first quantity to the total number of data items in the dataset.

5. The data recognition method as described in claim 3, characterized in that, The step of performing content matching on all data items in the dataset to determine the second similarity of the dataset includes: The content editing process is performed on the strings corresponding to any two data items in the dataset, and the edit distance between the strings corresponding to any two data items is calculated; wherein, the content editing process includes character deletion, character insertion and character replacement; The second similarity is determined based on the edit distance between the strings corresponding to any two data items in the dataset.

6. The data recognition method as described in claim 3, characterized in that, The step of performing rule matching on all data items in the dataset to determine the third similarity of the dataset includes: Based on a preset regular expression, the strings corresponding to all data items in the dataset are matched according to the rules to determine a second quantity; wherein, the second quantity is the number of data items corresponding to strings that conform to the regular expression; The third similarity is determined based on the ratio of the second quantity to the total number of data items in the dataset.

7. The data recognition method as described in claim 1, characterized in that, The step of verifying and evaluating multiple candidate master data items based on a pre-constructed identification and evaluation matrix to determine the target master data item includes: Based on the first description of the table containing each candidate master data item and the second description of the metadata, the data characteristics of each candidate master data item are scored according to the data lineage and the identification evaluation matrix to determine the initial score of each candidate master data item on each data characteristic; the data characteristics include: business value characteristics, data uniqueness characteristics, data sharing characteristics, data stability characteristics, and data periodicity characteristics; For each candidate master data item, the initial scores of the candidate master data item on each data feature are weighted and summed to obtain the verification evaluation score of the candidate master data item; Candidate master data items whose verification evaluation scores are greater than or equal to a preset evaluation score threshold are selected as target master data items.

8. A data identification device, characterized in that, include: The data preprocessing module is used to preprocess master data from different pre-collected data sources to obtain multiple datasets; wherein each dataset includes a data item corresponding to a column of data elements; The similarity matching module is used to perform similarity matching on each data item in the dataset to determine multiple candidate master data items; The verification and evaluation module is used to verify and evaluate multiple candidate master data items based on a pre-constructed identification and evaluation matrix to determine the target master data item.

9. A data recognition device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data identification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data identification method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data recognition method according to any one of claims 1 to 7.

Images