A method and device for intelligently recommending data standards

Abstract

The present invention discloses an intelligent recommendation method and device for data standards. The method first extracts keywords corresponding to the source table information by performing word segmentation on source table information, matches relevant data standard names in a database through the keywords, and obtains multiple groups of relevant data standard tables; then, a first screening is performed on the searched multiple groups of data standard tables through a preset scoring rule, and a preset number of relatively relevant data standard tables are selected; then, all data items of each selected data standard table are subjected to a second screening with all data items in the source table information using a similarity accumulation calculation method, and the data standard table that best matches the source table information is obtained, thereby successively improving the similarity comparison efficiency and accuracy through the second screening of different degrees, while also reducing the number of computer comparison operations, greatly alleviating the workload of access personnel, and the problem of mismatching occurring during the data standard matching process.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and device for intelligently recommending data standards. Background Art

[0002] Intelligent recommendations generate user profiles based on individual user preferences, providing personalized recommendations tailored to each user. This effectively improves click-through rates, conversion rates, and user retention in industries like media and e-commerce. For example, if a customer wants to buy a car, travel, or renovate their home, intelligent recommendations can be used to recommend the optimal vehicle model, travel route, or renovation plan based on their needs.

[0003] A common implementation of intelligent recommendation involves entering a phrase into a search box on an internet website. The system then generates search terms based on the phrase and presents them on the page, thereby facilitating a quick search. The more phrases a user enters, and the more accurate their semantics, the more precise the search terms are. However, search engine-based intelligent recommendation methods are not effectively applied to big data standard benchmarking. While this approach can perform data exploration (a data scanning method that comprehensively profiles data resources) upon data access and benchmark data standards based on the exploration results, the resource names of the accessed data often differ significantly from the corresponding standard names. For example, the data standard may be for permanent population, while the accessed data may be for regular population data or population registration data. This makes it impossible to accurately match the data standard name with the accessed data through intelligent recommendation, requiring manual intervention to screen out large amounts of invalid data.

[0004] Existing data block standard query methods, such as using SQL-like statements to perform fuzzy matching in a data standard library based on the name of the data table being accessed, are limited by the matching rule (fuzzy value + specific value + fuzzy value). As the name (specific value) contains more characters, the SQL-like statement will produce fewer matching results, and there's a high probability of not matching the corresponding data standard table. Therefore, manual specification is the only way to achieve this, but manual methods rarely produce recommended data standards in real-world environments.

[0005] Another example is keyword retrieval: Each data standard is extracted from the database with at least four keywords. When data access personnel access data, they set the corresponding data name as the keyword and use the keyword to perform SQL-like keyword matching in the database to find the corresponding data standard. However, this method has a high error rate in matching data standards and will match a large number of useless data standards. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an intelligent recommendation method for data standards to improve the accuracy of data standard recommendations.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A method for intelligently recommending data standards, comprising the steps of:

[0009] Acquire source table information, and perform word segmentation on the source table information to obtain at least one group of keywords;

[0010] Matching data standard names related to the keyword in a database according to the keyword to obtain a plurality of the data standard names and a plurality of data standard tables corresponding to the data standard names;

[0011] Filtering the data standard names according to a preset scoring rule to obtain a preset number of data standard tables;

[0012] A total similarity is obtained by accumulating similarities between all data items in each data standard table and all data items in the source table information, and an optimal data standard table is obtained based on the total similarity.

[0013] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0014] A device for intelligently recommending data standards includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the method for intelligently recommending data standards as described above is implemented.

[0015] The beneficial effects of the present invention are as follows: first, keywords corresponding to the source table information are extracted by performing word segmentation on the source table information, and relevant data standard names are matched in the database through the keywords to obtain multiple sets of relevant data standard tables; then, the multiple sets of data standard tables searched out are screened for the first time through a preset scoring rule, and a preset number of relatively relevant data standard tables are selected, and then all data items of each selected data standard table are screened for the second time with all data items in the source table information using a similarity accumulation calculation method to obtain the data standard table that best matches the source table information, thereby improving the similarity comparison efficiency and accuracy through the secondary screening of different degrees, while also reducing the number of computer comparison operations, greatly reducing the workload of access personnel, and the problem of mismatching occurring during the data standard matching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A flowchart of a method for intelligently recommending data standards according to an embodiment of the present invention;

[0017] Figure 2 The figure is a structural diagram of an intelligent recommendation device for data standards according to an embodiment of the present invention.

[0018] Figure 3 This is a flowchart of another step of a method for intelligently recommending data standards according to an embodiment of the present invention;

[0019] Figure 4 This is a flowchart of another step of a method for intelligently recommending data standards according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0021] Please refer to Figure 1 , an intelligent recommendation method for data standards, comprising the steps of:

[0022] Acquire source table information, and perform word segmentation on the source table information to obtain at least one group of keywords;

[0023] Matching data standard names related to the keyword in a database according to the keyword to obtain a plurality of the data standard names and a plurality of data standard tables corresponding to the data standard names;

[0024] Filtering the data standard names according to a preset scoring rule to obtain a preset number of data standard tables;

[0025] A total similarity is obtained by accumulating similarities between all data items in each data standard table and all data items in the source table information, and an optimal data standard table is obtained based on the total similarity.

[0026] From the above description, it can be seen that the beneficial effects of the present invention are: first, by performing word segmentation on the source table information to extract the keywords corresponding to the source table information, the relevant data standard names are matched in the database through the keywords to obtain multiple sets of relevant data standard tables; then, the multiple sets of data standard tables searched out are screened for the first time through the preset scoring rules, and a preset number of more relevant data standard tables are selected, and then all data items of each selected data standard table are screened for the second time with all data items in the source table information using the similarity accumulation calculation method to obtain the data standard table that best matches the source table information, thereby improving the similarity comparison efficiency and accuracy through secondary screening of different degrees, while also reducing the number of computer comparison operations, greatly reducing the workload of access personnel, and the mismatching problems that occur during the data standard matching process.

[0027] Furthermore, the data standard names are screened according to a preset scoring rule to obtain a preset number of groups of data standard tables, including:

[0028] Comparing the lengths of the data standard name and the keyword, and taking the longer one as the maximum length;

[0029] Obtaining the Levenshtein distance between the data standard name and the keyword;

[0030] Perform similarity calculation based on the maximum length and the Levenshtein distance to obtain a similarity score corresponding to each of the data standard names;

[0031] The data standard names are screened according to the similarity scores to obtain a preset number of groups of data standard names, and the data standard table corresponding to the preset number of groups of data standard names is obtained.

[0032] As can be seen from the above description, by obtaining the maximum length of the data standard name and the keyword, as well as the Levenshtein distance between the data standard name and the keyword, the similarity score corresponding to each data standard table is calculated, and then multiple groups of data standard tables with higher similarity scores are obtained based on the similarity scores. This improves the matching degree between the source table information and the searched data standard tables, thereby improving the accuracy of data standard recommendations.

[0033] Furthermore, the similarity calculation based on the maximum length and the Levenshtein distance to obtain a similarity score corresponding to each of the data standard names includes:

[0034] Calculate the similarity score S(x):

[0035] S(x)=(max(len a ,len b )-lev a，b (i, j))÷max(len a ,len b );

[0036] Among them, len a and len b Respectively represents the length of the string of data standard name and keyword; lev a，b (i, j) represents the Levenshtein distance between the data standard name and the keyword.

[0037] As can be seen from the above description, the similarity score calculation formula is used to calculate the maximum string length of the data standard name and the keyword, as well as the Levenshtein distance between the two, to obtain the similarity score of each data standard name and the keyword. This can more accurately search for data standard tables with higher similarity from the database, thereby improving the matching degree of the data standard tables.

[0038] Furthermore, the cumulative similarity calculation of all data items in each of the data standard tables and all data items in the source table information to obtain a total similarity includes:

[0039] Obtain all data items in the data standard table and all data items in the source table information;

[0040] Calculate the similarity between each data item in the data standard table and all data items in the source table information to obtain the maximum similarity of each data item in the data standard table;

[0041] The maximum similarity of each data item in the data standard table is added to obtain the total similarity of the data standard table.

[0042] From the above description, it can be seen that by calculating the similarity between each data item in the data standard table and all data items in the source table information, the total similarity of each data standard table is obtained, so that each data item in the data standard table can be compared with each data item in the source table information, and a data standard table with a higher matching degree with each data item in the source table information can be searched.

[0043] Furthermore, the optimal data standard table obtained according to the total similarity includes:

[0044] Obtaining the number of data items in the data standard table;

[0045] Obtaining an average data item similarity of each of the data standard tables according to the number of data items in the data standard table and the total similarity;

[0046] The data standard table is sorted according to the average data item similarity to obtain the optimal data standard table.

[0047] From the above description, it can be seen that by obtaining the number of data items in the data standard table and obtaining the average data item similarity based on the number of data items in the data standard table and the total similarity, the average data item similarity is used as the scoring standard to avoid the situation where the data standard table has too many data items and its final total similarity is much higher than that of other data standard tables, thereby improving the accuracy of data standard recommendation.

[0048] Furthermore, the optimal data standard table obtained according to the total similarity also includes:

[0049] Obtaining a similarity variance based on the maximum similarity of each data item in the data standard table and the corresponding average data item similarity;

[0050] The data standard table is sorted according to the similarity variance to obtain the optimal data standard table.

[0051] From the above description, it can be seen that the similarity variance is obtained by calculating the variance of the maximum similarity of each data item in the obtained data standard table and the corresponding average data item similarity. The matching degree of the data standard table is scored by the similarity variance, which can reflect the deviation of the data items in the data standard table, thereby selecting the data standard table with smaller deviation.

[0052] Furthermore, the step of matching the data standard name related to the keyword in the database according to the keyword includes:

[0053] Performing word segmentation on each of the data standard names to obtain word segmentation phrases;

[0054] Filtering the segmented phrases to obtain related phrases;

[0055] An index is created in the database according to the related phrases.

[0056] From the above description, it can be seen that when establishing a data standard name, the data standard name is first segmented to obtain segmented phrases, and then the segmented phrases are filtered, so as to avoid using some popular and feature-lacking phrases as search indexes, so that when indexing in the database through keywords, it is possible to more effectively match the valid data standard table.

[0057] Furthermore, before creating an index in the database based on the related phrases, the method further includes:

[0058] Obtaining a series of phrases associated with the related phrases;

[0059] The series of word groups are merged into the related word group to obtain an updated related word group.

[0060] As can be seen from the above description, by obtaining a series of phrases associated with the relevant phrases when generating an index, the possibility of the data standard table being indexed can be expanded, thereby improving the indexing accuracy of the data standard table.

[0061] Furthermore, the acquiring of source table information and performing word segmentation on the source table information to obtain at least one group of keywords includes:

[0062] Determine whether the keyword is valid; if not, obtain the data item in the source table information and generate the keyword;

[0063] From the above description, it can be seen that when valid keyword information cannot be obtained from the table name of the source table information, by further obtaining the data items in the source table information as keywords for retrieval, although the burden of generating keywords is increased, it is possible to more effectively search for data standard tables with higher similarity, reduce the burden of retrieval and improve the accuracy of retrieval.

[0064] Please refer to Figure 2 The present invention also provides an intelligent recommendation device for data standards, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned intelligent recommendation method for data standards is implemented.

[0065] The intelligent data standard recommendation method and device of the present invention can be applied to matching data standard tables of various types of source table information, and are described below through specific implementation methods:

[0066] Example 1

[0067] Please refer to Figure 1 and Figure 3 , a data standard intelligent recommendation method, characterized by comprising the steps of:

[0068] S1. Obtain source table information, and perform word segmentation on the source table information to obtain at least one group of keywords;

[0069] Since the table name of the source table information may be significantly different from the standard name, extracting keywords from the table name of the source table information for comparison can improve the accuracy;

[0070] S2. Matching data standard names related to the keyword in a database according to the keyword to obtain a plurality of data standard names and a plurality of data standard tables corresponding to the data standard names;

[0071] S3. Filter the data standard names according to a preset scoring rule to obtain a preset number of data standard tables, specifically:

[0072] S31, comparing the lengths of the data standard name and the keyword, and taking the longer one as the maximum length;

[0073] S32, obtaining the Levenshtein distance between the data standard name and the keyword;

[0074] S33, performing similarity calculation based on the maximum length and the Levenshtein distance to obtain a similarity score corresponding to each of the data standard names;

[0075] Calculate the similarity score S(x):

[0076] S(x)=(max(len a ,len b )-lev a，b (i, j))÷max(len a ,len b );

[0077] Among them, len a and len b Respectively represents the length of the string of data standard name and keyword; lev a，b (i, j) represents the Levenshtein distance between the data standard name and the keyword; for example, if the keyword is "transaction record", and the data standard names of the data standard existing in the database include "bank transaction record", "merchant transaction record", "WeChat transaction record" and "Alipay transaction record", etc., the corresponding similarity score is obtained by calculating the similarity score between each data standard name and the keyword "transaction record";

[0078] Among them, the Levenshtein distance lev a，b (i, j):

[0079]

[0080] S34. Filter the data standard names according to the similarity scores to obtain a preset number of groups of data standard names, and obtain the data standard table corresponding to the preset number of groups of data standard names; wherein the preset number of groups can be set according to the required accuracy. If higher accuracy is required, the number of groups can be set to more than 10 groups;

[0081] S4. Calculate the cumulative similarity between all data items in each data standard table and all data items in the source table information to obtain a total similarity, and obtain the optimal data standard table based on the total similarity. The similarity calculation adopts formula S(x), specifically:

[0082] S41. Acquire all data items in the data standard table and all data items in the source table information; for example, if the source table information is "ICBC transaction records", the corresponding data items include "card number", "remitter name", "remittee name", and "transaction amount", etc.; the first data standard table is "merchant transaction records", and the corresponding data items include "merchant name", "card number", "remittance amount", and "remittance source", etc.; the second data standard table is "WeChat transaction records", and the corresponding data items include "paying account", "receiving account", "payment amount", and "payment time", etc.; the third data standard table is "bank transaction records", and the corresponding data items include "receiving bank account", "paying bank account", "remittance amount", and "remittance time", etc.;

[0083] S42. Perform similarity calculation on each data item in the data standard table and all data items in the source table information to obtain the maximum similarity of each data item in the data standard table; if the similarity currently calculated is the similarity between the first data standard table and the source table information, perform similarity calculation on the data item "card number" and the data items "card number", "remitter name", "payee name" and "transaction amount" in the "Industrial and Commercial Bank of China transaction record" to obtain the maximum similarity corresponding to the data item "card number"; then perform similarity calculation on the data item "merchant name" and the data items "card number", "remitter name", "payee name" and "transaction amount" in the "Industrial and Commercial Bank of China transaction record" to obtain the maximum similarity corresponding to the data item "merchant name", and so on, perform similarity calculation on each data item in the first data standard table "merchant transaction record"; similarly, perform the same calculation on the second data standard table "WeChat transaction record" and the third data standard table "bank transaction record";

[0084] S43. Adding the maximum similarities of each data item in the data standard table to obtain the total similarity of the data standard table; for example, adding the maximum similarities corresponding to the data items "Merchant Name", "Card Number", "Remittance Amount", and "Remittance Source" in the first data standard table "Industrial and Commercial Bank of China Transaction Records" to obtain the total similarity corresponding to the first data standard table "Industrial and Commercial Bank of China Transaction Records"; selecting the optimal data standard table based on the total similarity;

[0085] The specific calculation formula is:

[0086] SS(x)=(max(S(x1))+max(S(x2))+max(S(x3))+…+max(S(x n )));

[0087] Among them, x1, x2, x3 and x nRespectively represent the first data item to the last data item in the data standard table.

[0088] Example 2

[0089] The difference between this embodiment and the first embodiment is that the total similarity is calculated to obtain a more accurate similarity ranking;

[0090] After completing step S43, the method further includes step S44:

[0091] S441. Obtain the number of data items in the data standard table; if the first data standard table "Merchant Transaction Record" contains 10 data items, then record the number of data items in the first data standard table as 10; if the second data standard table "WeChat Transaction Record" contains 13 data items, then record the number of data items in the second data standard table as 13;

[0092] S442. Obtain an average data item similarity of each of the data standard tables based on the number of data items in the data standard tables and the total similarity. If the total similarity of the first data standard table "Merchant Transaction Records" calculated in step S43 is 5, then the corresponding average data item similarity is 1 / 2. If the total similarity of the second data standard table "WeChat Transaction Records" calculated in step S43 is 7, then the corresponding average data item similarity is 7 / 13.

[0093] E(x)=SS(x)÷n

[0094] S443. Sort the data standard tables according to the average data item similarities to obtain the optimal data standard tables. The data standard tables may be sorted from high to low scores or from low to high scores according to the average data item similarities, such as the first data standard table 1 / 2, the second data standard table 7 / 13, or the second data standard table 7 / 13, the first data standard table 1 / 2. When extracting the optimal data standard table, the data standard table may be extracted from the first position or the last position according to the sequence sorting rules.

[0095] In another embodiment, the variance may be calculated by combining the average data item similarity calculated in step S442 and the maximum similarity of each data item calculated in step S42 to obtain the similarity variance.

[0096] Calculate the similarity variance:

[0097] S 2 (x)=((max(S(x1))-E(x))+(max(S(x2))-E(x))+(max(S(x3))-E(x))+…+(max(S(x) n ))-E(x)))÷n

[0098] Then, the data standard table is sorted according to the similarity variance to obtain the optimal data standard table;

[0099] In an optional embodiment, the readability of the result can be increased by converting it into a score, such as: EScore = (1-S 2 )×100; and set a corresponding score threshold, such as 50 points. When none of the filtered data standard tables meet the corresponding score threshold, manual review is performed.

[0100] Example 3

[0101] The difference between this embodiment and the first or second embodiment is that the data standard table is preprocessed;

[0102] Please refer to Figure 4 , step S2 includes step S02 before:

[0103] S021. Segment each of the data standard names to obtain segmented phrases. For example, if the data standard table is "resident population data," the data standard name is segmented into "resident," "population," and "data" using a segmentation tool.

[0104] S022. Filter the segmented phrases to obtain related phrases; among the three phrases "resident," "population," and "data," "data" is a popular phrase that appears in most data standard names, so it is removed during phrase filtering, while the two phrases "population" and "resident" are retained;

[0105] S023. Obtain a series of phrases associated with the relevant phrases; based on the two phrases "population" and "resident", obtain corresponding series of phrases such as "population registration", "personnel information" and other related phrases. The number of series of phrases can be controlled according to the set total number of phrases;

[0106] S024. Merge the series of phrases into the related phrases to obtain updated related phrases. If the total number of phrases set is 4, add the series of phrases "population registration" and "personnel information" to the original related phrases to obtain new related phrases: "population", "permanent", "population registration" and "personnel information";

[0107] S025. Creating an index in a database based on the relevant phrases;

[0108] The relevant phrases "population", "permanent residence", "population registration" and "personnel information" are stored in the database and indexed;

[0109] Wherein, step S1 further includes:

[0110] Determine whether the keyword is valid. If not, obtain the data items in the source table information and generate the keyword; if the source table information to be accessed is "bill", and the word segmentation tool cannot effectively split it into word segmentation phrases, that is, the current keyword is invalid; further obtain data items in the source table information "bill", such as "bank", "transaction" and "flow", and search through the obtained data items "bank", "transaction" and "flow" as keywords; or obtain corresponding keywords through the method in step S023 for search; or generate by manually marking keywords.

[0111] Example 4

[0112] Please refer to Figure 2 , an intelligent recommendation device for data standards, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the intelligent recommendation method for data standards as described in any one of embodiments one to three is implemented.

[0113] In summary, the present invention provides a method and device for intelligently recommending data standards. After source table information is input, keywords corresponding to the source table information are first extracted by segmenting the source table information, and a determination is made as to whether valid keywords can be extracted from the source table information name. If valid keywords cannot be extracted, similar keywords are further extracted or generated from the data items of the source table information. The keywords are then matched with the relevant data standard names in the database to obtain multiple sets of relevant data standard tables. The searched multiple sets of data standard tables are then screened for the first time by calculating the Levenshtein distance between the data standard names and the keywords to select a preset number of relatively relevant data standard tables. A similarity accumulation calculation method, including the total similarity, average, and variance, is then used to calculate all data items of each selected data standard table with all data items in the source table information to obtain a secondary screening result, thereby obtaining the data standard table that best matches the source table information. This secondary screening improves the efficiency and accuracy of similarity comparison through different degrees of screening, while also reducing the number of computer comparison operations, greatly alleviating the workload of access personnel, and preventing mismatches that occur during the data standard alignment process.

[0114] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An intelligent recommendation method for data standards, characterized in that: Including steps: Acquire source table information, and perform word segmentation on the source table information to obtain at least one group of keywords, including: Determine whether the keyword is valid; if not, obtain the data item in the source table information and generate the keyword; Matching data standard names related to the keyword in a database according to the keyword to obtain a plurality of the data standard names and a plurality of data standard tables corresponding to the data standard names; The data standard names are screened according to a preset scoring rule to obtain a preset number of data standard tables, including: Comparing the lengths of the data standard name and the keyword, and taking the longer one as the maximum length; Obtaining the Levenshtein distance between the data standard name and the keyword; Perform similarity calculation based on the maximum length and the Levenshtein distance to obtain a similarity score corresponding to each of the data standard names; Calculate the similarity score S(x): S(x)=(max(lena, lenb)-leva, b(i, j))÷max(lena, lenb), Where lena and lenb represent the string lengths of the data standard name and keyword respectively, and leva,b(i, j) represents the Levenshtein distance between the data standard name and the keyword; Filtering the data standard names according to the similarity scores to obtain a preset number of groups of data standard names, and obtaining the data standard table corresponding to the preset number of groups of data standard names; Performing cumulative similarity calculation on all data items in each of the data standard tables and all data items in the source table information to obtain a total similarity, and obtaining the optimal data standard table based on the total similarity; The accumulating similarity calculation of all data items in each of the data standard tables and all data items in the source table information to obtain a total similarity includes: Obtain all data items in the data standard table and all data items in the source table information; Calculate the similarity between each data item in the data standard table and all data items in the source table information to obtain the maximum similarity of each data item in the data standard table; The maximum similarity of each data item in the data standard table is added to obtain the total similarity of the data standard table.

2. The intelligent recommendation method for data standards according to claim 1, characterized in that: The optimal data standard table obtained according to the total similarity includes: Obtaining the number of data items in the data standard table; Obtaining an average data item similarity of each of the data standard tables according to the number of data items in the data standard table and the total similarity; The data standard table is sorted according to the average data item similarity to obtain the optimal data standard table.

3. The intelligent recommendation method for data standards according to claim 2, characterized in that: The optimal data standard table obtained according to the total similarity also includes: Obtaining a similarity variance based on the maximum similarity of each data item in the data standard table and the corresponding average data item similarity; The data standard table is sorted according to the similarity variance to obtain the optimal data standard table.

4. The intelligent recommendation method for data standards according to claim 1, characterized in that: The step of matching the data standard name related to the keyword in the database according to the keyword includes: Performing word segmentation on each of the data standard names to obtain word segmentation phrases; Filtering the segmented phrases to obtain related phrases; An index is created in the database according to the related phrases.

5. The intelligent recommendation method for data standards according to claim 4, characterized in that: Before creating an index in the database according to the related phrases, the method further includes: Obtaining a series of phrases associated with the related phrases; The series of word groups are merged into the related word group to obtain an updated related word group.

6. An intelligent data standard recommendation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of the intelligent recommendation method for data standards according to any one of claims 1 to 5.

Images