Multi-source data dictionary fusion method and device, medium and equipment

Abstract

The application discloses a multi-source data dictionary fusion method. The method comprises the following steps: obtaining to-be-fused data dictionaries and data logic description information; extracting first keywords in the data dictionary table of the to-be-fused data dictionaries based on the data logic description information; extracting second keywords in the data dictionary table based on the word frequency of the fields in the data dictionary table, and fusing the first keywords and the second keywords to obtain target keywords; calculating the two-dimensional weighted edit distance of the target keywords, and performing primary classification on the data dictionary table to obtain coarse categories of the data dictionary table; in the same coarse category, calculating the multi-dimensional weighted edit distance of the fields, performing secondary classification on the data dictionary table to obtain fine categories of the data dictionary table; in the same fine category, calculating the table similarity of the data dictionary table, and fusing the data dictionary table according to the table similarity. The application solves the problems of high personnel cost and long cycle in the scheme of relying on the design personnel to modify the design documents of the data dictionary.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, medium and device for multi-source data dictionary fusion. Background Technology

[0002] In addition to user data, database design also requires the design of a significant amount of non-user data. This includes data item lengths, types, user identifiers, primary / foreign keys, and relationships between data and files. This non-user data forms the intelligence system of the entire database system. To provide a common standard and basis for the design, implementation, operation, maintenance, and expansion of the database, and to ensure its sharing, security, integrity, consistency, validity, recoverability, and scalability, this information is centrally stored in a data dictionary. In recent years, data dictionaries have been widely used as an auxiliary governance tool in grassroots social governance both domestically and internationally.

[0003] However, when the application scope of a data dictionary changes or when two or more data dictionaries are needed to jointly complete business tasks, issues such as data inconsistency and redundancy arise due to the lack of unified data standards and naming conventions. For example, the data dictionary designed for a database in a government department or company may be ambiguous and redundant with the design for a cross-departmental collaborative application environment. Therefore, during the deployment phase of a joint system, the data dictionary already designed and applied needs to be adapted to the new system to ensure the accuracy of system operation. In actual business scenarios, a common solution is to rely on the original data dictionary designers to modify the data dictionary design documents and re-adapt and optimize the system to address the issue of decreased system capabilities when the data dictionary is adapted to the new system. However, this solution suffers from high personnel costs and long development cycles, making it difficult to meet actual business needs. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, medium and equipment for multi-source data dictionary fusion, which solves the problems of high personnel costs and long cycle in the scheme that relies on the original data dictionary designers to modify the design documents of the data dictionary.

[0005] According to one aspect of this application, a multi-source data dictionary fusion method is provided, comprising: Obtain multiple data dictionaries to be merged and data logic description information corresponding to each of the data dictionaries to be merged; Based on the data logic description information, the first keyword is extracted from the data dictionary table of each of the data dictionaries to be merged, wherein the data logic description information includes at least one of the following: the table name, primary key, entity relationship, and tag of the data dictionary table; Based on the word frequency of each field in the data dictionary table, a second keyword is extracted from the data dictionary table, and the first keyword and the second keyword are combined to obtain the target keyword; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and perform an initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain the coarse category corresponding to each data dictionary table; Within the same coarse category, the multidimensional weighted edit distance between the fields is calculated, and the data dictionary tables corresponding to the fields are reclassified based on the multidimensional weighted edit distance to obtain the fine category corresponding to each data dictionary table; Within the same subcategory, the table similarity between different data dictionary tables is calculated, and the data dictionary tables are merged based on the table similarity to obtain the target data dictionary.

[0006] Optionally, calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, including: Read the preset weight mapping information to obtain the preset weight corresponding to the target keyword; Based on the data logic description information, the preset weights are adjusted to obtain the first weight corresponding to each target keyword; The two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged is calculated based on the first weight.

[0007] Optionally, within the same coarse category, calculate the multidimensional weighted edit distance between the fields, including: Within the same coarse category, it is determined whether each field in the data dictionary table is the target keyword, and the design information of each field is obtained, wherein the design information includes at least the field type; Based on the judgment result corresponding to each field and the design information, the second weight of each field is determined, and the multidimensional weighted edit distance between the fields is calculated based on the second weight.

[0008] Optionally, before calculating the multidimensional weighted edit distance between the fields based on the second weight, the method further includes: Determine the adjustment factor for each target keyword, and adjust the second weight according to the adjustment factor, wherein the adjustment factor includes a language dimension adjustment factor and / or a text dimension adjustment factor.

[0009] Optionally, the data dictionary tables are fused based on the table similarity to obtain the target data dictionary, including: If the similarity of the tables is greater than the first similarity threshold, then the data dictionary tables are merged, and the target data dictionary is obtained based on the merged target data dictionary table; If the similarity of the tables is less than the first similarity threshold but greater than the second similarity threshold, then the intersection and difference of the data dictionary tables are extracted, and the target data dictionary is obtained based on the intersection and the difference.

[0010] Optionally, to obtain the target data dictionary by fusing the data dictionary tables based on the table similarity, the method further includes: If the table similarity is less than the second similarity threshold, the fusion process ends and a prompt message is generated, wherein the prompt message indicates that the data dictionary similarity is low; or, If the table similarity is less than the second similarity threshold, then return to the step of obtaining the design information of each field to obtain new design information, wherein the new design information includes field type, field length and default value.

[0011] Optionally, after obtaining multiple data dictionaries to be merged, the method further includes: The data dictionary to be merged is preprocessed, wherein the preprocessing includes at least entity disambiguation.

[0012] According to another aspect of this application, a multi-source data dictionary fusion apparatus is provided, the apparatus comprising: The acquisition module is used to acquire multiple data dictionaries to be merged and data logic description information corresponding to each of the data dictionaries to be merged. The feature extraction module is used to extract a first keyword from the data dictionary table of each of the data dictionaries to be merged, based on the data logic description information, wherein the data logic description information includes at least one of the following: the table name, primary key, entity relationship, and tag of the data dictionary table; and to extract a second keyword from the data dictionary table based on the word frequency of each field in the data dictionary table, and to merge the first keyword and the second keyword to obtain the target keyword. The classification module is used to calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and to perform an initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain a coarse category for each data dictionary table; and, within the same coarse category, to calculate the multi-dimensional weighted edit distance between the fields, and to further classify the data dictionary tables corresponding to the fields based on the multi-dimensional weighted edit distance to obtain a fine category for each data dictionary table; The fusion module is used to calculate the table similarity between different data dictionary tables within the same subcategory, and to fuse the data dictionary tables based on the table similarity to obtain the target data dictionary.

[0013] Optionally, the classification module is used to: read preset weight mapping information to obtain the preset weight corresponding to the target keyword; Based on the data logic description information, the preset weights are adjusted to obtain the first weight corresponding to each target keyword; The first weighted edit distance between target keywords in different data dictionaries to be merged is calculated based on the first weight.

[0014] Optionally, the classification module is used for: Within the same coarse category, it is determined whether each field in the data dictionary table is the target keyword, and the design information of each field is obtained, wherein the design information includes at least the field type; Based on the judgment result corresponding to each field and the design information, the second weight of each field is determined, and the multidimensional weighted edit distance between the fields is calculated based on the second weight.

[0015] Optionally, the classification module is used for: Determine the adjustment factor for each target keyword, and adjust the second weight according to the adjustment factor, wherein the adjustment factor includes a language dimension adjustment factor and / or a text dimension adjustment factor.

[0016] Optionally, the fusion module is used for: If the similarity of the tables is greater than the first similarity threshold, then the data dictionary tables are merged, and the target data dictionary is obtained based on the merged target data dictionary table; If the similarity of the tables is less than the first similarity threshold but greater than the second similarity threshold, then the intersection and difference of the data dictionary tables are extracted, and the target data dictionary is obtained based on the intersection and the difference.

[0017] Optionally, to obtain the target data dictionary by fusing the data dictionary tables based on the table similarity, the method further includes: If the table similarity is less than the second similarity threshold, the fusion process ends and a prompt message is generated, wherein the prompt message indicates that the data dictionary similarity is low; or, If the table similarity is less than the second similarity threshold, then return to the step of obtaining the design information of each field to obtain new design information, wherein the new design information includes field type, field length and default value.

[0018] Optionally, the apparatus further includes a preprocessing module for: The data dictionary to be merged is preprocessed, wherein the preprocessing includes at least entity disambiguation.

[0019] According to another aspect of this application, a medium is provided that stores a program or instructions thereon, which, when executed by a processor, implement the above-described multi-source data dictionary fusion method.

[0020] According to another aspect of this application, an apparatus is provided, including a storage medium and a processor, wherein the storage medium stores a computer program, and the processor executes the computer program to implement the above-described multi-source data dictionary fusion method.

[0021] By utilizing the aforementioned technical solution, this application takes into account the unique characteristics of data dictionaries as a special data structure. It achieves feature extraction through both data logical description information and TF-IDF, obtaining the first keyword and the second keyword respectively. The feature extraction method based on data logical description information considers the characteristics at the data structure level, while the feature extraction method based on TF-IDF considers the fact that the extracted object is text. The combination of these two methods is more suitable for the specific feature extraction scenario of keyword extraction in data dictionaries. Subsequently, the extracted keywords are used to classify the data dictionary table. In the specific classification process, a two-stage weighted edit distance algorithm is used for initial classification and secondary classification, respectively, and then data dictionary fusion is performed within the same category. By performing two-dimensional and then multi-dimensional weighted edit distance calculations, the computational efficiency is greatly improved, while avoiding the problem of introducing unsuitable dimensions for similarity calculations and causing bias in the calculation results due to directly using weighted edit distance. Therefore, the automatic data dictionary fusion method provided in this embodiment does not require manual modification of the data dictionary design document, effectively solving the problems of high personnel costs and long cycles, and reducing labor and time costs.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a multi-source data dictionary fusion method provided in an embodiment of this application is shown. Figure 2A flowchart illustrating another multi-source data dictionary fusion method provided in an embodiment of this application is shown; Figure 3 This illustration shows a coarse classification result of another multi-source data dictionary fusion method provided in an embodiment of this application; Figure 4 This illustration shows a detailed classification result diagram of another multi-source data dictionary fusion method provided in an embodiment of this application; Figure 5 A flowchart illustrating another multi-source data dictionary fusion method provided in an embodiment of this application is shown; Figure 6 The diagram shows a structural block diagram of a multi-source data dictionary fusion device provided in an embodiment of this application. Detailed Implementation

[0024] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0025] This embodiment provides a multi-source data dictionary fusion method, such as... Figure 1 As shown, the method includes: Step 101: Obtain multiple data dictionaries to be merged and the data logic description information corresponding to each data dictionary to be merged; Step 102: Based on the data logic description information, extract the first keyword from the data dictionary table of each data dictionary to be merged. The data logic description information includes at least one of the following: table name, primary key, entity relationship, and tag of the data dictionary table. Step 103: Based on the word frequency of each field in the data dictionary table, extract the second keyword from the data dictionary table, and merge the first keyword and the second keyword to obtain the target keyword; Step 104: Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and perform initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain the coarse category corresponding to each data dictionary table; Step 105: Within the same coarse category, calculate the multidimensional weighted edit distance between fields, and reclassify the data dictionary tables corresponding to the fields based on the multidimensional weighted edit distance to obtain the fine category corresponding to each data dictionary table; Step 106: Within the same subcategory, calculate the table similarity between different data dictionary tables, and merge the data dictionary tables based on the table similarity to obtain the target data dictionary.

[0026] The multi-source data dictionary fusion method provided in this application is used to fuse different data dictionaries. For example, in the field of grassroots social governance at home and abroad, the data dictionaries used by different cities may have inconsistent table designs. Therefore, the data dictionary fusion method can be used to merge the tables with different designs into a whole design to ensure that the original data dictionary is compatible with the new integrated system.

[0027] In this embodiment, at least two data dictionaries to be merged are first obtained, and data logic description information corresponding to each data dictionary to be merged is obtained. The data logic description information may include the table name, primary key, entity relationship, and tags of the data dictionary table.

[0028] This embodiment utilizes data logic description information and keyword extraction techniques for feature extraction. Specifically, the first keyword can be extracted using the data logic description information. Furthermore, the second keyword is extracted using the keyword extraction technique TF-IDF (Term Frequency-Inverse Document Frequency). TF-IDF is a commonly used weighted technique for finding keywords. It can be used to assess the importance of a field in text. By using TF-IDF to filter field keywords, words with high weight and importance in the field can be identified, thus assisting in tasks such as data analysis, search, and text mining.

[0029] After extracting keywords, the similarity of data dictionary tables is determined based on these keywords, and then data dictionary tables with high similarity are merged. Due to significant differences in presentation between data dictionaries and traditional text—traditional text is characterized by overall coherence, context relevance, repeatable fields, and numerous stop words; while data dictionaries are characterized by generally non-repeating fields and line-of-speech irrelevance—this embodiment employs a weighted edit distance algorithm to calculate similarity and achieve fusion. Specifically, a two-stage weighted edit distance calculation method can be used. The first calculation performs initial classification by calculating the two-dimensional weighted edit distance between target keywords to obtain the coarse category to which the data dictionary table corresponding to each target keyword belongs. The second calculation performs refined classification by further calculating the multi-dimensional weighted edit distance within each coarse category obtained from the first classification to obtain the fine category corresponding to each data dictionary table. Then, data dictionary tables within the same fine category are merged. For example, data dictionary tables within the same fine category can be merged, while those in different fine categories are not merged; alternatively, within the same fine category, table similarity can be further calculated, and tables with high similarity can be merged. This method reduces computational load and improves operational efficiency.

[0030] Understandably, the minimum edit distance algorithm is a classic algorithm for solving text similarity problems. Its principle is to calculate the minimum number of editing operations required to transform one string into another. For unweighted minimum edit distance, the proportion of adding, deleting, and modifying characters is the same across strings. Considering the different importance of words, a weighted minimum edit distance is used, assigning weights to different words.

[0031] The formula for weighted edit distance is as follows:

[0032] in, This represents the weighted edit distance between the first i characters of 'a' and the first j characters of 'b'. 'del' represents deletion, 'ins' represents insertion, and 'sub' represents replacement.

[0033] In practical applications, since not all dimensions are suitable for calculating similarity, different dimensions can be used for the two weighted edit distance calculations. For example, the first calculation uses two-dimensional weighted edit distance to analyze only the dimensions suitable for calculating the similarity of all tables. The second calculation uses multi-dimensional weighted edit distance to analyze the commonalities of the data dictionary tables in this coarse category based on the results of the first classification, determine the dimensions suitable for calculating the similarity of this type of data dictionary table, and then analyze these dimensions.

[0034] This embodiment proposes a data dictionary fusion method for a specific professional field, applicable to areas such as grassroots social governance both domestically and internationally. Considering the unique characteristics of the data dictionary's structure, feature extraction is achieved through both data logical description information and TF-IDF, yielding the first and second keywords respectively. The feature extraction method based on data logical description information takes into account the characteristics of the data structure, while the TF-IDF-based method considers the fact that the extracted object is text. The combination of these two methods is more suitable for the specific feature extraction scenario of keyword extraction in data dictionaries. Subsequently, the extracted keywords are used to classify the data dictionary table. A two-stage weighted edit distance algorithm is employed in the classification process, used for initial classification and subsequent classification, and then data dictionary fusion is performed within the same category. By performing two-dimensional and then multi-dimensional weighted edit distance calculations, computational efficiency is greatly improved, while avoiding the problem of introducing unsuitable dimensions for similarity calculations and resulting biases in the calculation results due to directly using weighted edit distance. Therefore, the automatic data dictionary fusion method provided in this embodiment eliminates the need for manual modification of the data dictionary design document, effectively solving problems such as high personnel costs and long cycles, and reducing labor and time costs.

[0035] Furthermore, as a refinement and extension of the specific implementation methods of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another multi-source data dictionary fusion method is provided, such as... Figure 2 As shown, the method includes the following steps: Step 201: Obtain multiple data dictionaries to be merged and the data logic description information corresponding to each data dictionary to be merged, and preprocess the data dictionaries to be merged, wherein the preprocessing includes at least entity disambiguation.

[0036] In this step, after obtaining the data dictionary to be fused and its corresponding data logical description information, the data dictionary to be fused also needs to be preprocessed to improve the accuracy and effectiveness of the fusion. Preprocessing includes entity disambiguation, which standardizes ambiguous terms in the data dictionary to be processed. It is understandable that the same real entity may have different names or aliases, such as phone number and contact information. In the community grid, both usually refer to the real entity "phone number," but they point to two different fields. Therefore, it is necessary to standardize these two entity references. Specifically, this can be achieved through algorithms based on word similarity or contextual information, pointing entities with different names to the same unique identifier, thereby eliminating entity ambiguity.

[0037] In addition, preprocessing may include data cleaning and normalization operations. For non-standard fields or words containing errors or redundancy, operations such as correcting spelling errors, removing special characters, standardizing capitalization, and resolving format inconsistencies can be performed to obtain high-quality and consistent field data. Preprocessing may also include data standardization operations, such as standardizing field units and mapping value ranges for data from different sources or in different formats. Through data standardization, the merged data becomes easier to manage and compare uniformly. Furthermore, preprocessing may include operations such as building a thesaurus. Based on relevant documents, domain knowledge, or expert opinions, a thesaurus is built to specify the relationships between words. This allows for the handling of special words or phrases through the thesaurus, ensuring that related fields in the data dictionary can be correctly identified and matched.

[0038] Step 202: Based on the data logic description information, extract the first keyword from the data dictionary table of each data dictionary to be merged. The data logic description information includes at least one of the following: table name, primary key, entity relationship, and tag of the data dictionary table. Based on the word frequency of each field in the data dictionary table, extract the second keyword from the data dictionary table and merge the first keyword and the second keyword to obtain the target keyword.

[0039] Step 203: Read the preset weight mapping information to obtain the preset weights corresponding to the target keywords; adjust the preset weights based on the data logic description information to obtain the first weights corresponding to each target keyword; calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged based on the first weights.

[0040] Step 204: Perform initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain the coarse category corresponding to each data dictionary table.

[0041] In steps 203-204, each target keyword is assigned a corresponding weight, i.e., a first weight. Then, a weighted average is performed using these first weights to obtain the two-dimensional weighted edit distance between every two target keywords. The data dictionary table is then categorized based on this two-dimensional weighted edit distance. It can be understood that the smaller the two-dimensional weighted edit distance between two target keywords, the higher the probability that they are of the same type. Based on this, a distance threshold can be pre-set, and target keywords with a two-dimensional weighted edit distance less than the threshold are extracted and their corresponding data dictionary tables are assigned to the same coarse category.

[0042] Figure 3 The diagram illustrates the coarse classification results of one embodiment of this application. As shown, the circled cells represent the target keywords assigned to the same coarse category. For example, the two-dimensional weighted edit distance between "residential address / building address code" and "ID card number" is 12, which is too large and therefore not assigned to the same coarse category. However, the two-dimensional weighted edit distance between "basic ID number" and "ID card number" is 6, which is relatively small and therefore assigned to the same coarse category. The final samples are divided into three coarse categories: residents, houses, and legal entities.

[0043] In practical applications, weight mapping information can be based on statistical weight tables or pre-set calculation rules. For example, in a community grid data dictionary, fields derived from national standards have higher weights than those derived from local standards. For instance, national standard fields such as community code, community name, and community boundary keywords have higher weights than local standard fields such as community classification, community leader, and community facilities. After determining the preset weights based on the weight mapping information, the preset information can be adjusted according to the data logic description information. For example, when faced with a two-dimensional representation of table name + primary key, the table name can be assigned 1.5 times the weight of the primary key to obtain the final first weight.

[0044] Step 205: In the same coarse category, determine whether each field in the data dictionary table is a target keyword, and obtain the design information for each field, wherein the design information includes at least the field type; determine the second weight of each field based on the judgment result and design information, and calculate the multidimensional weighted edit distance between fields based on the second weight.

[0045] Step 206: Reclassify the data dictionary tables corresponding to the fields based on the multidimensional weighted edit distance to obtain the subcategories corresponding to each data dictionary table.

[0046] In steps 205-206, each field is assigned a corresponding weight, also known as a second weight. Then, a weighted average is calculated using these second weights to obtain the multidimensional weighted edit distance between every two fields. The data dictionary table is then categorized based on this multidimensional weighted edit distance. It can be understood that the smaller the multidimensional weighted edit distance between two fields, the higher the probability that they are of the same type. Based on this, a distance threshold can be pre-set to extract fields whose multidimensional weighted edit distance is less than the threshold, and their corresponding data dictionary tables are assigned to the same sub-category. The distance threshold for the coarse category and the distance threshold for the sub-category can be the same or different. For example, if the distance weight is set to 5, a multidimensional weighted edit distance of 4.5 is sufficient to classify them as similar, thus placing them in the same sub-category.

[0047] Figure 4 This diagram illustrates the detailed classification results of one embodiment of this application. The diagram shows target keywords under the coarse resident category. The circled cells represent the target keywords assigned to the same detailed category. For example, the multidimensional weighted edit distance between "ID number in the basic population information table" and "ID number in the ID document information table" is 4.5, which is relatively small and therefore they are not assigned to the same detailed category. However, the multidimensional weighted edit distance between "ID number in the basic population information table" and "ID number in the extended resident information table" is 14.5, which is relatively large and therefore they are not assigned to the same coarse category.

[0048] In practical applications, the second weight can be determined based on whether the field is a target keyword and the field's design information. For example, the weight can be assigned according to a weight ratio of keyword: ordinary field: field type = 6:3:1.

[0049] In step 205, before calculating the multidimensional weighted edit distance between fields based on the second weight, the second weight can be adjusted. For example, an adjustment factor for each target keyword is determined, and the second weight is adjusted based on the adjustment factor, wherein the adjustment factor includes a language dimension adjustment factor and / or a text dimension adjustment factor.

[0050] In this step, due to the significant differences in presentation between long texts and data dictionary tables, weighted calculations can be applied across different dimensions to adjust the second weights from multiple perspectives. For example, a language family adjustment factor can be determined, and the second weights can be adjusted based on this factor, increasing the weight of Chinese characters and English words while decreasing the weight of numbers. Similarly, a text dimension adjustment factor can be determined, and the second weights can be adjusted based on this factor, increasing the weight of vocabulary words while decreasing the weight of short texts.

[0051] Step 207: Within the same subcategory, calculate the table similarity between different data dictionary tables.

[0052] Step 208: If the table similarity is greater than the first similarity threshold, then merge the data dictionary tables and obtain the target data dictionary based on the merged target data dictionary table; if the table similarity is less than the first similarity threshold but greater than the second similarity threshold, then extract the intersection and difference of the data dictionary tables and obtain the target data dictionary based on the intersection and difference; if the table similarity is less than the second similarity threshold, then end the fusion and generate a prompt message, wherein the prompt message is used to indicate that the data dictionary similarity is low; or, if the table similarity is less than the second similarity threshold, return to the step of obtaining the design information of each field to obtain new design information, wherein the new design information includes field type, field length and default value.

[0053] In steps 207-208, the table similarity between every two data dictionary tables in the same sub-category is determined based on the multidimensional weighted edit distance between fields, and table fusion is performed based on the sub-category results. Specifically, based on historical experience and actual application scenarios, a first similarity threshold and a second similarity threshold can be preset. The first and second similarity thresholds are used to divide the data into three different intervals, and it is determined which interval the second similarity falls into, and then the corresponding fusion method is adopted.

[0054] For example, we can set the first similarity to 70% and the second similarity to 30%. If the similarity between data dictionary table a and data dictionary table b is greater than or equal to 70%, then these two data tables are considered to have high similarity, and data dictionary tables a and b are directly merged into data dictionary table c. In this case, the change in the number of data dictionary tables is 1+1=1, meaning that the two data dictionary tables are merged, resulting in a new data dictionary table. If the similarity between data dictionary table a and data dictionary table b is between 30% and 70%, then these two data tables are considered to have some similarity. In this case, we take the intersection d of data dictionary tables a and data dictionary table b, the difference a' between a and d, and the difference b' between b and d, and use d, a', and d' as a new data dictionary table. In this case, the change in the number of data dictionary tables is 1+1=3, meaning that the two data dictionary tables are merged, resulting in 3 new data dictionary tables.

[0055] If the similarity between data dictionary table a and data dictionary table b is less than 30%, it is considered that the similarity between the two data dictionary tables is not high. At this time, the fusion operation for the two data dictionary tables can be ended directly and corresponding prompt information can be generated. Alternatively, a new design type can be obtained, such as field type, field length, and default value, and the similarity of the target keywords can be recalculated. When the similarity meets the condition, the fusion operation of the data dictionary tables can be re-executed.

[0056] Figure 5A flowchart of a data dictionary fusion process according to an embodiment of this application is shown. As shown in the figure, the overall process includes inputting the data dictionary to be fused, data dictionary data item deduplication and disambiguation, data dictionary semantic feature extraction, similarity calculation, and data dictionary fusion steps. In the data dictionary input step, multiple data dictionaries and corresponding data logic description files can be input, and the similarity algorithm to be used in the similarity calculation step can also be determined. In the data dictionary data item deduplication and disambiguation step, a pre-trained model can be used to perform specific deduplication and disambiguation operations, including constructing a thesaurus, training word vectors, nearest neighbor search, and entity disambiguation, ultimately obtaining a unified semantic entity model. In the data dictionary semantic feature extraction step, keywords are selected based on the unified semantic entity model. Keywords can be selected using the TF-IDF method or based on data logic description information such as table names, primary keys, and entities. Keywords such as relationships and tags can be selected, and two extraction methods can be combined. In the similarity calculation stage, similarity is calculated based on the extracted keywords. Keywords can be weighted and preliminarily calculated using two multi-dimensional weighted edit distances to achieve the initial classification of the data dictionary table and improve classification efficiency. Similarity is determined within the same category. If the similarity is below the threshold, no fusion operation is performed; if the similarity is above the threshold, the data dictionary fusion step is initiated. In the data dictionary fusion step, based on different similarity levels, SQL statements can be used to process the original table and construct a new data dictionary table to achieve data dictionary fusion.

[0057] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0058] Furthermore, as a specific implementation of the above-mentioned multi-source data dictionary fusion method, this application embodiment provides a multi-source data dictionary fusion apparatus, such as... Figure 6 As shown, the device includes: an acquisition module, a feature extraction module, a classification module, and a fusion module.

[0059] The acquisition module is used to acquire multiple data dictionaries to be merged and the data logic description information corresponding to each data dictionary to be merged. The feature extraction module is used to extract the first keyword from the data dictionary table of each data dictionary to be merged based on the data logic description information. The data logic description information includes at least one of the following: the table name, primary key, entity relationship, and tag of the data dictionary table; and to extract the second keyword from the data dictionary table based on the word frequency of each field in the data dictionary table, and merge the first keyword and the second keyword to obtain the target keyword. The classification module is used to calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and to perform an initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain the coarse category corresponding to each data dictionary table; and, within the same coarse category, to calculate the multi-dimensional weighted edit distance between fields, and to further classify the data dictionary tables corresponding to the fields based on the multi-dimensional weighted edit distance to obtain the fine category corresponding to each data dictionary table. The fusion module is used to calculate the table similarity between different data dictionary tables within the same subcategory, and to fuse the data dictionary tables based on the table similarity to obtain the target data dictionary.

[0060] In specific application scenarios, the classification module may optionally be used for: Read the preset weight mapping information to obtain the preset weight corresponding to the target keyword; Based on the data logic description information, the preset weights are adjusted to obtain the first weight corresponding to each target keyword; The two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged is calculated based on the first weight.

[0061] In specific application scenarios, the classification module may optionally be used for: Within the same coarse category, determine whether each field in the data dictionary table is a target keyword, and obtain the design information for each field, which includes at least the field type; Based on the judgment results and design information corresponding to each field, determine the second weight of each field, and calculate the multidimensional weighted edit distance between fields based on the second weight.

[0062] In specific application scenarios, the classification module may optionally be used for: Determine the adjustment factor for each target keyword and adjust the second weight based on the adjustment factor, wherein the adjustment factor includes a language dimension adjustment factor and / or a text dimension adjustment factor.

[0063] In specific application scenarios, the fusion module may optionally be used for: If the table similarity is greater than the first similarity threshold, the data dictionary tables are merged, and the target data dictionary is obtained based on the merged target data dictionary table. If the table similarity is less than the first similarity threshold but greater than the second similarity threshold, then the intersection and difference of the data dictionary tables are extracted, and the target data dictionary is obtained based on the intersection and difference.

[0064] In specific application scenarios, optionally, the target data dictionary can be obtained by fusing data dictionary tables based on table similarity, and may also include: If the table similarity is less than the second similarity threshold, the fusion process ends and a prompt message is generated, indicating that the data dictionary similarity is low; or, If the table similarity is less than the second similarity threshold, return to the step of obtaining the design information for each field to obtain new design information, which includes field type, field length, and default value.

[0065] In specific application scenarios, the device may optionally include a preprocessing module for: The data dictionary to be merged is preprocessed, including at least entity disambiguation.

[0066] According to another aspect of this application, a medium is provided that stores a program or instructions thereon, which, when executed by a processor, implement the above-described multi-source data dictionary fusion method.

[0067] It should be noted that other corresponding descriptions of the functional modules involved in the multi-source data dictionary fusion device provided in this application embodiment can be found in the corresponding descriptions in the above method, and will not be repeated here.

[0068] Based on the above method, the present application also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described multi-source data dictionary fusion method.

[0069] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause an electronic device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0070] Based on the above, Figures 1 to 5 The method shown, and Figure 6 To achieve the above objectives, the present application also provides a device, specifically a personal computer, server, network device, etc., as shown in the virtual device embodiment. This electronic device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figures 1 to 5 The multi-source data dictionary fusion method is shown.

[0071] Optionally, the electronic device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0072] Those skilled in the art will understand that the electronic device structure provided in this embodiment does not constitute a limitation on the electronic device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0073] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of the electronic device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between various controls within the storage medium, as well as communication with other hardware and software within the physical device.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0075] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the units or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the units in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be located in one or more apparatuses different from this embodiment, with corresponding changes. The units of the above-described embodiment can be combined into one unit, or further divided into multiple sub-units.

[0076] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for fusing multi-source data dictionaries, characterized in that, The method includes: Obtain multiple data dictionaries to be merged and data logic description information corresponding to each of the data dictionaries to be merged; Based on the data logic description information, the first keyword is extracted from the data dictionary table of each of the data dictionaries to be merged, wherein the data logic description information includes at least one of the following: the table name, primary key, entity relationship, and tag of the data dictionary table; Based on the word frequency of each field in the data dictionary table, a second keyword is extracted from the data dictionary table, and the first keyword and the second keyword are combined to obtain the target keyword; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and perform an initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain the coarse category corresponding to each data dictionary table; Within the same coarse category, the multidimensional weighted edit distance between the fields is calculated, and the data dictionary tables corresponding to the fields are reclassified based on the multidimensional weighted edit distance to obtain the fine category corresponding to each data dictionary table; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, including: Read the preset weight mapping information to obtain the preset weight corresponding to the target keyword. The weight mapping information is based on a statistical weight table or a preset calculation rule. Based on the data logic description information, the preset weights are adjusted to obtain the first weight corresponding to each target keyword; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be fused based on the first weight; Within the same subcategory, the table similarity between different data dictionary tables is calculated, and the data dictionary tables are merged based on the table similarity to obtain the target data dictionary.

2. The method according to claim 1, characterized in that, Within the same coarse category, the multidimensional weighted edit distance between the fields is calculated, including: Within the same coarse category, it is determined whether each field in the data dictionary table is the target keyword, and the design information of each field is obtained, wherein the design information includes at least the field type; Based on the judgment result corresponding to each field and the design information, the second weight of each field is determined, and the multidimensional weighted edit distance between the fields is calculated based on the second weight.

3. The method according to claim 2, characterized in that, Before calculating the multidimensional weighted edit distance between the fields based on the second weight, the method further includes: Determine the adjustment factor for each target keyword, and adjust the second weight according to the adjustment factor, wherein the adjustment factor includes a language dimension adjustment factor and / or a text dimension adjustment factor.

4. The method according to claim 3, characterized in that, The target data dictionary is obtained by fusing the data dictionary tables based on the table similarity, including: If the similarity of the tables is greater than the first similarity threshold, then the data dictionary tables are merged, and the target data dictionary is obtained based on the merged target data dictionary table; If the similarity of the tables is less than the first similarity threshold but greater than the second similarity threshold, then the intersection and difference of the data dictionary tables are extracted, and the target data dictionary is obtained based on the intersection and the difference.

5. The method according to claim 4, characterized in that, The target data dictionary is obtained by fusing the data dictionary tables based on the table similarity, and also includes: If the table similarity is less than the second similarity threshold, the fusion process ends and a prompt message is generated, wherein the prompt message indicates that the data dictionary similarity is low; or, If the table similarity is less than the second similarity threshold, then return to the step of obtaining the design information of each field to obtain new design information, wherein the new design information includes field type, field length and default value.

6. The method according to claim 1, characterized in that, After acquiring multiple data dictionaries to be merged, the method further includes: The data dictionary to be merged is preprocessed, wherein the preprocessing includes at least entity disambiguation.

7. A multi-source data dictionary fusion device, characterized in that, The device includes: The acquisition module is used to acquire multiple data dictionaries to be merged and data logic description information corresponding to each of the data dictionaries to be merged. The feature extraction module is used to extract a first keyword from the data dictionary table of each of the data dictionaries to be merged, based on the data logic description information, wherein the data logic description information includes at least one of the following: the table name, primary key, entity relationship, and tag of the data dictionary table; and to extract a second keyword from the data dictionary table based on the word frequency of each field in the data dictionary table, and to merge the first keyword and the second keyword to obtain the target keyword. The classification module is used to calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, and to perform an initial classification of the data dictionary tables corresponding to the target keywords based on the two-dimensional weighted edit distance to obtain a coarse category for each data dictionary table; and, within the same coarse category, to calculate the multi-dimensional weighted edit distance between the fields, and to further classify the data dictionary tables corresponding to the fields based on the multi-dimensional weighted edit distance to obtain a fine category for each data dictionary table; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be merged, including: Read the preset weight mapping information to obtain the preset weight corresponding to the target keyword. The weight mapping information is based on a statistical weight table or a preset calculation rule. Based on the data logic description information, the preset weights are adjusted to obtain the first weight corresponding to each target keyword; Calculate the two-dimensional weighted edit distance between target keywords in different data dictionaries to be fused based on the first weight; The fusion module is used to calculate the table similarity between different data dictionary tables within the same subcategory, and to fuse the data dictionary tables based on the table similarity to obtain the target data dictionary.

8. A storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the method as described in any one of claims 1 to 6.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1 to 6.

Images