Data mapping method and electronic equipment

Abstract

The invention discloses a data mapping method and electronic equipment, which are used for improving the efficiency and accuracy of data mapping. The method comprises the steps of generating a source text according to first attribute information of a source data set, and generating a target text according to first attribute information of a target data set; and establishing a mapping relationship between the source data set and the target data set according to the similarity between the source text and the target text.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data mapping method and electronic equipment. Background Art

[0002] In the field of big data, the ETL (Extract-Transform-Load) process is the process of data integration, cleaning, conversion and loading, and is an important step in data processing in data warehouses and data lakes. It is usually used to extract data from various data sources, clean and transform it, and then load the data into a data warehouse or data lake. In the ETL process, it is necessary to establish an association relationship between source data elements and corresponding elements in the target data warehouse, that is, to establish a mapping relationship between non-standard source data and data in the target data warehouse that meets industry standards, which is called data mapping.

[0003] In a big data environment, data mapping is particularly important because it directly affects the efficiency of data integration, the reliability and accuracy of data analysis. Existing technical solutions usually perform manual configuration when performing data mapping, and only focus on the mapping of a single source data element. Manual mapping is required for each data element in the database, resulting in low efficiency and low accuracy of data mapping. Summary of the invention

[0004] The present invention provides a data mapping method and electronic equipment, which are used to improve the efficiency and accuracy of data mapping.

[0005] In a first aspect, an embodiment of the present invention provides a data mapping method, the method comprising:

[0006] Generate a source text according to the first attribute information of the source data set, and generate a target text according to the first attribute information of the target data set;

[0007] According to the similarity between the source text and the target text, a mapping relationship between the source data set and the target data set is established.

[0008] The data mapping method provided in this embodiment uses a text generation method to determine the similarity between data sets by comparing the similarity between texts, thereby automatically realizing the mapping between data sets, improving the efficiency of data mapping, and saving labor costs.

[0009] As an optional implementation manner, the first attribute information includes data set attributes and / or data item attributes; the source text and the target text are generated by any one or more of the following methods:

[0010] Generate a source text according to at least one data set attribute of a source data set, and generate a target text according to at least one data set attribute of a target data set; or,

[0011] A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

[0012] This embodiment generates text through data set attributes or data item attributes, and can combine the attribute information of the data set into the text, so as to determine the mapping relationship between different data sets by comparing the similarity of the text, thereby improving the efficiency and accuracy of data set mapping.

[0013] As an optional implementation, the generating of the source text according to at least one data set attribute of the source data set, and the generating of the target text according to at least one data set attribute of the target data set, include:

[0014] Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

[0015] This embodiment can also concatenate multiple data set attributes into one text, thereby generating text by combining different attributes of the data set, making the comparison between texts more accurate, that is, improving the accuracy of similarity calculation between source text and target file, thereby improving the accuracy of data mapping.

[0016] As an optional implementation, generating the source text according to at least one data item attribute of the source data set, and generating the target text according to at least one data item attribute of the target data set, includes:

[0017] For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0018] This embodiment can also splice the data item attributes of the data items in the data set, thereby generating text in combination with the attribute information of the data items in the data set, making the similarity calculation between texts more accurate, and can improve the accuracy of the similarity calculation between the source text and the target file, thereby improving the accuracy of data mapping.

[0019] As an optional implementation manner, establishing a mapping relationship between a source data set and a target data set according to the similarity between the source text and the target text includes:

[0020] Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text;

[0021] A mapping relationship between a source data set and at least one matching target data set is established.

[0022] This embodiment establishes a mapping relationship between the source data set and the target data set by comparing the similarity between the source text and the target text, generates text in combination with the attribute information of the data set, and establishes a mapping relationship between different data sets by comparing the similarities between different texts, thereby improving mapping efficiency.

[0023] As an optional implementation, the method further includes:

[0024] Generate data item text according to the second attribute information of the data item of the source data set;

[0025] Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set;

[0026] According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0027] This embodiment can also automatically establish a mapping relationship between data items in different data sets, generate text in combination with the second attribute information of the data item, include the attribute information of the data item in the text, and establish a mapping relationship between different data items by comparing the similarities between the texts.

[0028] As an optional implementation manner, the second attribute information includes at least one data item attribute; the data item text and the target data item text are generated by any one or more of the following methods:

[0029] generating a data item text according to at least one data item attribute of a source data set;

[0030] A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

[0031] This embodiment integrates the data item attributes into the data item text, so that the generated data item text contains the attribute information of the data item. By comparing the texts of different data items and establishing the mapping relationship between different data items, the efficiency of data mapping can be improved.

[0032] As an optional implementation manner, the second attribute information includes a data item name; the method further includes:

[0033] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0034] This embodiment directly establishes a mapping relationship between data items with the same name by comparing the names of the data items, without the need to perform a similarity matching operation, thereby simplifying the operation and improving data matching efficiency.

[0035] In a second aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0036] Generate a source text according to the first attribute information of the source data set, and generate a target text according to the first attribute information of the target data set;

[0037] According to the similarity between the source text and the target text, a mapping relationship between the source data set and the target data set is established.

[0038] As an optional implementation manner, the first attribute information includes data set attributes and / or data item attributes; the processor is specifically configured to generate the source text and the target text in any one or more of the following ways:

[0039] Generate a source text according to at least one data set attribute of a source data set, and generate a target text according to at least one data set attribute of a target data set; or,

[0040] A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

[0041] As an optional implementation, the processor is specifically configured to execute:

[0042] Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

[0043] As an optional implementation, the processor is specifically configured to execute:

[0044] For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0045] As an optional implementation, the processor is specifically configured to execute:

[0046] Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text;

[0047] A mapping relationship between a source data set and at least one matching target data set is established.

[0048] As an optional implementation, the processor is further configured to execute:

[0049] Generate data item text according to the second attribute information of the data item of the source data set;

[0050] Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set;

[0051] According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0052] As an optional implementation, the second attribute information includes at least one data item attribute; the processor is specifically configured to generate the data item text and the target data item text in any one or more of the following ways:

[0053] generating a data item text according to at least one data item attribute of a source data set;

[0054] A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

[0055] As an optional implementation manner, the second attribute information includes a data item name; and the processor is further configured to execute:

[0056] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0057] In a third aspect, an embodiment of the present invention further provides a data mapping device, the device comprising:

[0058] A text generation module, used to generate a source text according to the first attribute information of the source data set, and to generate a target text according to the first attribute information of the target data set;

[0059] The mapping establishment module is used to establish a mapping relationship between the source data set and the target data set according to the similarity between the source text and the target text.

[0060] As an optional implementation manner, the first attribute information includes data set attributes and / or data item attributes; the text generation module is specifically used to generate the source text and the target text in any one or more of the following ways:

[0061] Generate a source text according to at least one data set attribute of a source data set, and generate a target text according to at least one data set attribute of a target data set; or,

[0062] A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

[0063] As an optional implementation, the text generation module is specifically used for:

[0064] Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

[0065] As an optional implementation, the text generation module is specifically used for:

[0066] For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0067] As an optional implementation manner, the mapping establishment module is specifically used for:

[0068] Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text;

[0069] A mapping relationship between a source data set and at least one matching target data set is established.

[0070] As an optional implementation, the device further includes a data item mapping module specifically configured to:

[0071] Generate data item text according to the second attribute information of the data item of the source data set;

[0072] Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set;

[0073] According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0074] As an optional implementation manner, the second attribute information includes at least one data item attribute; the data item mapping module is specifically used to generate the data item text and the target data item text by any one or more of the following methods:

[0075] generating a data item text according to at least one data item attribute of a source data set;

[0076] A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

[0077] As an optional implementation manner, the second attribute information includes a data item name; and the data item mapping module is further configured to:

[0078] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0079] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of any one of the methods described in the first aspect above.

[0080] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes any one of the methods described in the first aspect.

[0081] These and other aspects of the present application will be more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0083] Figure 1 A data mapping method implementation flow chart provided in an embodiment of the present invention;

[0084] Figure 2 A schematic diagram of a data set mapping process flow provided by an embodiment of the present invention;

[0085] Figure 3 A flowchart of data item text generation provided by an embodiment of the present invention;

[0086] Figure 4 A flowchart of establishing a mapping relationship of data items provided by an embodiment of the present invention;

[0087] Figure 5 A flowchart of establishing a mapping relationship through data item names provided by an embodiment of the present invention;

[0088] Figure 6 A flowchart of hierarchical mapping of data sets and data items provided in an embodiment of the present invention;

[0089] Figure 7 A schematic diagram of an electronic device provided by an embodiment of the present invention;

[0090] Figure 8 A schematic diagram of a data mapping device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] In the embodiments of the present invention, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0093] In the embodiment of the present invention, the term "text similarity" refers to the degree of similarity between two or more entities (words, short texts, documents), which can also be understood as the commonalities or differences between texts. The greater the commonalities and the smaller the differences, the higher the similarity. When two text fragments are exactly the same, the similarity is the highest.

[0094] The term "multithreading" in the embodiments of the present invention refers to a technology that implements concurrent execution of multiple threads from software or hardware.

[0095] The term "data set" in the embodiment of the present invention is a finite set of data item values.

[0096] The term "source text" in the embodiment of the present invention refers to the text used to find the mapping.

[0097] The term "target text" in the embodiment of the present invention refers to text used to match the source text.

[0098] The term "source data set" in the embodiment of the present invention refers to a data set used to find a mapping relationship.

[0099] The term "target data set" in the embodiment of the present invention refers to a data set that can form a mapping relationship with a source data set.

[0100] The term "mapping" in the embodiment of the present invention refers to the existence of a mapping relationship between a data item in a data set and a data item in another set, and the mapping relationship indicates that the two data items have the same business meaning.

[0101] The application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0102] Before introducing the data mapping method provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail below.

[0103] In the field of big data, the ETL (Extract-Transform-Load) process is the process of data integration, cleaning, conversion and loading, and is an important step in data processing in data warehouses and data lakes. It is usually used to extract data from various data sources, clean and transform it, and then load the data into a data warehouse or data lake. In the ETL process, it is necessary to establish an association relationship between source data elements and corresponding elements in the target data warehouse, that is, to establish a mapping relationship between non-standard source data and data in the target data warehouse that meets industry standards, which is called data mapping.

[0104] In a big data environment, data mapping is particularly important because it directly affects the efficiency of data integration, the reliability and accuracy of data analysis. Existing technical solutions usually perform manual configuration when performing data mapping, and only focus on the mapping of a single source data element. Manual mapping is required for each data element in the database, resulting in low efficiency and low accuracy of data mapping.

[0105] In order to solve the above technical problems, an embodiment of the present invention provides a data mapping method. The core idea is to use the first attribute information of the data set to generate the text of the source data set and the target data set respectively, and establish a mapping relationship between the source data set and the target data set according to the similarity between the texts of the source data set and the target data set respectively. The mapping between data sets can be automatically performed, the efficiency of data mapping can be improved, and labor costs can be saved.

[0106] like Figure 1 As shown, this embodiment provides a data mapping method, and the specific implementation process of the method is as follows:

[0107] Step 100: Generate a source text according to the first attribute information of the source data set, and generate a target text according to the first attribute information of the target data set;

[0108] Optionally, the first attribute information in this embodiment includes but is not limited to at least one of a data set attribute and a data item attribute. Optionally, the data set attribute includes but is not limited to at least one of a data set name, a data set code, and a data set description information. Optionally, the data item attribute includes but is not limited to at least one of a data item name, a data item code, and a data item description information.

[0109] In implementation, data set attributes include but are not limited to data set name, data set code, and data set description information. This embodiment provides an example of data set attribute fields, as shown in Table 1 below. Data set attributes include multiple fields, namely name, code, and data set description information, and each field includes a field name, a field description, and a field type.

[0110] Table 1 Dataset attribute fields

[0111]

[0112] The role of a data set may refer to the role of the data stored in the data set.

[0113] Optionally, the present embodiment can generate source text according to one first attribute information of the source data set, or can generate source text according to multiple first attribute information of the source data set. Similarly, the present embodiment can generate target text according to one first attribute information of the target data set, or can generate target text according to multiple first attribute information of the target data set. In implementation, multiple source texts can be generated in different ways, and multiple target texts can be generated in different ways. It should be noted that when comparing the similarity between the source text and the target text, the source text and the target text generated in the same way are compared.

[0114] In some embodiments, the first attribute information includes data set attributes and / or data item attributes; the source text and the target text are generated by any one or more of the following methods:

[0115] Mode 1: generating a source text according to at least one data set attribute of a source data set, and generating a target text according to at least one data set attribute of a target data set;

[0116] In implementation, the source text and the target text may be generated by any one or a combination of the following methods:

[0117] Method 1a: generating a source text according to multiple data set attributes of a source data set, and generating a target text according to multiple data set attributes of a target data set;

[0118] Optionally, multiple dataset attributes of the source dataset are concatenated to obtain the source text, and multiple dataset attributes of the target dataset are concatenated to obtain the target text.

[0119] During implementation, the source text is generated according to the dataset name, dataset code and dataset description information of the source dataset, and the target text is generated according to the dataset name, dataset code and dataset description information of the target dataset.

[0120] For example, the dataset name, dataset code, and dataset description information of the source dataset may be concatenated to obtain the source text, and the dataset name, dataset code, and dataset description information of the target dataset may be concatenated to obtain the target text.

[0121] Mode 1b: generating source text according to a data set attribute of the source data set, and generating target text according to a data set attribute of the target data set.

[0122] For example, the dataset name of the source dataset can be used as the source text, and the dataset name of the target dataset can be used as the target text; or, the dataset code of the source dataset can be used as the source text, and the dataset code of the target dataset can be used as the target text; or, the dataset description information of the source dataset can be used as the source text, and the dataset description information of the target dataset can be used as the target text.

[0123] Mode 2: Generate source text according to at least one data item attribute of the source data set, and generate target text according to at least one data item attribute of the target data set.

[0124] In implementation, data item attributes include, but are not limited to, data item name, data item code, and data item description information. This embodiment provides an example of a data item attribute field, as shown in Table 2 below. The data item attribute includes multiple fields, namely, data item name, data item code, and data item description information, and each field includes a field name, a field description, and a field type.

[0125] Table 2 Data item attribute fields

[0126]

[0127]

[0128] In implementation, the source text and the target text may be generated by any one or a combination of the following methods:

[0129] Mode 2a: generating source text according to a data item attribute of a source data set, and generating target text according to a data item attribute of a target data set;

[0130] Optionally, for a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0131] In implementation, the data item names of the data items of the source data set may be concatenated to obtain the source text, and the data item names of the data items of the target data set may be concatenated to obtain the target text. The data item description information of the data items of the source data set may be concatenated to obtain the source text, and the data item description information of the data items of the target data set may be concatenated to obtain the target text. The data item codes of the data items of the source data set may be concatenated to obtain the source text, and the data item codes of the data items of the target data set may be concatenated to obtain the target text.

[0132] Mode 2b: Generate source text according to multiple data item attributes of a source data set, and generate target text according to multiple data item attributes of a target data set.

[0133] In implementation, the data item name and data item description information of each data item of the source data set may be concatenated to obtain the source text, and the data item name and data item description information of each data item of the target data set may be concatenated to obtain the target text.

[0134] Optionally, this embodiment may determine the type of the generated source text or target text according to the elements in the first attribute information required to generate the source text or target text. For example, the type of the source text or target text is as follows:

[0135] Type 1: text obtained by concatenating the dataset name, dataset code, and dataset description information;

[0136] Type 2: text obtained by concatenating the data item names of each data item in the data set;

[0137] Type 3: text obtained by concatenating the data item description information of each data item in the data set.

[0138] Step 101: Establish a mapping relationship between a source data set and a target data set according to the similarity between the source text and the target text.

[0139] In the implementation, calculating the similarity between the source text and the target text specifically includes: calculating the similarity between the source text and the target text of the same type. The source text and the target text generated by the same first attribute information belong to the same type, that is, the elements contained in the first attribute information of the source data set and the first attribute information of the target data set required when generating the source text or the target text are consistent, for example, the first attribute information of the source data set used when generating the source text includes the data set name, data set code and data set description information, and the first attribute information of the target data set used when generating the target text includes the data set name, data set code and data set description information.

[0140] In some embodiments, this embodiment establishes a mapping relationship between a source data set and a target data set according to the similarity between the source text and the target text in the following manner:

[0141] According to the similarity between the source text and the target text, at least one target data set that matches the source data set is determined; and a mapping relationship between the source data set and the at least one matching target data set is established.

[0142] It should be noted that the target data set that matches the source data set in this embodiment may be one or more. When calculating the similarity between the source text and the target text, the similarity between the source text and the target text of the same type is calculated. When there are multiple types of source texts and target texts, multiple independent threads can be used for parallel calculation, and one thread calculates the similarity between the source text and the target text of the same type.

[0143] Optionally, when calculating the similarity between multiple types of source texts and target texts, multiple similarities corresponding to the multiple types are obtained, the average of the multiple similarities is calculated, the average is determined as the similarity score of the current target data set, the top N target data sets with the highest similarity scores are selected as target data sets that match the source data sets, and a mapping relationship between the source data sets and the matched target data sets is established.

[0144] Optionally, when calculating the similarity between multiple types of source texts and target texts, multiple similarities corresponding to the multiple types are obtained, and weighted summation is performed on the corresponding similarities using the weights corresponding to the multiple source texts to obtain a weighted sum value, and the weighted sum value is determined as the similarity score of the current target data set, and the first N target data sets with the highest similarity scores are selected as target data sets matched by the source data sets, and a mapping relationship between the source data sets and the matched target data sets is established. N is an integer greater than or equal to 1.

[0145] like Figure 2 As shown, this embodiment provides a schematic diagram of a data set mapping process flow, wherein the specific process is as follows:

[0146] Step 200: Obtain first attribute information of a source data set;

[0147] Step 201a, concatenate the dataset name, dataset code and dataset description information of the source dataset to obtain source text A1;

[0148] Step 201b, concatenate the data item names of the data items of the source data set to obtain source text A2;

[0149] Step 201c, concatenating the data item description information of each data item of the source data set to obtain source text A3;

[0150] Step 202: traverse the target data set to obtain the first attribute information of the current target data set;

[0151] Step 203a, concatenate the dataset name, dataset code and dataset description information of the target dataset to obtain the target text B1;

[0152] Step 203b, concatenate the data item names of the data items of the target data set to obtain the target text B2;

[0153] Step 203c, concatenating the data item description information of each data item of the target data set to obtain the target text B3;

[0154] Step 204a, calculating the similarity between the source text A1 and the target text B1 to obtain a similarity value AB1;

[0155] Step 204b, calculating the similarity between the source text A2 and the target text B2 to obtain a similarity value AB2;

[0156] Step 204c, calculating the similarity between the source text A3 and the target text B3 to obtain a similarity value AB3;

[0157] In implementation, the SimHash algorithm may be used to calculate the similarity between different texts, or other algorithms may be used, and this embodiment does not impose too many limitations on this.

[0158] Step 205, calculating the average value of AB1, AB2 and AB3;

[0159] Step 206, record the average value;

[0160] Step 207, determine whether the target data set has been traversed, if yes, execute step 208, otherwise execute step 202;

[0161] Step 208: Determine the target data set corresponding to the largest first N average values ​​as the target data set mapped by the source data set.

[0162] In some embodiments, after determining the mapping relationship between the source data set and the target data set, that is, after determining the target data set mapped by the source data set, the present embodiment may also perform mapping processing on the data items between the two data sets. The specific execution process is as follows:

[0163] Process 1) Generate data item text according to the second attribute information of the data item of the source data set;

[0164] Optionally, the second attribute information in this embodiment includes at least one data item attribute; the data item attributes include but are not limited to: at least one of a data item name, a data item code, and a data item description information.

[0165] Process 2) Generate a target data item text according to the second attribute information of the data item of the target data set that has a mapping relationship with the source data set;

[0166] In some embodiments, the second attribute information in this embodiment includes at least one data item attribute; the data item text and the target data item text are generated by any one or more of the following methods:

[0167] A data item text is generated according to at least one data item attribute of a source data set; and a target data item text is generated according to at least one data item attribute of a target data set that has a mapping relationship with the source data set.

[0168] Optionally, a data item text is generated according to a data item attribute of a source data set; a target data item text is generated according to a data item attribute of a target data set that has a mapping relationship with the source data set; or,

[0169] A data item text is generated according to multiple data item attributes of a source data set; and a target data item text is generated according to multiple data item attributes of a target data set that has a mapping relationship with the source data set.

[0170] It should be noted that when comparing the similarity between a data item text and a target data item text, data item text and target data item text of the same type are compared.

[0171] Optionally, this embodiment may determine the types of the generated data item text and the target data item text according to the elements in the second attribute information required to generate the data item text and the target data item text. For example, the types of the data item text and the target data item text are as follows:

[0172] Type 1: Text obtained by concatenating the data item name and data item code of each data item;

[0173] Type 2: Text generated based on the data item description information of each data item.

[0174] In implementation, the data item text and the target data item text may be generated by any one or a combination of the following methods:

[0175] Method a: Generate data item text according to data item description information of each data item of the source data set; generate target data item text according to data item description information of each data item of the target data set that has a mapping relationship with the source data set.

[0176] Method b: concatenate the data item name and data item code of each data item in the source data set to obtain the data item text, and concatenate the data item name and data item code of each data item in the target data set that has a mapping relationship with the source data set to obtain the target data item text.

[0177] Process 3) According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0178] In implementation, calculating the similarity between the data item text and the target data item text specifically includes: calculating the similarity between the data item text and the target data item text of the same type. The data item text and the target data item text generated by the same second attribute information belong to the same type, that is, the elements contained in the second attribute information of the source data set and the second attribute information of the target data set required when generating the data item text and the target data item text are consistent, for example, the second attribute information of the source data set used when generating the data item text includes the data item name and the data item code, and the second attribute information of the target data set used when generating the target data item text includes the data item name and the data item code.

[0179] In some embodiments, this embodiment establishes a mapping relationship between data items in a source data set and a target data set according to the similarity between the data item text and the target data item text in the following manner:

[0180] According to the similarity between the data item text and the target data item text, at least one target data item matching the data item of the source data set is determined; and a mapping relationship between the data item of the source data set and the matching at least one target data item is established.

[0181] It should be noted that the target data item for data item matching in this embodiment may be one or more. When calculating the similarity between different types of data item texts and the target data item texts, multiple independent threads may be used for parallel calculation, with one thread calculating the similarity between one type of data item text and the target data item text.

[0182] In implementation, when traversing the data items in the data set for comparison, the similarities between different types of texts are compared each time, and each type of text corresponds to a separate data item text set. The element fields in the data item text set are shown in Table 3. When the target data items that match the data items in the source data set are screened out from different types of text, the output final matching text set list is shown in Table 4.

[0183] Table 3 Fields of the data item text collection

[0184]

[0185] Table 4 Fields of the target data item text set for data item text matching

[0186]

[0187]

[0188] like Figure 3 As shown, this embodiment provides a process for generating data item text, which is specifically as follows:

[0189] Step 300, traverse the data items of the data set;

[0190] Optionally, the dataset includes but is not limited to a source dataset or a target dataset.

[0191] Step 301: Obtain second attribute information of a data item of a data set;

[0192] The second attribute information includes but is not limited to the data item name, data item code, and data item description information.

[0193] Step 302a, concatenate the data item name and the data item code of the data item to obtain the data item text A1;

[0194] Step 302b, determining the description information of the data item as data item text A2;

[0195] Step 303a, entering the data item text A1 into the text set of type 1;

[0196] Step 303b, entering the data item text A2 into the text set of type 2;

[0197] Step 304, determine whether the data item has been traversed, if yes, execute step 305, otherwise execute step 301;

[0198] Step 305: Output the text set list.

[0199] The text set list includes different types of data item texts corresponding to each data item.

[0200] This embodiment can determine the data item text of the source data set and the target data item text of the target data set in the above manner.

[0201] In some embodiments, after determining the text set, the similarity between the data item text and the target data item text can be calculated using the SimHash algorithm. When the data item text and the target data item text include multiple types, the similarity between the data item text and the target data item text under each type can be calculated to obtain the similarities under multiple types, and the average of multiple similarities is calculated. For each data item text, a target data item text with the highest average value is selected, thereby establishing a mapping relationship between the data item and the target data item.

[0202] Optionally, after determining the text set, the similarity between the data item text and the target data item text can be calculated using the SimHash algorithm. When the data item text and the target data item text include multiple types, the similarity between the data item text and the target data item text under each type can be calculated to obtain the similarities under multiple types. The similarities under different types are weighted and summed using different types of weights to obtain a weighted sum value. For each data item text, a target data item text with the highest weighted sum value is selected to establish a mapping relationship between the data item and the target data item.

[0203] In some embodiments, the comparison loop process can also be simplified by a fork-join algorithm, where each data item of a source data set selects the data item with the highest text similarity to its own data item as the target data item for mapping the data item of the source data set, thereby forming a mapping relationship between the data item of the source data set and the data item of the target data set. The goal of the fork-join algorithm is to decompose a large task into multiple small tasks, and execute these small tasks simultaneously on multiple processors, and finally merge their results to obtain the final result.

[0204] like Figure 4 As shown, this embodiment also provides a process for establishing a mapping relationship of data items, which is specifically as follows:

[0205] Step 400: Obtain a data item text set of data items in a source data set;

[0206] The data item text set includes multiple types of data item texts for each data item;

[0207] Step 401, determine whether the data item text set of the source data set is greater than a threshold, if so, execute step 402, otherwise execute step 403;

[0208] Step 402, split the data item text set into halves, and execute step 400;

[0209] Splitting the data item text set in half can break down large tasks into small tasks and reduce the amount of computation.

[0210] Step 403: Obtain a data item text set of data items in the target data set;

[0211] Step 404, determine whether the data item text set of the target data set is greater than a threshold, if so, execute step 405, otherwise execute step 406;

[0212] Step 405, split the data item text set into halves and execute step 403;

[0213] Step 406: Select two types of data item texts generated by data items of the source data set;

[0214] Step 407, selecting two types of target data item texts generated by data items of the target data set;

[0215] Step 408: Calculate the similarity between each type of data item text and the target data item text respectively through two independent threads to obtain two types of similarities;

[0216] Step 409, determine whether the target data item text has been selected, if yes, execute step 410, otherwise execute step 407;

[0217] Step 410, determine whether the data item text is selected, if yes, execute step 411, otherwise execute step 406;

[0218] Step 411, calculating the similarity between each data item text and each target data item text;

[0219] Step 412: Select the target data item corresponding to the target data item text having the highest similarity to the data item text of each data item;

[0220] Step 413: Generate a mapping relationship between the data item and the target data item.

[0221] In some embodiments, the second attribute information of this embodiment includes a data item name; the method further includes:

[0222] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0223] During implementation, after obtaining one or more target data sets mapped to the source data set, this embodiment can directly establish a mapping relationship between data items with the same name based on the data item names of the source data set and the target data set. At this time, there is no need to perform similarity calculation, and the mapping relationship can be directly established to save computing scale.

[0224] like Figure 5 As shown, this embodiment provides a process for establishing a mapping relationship through data item names, which is specifically as follows:

[0225] Step 500: Obtain second attribute information of a data item in a source data set;

[0226] Wherein, the second attribute information includes the data item name;

[0227] Step 501: determine a data item set of a source data set;

[0228] Step 502, determine whether the number of data items in the data item set is greater than a threshold, if yes, execute step 503, otherwise execute step 504;

[0229] Step 503, split the data item set into half, and execute step 501;

[0230] Step 504: Obtain second attribute information of the data item in the target data set;

[0231] Step 505: determine a data item set of the target data set;

[0232] Step 506, determine whether the number of data items in the data item set is greater than a threshold, if yes, execute step 507, otherwise execute step 508;

[0233] Step 507, split the data item set into halves and execute step 505;

[0234] Step 508: Select data items of the source data set;

[0235] Step 509: Select data items of the target data set;

[0236] Step 510: Determine, through an independent thread, whether the data item names of the source data set and the target data set are the same. If yes, execute step 511; otherwise, execute step 512.

[0237] Step 511, record the judgment result;

[0238] Step 512, determine whether the data item set of the target data set has been selected, if so, execute step 513, otherwise execute step 509;

[0239] Step 513, determine whether the data item set of the source data set has been selected, if so, execute step 514, otherwise execute step 508;

[0240] Step 514: output the judgment result, and integrate the data items with the same data item names in the source data set and the target data set;

[0241] Step 515: Establish a mapping relationship between data items with the same name in the source data set and the target data set.

[0242] like Figure 6As shown, this embodiment provides a hierarchical mapping process of data sets and data items, wherein, in the data set recommendation stage, it is used to generate a source text according to the first attribute information of the source data set, and to generate a target text according to the first attribute information of the target data set; calculate the similarity between the source text and the target text; establish a mapping relationship between the source data set and the target data set according to the similarity between the source text and the target text, and determine multiple target data sets that have a mapping relationship with the source data set. In the data item recommendation stage, it is used to determine whether the data item names of the source data set and the target data set are the same, and if the data item names are the same, establish a mapping relationship between the data items of the same name in the source data set and the target data set. Generate a data item text according to at least one data item attribute of the source data set; generate a target data item text according to at least one data item attribute of the target data set that has a mapping relationship with the source data set. Calculate the similarity between the data item text and the target data item text, establish a mapping relationship between the data items in the source data set and the target data set according to the similarity, and finally obtain a mapping relationship between the data items in the source data set and the target data set.

[0243] This embodiment can simplify the configuration process of dataset recommendation and effectively improve the performance of dataset recommendation. It supports mapping recommendations for larger datasets, makes more reasonable use of the multi-threading mechanism and splits the task to reduce the complexity of the calculation task, so that the mapping recommendation task for larger datasets can be calculated faster. It recommends dataset mapping more accurately, recommends datasets in a hierarchical manner, first recommends the mapping of the dataset and then recommends the mapping of the data items in the dataset, and includes the descriptive information of the dataset in the calculation scope. It simplifies some obvious mapping relationship recommendation operations. Since the data items are short texts, there is a high probability that the data item names of the source dataset and the data item names of the target dataset may have the same value. By simplifying the similarity operation, the operation time can be saved and the operation speed can be accelerated.

[0244] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device. Since the electronic device is the electronic device in the method in the embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0245] like Figure 7 As shown, the electronic device includes a processor 700 and a memory 701, wherein the memory 701 is used to store a program executable by the processor 700, and the processor 700 is used to read the program in the memory 701 and perform the following steps:

[0246] Generate a source text according to the first attribute information of the source data set, and generate a target text according to the first attribute information of the target data set;

[0247] According to the similarity between the source text and the target text, a mapping relationship between the source data set and the target data set is established.

[0248] As an optional implementation manner, the first attribute information includes data set attributes and / or data item attributes; the processor 700 is specifically configured to generate the source text and the target text in any one or more of the following ways:

[0249] Generate a source text according to at least one data set attribute of a source data set, and generate a target text according to at least one data set attribute of a target data set; or,

[0250] A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

[0251] As an optional implementation manner, the processor 700 is specifically configured to execute:

[0252] Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

[0253] As an optional implementation manner, the processor 700 is specifically configured to execute:

[0254] For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0255] As an optional implementation manner, the processor 700 is specifically configured to execute:

[0256] Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text;

[0257] A mapping relationship between a source data set and at least one matching target data set is established.

[0258] As an optional implementation manner, the processor 700 is further configured to execute:

[0259] Generate data item text according to the second attribute information of the data item of the source data set;

[0260] Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set;

[0261] According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0262] As an optional implementation manner, the second attribute information includes at least one data item attribute; the processor 700 is specifically configured to generate the data item text and the target data item text in any one or more of the following ways:

[0263] generating a data item text according to at least one data item attribute of a source data set;

[0264] A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

[0265] As an optional implementation manner, the second attribute information includes a data item name; the processor 700 is further configured to execute:

[0266] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0267] Based on the same inventive concept, an embodiment of the present invention also provides a data mapping device. Since the device is the device in the method in the embodiment of the present invention, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0268] like Figure 8 As shown, the device comprises:

[0269] A text generation module 800, configured to generate a source text according to the first attribute information of the source data set, and to generate a target text according to the first attribute information of the target data set;

[0270] The mapping establishment module 801 is used to establish a mapping relationship between a source data set and a target data set according to the similarity between the source text and the target text.

[0271] As an optional implementation, the first attribute information includes data set attributes and / or data item attributes; the text generation module 800 is specifically used to generate the source text and the target text in any one or more of the following ways:

[0272] Generate a source text according to at least one data set attribute of a source data set, and generate a target text according to at least one data set attribute of a target data set; or,

[0273] A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

[0274] As an optional implementation, the text generation module 800 is specifically used for:

[0275] Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

[0276] As an optional implementation, the text generation module 800 is specifically used for:

[0277] For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

[0278] As an optional implementation manner, the mapping establishment module 801 is specifically used for:

[0279] Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text;

[0280] A mapping relationship between a source data set and at least one matching target data set is established.

[0281] As an optional implementation, the device further includes a data item mapping module specifically configured to:

[0282] Generate data item text according to the second attribute information of the data item of the source data set;

[0283] Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set;

[0284] According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

[0285] As an optional implementation manner, the second attribute information includes at least one data item attribute; the data item mapping module is specifically used to generate the data item text and the target data item text by any one or more of the following methods:

[0286] generating a data item text according to at least one data item attribute of a source data set;

[0287] A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

[0288] As an optional implementation manner, the second attribute information includes a data item name; and the data item mapping module is further configured to:

[0289] If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

[0290] Based on the same inventive concept, an embodiment of the present disclosure provides a computer storage medium, the computer storage medium includes: a computer program code, when the computer program code is executed on a computer, the computer executes any of the data mapping methods discussed above. Since the principle of solving the problem by the above-mentioned computer storage medium is similar to that of the data mapping method, the implementation of the above-mentioned computer storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0291] In the specific implementation process, the computer storage medium may include: Universal Serial Bus Flash Drive (USB), mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other storage media that can store program codes.

[0292] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data mapping methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data mapping method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0293] The computer program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0294] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0295] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0296] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0297] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0298] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data mapping method, characterized in that: The method includes: Generate a source text according to the first attribute information of the source data set, and generate a target text according to the first attribute information of the target data set; According to the similarity between the source text and the target text, a mapping relationship between the source data set and the target data set is established.

2. The method according to claim 1, characterized in that The first attribute information includes data set attributes and / or data item attributes; the source text and the target text are generated by any one or more of the following methods: generating a source text according to at least one data set attribute of a source data set, and generating a target text according to at least one data set attribute of a target data set; or, A source text is generated according to at least one data item attribute of a source data set, and a target text is generated according to at least one data item attribute of a target data set.

3. The method according to claim 2, characterized in that The generating of the source text according to at least one data set attribute of the source data set, and the generating of the target text according to at least one data set attribute of the target data set, include: Multiple dataset attributes of a source dataset are concatenated to obtain a source text, and multiple dataset attributes of a target dataset are concatenated to obtain a target text.

4. The method according to claim 2, characterized in that: The generating of the source text according to at least one data item attribute of the source data set, and the generating of the target text according to at least one data item attribute of the target data set, comprises: For a data item attribute, the data item attributes of each data item of the source data set are spliced ​​to obtain the source text, and the data item attributes of each data item of the target data set are spliced ​​to obtain the target text.

5. The method according to claim 1, characterized in that The step of establishing a mapping relationship between a source data set and a target data set according to the similarity between the source text and the target text includes: Determining at least one target data set that matches the source data set based on the similarity between the source text and the target text; A mapping relationship between a source data set and at least one matching target data set is established.

6. The method according to any one of claims 1 to 5, characterized in that: The method further includes: Generate data item text according to the second attribute information of the data item of the source data set; Generate a target data item text according to the second attribute information of the data item of the target data set having a mapping relationship with the source data set; According to the similarity between the data item text and the target data item text, a mapping relationship between the data items of the source data set and the target data set is established.

7. The method according to claim 6, characterized in that The second attribute information includes at least one data item attribute; the data item text and the target data item text are generated by any one or more of the following methods: generating a data item text according to at least one data item attribute of a source data set; A target data item text is generated according to at least one data item attribute of a target data set having a mapping relationship with a source data set.

8. The method according to claim 6, characterized in that The second attribute information includes a data item name; the method further includes: If the data item names of the source data set and the target data set are the same, a mapping relationship between the data items with the same names in the source data set and the target data set is established.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images