Data storage method, device, electronic device, and readable medium
By receiving table metadata and task metadata, using the table similarity identification model to judge and merge duplicate or similar data tables, the problem of waste of storage space is solved and storage resource saving is achieved.
Patent Information
- Application Number
- CN202011447767.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-12-11
AI Technical Summary
In the prior art, there are a large number of duplicate or similar data in the stored data, resulting in wasted storage space and the inability to effectively reduce storage resource usage.
By receiving table metadata and task metadata, inputting the table similarity identification model, and determining whether the similarity between the table to be merged and the table to be matched exceeds the preset threshold. If so, merge to generate the target data table and store it.
It reduces the storage space occupied by data table storage, saves storage resources, and improves the efficiency of data storage.
Smart Images

Figure CN112597154B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data storage, and in particular to a data storage method and a data storage device, an electronic device, and a readable medium. Background Art
[0002] At present, in order to support more and more sophisticated data analysis and more and more extensive vehicle functions, more and more data needs to be stored, and the storage space occupied is also increasing. Among the stored data, there are many duplicate or similar data. For these data, they occupy storage space but cannot provide corresponding functions, which is undoubtedly a huge waste of storage space. Summary of the invention
[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data storage method and a corresponding data storage device that overcome the above problems or at least partially solve the above problems.
[0004] In order to solve the above problem, an embodiment of the present invention discloses a method for storing data, wherein the data is stored in the form of a data table, the data table includes a table to be merged and a table to be matched, and the data table has corresponding table metadata and task metadata. The method includes:
[0005] receiving the table metadata and the task metadata;
[0006] Inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarity between the table to be merged and the plurality of tables to be matched;
[0007] Determining whether the table similarity exceeds a preset threshold;
[0008] If yes, then merge the table to be merged with the table to be matched to generate a target data table;
[0009] The target data table is stored.
[0010] Optionally, after the step of receiving the table metadata and the task metadata, the step further includes:
[0011] Performing data cleaning on the table metadata to obtain target table metadata;
[0012] The task metadata is cleaned to obtain target task metadata.
[0013] Optionally, the step of performing data cleaning on the table metadata to obtain target table metadata includes:
[0014] Filtering the table metadata, retaining preset fields and constructing field names for the preset fields;
[0015] Calculating the amount of data in the data table within a preset time;
[0016] The target table metadata is generated using the data volume, the preset fields, and the field names.
[0017] Optionally, the step of performing data cleaning on the task metadata to obtain target task metadata includes:
[0018] Removing non-periodic tasks from the task metadata to obtain periodic task metadata;
[0019] Decomposing the periodic task metadata to obtain single-task metadata; the single-task metadata includes self-dependent tasks;
[0020] Filtering the single-task metadata, removing self-dependent tasks, and obtaining target single-task metadata;
[0021] Based on the target single task metadata, obtaining a data lineage table corresponding to the data table;
[0022] The data lineage table is used to generate target task metadata.
[0023] Optionally, the table similarity includes table name similarity, field similarity, data volume similarity and blood relationship similarity, the target table metadata includes the table name, preset fields and data volume of the data table, the target task metadata includes the data blood relationship table, and the step of inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarity between the table to be merged and the multiple tables to be matched includes:
[0024] Using the first table name of the table to be merged and the second table name of the table to be matched, obtaining the table name similarity through a table similarity recognition model;
[0025] Using the first preset field of the table to be merged and the second preset field of the table to be matched, the field similarity is acquired through a table similarity recognition model;
[0026] Using the first data volume of the table to be merged and the second data volume of the table to be matched, obtaining the data volume similarity through a table similarity recognition model;
[0027] The first data kinship table of the table to be merged and the second data kinship table of the table to be matched are used to obtain the kinship similarity through a table similarity recognition model.
[0028] Optionally, after the step of inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarities between the table to be merged and the plurality of tables to be matched, the step further includes:
[0029] Based on the table similarities, the to-be-matched tables are arranged in a preset order.
[0030] Optionally, the data lineage table includes a root node, and the step of using the first data lineage table of the table to be merged and the second data lineage table of the table to be matched to obtain the lineage similarity through a table similarity recognition model includes:
[0031] Traversing the first data lineage table and the second data lineage table;
[0032] Determine a first bloodline root node in the first data bloodline table and a second bloodline root node in the second data bloodline table;
[0033] The first blood relationship root node and the second blood relationship root node are used to obtain the blood relationship similarity through a table similarity recognition model.
[0034] The embodiment of the present invention further discloses a data storage device, wherein the data is stored in the form of a data table, the data table includes a table to be merged and a table to be matched, the data table has corresponding table metadata and task metadata, and the device includes:
[0035] A receiving module, used for receiving the table metadata and the task metadata;
[0036] A table similarity acquisition module, used for inputting the table metadata and the task metadata into the constructed table similarity recognition model to acquire the table similarity between the table to be merged and the plurality of tables to be matched;
[0037] A judging module, used to judge whether the table similarity exceeds a preset threshold;
[0038] A target data table generating module, configured to merge the table to be merged with the table to be matched to generate a target data table if the table similarity exceeds a preset threshold;
[0039] A storage module is used to store the target data table.
[0040] The embodiment of the present invention further discloses an electronic device, including:
[0041] one or more processors; and
[0042] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the processors to perform one or more of the methods described above.
[0043] The embodiment of the present invention further discloses one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute one or more of the methods described above.
[0044] The embodiments of the present invention include the following advantages:
[0045] In the embodiment of the present invention, table metadata and task metadata are received, the table metadata and task metadata are input into the constructed table similarity recognition model, the table similarity between the table to be merged and the multiple tables to be matched is obtained, and it is determined whether the table similarity exceeds a preset threshold. If so, the table to be merged and the table to be matched are merged to generate a target data table, and the target data table is stored. Thus, repeated or similar data tables can be merged and stored, reducing the storage space occupied by data table storage and saving storage resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of steps of an embodiment of a data storage method of the present invention;
[0047] Figure 2 is a flowchart of another data storage method embodiment of the present invention;
[0048] Figure 3 is a schematic diagram of target task metadata of the present invention;
[0049] Figure 4 It is a schematic diagram of table similarity between a table to be merged and a table to be matched according to the present invention;
[0050] Figure 5 It is a data flow diagram of the present invention;
[0051] Figure 6 It is a structural block diagram of an embodiment of a data storage device of the present invention. DETAILED DESCRIPTION
[0052] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] One of the main inventive points of the present invention is to receive table metadata and task metadata, input the table metadata and task metadata into the constructed table similarity recognition model, obtain the table similarity between the table to be merged and multiple tables to be matched, determine whether the table similarity exceeds a preset threshold, and if so, merge the table to be merged with the table to be matched to generate a target data table and store the target data table. Thus, duplicate or similar data tables can be merged and stored, reducing the storage space occupied by data table storage and saving storage resources.
[0054] Reference Figure 1 , shows a flowchart of a data storage method embodiment of the present invention, wherein the data is stored in the form of a data table, wherein the data table includes a table to be merged and a table to be matched, and the data table has corresponding table metadata and task metadata, and specifically may include the following steps:
[0055] Step 101, receiving the table metadata and the task metadata;
[0056] Specifically, the terminal can initiate a request to obtain data from a storage device that stores data. After receiving the request, the storage device sends the table metadata and task metadata corresponding to the stored data table to the terminal. The data table can be divided into a table to be merged and a table to be matched. The table metadata is used to describe the various attributes of the data table, such as the name of the data table, the number of rows, the number of columns, the amount of data, etc. The task metadata is used to describe the tasks in which the data table participates, such as other data tables generated through the data table, or the data table is modified. The terminal determines a data table from multiple data tables as the table to be merged, and the remaining data tables can be used as the table to be matched, and receives the table metadata and task metadata of the table to be merged and the table to be matched.
[0057] Step 102, inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarity between the table to be merged and the plurality of tables to be matched;
[0058] After obtaining the table metadata and task metadata of the table to be merged and the table to be matched, the table metadata and task metadata can be input into the constructed table similarity recognition model, and the table to be merged is used as the standard to traverse all the tables to be matched, and the table metadata and task metadata are used to calculate the table similarity between the table to be merged and each table to be matched.
[0059] Step 103, determining whether the table similarity exceeds a preset threshold;
[0060] After obtaining the table similarity, it can be determined whether the table similarity exceeds a preset threshold. If it exceeds the preset threshold, it proves that there is more duplicate or similar data between the table to be merged and the table to be matched.
[0061] Step 104: If yes, merge the table to be merged with the table to be matched to generate a target data table;
[0062] When the similarity of the judgment tables exceeds a preset threshold, the table to be merged is merged with the table to be matched. Specifically, different fields in the data tables can be retained, and only one of the same fields can be retained, thereby obtaining a target data table formed by merging the table to be merged and at least one table to be matched.
[0063] Step 105: store the target data table.
[0064] After obtaining the target data table, the target data table may be stored, and the table to be merged and the table to be matched used to generate the target data table may be deleted.
[0065] In an embodiment of the present invention, table metadata and task metadata are received, the table metadata and task metadata are input into a constructed table similarity recognition model, the table similarity between the table to be merged and the multiple tables to be matched is obtained, and it is determined whether the table similarity exceeds a preset threshold. If so, the table to be merged and the table to be matched are merged to generate a target data table, and the target data table is stored. Thus, repeated or similar data tables can be merged and stored, reducing the storage space occupied by data table storage and saving storage resources.
[0066] refer to Figure 2 , shows a flowchart of another data storage method embodiment of the present invention, wherein the data is stored in the form of a data table, wherein the data table includes a table to be merged and a table to be matched, and wherein the data table has corresponding table metadata and task metadata, and specifically may include the following steps:
[0067] Step 201, receiving the table metadata and the task metadata;
[0068] Since step 201 is similar to step 101 in the previous embodiment, the specific description may refer to step 101 in the previous embodiment, which will not be repeated here.
[0069] Step 202, clean the table metadata to obtain target table metadata;
[0070] Since the received table metadata is unfiltered raw data, it contains not only the data required for calculating table similarity but also unnecessary data, and the raw data is arranged in a scattered and messy manner, which is not conducive to determining the data required for calculating similarity. Therefore, the table metadata can be cleaned to filter out unnecessary data and organize the required data to obtain the target table metadata.
[0071] In an optional embodiment of the present invention, step 202 further includes the following sub-steps:
[0072] Filtering the table metadata, retaining preset fields and constructing field names for the preset fields;
[0073] Calculating the amount of data in the data table within a preset time;
[0074] The target table metadata is generated using the data volume, the preset fields, and the field names.
[0075] Specifically, the data tables that need to be merged are permanent, daily periodic data tables. Such data tables need to be stored in storage devices for a long time, so there is a need for merging. The table metadata is filtered, and the unnecessary fields in the data tables that are temporarily generated and will not be retained for a long time and the permanent data tables are removed, and the fields required for calculating the similarity of the tables are retained. For example, the fields project_name (project name), table_name (table name), and column_name (column name) can be retained. After the fields are retained, in order to facilitate reading of the data, an ID (Identity document) is constructed, such as project_name and table_name, and the fields belonging to the same category in the retained fields are placed in the constructed ID, thereby classifying the fields. Since the amount of data in the data table may be different at different times, the average amount of data in the data table within a preset time can be calculated, for example, the average amount of data in the data table within 7 days can be calculated. The target table metadata is generated using the retained fields, the constructed ID, and the calculated average amount of data.
[0076] Step 203, performing data cleaning on the task metadata to obtain target task metadata;
[0077] Since the received task metadata is unfiltered raw data, it contains not only the data required for calculating the table similarity, but also unnecessary data. In addition, the raw data is arranged in a scattered and messy manner, which is not conducive to determining the data required for calculating the similarity. Therefore, the task metadata can be cleaned to filter out unnecessary data and organize the required data to obtain the target task metadata. Figure 3 The figure shows a schematic diagram of a target task metadata, which is the target task metadata of a data table, including id, table_name, cols and rows, where id is the id of the data table, table_name is the table name of the data table, cols includes all the fields in the data table, and rows is the average value of the data volume within a preset time.
[0078] In an optional embodiment of the present invention, the step 203 further includes the following sub-steps:
[0079] Removing non-periodic tasks from the task metadata to obtain periodic task metadata;
[0080] Decomposing the periodic task metadata to obtain single-task metadata; the single-task metadata includes self-dependent tasks;
[0081] Filtering the single-task metadata, removing self-dependent tasks, and obtaining target single-task metadata;
[0082] Based on the target single task metadata, obtaining a data lineage table corresponding to the data table;
[0083] The data lineage table is used to generate target task metadata.
[0084] Specifically, the task metadata can be cleaned to obtain the data lineage table of the data table. The data lineage table is used to indicate the relationship between data tables. For example, there are data tables A and B, and the task inputs data table A and then data table B. It is considered that there is a data lineage relationship between data tables A and B, that is, the data lineage table is a table indicating the input and output relationship between data tables. The tasks executed on the data table can be divided into periodic tasks and non-periodic tasks according to the execution time and number of times. Periodic tasks are tasks that are executed regularly for a fixed number of times, such as twice a day, while non-periodic tasks are tasks that are executed irregularly, such as temporary operations on the data table, with an indefinite number of executions and no specified execution time. Since non-periodic tasks are generally less in number and have no obvious rules compared to periodic tasks, they are not of reference significance. Therefore, the non-periodic tasks in the task metadata can be removed, leaving only the periodic task metadata. As for periodic task metadata, the tasks executed on the data table are often not one-time, that is, the data lineage relationship between data tables includes multiple layers. Therefore, the periodic task metadata can be decomposed to obtain multiple single-task metadata with only one layer. For example, the existing data table A, data table B and data table C, and after inputting data table A, data table B is output, and after data table B is processed, data table C is obtained, which can be decomposed into single-task metadata: "Input: data table A; output: data table B" and "Input: data table B; output: data table C". It should be noted that the single-task metadata also includes self-dependent tasks. Self-dependent tasks refer to tasks that do not have data lineage relationships between data tables and other data tables. Executing self-dependent tasks on data tables will not output other data tables. Therefore, the self-dependent tasks are removed to obtain the target single-task metadata. All target single-task metadata are used to form a data lineage table, and the data lineage table is used as the target task metadata.
[0085] Step 204, inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarities between the table to be merged and the plurality of tables to be matched;
[0086] In an optional embodiment of the present invention, the table familiarity includes table name similarity, field similarity, data volume similarity and blood relationship similarity, the target table metadata includes the table name, preset fields and data volume of the data table, the target task metadata includes the data blood relationship table, and step 204 includes the following sub-steps:
[0087] Using the first table name of the table to be merged and the second table name of the table to be matched, obtaining the table name similarity through a table similarity recognition model;
[0088] Using the first preset field of the table to be merged and the second preset field of the table to be matched, the field similarity is acquired through a table similarity recognition model;
[0089] Using the first data volume of the table to be merged and the second data volume of the table to be matched, obtaining the data volume similarity through a table similarity recognition model;
[0090] The first data kinship table of the table to be merged and the second data kinship table of the table to be matched are used to obtain the kinship similarity through a table similarity recognition model.
[0091] Specifically, in order to measure the similarity between the table to be merged and the table to be matched, four dimensions can be considered, namely, table name similarity, field similarity, data volume similarity, and blood relationship similarity. Therefore, the table similarity recognition model can use the table name of the table to be merged and the table name of the table to be matched to calculate the table name similarity between the table to be merged and the table to be matched. Specifically, the calculation method of the table name similarity in the table similarity recognition model can be "table name similarity = 1-minimum edit distance / Max(Len(A table name), Len(B table name))", where the minimum edit distance is a user-defined function, and the Apache Commons library can be used to implement the customization of the Levenshtein Distance function, where the Apache Commons library is a java toolkit, and the Levenshtein Distance function refers to the minimum number of edit operations required to convert one string into another between two strings. Generally speaking, the smaller the edit distance, the greater the similarity of the two strings. "Max(Len(A table name), Len(B table name))" means returning the table name with the larger number of characters in data table A and data table B. According to this calculation method, a self-cross-correlation calculation is performed on the table_name field of the data table to obtain the table name similarity between the table to be merged and the table to be matched.
[0092] As for field similarity, the table similarity recognition model can use the preset fields of the table to be merged and the preset fields of the table to be matched to calculate the field similarity between the table to be merged and the table to be matched. The calculation method of field similarity in the table similarity recognition model can be "field similarity = 1-Len (field intersection) / Len (field of table A union field of table B)", where "Len (field intersection)" refers to the number of fields with intersection in data table A and data table B, and "Len (field of table A union field of table B)" refers to the total number of fields obtained by merging the fields of table A and table B. This calculation method can be implemented by performing self-cross correlation calculation on the table_name field of the data table through the array_intersect user-defined function, which can return the value of the same parameter in different data table fields.
[0093] For data volume similarity, the table similarity recognition model can use the data volume of the table to be merged and the data volume of the table to be matched to calculate the data volume similarity between the table to be merged and the table to be matched. The calculation method of data volume similarity in the table similarity recognition model can be "data volume similarity = 1-Abs (data volume of table A-data volume of table B) / MAX (data volume of table A, data volume of table B)", where "Abs (data volume of table A-data volume of table B)" is the absolute value of the value obtained by subtracting the data volume of table B from the data volume of table A, and "MAX (data volume of table A, data volume of table B)" is the larger value of the data volume of table A and the data volume of table B. According to this calculation method, the data volume similarity between the table to be merged and the table to be matched can be calculated.
[0094] For the similarity of blood relationship, the table similarity recognition model can use the data blood relationship table of the table to be merged and the data blood relationship table of the table to be matched to calculate the similarity of blood relationship between the table to be merged and the table to be matched. The calculation method of blood relationship similarity in the table similarity recognition model can be "1-Len (blood relationship root node intersection) / Len (blood relationship root node of table A union blood relationship root node of table B)", where "Len (blood relationship root node intersection)" refers to the number of intersections between the blood relationship root nodes in the data blood relationship table of data table A and the blood relationship root nodes in the data blood relationship table of data table B, and "Len (blood relationship root node of table A union blood relationship root node of table B)" refers to the blood relationship root nodes in the data blood relationship table of data table A and the blood relationship root nodes in the data blood relationship table of data table B. The total number of blood relationship root nodes is obtained after the blood relationship root nodes in the data blood relationship table of data table A and the blood relationship root nodes in the data blood relationship table of data table B are merged. This calculation method can be implemented through the array_intersect user-defined function to obtain the similarity of blood relationship between the table to be merged and the table to be matched. Figure 4As shown, it is a schematic diagram of table similarity between a table to be merged and a table to be matched. Column 301 is the table to be merged, and column 302 is multiple tables to be matched. For each table to be matched, the table name similarity, field similarity, data volume similarity and blood relationship similarity between it and the table to be merged are calculated through the table similarity recognition model.
[0095] In an optional embodiment of the present invention, the step of using the first data kinship table of the table to be merged and the second data kinship table of the table to be matched to obtain the kinship similarity through a table similarity recognition model further includes the following sub-steps:
[0096] Traversing the first data lineage table and the second data lineage table;
[0097] Determine a first bloodline root node in the first data bloodline table and a second bloodline root node in the second data bloodline table;
[0098] The first blood relationship root node and the second blood relationship root node are used to obtain the blood relationship similarity through a table similarity recognition model.
[0099] Specifically, since the data blood relationship table is generally a tree diagram, in order to determine the blood relationship root node, it is necessary to traverse all the branches of the data blood relationship table. Specifically, Python can be used to simulate the queue to implement the multi-branch tree full traversal search_roots function. Through this function, the blood relationship is continuously explored upward until all blood relationship root nodes are found. Through the table similarity recognition model, the blood relationship similarity can be calculated using the found blood relationship root node.
[0100] Step 205, arranging the to-be-matched tables in a preset order based on the table similarities;
[0101] Specifically, after obtaining the table similarity, the tables to be matched can be arranged in order of table similarity and displayed to the user. The user can also select one from table name similarity, field similarity, data volume similarity and blood relationship similarity, so that the tables to be matched can be arranged and displayed in order of the selected similarity.
[0102] Step 206, determining whether the table similarity exceeds a preset threshold;
[0103] After obtaining the table similarity, it can be determined whether the table similarity exceeds a preset threshold. If it exceeds the preset threshold, it proves that there are many duplicate or similar data between the table to be merged and the table to be matched. Specifically, different weights can be set for table name similarity, field similarity, data volume similarity, and blood relationship similarity. By weighting the four similarities, it is determined whether the weighted total table similarity exceeds the preset threshold, thereby deciding whether to merge the data tables.
[0104] Step 207: If yes, merge the table to be merged with the table to be matched to generate a target data table;
[0105] Step 208: store the target data table.
[0106] Since steps 207 and 208 are similar to steps 104 and 105 in the previous embodiment, the specific description may refer to steps 104 and 105 in the previous embodiment, which will not be repeated here.
[0107] In an embodiment of the present invention, table metadata and task metadata are received, data cleaning is performed on the table metadata, target table metadata is obtained, data cleaning is performed on the task metadata, target task metadata is obtained, the table metadata and task metadata are input into a constructed table similarity recognition model, table similarities between a table to be merged and multiple tables to be matched are obtained, and based on the table similarities, the tables to be matched are arranged in a preset order, and it is determined whether the table similarity exceeds a preset threshold. If so, the table to be merged and the table to be matched are merged to generate a target data table, and the target data table is stored. By cleaning the data, the efficiency of obtaining table similarity is improved, thereby improving the processing speed of the table to be merged and the table to be matched.
[0108] In order to facilitate technical personnel to further deepen their understanding of the present invention, a data flow diagram of the present invention is shown below.
[0109] like Figure 5 As shown, it is a data flow diagram of the present invention. After obtaining the table metadata and task metadata of the data table, the table metadata and task metadata are cleaned to obtain the target table metadata and target task metadata. The target table metadata and target task metadata are input into the table similarity recognition model to obtain the table name similarity, field similarity, data volume similarity and blood relationship similarity. According to multiple similarities, it is determined whether the table to be merged and the table to be matched need to be merged. After merging the data tables to be merged, the target data table is obtained and stored, so that the user can use the stored target data table for daily work and decision support, which saves storage space and does not affect the user's use of the data table.
[0110] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0111] Reference Figure 6 , shows a structural block diagram of an embodiment of a data storage device of the present invention, wherein the data is stored in the form of a data table, wherein the data table includes a table to be merged and a table to be matched, and wherein the data table has corresponding table metadata and task metadata, and specifically may include the following modules:
[0112] Receiving module 401, used for receiving the table metadata and the task metadata;
[0113] A table similarity acquisition module 402 is used to input the table metadata and the task metadata into the constructed table similarity recognition model to acquire the table similarity between the table to be merged and the plurality of tables to be matched;
[0114] A judgment module 403 is used to judge whether the table similarity exceeds a preset threshold;
[0115] A target data table generating module 404 is used to merge the table to be merged with the table to be matched to generate a target data table if the table similarity exceeds a preset threshold;
[0116] The storage module 405 is used to store the target data table.
[0117] In one embodiment of the present invention, the device further comprises:
[0118] A table metadata cleaning module, used to clean the table metadata and obtain target table metadata;
[0119] The task metadata cleaning module is used to clean the task metadata and obtain target task metadata.
[0120] In one embodiment of the present invention, the table metadata cleaning module includes:
[0121] A table metadata filtering submodule, used to filter the table metadata, retain preset fields and construct field names for the preset fields;
[0122] A data volume calculation submodule, used to calculate the data volume of the data table within a preset time;
[0123] Target table metadata generation submodule.
[0124] In one embodiment of the present invention, the task metadata cleaning module includes:
[0125] A periodic task metadata acquisition submodule, used to remove non-periodic tasks from the task metadata to obtain periodic task metadata;
[0126] A single-task metadata acquisition submodule is used to decompose the periodic task metadata to obtain single-task metadata; the single-task metadata includes self-dependent tasks;
[0127] A target single-task metadata acquisition submodule is used to filter the single-task metadata, remove self-dependent tasks, and obtain the target single-task metadata;
[0128] A data lineage table acquisition submodule, used to acquire a data lineage table corresponding to the data table based on the target single task metadata;
[0129] The target task metadata generation submodule is used to generate the target task metadata using the data lineage table.
[0130] In one embodiment of the present invention, the table familiarity includes table name similarity, field similarity, data volume similarity and blood relationship similarity, the target table metadata includes the table name, preset fields and data volume of the data table, the target task metadata includes the data blood relationship table, and the table similarity acquisition module 402 also includes:
[0131] A table name similarity acquisition submodule, used to use the first table name of the table to be merged and the second table name of the table to be matched to acquire the table name similarity through a table similarity recognition model;
[0132] A field similarity acquisition submodule, configured to use the first preset field of the table to be merged and the second preset field of the table to be matched to acquire the field similarity through a table similarity recognition model;
[0133] A data volume similarity acquisition submodule, configured to use the first data volume of the table to be merged and the second data volume of the table to be matched to acquire the data volume similarity through a table similarity recognition model;
[0134] The blood relationship similarity acquisition submodule is used to use the first data blood relationship table of the table to be merged and the second data blood relationship table of the table to be matched to acquire the blood relationship similarity through a table similarity recognition model.
[0135] In one embodiment of the present invention, the device further comprises:
[0136] The arrangement module is used to arrange the to-be-matched tables in a preset order based on the table similarities.
[0137] In one embodiment of the present invention, the blood relationship similarity acquisition submodule further includes:
[0138] A traversal unit, used for traversing the first data lineage table and the second data lineage table;
[0139] A lineage root node determination unit, used to determine a first lineage root node in the first data lineage table and a second lineage root node in the second data lineage table;
[0140] An acquisition unit is used to adopt the first blood relationship root node and the second blood relationship root node to acquire the blood relationship similarity through a table similarity recognition model.
[0141] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0142] The embodiment of the present invention further discloses an electronic device, including:
[0143] one or more processors; and
[0144] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the processors to perform one or more of the methods described above.
[0145] The embodiment of the present invention further discloses one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, enable the processors to execute one or more of the methods described above.
[0146] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0147] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0148] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0151] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0152] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0153] The above is a detailed introduction to a data storage method and a data storage device, an electronic device and a readable medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data storage method, characterized in that: The data is stored in the form of a data table, the data table includes a table to be merged and a table to be matched, the data table has corresponding table metadata and task metadata, and the method includes: Receive the table metadata and the task metadata; wherein the table metadata is used to describe various attributes of the data table, and the task metadata is used to describe the tasks involved in the data table; Inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarity between the table to be merged and the plurality of tables to be matched; Determining whether the table similarity exceeds a preset threshold; If yes, then merge the table to be merged with the table to be matched to generate a target data table; storing the target data table; After the step of receiving the table metadata and the task metadata, the method further includes: Performing data cleaning on the task metadata to obtain target task metadata; The step of performing data cleaning on the task metadata to obtain target task metadata includes: Removing non-periodic tasks from the task metadata to obtain periodic task metadata; Decomposing the periodic task metadata to obtain single-task metadata; the single-task metadata includes self-dependent tasks; Filtering the single-task metadata, removing self-dependent tasks, and obtaining target single-task metadata; Based on the target single task metadata, obtaining a data lineage table corresponding to the data table; The data lineage table is used to generate target task metadata.
2. The method according to claim 1, characterized in that After the step of receiving the table metadata and the task metadata, the method further includes: The table metadata is cleaned to obtain target table metadata.
3. The method according to claim 2, characterized in that The step of performing data cleaning on the table metadata to obtain the target table metadata includes: Filtering the table metadata, retaining preset fields and constructing field names for the preset fields; Calculating the amount of data in the data table within a preset time; The target table metadata is generated using the data volume, the preset fields, and the field names.
4. The method according to claim 2, characterized in that: The table similarity includes table name similarity, field similarity, data volume similarity and lineage relationship similarity; the target table metadata includes the table name, preset fields and data volume of the data table; the target task metadata includes the data lineage table; the step of inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarity between the table to be merged and the multiple tables to be matched includes: Using the first table name of the table to be merged and the second table name of the table to be matched, obtaining the table name similarity through a table similarity recognition model; Using the first preset field of the table to be merged and the second preset field of the table to be matched, the field similarity is acquired through a table similarity recognition model; Using the first data volume of the table to be merged and the second data volume of the table to be matched, obtaining the data volume similarity through a table similarity recognition model; The first data kinship table of the table to be merged and the second data kinship table of the table to be matched are used to obtain the kinship similarity through a table similarity recognition model.
5. The method according to claim 4, characterized in that After the step of inputting the table metadata and the task metadata into the constructed table similarity recognition model to obtain the table similarities between the table to be merged and the plurality of tables to be matched, the method further includes: Based on the table similarities, the to-be-matched tables are arranged in a preset order.
6. The method according to claim 4, characterized in that The data lineage table includes a root node, and the step of using the first data lineage table of the table to be merged and the second data lineage table of the table to be matched to obtain the lineage similarity through a table similarity recognition model includes: Traversing the first data lineage table and the second data lineage table; Determine a first bloodline root node in the first data bloodline table and a second bloodline root node in the second data bloodline table; The first blood relationship root node and the second blood relationship root node are used to obtain the blood relationship similarity through a table similarity recognition model.
7. A data storage device, characterized in that: The data is stored in the form of a data table, the data table includes a table to be merged and a table to be matched, the data table has corresponding table metadata and task metadata, and the device includes: A receiving module, used to receive the table metadata and the task metadata; wherein the table metadata is used to describe various attributes of the data table, and the task metadata is used to describe the tasks involved in the data table; A table similarity acquisition module, used for inputting the table metadata and the task metadata into the constructed table similarity recognition model to acquire the table similarity between the table to be merged and the plurality of tables to be matched; A judging module, used to judge whether the table similarity exceeds a preset threshold; A target data table generating module, configured to merge the table to be merged with the table to be matched to generate a target data table if the table similarity exceeds a preset threshold; A storage module, used for storing the target data table; The device also includes: A task metadata cleaning module is used to clean the task metadata and obtain target task metadata; The task metadata cleaning module includes: A periodic task metadata acquisition submodule, used to remove non-periodic tasks from the task metadata to obtain periodic task metadata; A single-task metadata acquisition submodule is used to decompose the periodic task metadata to obtain single-task metadata; the single-task metadata includes self-dependent tasks; A target single-task metadata acquisition submodule is used to filter the single-task metadata, remove self-dependent tasks, and obtain the target single-task metadata; A data lineage table acquisition submodule, used to acquire a data lineage table corresponding to the data table based on the target single task metadata; The target task metadata generation submodule is used to generate the target task metadata using the data lineage table.
8. An electronic device, characterized in that: include: one or more processors; and One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, cause the processors to perform one or more methods as claimed in any one of claims 1-6.
9. One or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform one or more methods as claimed in any one of claims 1 to 6.
Citation Information
Patent Citations
Duplicate data identification method and device
CN106294429A
Correlation matrix-based uncertain data connection and combination algorithm
CN106897385A
Medical streaming data blood relationship analysis and storage methods and devices
CN111627552A
Data table storage method and device, computer equipment and readable storage medium
CN111723210A
Data processing method and device
CN111831629A
