Redundant asset identification methods, data processing systems, and storage media in data lakes

Abstract

This invention relates to the field of data processing technology, and particularly to a method, data processing system, and storage medium for identifying redundant assets in a data lake. The method includes the following steps: determining the number of upstream objects referenced by a target data table in the data lake; if the number of upstream objects is single, determining that the data source type of the target data table is a single data source type, and invoking a single data source redundant asset identification strategy to determine whether the target data table is a redundant asset; if the number of upstream objects is multiple, determining that the data source type of the target data table is a multi-data source type, and invoking a multi-data source redundant asset identification strategy to determine whether the target data table is a redundant asset. This invention enables the identification of redundancy in data tables across multiple business modules, both within and across modules, when cleaning up redundant assets in a data lake, aiming to solve the problem of identifying redundant assets between multiple business functional modules.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for identifying redundant assets in a data lake, a data processing system, and a storage medium. Background Technology

[0002] A data lake is a highly scalable data storage area. Due to the complex reference relationships between data structures within a data lake, redundant assets such as tables, jobs, and fields are prone to emerge. These redundant assets not only waste storage space but also make data traceability difficult. To avoid these negative impacts, appropriate cleanup strategies are needed to regularly remove redundant assets from the data lake.

[0003] In relevant technical solutions, redundant assets in a data lake are usually identified through business analysis. That is, from the perspective of a certain business function in a data lake, similar data tables under that business function are identified as redundant assets and cleaned up.

[0004] However, this approach can only identify redundant assets within a single business function module, and cannot identify redundant assets across multiple business function modules. When the data lake is large and has many business function partitions, the redundant assets that traditional business analysis methods can identify are limited, resulting in poor redundancy cleanup.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a method for identifying redundant assets in a data lake, aiming to solve the problem of how to identify redundant assets among multiple business function modules.

[0007] To achieve the above objectives, the present invention provides a method for identifying redundant assets in a data lake, the method comprising the following steps:

[0008] Determine the number of upstream objects referenced by the target data table in the data lake;

[0009] If the number of upstream objects is single, the data source type of the target data table is determined to be a single data source type, and the single data source redundant asset identification strategy is called to determine whether the target data table is a redundant asset.

[0010] If there are multiple upstream objects, the data source type of the target data table is determined to be a multi-data-source type, and the multi-data-source redundant asset identification strategy is invoked to determine whether the target data table is a redundant asset.

[0011] Optionally, the step of using a single-data-source redundant asset identification strategy to determine whether the data table is a redundant asset includes:

[0012] Obtain the job read / write table corresponding to the target data table;

[0013] Determine whether the job read / write table meets the single data source redundancy judgment condition;

[0014] If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table;

[0015] Based on the frequency of each word, determine whether the target data table meets the single data source redundancy judgment condition;

[0016] If the conditions are met, the target data table is determined to be the redundant asset.

[0017] Optionally, the step of determining whether the job read / write table meets the single data source redundancy judgment condition includes:

[0018] Determine whether the job read / write table contains aggregate functions and window functions;

[0019] If not included, determine whether the job read / write table is a single data source job read / write table, wherein the single data source job read / write table is a job read / write table that only includes insert operators, filter operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0020] If so, determine that the job read / write table satisfies the single data source redundancy judgment condition.

[0021] Optionally, the step of determining whether the target data table satisfies the single data source redundancy judgment condition based on the frequency of each of the aforementioned words includes:

[0022] Determine whether the frequency of the word corresponding to the insertion operator is less than or equal to two;

[0023] If so, determine whether at least one of the following is true: the word frequency of the filtering operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function;

[0024] If so, determine that the target data table satisfies the single data source redundancy judgment condition.

[0025] Optionally, the step of invoking a multi-data-source redundant asset identification strategy to determine whether the target data table is a redundant asset includes:

[0026] Determine whether the target data table has the same primary key as the upstream data table;

[0027] If it exists, obtain the job read / write table corresponding to the target data table;

[0028] Determine whether the job read / write table meets the multi-data-source redundancy judgment condition;

[0029] If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table;

[0030] Based on the frequency of each word, determine whether the target data table meets the redundancy judgment condition of multiple data sources;

[0031] If the conditions are met, the target data table is determined to be the redundant asset.

[0032] Optionally, the step of determining whether the job read / write table meets the multi-data-source redundancy judgment condition includes:

[0033] Determine whether the job read / write table contains aggregate functions and window functions;

[0034] If not included, determine whether the job read / write table is a multi-data-source job read / write table, wherein the multi-data-source job read / write table is a job read / write table that only includes insert operators, filter operators, association operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0035] If so, determine that the job read / write table satisfies the multi-data-source redundancy judgment condition.

[0036] Optionally, the step of determining whether the target data table satisfies the multi-data-source redundancy judgment condition based on the frequency of each of the aforementioned words includes:

[0037] Determine whether at least one of the following is true or false: the word frequency of the filtering operator, the word frequency of the association operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function;

[0038] If so, determine that the target data table satisfies the multi-data-source redundancy judgment condition.

[0039] Optionally, the method for identifying redundant assets in the data lake further includes:

[0040] Obtain the job read / write table corresponding to the target data table, and determine whether the target data table is the redundant asset based on the job read / write table; or,

[0041] Determine the keyword frequency of each script in each of the target data tables, and determine whether the target data table is a redundant asset based on the keyword frequency.

[0042] Furthermore, to achieve the above objectives, the present invention also provides a data processing system, the data processing system comprising: a memory, a processor, and a redundant asset identification program in a data lake stored in the memory and executable on the processor, wherein when the redundant asset identification program in the data lake is executed by the processor, it implements the steps of the redundant asset identification method in the data lake as described above.

[0043] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a redundant asset identification program in a data lake, wherein the redundant asset identification program in the data lake, when executed by a processor, implements the steps of the redundant asset identification method in the data lake as described above.

[0044] This invention provides a method, data processing system, and storage medium for identifying redundant assets in a data lake. By determining the number of upstream objects of a data table in the data lake, different redundant asset identification strategies are selected to identify the data table. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the hardware operating environment of the data processing system involved in the embodiments of the present invention;

[0046] Figure 2 This is a flowchart illustrating a single-data-source embodiment of the redundant asset identification method in a data lake according to the present invention.

[0047] Figure 3 This is a flowchart illustrating a multi-data-source embodiment of the redundant asset identification method in a data lake according to the present invention.

[0048] Figure 4 This is a flowchart illustrating the third embodiment of the redundant asset identification method in a data lake according to the present invention.

[0049] Figure 5 This is a flowchart illustrating the fourth embodiment of the redundant asset identification method in a data lake according to the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] This application identifies data tables by determining the number of upstream objects referenced by the data tables in the data lake, and selects different redundant asset identification strategies to identify the data tables.

[0052] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0053] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the data processing system involved in the embodiments of the present invention.

[0054] like Figure 1 As shown, the data processing system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The data processing system architecture shown does not constitute a limitation on the data processing system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a redundant asset identification program for the data lake. The operating system is a program that manages and controls the hardware and software resources of the data processing system, while the redundant asset identification program for the data lake and other software or programs are also included.

[0057] exist Figure 1 In the data processing system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the redundant asset identification program in the data lake stored in the memory 1005.

[0058] In this embodiment, the data processing system includes: a memory 1005, a processor 1001, and a redundant asset identification program in a data lake stored on the memory and executable on the processor, wherein:

[0059] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0060] Determine the number of upstream objects referenced by the target data table in the data lake;

[0061] If the number of upstream objects is single, the data source type of the target data table is determined to be a single data source type, and the single data source redundant asset identification strategy is called to determine whether the target data table is a redundant asset.

[0062] If there are multiple upstream objects, the data source type of the target data table is determined to be a multi-data-source type, and the multi-data-source redundant asset identification strategy is invoked to determine whether the target data table is a redundant asset.

[0063] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0064] Obtain the job read / write table corresponding to the target data table;

[0065] Determine whether the job read / write table meets the single data source redundancy judgment condition;

[0066] If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table;

[0067] Based on the frequency of each word, determine whether the target data table meets the single data source redundancy judgment condition;

[0068] If the conditions are met, the target data table is determined to be the redundant asset.

[0069] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0070] Determine whether the job read / write table contains aggregate functions and window functions;

[0071] If not included, determine whether the job read / write table is a single data source job read / write table, wherein the single data source job read / write table is a job read / write table that only includes insert operators, filter operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0072] If so, determine that the job read / write table satisfies the single data source redundancy judgment condition.

[0073] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0074] Determine whether the frequency of the word corresponding to the insertion operator is less than or equal to two;

[0075] If so, determine whether at least one of the following is true: the word frequency of the filtering operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function;

[0076] If so, determine that the target data table satisfies the single data source redundancy judgment condition.

[0077] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0078] Determine whether the target data table has the same primary key as the upstream data table;

[0079] If it exists, obtain the job read / write table corresponding to the target data table;

[0080] Determine whether the job read / write table meets the multi-data-source redundancy judgment condition;

[0081] If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table;

[0082] Based on the frequency of each word, determine whether the target data table meets the redundancy judgment condition of multiple data sources;

[0083] If the conditions are met, the target data table is determined to be the redundant asset.

[0084] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0085] Determine whether the job read / write table contains aggregate functions and window functions;

[0086] If not included, determine whether the job read / write table is a multi-data-source job read / write table, wherein the multi-data-source job read / write table is a job read / write table that only includes insert operators, filter operators, association operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0087] If so, determine that the job read / write table satisfies the multi-data-source redundancy judgment condition.

[0088] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0089] Determine whether at least one of the following is true or false: the word frequency of the filtering operator, the word frequency of the association operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function;

[0090] If so, determine that the target data table satisfies the multi-data-source redundancy judgment condition.

[0091] When processor 1001 calls the redundant asset identification program in the data lake stored in memory 1005, it performs the following operations:

[0092] Obtain the job read / write table corresponding to the target data table, and determine whether the target data table is the redundant asset based on the job read / write table; or,

[0093] Determine the keyword frequency of each script in each of the target data tables, and determine whether the target data table is a redundant asset based on the keyword frequency.

[0094] Based on the hardware architecture of the data processing system based on the above data processing technology, an embodiment of the redundant asset identification method in the data lake of the present invention is proposed.

[0095] Reference Figure 2 In a single data source embodiment, the method for identifying redundant assets in the data lake includes the following steps:

[0096] Step S10: Determine the number of upstream objects referenced by the target data table in the data lake;

[0097] In this embodiment, when performing redundant asset identification, the data processing system selects a data table from the data lake as the target data table to execute the step. It is understood that this embodiment describes whether a single data table is a redundant asset; however, in actual redundant asset identification, the data processing system can simultaneously identify multiple data tables as target data tables and execute corresponding steps for each target data table.

[0098] In this embodiment, the number of upstream objects referenced by the target data table refers to the calling relationship between the target data table within a module or between multiple business modules. If there are multiple upstream objects referenced, the target data table is characterized as a multi-source data table; if there is only one upstream object referenced, the target data table is characterized as a single-source data table.

[0099] Optionally, the number of upstream objects referenced can be determined by the reference relationships between target data tables. These reference relationships can be determined by analyzing the table structure; specifically, if a field in one data table uses a field from another table, then that table is considered to have a reference relationship.

[0100] Alternatively, the reference relationship can be determined by pre-assigning a foreign key constraint identifier to the table when it is created, and the system can directly check whether the foreign key constraint identifier exists in the data table to determine whether the data table has a reference relationship.

[0101] Step S20: If the number of upstream objects is single, determine that the data source type of the target data table is a single data source type, and call the single data source redundant asset identification strategy to determine whether the target data table is a redundant asset;

[0102] Step S30: If there are multiple upstream objects, determine that the data source type of the target data table is a multi-data source type, and call the multi-data source redundant asset identification strategy to determine whether the target data table is a redundant asset.

[0103] In this embodiment, the single-data-source redundant asset identification strategy is characterized as a strategy for identifying redundancy in a single-data-source data table, while the multi-data-source redundant asset identification strategy is characterized as a strategy for identifying redundancy in a multi-data-source data table. The identification logic for the two strategies is different.

[0104] In this embodiment, for a target data table that references a single upstream object, the system determines that the target data table is a single-source data table and calls the single-source redundant asset identification strategy to determine whether the target data table is a redundant asset; for a target data table that references multiple upstream objects, the system determines that the target data table is a multi-source data table and calls the single-source redundant asset identification strategy to determine whether the target data table is a redundant asset.

[0105] Optionally, in this embodiment, after determining whether the target data table is a redundant asset based on different strategies, if the target data table is identified as a redundant asset, a cleanup action is performed on the target data table to reduce redundant assets in the data lake and avoid excessive redundant assets in the data lake, which would cause data redundancy in the data lake.

[0106] In the technical solution provided in this embodiment, different redundant asset identification strategies are selected to identify data tables by determining the number of upstream objects referenced by the data tables in the data lake. For data tables with different numbers of upstream object references, the identification strategy used during redundancy identification will distinguish them from data tables without reference relationships. This enables the redundancy identification of data tables from multiple business segments when cleaning up redundant assets in the data lake, thereby improving the effectiveness of redundant asset cleanup in large-scale data lakes.

[0107] Reference Figure 3 Based on the single data source embodiment, step S20 includes:

[0108] Step S21: Obtain the job read / write table corresponding to the target data table;

[0109] Step S22: Determine whether the job read / write table meets the single data source redundancy judgment condition;

[0110] Step S23: If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table;

[0111] Step S24: Determine whether the target data table meets the single data source redundancy judgment condition based on the frequency of each word.

[0112] Step S25: If the conditions are met, determine the target data table as the redundant asset.

[0113] As an optional embodiment, in this embodiment, for a target data table with a single upstream object and a single data source table type, a single data source redundant asset identification strategy is used to determine whether the target data table is redundant.

[0114] In this embodiment, the job read / write table corresponding to the target data table is first obtained. The job read / write table refers to the operation table in the database that performs read and write operations on the data table. Based on the job read / write table, it is determined whether there is a redundancy flag in the target data table. If a redundancy flag exists, it means that the job read / write table meets the single data source redundancy judgment condition.

[0115] It should be noted that the reason for determining whether a target data table is redundant in this embodiment is that redundant assets are characterized by repetitive or invalid write and read operations. By analyzing the job read and write table of the read and write data table, we can determine whether the read and write operation causes repeated changes to the data or whether the data does not change. In this case, we can determine that the data table may be a redundant asset.

[0116] In this embodiment, the job read / write table of the target data table is first judged by the single data source redundancy judgment condition. Then, the word frequency of the keywords of each script in the target data table is determined. Based on the number of word frequencies, a second judgment is made on whether the target data table meets the single data source redundancy judgment condition.

[0117] Optionally, the word frequency of the target keywords in each script in the target data table can be determined by the system's keyword frequency statistics module. The function of the keyword frequency statistics module is to detect the target keywords that are the same as the reference keywords in the keyword dictionary among the keywords that appear in the script, and count the number of times the target keyword appears, which is taken as the word frequency of the target keyword.

[0118] It should be noted that the keyword dictionary is a keyword dataset pre-compiled in the system in this embodiment. For example, the target keywords and their functions in the keyword dictionary can be found in the following table:

[0119] Insertion operator INSERT Filter Operators WHERE Update operator UPDATE Union operator UNION Field conversion function CASE Field concatenation function RAWCAT, CONCAT, CONCAT_WS Field extraction function SUBSTRING, SUBSTR, LEFT, RIGHT, SPLIT_PART Field replacement function REPLACE

[0120] In the technical solution provided in this embodiment, for a target data table with only one upstream object and a single data source table type, a single data source redundant asset identification strategy is used to determine whether the target data table is redundant. The single data source redundant asset identification strategy assesses the redundancy of the target data table from two dimensions: job read / write table and word frequency, thereby determining whether the data table is a redundant asset.

[0121] Furthermore, in this embodiment, step S22 includes:

[0122] Step S221: Determine whether the job read / write table contains aggregate functions and window functions;

[0123] Step S222: If not included, determine whether the job read / write table is a single data source job read / write table, wherein the single data source job read / write table is a job read / write table that only includes insertion operators, filtering operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0124] Step S223: If yes, determine that the job read / write table satisfies the single data source redundancy judgment condition.

[0125] As an optional implementation, in this embodiment, an aggregation function is a function used to calculate and evaluate a set of data. It typically takes a set of values ​​as input and returns a single value as output. For example, aggregation functions may include, but are not limited to, functions of type SUM, AVG, MAX, and MIN. A window function, on the other hand, refers to a function that performs aggregation operations on a subset of the data in a dataset, rather than aggregating the entire dataset. Window functions can perform operations such as sorting, grouping, and filtering on the data, and return the aggregation result along with the original dataset. For example, window functions may include, but are not limited to, functions of type ROW_NUMBER, RANK, DENSE_RANK, LAG, and LEAD.

[0126] In this embodiment, it is first determined whether the job read / write table contains aggregate functions and window functions. If it contains the above aggregate functions and window functions, the system directly determines that the data table is not a redundant asset and stops making subsequent judgments; if it does not contain the above functions, further judgments are made.

[0127] If the job read / write table does not contain aggregate functions and window functions, then it is determined whether the job read / write table is a single data source job read / write table. The single data source job read / write table is a job read / write table that only contains insert operators, filter operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions.

[0128] It should be emphasized that if the job read / write table contains any operator keywords other than the operators or functions mentioned above, then the job read / write table is not a single data source job read / write table.

[0129] In the technical solution provided in this embodiment, if the job read / write table does not contain aggregate functions and window functions, and the job read / write table is a single data source job read / write table, then the job read / write table corresponding to the target data table is determined to meet the single data source redundancy judgment condition.

[0130] Furthermore, in this embodiment, step S24 includes:

[0131] Step S241: Determine whether the frequency of the word corresponding to the insertion operator is less than or equal to two;

[0132] Step S242: If yes, determine whether at least one of the following is true: the word frequency of the filtering operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function;

[0133] Step S243: If yes, determine that the target data table satisfies the single data source redundancy judgment condition.

[0134] As an optional implementation, in this embodiment, the method for determining the word frequency of different target keywords differs for target data tables with a single data source type.

[0135] In this embodiment, the target keywords of the script corresponding to the target data table include insert operator, filter operator, field conversion function, field concatenation function, field truncation function, and field replacement function.

[0136] The word frequency determination of the target keywords mentioned above is divided into two types: determining the word frequency of the insertion operator, and determining the word frequency of other keywords besides the insertion operator.

[0137] In this embodiment, it is determined whether the frequency of the insertion operator is less than or equal to two. If there are more than two insertion operators, that is, there are multiple keywords of the insertion operator type in the script of the data table, the system directly determines that the data table is a non-redundant asset; if there are less than or equal to two insertion operators, further judgment is made.

[0138] Furthermore, if the number of word frequencies corresponding to the insertion operator is less than or equal to two, the word frequencies of other keywords are then evaluated. If the word frequency of any one of the filtering operator, the field conversion function, the field concatenation function, the field truncation function, and the field replacement function is one or more, the system determines that the data table meets the single data source redundancy judgment condition.

[0139] It should be noted that in the above scheme, the order of judgment between the insertion operator and other keywords can be replaced. It is also possible to first judge whether the frequency of other keywords is multiple, and then judge whether the frequency of the insertion operator is less than or equal to two. However, since judging the insertion operator first will reduce the computational cost for the system, this embodiment will use judging the insertion operator first as an example for explanation.

[0140] In the technical solution provided in this embodiment, if it is determined that the word frequency of the insertion operator in the target keyword is less than or equal to two, and the word frequency of any one of the other keywords in the target keyword other than the insertion operator is one or more, then the target data table is determined to meet the single data source redundancy judgment condition, and the target data table is considered to be a redundant asset.

[0141] Reference Figure 4 In the third embodiment, based on any embodiment, step S30 includes:

[0142] Step S31: Determine whether the target data table has the same primary key as the upstream data table;

[0143] Step S32: If it exists, obtain the job read / write table corresponding to the target data table;

[0144] Step S33: Determine whether the job read / write table meets the multi-data-source redundancy judgment condition;

[0145] Step S34: If satisfied, determine the word frequency of the keywords in each script of each target data table;

[0146] Step S35: Determine whether the target data table satisfies the multi-data-source redundancy judgment condition based on the frequency of each word.

[0147] Step S36: If the conditions are met, determine the target data table as the redundant asset.

[0148] As an optional embodiment, in this embodiment, for a target data table with multiple upstream objects and a multi-data-source table type, a multi-data-source redundant asset identification strategy is used to determine whether the target data table is redundant.

[0149] In this embodiment, in order to demonstrate the difference in identification logic between the multi-data-source redundant asset identification strategy and the single-data-source redundant asset identification strategy, the main differences between the two strategies will be explained below. The specific implementation of the steps is similar to that in the multi-data-source embodiment, and will not be repeated here.

[0150] In the multi-data-source redundant asset identification strategy, before obtaining the job read / write table corresponding to the target data table, the system will also determine whether there is a primary key in the target data table that is the same as the upstream data table. If so, the job read / write table acquisition step will be executed.

[0151] II. The job read / write table obtained in the multi-data-source redundant asset identification strategy differs from that in the single-data-source redundant asset identification strategy. The redundancy judgment conditions of the multi-data-source strategy are different from those of the single-data-source redundant asset identification strategy.

[0152] Third, the target keywords identified are different, and the redundancy judgment conditions for multiple data sources are also different from the single data source redundancy judgment conditions in the single data source redundancy asset identification strategy.

[0153] For example, for a target data table of a multi-source table type, the target keywords and their functions in the corresponding keyword dictionary can be found in the following table:

[0154] Insertion operator INSERT Filter Operators WHERE Associative operators JOIN Update operator UPDATE Union operator UNION Field conversion function CASE Field concatenation function RAWCAT, CONCAT, CONCAT_WS Field extraction function SUBSTRING, SUBSTR, LEFT, RIGHT, SPLIT_PART Field replacement function REPLACE

[0155] In this example, compared to a target data table of a single data source table type, this keyword dictionary adds association operators for judgment.

[0156] In the technical solution provided in this embodiment, for a target data table with multiple upstream objects and a multi-data-source table type, a multi-data-source redundant asset identification strategy is used to determine whether the target data table is redundant. The multi-data-source redundant asset identification strategy also performs redundancy assessment on the target data table from two dimensions: job read / write table and word frequency, thereby determining whether the data table is a redundant asset.

[0157] Furthermore, in this embodiment, step S33 includes:

[0158] Step S331: Determine whether the job read / write table contains aggregate functions and window functions;

[0159] Step S332: If not included, determine whether the job read / write table is a multi-data-source job read / write table, wherein the multi-data-source job read / write table is a job read / write table that only includes insertion operators, filtering operators, association operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions;

[0160] Step S333: If yes, determine that the job read / write table satisfies the multi-data-source redundancy judgment condition.

[0161] As an optional implementation, in this embodiment, similar to step S20 in the multi-data-source embodiment, it is first determined whether the job read / write table contains aggregate functions and window functions. If it contains the above aggregate functions and window functions, the system directly determines that the data table is not a redundant asset and stops making subsequent judgments; if it does not contain the above functions, further judgments are made.

[0162] Furthermore, if the job read / write table does not contain aggregate functions and window functions, then it is determined whether the job read / write table is a multi-data-source job read / write table. The multi-data-source job read / write table is a job read / write table that only contains insert operators, filter operators, association operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions.

[0163] It is important to emphasize that the difference between a multi-source job read / write table and a single-source job read / write table is that the multi-source job read / write table also includes an association operator.

[0164] In the technical solution provided in this embodiment, if the job read / write table does not contain aggregate functions and window functions, and the job read / write table is a multi-data-source job read / write table, then the job read / write table corresponding to the target data table is determined to satisfy the multi-data-source redundancy judgment condition.

[0165] Furthermore, in this embodiment, step S35 includes:

[0166] Step S351: Determine whether at least one of the following is true or false: the word frequency of the filtering operator, the word frequency of the association operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function.

[0167] Step S352: If yes, determine that the target data table satisfies the multi-data-source redundancy judgment condition.

[0168] As an optional implementation, the main difference between this embodiment and the multi-data-source embodiment regarding step S24 is:

[0169] First, do not check the frequency of the insertion operator. This is because data tables with multiple data sources may have multiple insertion operators.

[0170] Second, the frequency of the association operator has been added. If the frequency of the association operator is multiple, the target data table is determined to meet the redundancy judgment condition of multiple data sources.

[0171] In the technical solution provided in this embodiment, if it is determined that the word frequency of at least one of the target keyword's insertion operator, filtering operator, association operator, field conversion function, field concatenation function, field truncation function, and field replacement function is multiple, then the target data table is determined to meet the multi-data-source redundancy judgment condition, and the target data table is considered to be a redundant asset.

[0172] Reference Figure 5 In the fourth embodiment, based on any embodiment, the method for identifying redundant assets in the data lake further includes:

[0173] Step S101: Obtain the job read / write table corresponding to the target data table, and determine whether the target data table is the redundant asset based on the job read / write table;

[0174] Alternatively, in step S102, the frequency of keywords in each script of each target data table is determined, and whether the target data table is the redundant asset is determined based on the frequency of keywords.

[0175] As an optional embodiment, this embodiment also provides several other methods for identifying redundant assets based on the foregoing embodiments.

[0176] I. Identify the target data table from the perspective of the task read / write table.

[0177] For a target data table with a single data source, determine whether it meets the single data source redundancy judgment condition in the single data source redundancy asset identification strategy (see the relevant record involved in step S22 in the single data source embodiment). If it does not meet the single data source redundancy judgment condition, directly determine that the target data table is not a redundant asset.

[0178] For a target data table with multiple data sources, determine whether it meets the multiple data source redundancy judgment conditions in the multiple data source redundancy asset identification strategy (see the relevant records involved in step S33 in the third embodiment). If it does not meet the multiple data source redundancy judgment conditions, directly determine that the target data table is not a redundant asset.

[0179] 2. Identify the target data table from the perspective of keyword frequency.

[0180] For a target data table with a single data source, determine whether it meets the multi-data-source redundancy judgment condition in the single-data-source redundant asset identification strategy (see the relevant record in step S24 of the single-data-source embodiment). If it does not meet the single-data-source redundancy judgment condition, directly determine that the target data table is not a redundant asset.

[0181] For a target data table with multiple data sources, determine whether it meets the multiple data source redundancy judgment conditions in the multiple data source redundancy asset identification strategy (see the relevant records involved in step S35 in the third embodiment). If it does not meet the multiple data source redundancy judgment conditions, directly determine that the target data table is not a redundant asset.

[0182] In the technical solution provided in this embodiment, unlike the previous embodiment which judged whether the target data table was a redundant asset from both the job read / write table and keyword frequency dimensions, this embodiment can select only one dimension for judgment. Selecting a single dimension to judge redundant assets can effectively reduce the system's computing power overhead while judging the redundancy of data tables across business segments; while selecting multiple dimensions to judge redundant assets can achieve higher accuracy in judging the redundancy of data tables across business segments.

[0183] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a data processing system to implement the process steps of the embodiments of the above methods.

[0184] Therefore, the present invention also provides a computer-readable storage medium storing a redundant asset identification program in a data lake, which, when executed by a processor, implements the various steps of the redundant asset identification method in a data lake as described in the above embodiments.

[0185] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0186] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0187] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0189] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0190] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0191] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0192] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0193] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for identifying redundant assets in a data lake, characterized in that, The method for identifying redundant assets in the data lake includes the following steps: Determine the number of upstream objects referenced by the target data table in the data lake; If the number of upstream objects is single, the data source type of the target data table is determined to be a single data source type, and the single data source redundant asset identification strategy is called to determine whether the target data table is a redundant asset. If there are multiple upstream objects, determine that the data source type of the target data table is a multi-data source type, and call the multi-data source redundant asset identification strategy to determine whether the target data table is a redundant asset; The step of using the single-data-source redundant asset identification strategy to determine whether the data table is a redundant asset includes: Obtain the job read / write table corresponding to the target data table; Determining whether the job read / write table meets the single data source redundancy judgment condition includes: determining whether the job read / write table contains aggregate functions and window functions; if not, determining whether the job read / write table is a single data source job read / write table, wherein the single data source job read / write table is a job read / write table that only contains insertion operators, filter operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions; if so, determining that the job read / write table meets the single data source redundancy judgment condition. If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table; Based on the frequency of each word, determine whether the target data table meets the single data source redundancy judgment condition; If the conditions are met, the target data table is determined to be the redundant asset; The step of using the multi-data-source redundant asset identification strategy to determine whether the target data table is a redundant asset includes: Determine whether the target data table has the same primary key as the upstream data table; If it exists, obtain the job read / write table corresponding to the target data table; Determining whether the job read / write table satisfies the multi-data-source redundancy judgment condition includes: determining whether the job read / write table contains aggregate functions and window functions; if not, determining whether the job read / write table is a multi-data-source job read / write table, wherein the multi-data-source job read / write table is a job read / write table that only contains insert operators, filter operators, join operators, field conversion functions, field concatenation functions, field truncation functions, and field replacement functions; if so, determining that the job read / write table satisfies the multi-data-source redundancy judgment condition. If satisfied, determine the word frequency of the target keywords for each script corresponding to the target data table; Based on the frequency of each word, determine whether the target data table meets the redundancy judgment condition of multiple data sources; If the conditions are met, the target data table is determined to be the redundant asset.

2. The method for identifying redundant assets in a data lake as described in claim 1, characterized in that, The step of determining whether the target data table satisfies the single data source redundancy judgment condition based on the frequency of each word includes: Determine whether the frequency of the word corresponding to the insertion operator is less than or equal to two; If so, determine whether at least one of the following is true: the word frequency of the filtering operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function; If so, determine that the target data table satisfies the single data source redundancy judgment condition.

3. The method for identifying redundant assets in a data lake as described in claim 1, characterized in that, The step of determining whether the target data table satisfies the multi-data-source redundancy judgment condition based on the frequency of each word includes: Determine whether at least one of the following is true or false: the word frequency of the filtering operator, the word frequency of the association operator, the word frequency of the field transformation function, the word frequency of the field concatenation function, the word frequency of the field truncation function, and the word frequency of the field replacement function; If so, determine that the target data table satisfies the multi-data-source redundancy judgment condition.

4. The method for identifying redundant assets in a data lake as described in any one of claims 1 to 3, characterized in that, The method for identifying redundant assets in the data lake also includes: Obtain the job read / write table corresponding to the target data table, and determine whether the target data table is the redundant asset based on the job read / write table; or, Determine the keyword frequency of each script in each of the target data tables, and determine whether the target data table is a redundant asset based on the keyword frequency.

5. A data processing system, characterized in that, The data processing system includes: a memory, a processor, and a redundant asset identification program in a data lake stored in the memory and executable on the processor. When executed by the processor, the redundant asset identification program in the data lake implements the steps of the redundant asset identification method in a data lake as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a redundant asset identification program in the data lake, which, when executed by a processor, implements the steps of the redundant asset identification method in the data lake as described in any one of claims 1 to 4.

Images