Data lake management method, apparatus, device, and storage medium

Abstract

The application discloses a data lake management method, device and equipment and a storage medium. The data lake management method comprises the following steps: acquiring date configuration information and a data retention slice quantity of a data flow area; generating a data lake management strategy according to the date configuration information and the data retention slice quantity; and managing data in the data flow area based on the data lake management strategy. The application scheme realizes discontinuous storage of data in each quantity flow area, maximally realizes on-demand allocation, avoids accumulation of non-use data, improves the management efficiency of the data flow area and optimizes the data storage cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a data lake management method and device, equipment and a storage medium. BACKGROUND

[0002] In order to meet different data processing requirements and reasonably allocate storage costs, a data lake is usually divided into three data flow areas: an online area, a nearline area and a historical data archiving area. The online area is also called operational data storage (ODS) and is mainly used to support daily operations of an enterprise. The data in the online area is usually new, such as data in the last week or the last month, to support data batch processing calculation and timely business decision-making. The nearline area is also called a data mart or operational data storage (ODM) and is mainly used to support business analysis requirements of an enterprise. The nearline area contains more historical data than the online area, but the data retention time is shorter than that of the historical data archiving area. The historical data archiving area is also called data archiving storage (DAS) and is mainly used for long-term retention of historical data and to meet legal, compliance and other requirements. The data in the historical data archiving area includes all data from the earliest time to the current time.

[0003] In order to make the assets in the data lake flow in an orderly manner in the data flow areas over time, a life cycle policy is generally configured for the data lake. However, the current data lake configuration policy is not flexible enough. When data of a daily snapshot or a special date needs to be retained, the data flow areas are prone to accumulate too much non-use data, which reduces the data management efficiency. For example, too much non-batch processing data is accumulated in the online area, which reduces the data processing batch efficiency of the data flow area. SUMMARY

[0004] The main purpose of the present application is to provide a data lake management method, device, equipment and storage medium, which aims to solve the problem of low data management efficiency of the data lake.

[0005] To achieve the above purpose, the present application provides a data lake management method, which comprises the following steps:

[0006] obtaining date configuration information and a data retention slice number of a data flow area;

[0007] generating a data lake management policy according to the date configuration information and the data retention slice number;

[0008] managing data of the data flow area based on the data lake management policy.

[0009] Optionally, the data flow area comprises at least an online area and an offline area, the data lake management policy comprises at least a split policy and a first clean-up policy, and the step of generating the data lake management policy according to the date configuration information and the data retention slice quantity comprises:

[0010] generating the split policy and the first clean-up policy according to the date configuration information and the data retention slice quantity of the online area, wherein the split policy is used to determine the data flow rule from the online area to the offline area, and the first clean-up policy is used to determine the data range to be retained or cleaned up in the online area.

[0011] Optionally, the step of managing the data in the data flow area based on the data lake management policy comprises:

[0012] splitting and retaining the data involved by the date configuration information and the data retention slice quantity of the online area to the online area according to the split policy, and flowing the data not involved by the date configuration information and the data retention slice quantity of the online area to the offline area;

[0013] cleaning up the data not involved by the date configuration information and the data retention slice quantity of the online area from the online area according to the first clean-up policy.

[0014] Optionally, the data flow area further comprises a historical data archiving area, the data lake management policy further comprises an archiving policy and a second clean-up policy, and the step of generating the data lake management policy according to the date configuration information and the data retention slice quantity further comprises:

[0015] generating the second clean-up policy according to the date configuration information and the data retention slice quantity of the offline area, wherein the second clean-up policy is used to determine the data range to be retained or cleaned up in the offline area;

[0016] generating the archiving policy according to the date configuration information, the data retention slice quantity of the offline area and the data retention slice quantity of the historical data archiving area, wherein the archiving policy is used to determine the data flow rule from the offline area to the historical data archiving area.

[0017] Optionally, the step of managing the data in the data flow area based on the data lake management policy comprises:

[0018] splitting and retaining the data involved by the date configuration information and the data retention slice quantity of the offline area to the offline area according to the archiving policy, and flowing the data not involved by the date configuration information and the data retention slice quantity of the online area to the historical data archiving area according to the data retention slice quantity of the historical data archiving area.

[0019] According to the second cleaning strategy, the date configuration information and the data in the near-line area that are not involved in the data retention slice quantity of the near-line area are cleaned.

[0020] Optionally, the data flow direction between the data flow areas is: from the online area to the near-line area, and from the near-line area to the historical data archiving area; the data retention slice quantity of the historical data archiving area includes: quantity 0 and quantity -1, wherein the quantity 0 indicates that the historical data archiving area does not retain any data, and the quantity -1 indicates that all data before the date retained by the near-line area is retained.

[0021] Optionally, the date configuration information includes: day configuration item information, month configuration item information, and year configuration item information.

[0022] Embodiments of the present application also provide a data lake management device, the data lake management device comprising:

[0023] The acquisition module is configured to acquire date configuration information and a data retention slice quantity of a data flow area;

[0024] The strategy module is configured to generate a data lake management strategy according to the date configuration information and the data retention slice quantity;

[0025] The management module is configured to manage data of the data flow area based on the data lake management strategy.

[0026] Embodiments of the present application also provide a data lake management device, the data lake management device comprising a memory, a processor, and a data lake management program stored in the memory and executable on the processor, and when the data lake management program is executed by the processor, the steps of the data lake management method described above are implemented.

[0027] Embodiments of the present application also provide a computer readable storage medium, the computer readable storage medium storing a data lake management program, and when the data lake management program is executed by a processor, the steps of the data lake management method described above are implemented.

[0028] The data lake management method, apparatus, device, and storage medium proposed in this application involve obtaining date configuration information and the number of data retention slices in each data flow area; generating a data lake management strategy based on the date configuration information and the number of data retention slices; and managing the data in the data flow area based on the data lake management strategy. Based on this application's solution, generating a data lake management strategy based on the date configuration information and the number of data retention slices in each data flow area, and managing the data according to the data lake management strategy, allows for non-contiguous storage of data in each data flow area, maximizing on-demand allocation, avoiding the accumulation of unused data, improving the management efficiency of the data flow area, and optimizing data storage costs. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the functional modules of the equipment belonging to the data lake management device of this application;

[0030] Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data lake management method of this application;

[0031] Figure 3 This is a flowchart illustrating a second exemplary embodiment of the data lake management method of this application;

[0032] Figure 4 This is a flowchart illustrating a third exemplary embodiment of the data lake management method of this application;

[0033] Figure 5 This is an example of the data representation intent for the data lake management method described in this application.

[0034] Figure 6 This is a schematic diagram illustrating the configuration strategy for an exemplary scenario one involved in the data lake management method of this application;

[0035] Figure 7 This is a schematic diagram illustrating the configuration strategy for an exemplary scenario two involved in the data lake management method of this application;

[0036] Figure 8 This is a schematic diagram of the configuration strategy for an exemplary scenario three involved in the data lake management method of this application.

[0037] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0038] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0039] The main solution of the embodiment of the application comprises: obtaining date configuration information and a data retention slice quantity of a data flow area; generating a data lake management strategy according to the date configuration information and the data retention slice quantity; and managing data of the data flow area based on the data lake management strategy. According to the solution of the application, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity of each data flow area, and the data is managed according to the data lake management strategy, so that the data is discontinuously stored in each quantity flow area, on-demand allocation is maximally realized, accumulation of non-use data is avoided, the management efficiency of the data flow area is improved, and the data storage cost is optimized.

[0040] Specifically, referring to Figure 1 , Figure 1 is a schematic diagram of functional modules of a device to which the data lake management apparatus of the application belongs. The data lake management apparatus can be an apparatus independent of the device, capable of data acquisition, strategy generation and data management, which can be carried on the device in the form of hardware or software. The device can be a smart mobile terminal such as a mobile phone or a tablet computer having a data processing function, and can also be a fixed device or a server having a data processing function.

[0041] In the embodiment, the device to which the data lake management apparatus belongs comprises at least an output module 110, a processor 120, a memory 130 and a communication module 140.

[0042] The memory 130 stores an operating system and a data lake management program, and the data lake management apparatus can store the obtained date configuration information and data retention slice quantity of the data flow area, and the data lake management strategy generated according to the date configuration information and the data retention slice quantity, etc. in the memory 130; the output module 110 can be a display screen, etc. The communication module 140 can include a WIFI module, a mobile communication module and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0043] When the data lake management program in the memory 130 is executed by the processor, the following steps are implemented:

[0044] obtaining date configuration information and a data retention slice quantity of a data flow area;

[0045] generating a data lake management strategy according to the date configuration information and the data retention slice quantity;

[0046] managing data of the data flow area based on the data lake management strategy.

[0047] Further, when the data lake management program in the memory 130 is executed by the processor, the following steps are also implemented:

[0048] generating the split policy and the first clean policy according to the date configuration information and the data retention slice quantity of the online zone, wherein the split policy is used to determine data flow transfer rules of the online zone to the nearline zone, and the first clean policy is used to determine a data range retained or cleaned in the online zone.

[0049] Further, the data lake management program in the memory 130, when executed by the processor, further implements the following steps:

[0050] splitting and retaining data involved in the date configuration information and the data retention slice quantity of the online zone to the online zone according to the split policy, and transferring data not involved in the date configuration information and the data retention slice quantity of the online zone to the nearline zone;

[0051] cleaning data not involved in the date configuration information and the data retention slice quantity of the online zone from the online zone according to the first clean policy.

[0052] Further, the data lake management program in the memory 130, when executed by the processor, further implements the following steps:

[0053] generating the second clean policy according to the date configuration information and the data retention slice quantity of the nearline zone, wherein the second clean policy is used to determine a data range retained or cleaned in the nearline zone;

[0054] generating the archiving policy according to the date configuration information, the data retention slice quantity of the nearline zone and the data retention slice quantity of the historical data archiving zone, wherein the archiving policy is used to determine data flow transfer rules of the nearline zone to the historical data archiving zone.

[0055] Further, the data lake management program in the memory 130, when executed by the processor, further implements the following steps:

[0056] splitting and retaining data involved in the date configuration information and the data retention slice quantity of the nearline zone to the nearline zone according to the archiving policy, and transferring data not involved in the date configuration information and the data retention slice quantity of the online zone to the historical data archiving zone according to the data retention slice quantity of the historical data archiving zone;

[0057] cleaning data not involved in the date configuration information and the data retention slice quantity of the nearline zone from the nearline zone according to the second clean policy.

[0058] The embodiment obtains date configuration information and a data retention slice quantity of a data flow area, generates a data lake management strategy according to the date configuration information and the data retention slice quantity, and manages data in the data flow area based on the data lake management strategy. According to the scheme, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity of each data flow area, and the data is managed according to the data lake management strategy, so that the data is discontinuously stored in each data flow area, on-demand allocation is maximally realized, accumulation of non-use data is avoided, management efficiency of the data flow area is improved, and data storage cost is optimized.

[0059] The data lake management method embodiment is based on the device architecture but is not limited to the above-mentioned architecture.

[0060] First embodiment

[0061] Reference Figure 2 , Figure 2 The figure is a flowchart of the first exemplary embodiment of the data lake management method. In the embodiment, the data lake management method comprises:

[0062] In step S10, date configuration information and a data retention slice quantity of a data flow area are obtained.

[0063] The execution subject of the method can be a data lake management device, a data lake management terminal device or a server. The embodiment takes the data lake management device as an example, which can be integrated on a terminal device with data processing function, such as a smart phone or a computer.

[0064] The embodiment mainly aims at the problem that the existing data lake configuration strategy is not flexible enough, and when data of a day snapshot or a special date needs to be retained, too much non-batch running data is accumulated in the data flow area, such as an online area, resulting in low data processing and batch running efficiency of the data flow area.

[0065] First, date configuration information of data to be retained and a data retention slice quantity of each data flow area are obtained. The date configuration information can be a specified year, month or day. The data retention slice quantity refers to the number of date data to be retained in each data flow area. For example, the date configuration information is specified as the end of each month, and the data retention slice quantity is 6, which means that the data to be retained includes data of the end of the current month and data of the end of the previous five months, i.e. data of the end of six months.

[0066] In the embodiment, the date configuration information and the data retention slice quantity of the data flow area can be obtained by pre-configuration. The date configuration information and the data retention slice quantity of the data flow area can be flexibly configured according to actual needs.

[0067] Further, in the embodiment, the date configuration information includes day configuration item information, month configuration item information, and year configuration item information. Specifically, the day configuration item information can be a specified date, such as the current date; the month configuration item information can be a specified date of each month, such as the end of each month, the beginning of each month, or the middle of each month; and the year configuration item information can be a specified date of each year, such as January 1 of each year or December 31 of each year. The day configuration item information, the month configuration item information, and the year configuration item information can be flexibly combined as needed, and better meet the data batch running and analysis requirements of different business scenarios. The data corresponding to the day configuration item information, the month configuration item information, and the year configuration item information is not repeatedly saved in each data flow area, that is, only one piece of data exists in a certain data flow area.

[0068] More specifically, the configured day configuration item information in combination with the data retention slice quantity refers to the number of consecutive days of day data that need to be retained in the corresponding data flow area, and the number of days of each data flow area starts from the number of days configured in the previous data flow area. The configured month configuration item information in combination with the data retention slice quantity refers to the number of consecutive months of specified dates (1-31) or end-of-month data that need to be retained in the corresponding data flow area, and the number of months of each data flow area starts from the number of months configured in the previous data flow area and is calculated forward. The configured year configuration item information in combination with the data retention slice quantity refers to the number of consecutive years of data of a specified day in a specified month that need to be retained in the corresponding data flow area, and the number of years of each data flow area starts from the number of years configured in the previous data flow area and is calculated forward.

[0069] Step S20, generating a data lake management strategy according to the date configuration information and the data retention slice quantity.

[0070] In the embodiment, a data lake management strategy is generated according to the obtained date configuration information and data retention slice data, so that subsequent data in the data lake can be processed according to the generated management strategy. The data lake management strategy can include but is not limited to a strategy of retaining data of a specified date and quantity, a strategy of clearing data of a specified date and quantity, or a strategy of transferring data of a specified date and quantity.

[0071] Step S30, managing the data of the data flow area based on the data lake management strategy.

[0072] In the embodiment, data management of the corresponding data flow area is performed according to the generated data lake management strategy, for example, retaining data of a specified date and quantity in the data flow area according to the data lake management strategy, clearing data of a specified date and quantity in the data flow area according to the data lake management strategy, and / or transferring data of a specified date and quantity in the data flow area according to the data lake management strategy.

[0073] The embodiment obtains date configuration information and a data retention slice quantity of a data flow area, generates a data lake management strategy according to the date configuration information and the data retention slice quantity, and manages data in the data flow area based on the data lake management strategy. According to the scheme, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity of each data flow area, and the data is managed according to the data lake management strategy, so that the data is discontinuously stored in each data flow area, on-demand allocation is maximally realized, accumulation of non-use data is avoided, and the management efficiency of the data flow area is improved and the data storage cost is optimized.

[0074] Second embodiment

[0075] Further, based on the first embodiment, in the embodiment, the data flow area at least includes an online area and an offline area, the data lake management strategy at least includes a split strategy and a first cleaning strategy, and the step S20 of generating the data lake management strategy according to the date configuration information and the data retention slice quantity can include:

[0076] In the step S201, the split strategy and the first cleaning strategy are generated according to the date configuration information and the data retention slice quantity of the online area, wherein the split strategy is used to determine a data flow rule from the online area to the offline area, and the first cleaning strategy is used to determine a data range to be retained or cleaned in the online area.

[0077] Specifically, the data flow area of the embodiment at least includes an online area and an offline area. The online area is mainly used to support the daily operation demand of an enterprise, and the data in the online area is usually new, such as data in the last week or the last month, to support data batch processing calculation and timely business decision-making. The offline area is mainly used to support the business analysis demand of the enterprise, and the offline area contains more historical data than the online area, such as data in the last few years, to support more in-depth data analysis.

[0078] In the embodiment, the split strategy and the first cleaning strategy are generated according to the obtained date configuration information and the data retention slice quantity of the online area. The split strategy is used to determine a data flow rule from the online area to the offline area. That is, according to the data flow rule specified by the split strategy, data of a specified date and quantity can be flowed from the online area to the offline area. The first cleaning strategy is used to determine a data range to be retained or cleaned in the online area. For example, the first cleaning strategy specifies a data range to be retained, and data of a corresponding date and quantity is retained according to the retained data range; or the first cleaning strategy specifies a data range to be cleaned, and data of a corresponding date and quantity is cleaned according to the cleaned data range.

[0079] Further, referring to Figure 3 , Figure 3 is a flowchart of a second exemplary embodiment of the data lake management method. In this embodiment, the step S30 of managing the data in the data flow area based on the data lake management policy can include:

[0080] Step S301, according to the splitting policy, split and retain the data involved in the date configuration information and the data retention slice quantity of the online area to the online area, and transfer the data not involved in the date configuration information and the data retention slice quantity of the online area to the nearline area.

[0081] Specifically, according to the generated splitting policy, the data involved in the date configuration information and the data retention slice quantity of the online area, and the data not involved in the date configuration information and the data retention slice quantity of the online area are split. The data involved in the date configuration information and the data retention slice quantity of the online area is retained in the online area, and the data not involved in the date configuration information and the data retention slice quantity of the online area is split and transferred to the nearline area.

[0082] Step S302, according to the first cleaning policy, clean the data not involved in the date configuration information and the data retention slice quantity of the online area from the online area.

[0083] Specifically, based on the split data not involved in the date configuration information and the data retention slice quantity of the online area, according to the generated first cleaning policy, the data not involved in the date configuration information and the data retention slice quantity of the online area is cleaned from the online area.

[0084] So far, compared with the prior art, in this embodiment, the online area only retains the data involved in the date configuration information and the data retention slice quantity of the online area required for daily batch running. The remaining data not required for batch running is transferred to the nearline area and cleaned in the online area, avoiding the accumulation of too much non-batch use data in the online area, and ensuring the efficiency of data processing and batch running.

[0085] Third embodiment

[0086] Further, based on the first embodiment or the second embodiment, in this embodiment, the data flow area further includes a historical data archiving area, the data lake management policy further includes an archiving policy and a second cleaning policy, and the step S20 of generating a data lake management policy according to the date configuration information and the data retention slice quantity can include:

[0087] In step S202, the second clean-up policy is generated according to the date configuration information and the data retention slice quantity of the nearline area, wherein the second clean-up policy is used to determine the data range to be retained or cleaned up in the nearline area.

[0088] In this embodiment, the second clean-up policy is generated according to the obtained date configuration information and the data retention slice quantity of the nearline area, wherein the second clean-up policy is used to determine the data range to be retained or cleaned up in the nearline area. For example, if the second clean-up policy specifies the data range to be retained, then the data corresponding to the date and quantity in the nearline area is retained according to the retained data range; or if the second clean-up policy specifies the data range to be cleaned up, then the data corresponding to the date and quantity in the nearline area is cleaned up according to the cleaned-up data range.

[0089] In step S203, the archiving policy is generated according to the date configuration information, the data retention slice quantity of the nearline area, and the data retention slice quantity of the historical data archiving area, wherein the archiving policy is used to determine the data flow transfer rule from the nearline area to the historical data archiving area.

[0090] Specifically, the data flow transfer area of this embodiment can also include a historical data archiving area. The historical data archiving area is mainly used for long-term retention of historical data and to meet legal, compliance, and other requirements. The data of the historical data archiving area includes all data, from the earliest to the current time. Since the historical data archiving area contains a large amount of historical data, the access and query efficiency of the data is low, and therefore these data are usually accessed only when necessary.

[0091] In this embodiment, the archiving policy is generated according to the obtained date configuration information, the data retention slice quantity of the nearline area, and the data retention slice quantity of the historical data archiving area. The archiving policy is used to determine the data flow transfer rule from the nearline area to the historical data archiving area. That is, according to the data flow transfer rule specified by the archiving policy, the data of the specified date and quantity can be transferred from the nearline area to the historical data archiving area.

[0092] Further, with reference to Figure 4 , Figure 4 This is a flowchart of the third exemplary embodiment of the data lake management method of the present application. In this embodiment, the step S30 of managing the data in the data flow area based on the data lake management policy can include:

[0093] In step S303, the data involved by the date configuration information and the data retention slice quantity of the nearline area is split and retained to the nearline area according to the archiving policy, and the data not involved by the date configuration information and the data retention slice quantity of the online area is transferred to the historical data archiving area according to the data retention slice quantity of the historical data archiving area.

[0094] Specifically, according to the generated archiving strategy, the data involved in the date configuration information and the data retention slice quantity of the nearline area is split out, and the data not involved in the date configuration information and the data retention slice quantity of the nearline area is split out. The data involved in the date configuration information and the data retention slice quantity of the nearline area is retained in the nearline area; and the data not involved in the date configuration information and the data retention slice quantity of the online area is transferred to the historical data archiving area according to the data retention slice quantity of the historical data archiving area.

[0095] In step S304, the data not involved in the date configuration information and the data retention slice quantity of the nearline area is cleaned from the nearline area according to the second cleaning strategy.

[0096] Specifically, based on the data not involved in the date configuration information and the data retention slice quantity of the nearline area split out above, the data not involved in the date configuration information and the data retention slice quantity of the nearline area is cleaned from the nearline area according to the generated second cleaning strategy.

[0097] At this point, compared with the prior art, the nearline area in the embodiment only retains the data involved in the date configuration information and the data retention slice quantity of the nearline area required to meet the business analysis needs of the enterprise. The remaining data not required for business analysis is transferred to the historical data archiving area and cleaned in the nearline area, which can avoid the backup progress delay hidden danger of the nearline area as the business grows.

[0098] Further, in the embodiment, the flow direction of the data between the data flow areas is: from the online area to the nearline area, and from the nearline area to the historical data archiving area; and the data retention slice quantity of the historical data archiving area includes: quantity 0 and quantity -1, wherein the quantity 0 indicates that the historical data archiving area does not retain any data, and the quantity -1 indicates that the full amount of data before the date retained by the nearline area is retained.

[0099] Specifically, the flow direction of the data between the data flow areas is: from the online area to the nearline area, and from the nearline area to the historical data archiving area, that is, online area-nearline area-historical data archiving area.

[0100] In the embodiment, the configuration item of the data retention slice quantity of the historical data archiving area can be set to: quantity 0 and quantity -1. Among them, quantity 0 indicates that the historical data archiving area does not retain any data, and quantity -1 indicates that the full amount of data before the date retained by the nearline area is retained.

[0101] Example scenario one, refer to Figure 5, suppose that a retail department needs to use the month-end and year-end data in the batch running of multiple tables. Based on this scenario, the configuration of the date configuration information and the number of data retention slices in the data flow area is as follows: Figure 6 , the day configuration item information is the current date, and the number of data retention slices in the online area is 7; the month configuration item information is the month-end, the number of data retention slices in the online area is 3, the number of data retention slices in the nearline area is 12, and the number of data retention slices in the historical data archive area is -1; the year configuration item information is the year-end (i.e., December 31), and the number of data retention slices in the online area is 2.

[0102] According to the above-mentioned configuration of the date configuration information and the number of data retention slices in each data flow area, the retained data in the online area is the data of the current date plus 7 days, 3 month-end days, and 2 year-end days (a total of 12 days), the retained data in the nearline area is the data of 12 month-end days before the data date of the online area, and the retained data in the historical data archive area is the data of all month-end days before the data date of the nearline area.

[0103] Therefore, the management strategy of the data lake generated according to the above-mentioned configuration of the date configuration information and the number of data retention slices in each data flow area includes the splitting strategy, the cleaning strategy, and the archiving strategy. The splitting strategy is that, in addition to retaining the data of the last 7 days, the online area also retains 3 month-end data and 2 year-end data, and splits the remaining data outside the data range to the nearline area. The cleaning strategy is to completely remove the data remaining outside the data range in the online area. The archiving strategy is to retain the data of 12 month-end days before the data date of the online area to the nearline area, and to split the data other than the data retained by the nearline area to the historical data archive area.

[0104] Example scenario two, suppose a regulatory audit business scenario, which involves gambling and fraud reporting period is from the 21st of the previous month to the 20th of the current month, that is, the actual reporting needs to use the snapshot of the 20th of the current month. Referring to Figure 7 , the configured day configuration item information is the current date, and the number of data retention slices in the online area is 7; the month configuration item information is the 20th of each month, the number of data retention slices in the online area is 3, the number of data retention slices in the nearline area is 12, and the number of data retention slices in the historical data archive area is -1; the year configuration item information is the 28th of each February, the number of data retention slices in the online area is 2, the number of data retention slices in the nearline area is 3, and the number of data retention slices in the historical data archive area is -1.

[0105] According to the date configuration information and the data retention slice quantity of each data flow area configured above, the retained data of the online area is the data of the current date, the data of the 20th day of the last 3 months, and the data of February 28 of the last 2 years (a total of 12 days), the retained data of the nearline area is the data of the 20th day of the last 12 months and the data of February 28 of the last 3 years before the data date of the online area (a total of 15 days), and the retained data of the historical data archiving area is the data of all the 20th day of each month and the data of February 28 of each year before the data date of the nearline area.

[0106] Therefore, the splitting strategy, the cleaning strategy, and the archiving strategy of the data lake are generated according to the date configuration information and the data retention slice quantity of each data flow area configured above. The splitting strategy is that, in addition to retaining the data of the last 7 days, the online area also retains the data of the 20th day of the last 3 months and the data of February 28 of the last 2 years, and splits the remaining data out of the data range to the nearline area. The cleaning strategy is that all the data out of the data range in the online area is cleaned. The archiving strategy is that the data of the 20th day of the last 12 months and the data of February 28 of the last 3 years before the data date of the online area are retained to the nearline area, and other data except the data retained by the nearline area are split to the historical data archiving area.

[0107] In the third example scenario, for historical asset data, in order to ensure that data can be traced back and verified when necessary, the related tables of the upstream host business system on which the historical asset data is generated need to be warehoused and backed up for at least nearly one year. Based on the above scenario, referring to Figure 8 , the date configuration information configured is the current date, the data retention slice quantity of the online area is 30, and the data retention slice quantity of the nearline area is 90; the month configuration information is the end of the month, the data retention slice quantity of the online area is 3, the data retention slice quantity of the nearline area is 12, and the data retention slice quantity of the historical data archiving area is -1.

[0108] According to the date configuration information and the data retention slice quantity of each data flow area configured above, the retained data of the online area is the data of the current date, the data of the 20th day of the last 3 months, and the data of February 28 of the last 2 years (a total of 12 days), the retained data of the nearline area is the data of the 20th day of the last 12 months and the data of February 28 of the last 3 years before the data date of the online area (a total of 15 days), and the retained data of the historical data archiving area is the data of all the 20th day of each month and the data of February 28 of each year before the data date of the nearline area.

[0109] Therefore, the splitting strategy, the cleaning strategy and the archiving strategy of the data lake are generated according to the date configuration information and the data retention slice quantity of each data flow area. The splitting strategy is that, in addition to retaining data of the last 30 days, the online area also retains data of the last 3 months, and the remaining data beyond the data range is split to the nearline area. The archiving strategy is that, data of 90 days before the data date of the online area + data of 12 months are retained to the nearline area, data of the last year except the data retained by the nearline area is split to the historical data archiving area, and all the data of the last month before the data date of the nearline area is retained to the historical data archiving area. The cleaning strategy is that, in addition to the data of the last year (that is, 365 days) retained by the online area, the nearline area and the historical data archiving area, and all the data of the last month before the data date of the nearline area, all other data is cleaned.

[0110] In the embodiment, the date configuration information and the data retention slice quantity of the data flow area are acquired, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity, and the data of the data flow area is managed based on the data lake management strategy. According to the scheme, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity of each data flow area, and the data is managed according to the data lake management strategy, so that the data is discontinuously stored in each data flow area, the on-demand allocation is maximally realized, the accumulation of non-use data is avoided, the management efficiency of the data flow area is improved, and the data storage cost is optimized.

[0111] In addition, the embodiment of the application further provides a data lake management device, which comprises:

[0112] The acquisition module is configured to acquire date configuration information and a data retention slice quantity of a data flow area.

[0113] The strategy module is configured to generate a data lake management strategy according to the date configuration information and the data retention slice quantity.

[0114] The management module is configured to manage data of the data flow area based on the data lake management strategy.

[0115] The principle and implementation process of the data lake management are realized in the embodiment, and details are not described herein again.

[0116] In addition, the embodiment of the application further provides a data lake management device, which comprises a memory, a processor and a data lake management program stored in the memory and executable on the processor, and the data lake management program implements the steps of the data lake management method when executed by the processor.

[0117] Since the data lake management program is executed by the processor, all the technical solutions of all the foregoing embodiments are adopted, and at least all the beneficial effects brought by all the technical solutions of all the foregoing embodiments are obtained, which will not be repeated here.

[0118] In addition, the embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores the data lake management program. The data lake management program is executed by the processor to implement the steps of the data lake management method.

[0119] Since the data lake management program is executed by the processor, all the technical solutions of all the foregoing embodiments are adopted, and at least all the beneficial effects brought by all the technical solutions of all the foregoing embodiments are obtained, which will not be repeated here.

[0120] Compared with the prior art, the data lake management method, device, equipment and storage medium provided by the embodiment of the present application have the following advantages. The data lake management method acquires date configuration information and a data retention slice quantity of a data flow area, generates a data lake management strategy according to the date configuration information and the data retention slice quantity, and manages data in the data flow area based on the data lake management strategy. According to the present application, the data lake management strategy is generated according to the date configuration information and the data retention slice quantity of each data flow area, and the data is managed according to the data lake management strategy, so that the data is discontinuously stored in each data flow area, on-demand allocation is maximally realized, accumulation of non-use data is avoided, the management efficiency of the data flow area is improved, and the data storage cost is optimized.

[0121] It should be noted that in this document, the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0122] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0123] The above application embodiment serial number is only for description, not representing the pros and cons of the embodiment.

[0124] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software product, which is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including a plurality of instructions for making a device (which can be a mobile phone, computer, server, controlled terminal, or network device, etc.) execute the method of each embodiment of the present application.

[0125] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data lake management method, characterized in that, The data lake management method comprises: obtaining date configuration information and data retention slice quantity of a data flow area; generating a data lake management strategy according to the date configuration information and the data retention slice quantity; managing data in the data flow area based on the data lake management strategy; wherein the data flow area comprises at least an online area and an offline area, the data lake management strategy comprises at least a splitting strategy and a first cleaning strategy, and the step of generating the data lake management strategy according to the date configuration information and the data retention slice quantity comprises: generating the splitting strategy and the first cleaning strategy according to the date configuration information and the data retention slice quantity of the online area, wherein the splitting strategy is used to determine data flow conversion rules from the online area to the offline area, and the first cleaning strategy is used to determine data ranges to be retained or cleaned in the online area.

2. The data lake management method of claim 1, wherein, The step of managing data in the data flow area based on the data lake management strategy comprises: splitting and retaining data involved in the date configuration information and the data retention slice quantity of the online area to the online area according to the splitting strategy, and converting data not involved in the date configuration information and the data retention slice quantity of the online area to the offline area; cleaning data not involved in the date configuration information and the data retention slice quantity of the online area from the online area according to the first cleaning strategy.

3. The data lake management method of claim 1, wherein, The data flow area further comprises a historical data archiving area, the data lake management strategy further comprises an archiving strategy and a second cleaning strategy, and the step of generating the data lake management strategy according to the date configuration information and the data retention slice quantity further comprises: generating the second cleaning strategy according to the date configuration information and the data retention slice quantity of the offline area, wherein the second cleaning strategy is used to determine data ranges to be retained or cleaned in the offline area; generating the archiving strategy according to the date configuration information, the data retention slice quantity of the offline area and the data retention slice quantity of the historical data archiving area, wherein the archiving strategy is used to determine data flow conversion rules from the offline area to the historical data archiving area.

4. The data lake management method of claim 3, wherein, The step of managing data in the data flow area based on the data lake management strategy comprises: splitting and retaining data involved in the date configuration information and the data retention slice quantity of the offline area to the offline area according to the archiving strategy, and converting data not involved in the date configuration information and the data retention slice quantity of the online area to the historical data archiving area according to the data retention slice quantity of the historical data archiving area; cleaning data not involved in the date configuration information and the data retention slice quantity of the offline area from the offline area according to the second cleaning strategy.

5. The data lake management method of claim 3, wherein, The flow direction of data among the data flow areas is from the online area to the offline area, and from the offline area to the historical data archiving area. The data retention slice quantity of the historical data archiving area includes quantity 0 and quantity -1, wherein the quantity 0 indicates that the historical data archiving area does not retain any data, and the quantity -1 indicates that all data before a date retained by the nearline area is retained.

6. The data lake management method of claim 1, wherein, The date configuration information includes day configuration item information, month configuration item information and year configuration item information.

7. A data lake management apparatus, characterized by, The data lake management apparatus includes: an acquisition module configured to acquire date configuration information and a data retention slice quantity of a data flow area; a policy module configured to generate a data lake management policy according to the date configuration information and the data retention slice quantity, wherein the data flow area includes at least an online area and a nearline area, and the data lake management policy includes at least a splitting policy and a first cleaning policy; the policy module is further configured to generate the splitting policy and the first cleaning policy according to the date configuration information and the data retention slice quantity of the online area, wherein the splitting policy is used to determine a data flow rule from the online area to the nearline area, and the first cleaning policy is used to determine a data range retained or cleaned in the online area; a management module configured to manage data of the data flow area based on the data lake management policy.

8. A data lake management device, characterized by, The data lake management device includes a memory, a processor and a data lake management program stored in the memory and executable on the processor, and the data lake management program, when executed by the processor, implements the steps of the data lake management method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a data lake management program, and the data lake management program, when executed by a processor, implements the steps of the data lake management method according to any one of claims 1-6.

Images