Hive data automatic offline method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for automatically logging off Hive data, comprising: obtaining Hive table data and task meta-information, and obtaining a data log corresponding to the Hive table data and task meta-information; determining a data flow chain diagram based on the association relationship between the data log, Hive table data and task meta-information, and determining a data status table based on the data flow chain diagram; obtaining a preset data offline strategy, and determining candidate offline data based on the data offline strategy and the data status table; and performing corresponding data offline operations on the candidate offline data according to the preset data offline operation strategy. Through the present application, the mining of useless data and the automation of the offline process can be realized, storage and computing resources can be released, the risk of misoperation can be greatly reduced, the efficiency of data cleaning can be improved, and the efficiency and accuracy of Hive data offline can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment, and storage medium for automatically logging off Hive data. Background Art

[0002] Big data technology has become widespread, and many companies are leveraging it to build their own Hive databases. However, data output often consumes computing and storage resources, and over time, databases accumulate large amounts of useless data, wasting resources.

[0003] In the Hive database, the scheduled task scheduling system uses or produces data when executing scheduled tasks. During business development and configuration, the data that the tasks depend on is also configured. With long-term iterations of the Hive database, a large amount of useless data will exist. Some data was developed to meet current business needs and is no longer used; some is intermediate data produced by temporary analysis; and some is data that can be replaced by new versions. This data consumes a large amount of valuable computing and storage resources, resulting in waste. Therefore, in related technologies, when processing useless data in the Hive database, a dependency graph between data and tasks is constructed to determine the dependencies between data and tasks. The data that is not dependent on will be sent to the relevant person in charge of the data, and after their confirmation, it will be taken offline.

[0004] However, there are several significant problems with existing technologies: 1. The dependencies and outputs of tasks and data require manual configuration, which may lead to problems such as omissions, mismatches, and outdated information, making it difficult to accurately determine whether the data is still dependent on other tasks; 2. Data is used in a variety of scenarios, including data analysis, service queries, etc. in addition to scheduled tasks. Therefore, using only the configuration of the scheduling system will result in omissions; 3. Manual judgment is required on whether the data is useless, and offline operations also require manual operations, resulting in low resource recovery efficiency and prone to errors; 4. The progress of data offline is not transparent, and it is easy to accidentally offline and affect business stability.

[0005] Therefore, no effective solution has been proposed for the problems of low offline efficiency and low offline accuracy of Hive data in the existing technology. Summary of the Invention

[0006] The embodiments of the present invention provide a method, apparatus, device and storage medium for automatically logging off Hive data, aiming to solve the problems of low efficiency and low accuracy of logging off Hive data in the prior art methods.

[0007] In a first aspect, an embodiment of the present invention provides a method for automatically logging off Hive data, the method comprising:

[0008] Obtain Hive table data and task meta information, and obtain data logs corresponding to the Hive table data and the task meta information;

[0009] Determine a data flow chain diagram based on the association relationship between the data log, the Hive table data, and the task meta information, and determine a data status table based on the data flow chain diagram;

[0010] Obtaining a preset data offline strategy, and determining candidate offline data based on the data offline strategy and the data status table;

[0011] According to a preset data offline operation strategy, a corresponding data offline operation is performed on the candidate offline data.

[0012] In a second aspect, an embodiment of the present invention provides a device for automatically logging off Hive data, comprising:

[0013] An acquisition module is used to obtain Hive table data and task meta information, and obtain data logs corresponding to the Hive table data and the task meta information;

[0014] An association module, configured to determine a data flow chain diagram based on the association relationship among the data log, the Hive table data, and the task meta-information, and determine a data status table based on the data flow chain diagram;

[0015] An extraction module, configured to obtain a preset data offline strategy, and determine candidate offline data based on the data offline strategy and the data status table;

[0016] The offline module is used to perform corresponding data offline operations on the candidate offline data according to a preset data offline operation strategy.

[0017] Embodiments of the present invention provide a method and apparatus for automatically shutting down Hive data. The method comprises obtaining Hive table data and task metadata, and obtaining data logs corresponding to the Hive table data and task metadata; determining a data flow chain diagram based on the association between the data logs, Hive table data, and task metadata, and determining a data status table based on the data flow chain diagram; obtaining a preset data offline policy, and determining candidate offline data based on the data offline policy and the data status table; and performing corresponding data offline operations on the candidate offline data according to the preset data offline operation policy.

[0018] This application provides a system for identifying and automatically deleting useless data. By parsing operation logs and HDFS read and write messages, it obtains the usage and output relationship between tasks and data, builds a complete data flow chain, and thus more accurately mines useless data based on the upstream and downstream relationships of data in the data flow chain and the set filtering rules. Finally, it automatically deleting useless data, freeing up storage and computing resources, significantly reducing the risk of manual operation errors, improving the efficiency of data cleaning, and improving the efficiency and accuracy of Hive data delisting. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of the automatic offline method for Hive data provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram illustrating the association between data, tasks, and various logs provided in an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a data transfer link provided by an embodiment of the present invention;

[0023] Figure 4 The data status representation provided by the embodiment of the present invention is schematic;

[0024] Figure 5 A schematic diagram of a process for determining candidate offline data provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the process of data offline operation provided by an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of a specific process of data offline operation provided by an embodiment of the present invention;

[0027] Figure 8 A schematic block diagram of a device for automatically logging off Hive data provided by an embodiment of the present invention;

[0028] Figure 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0030] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0031] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0032] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] The embodiment of the present invention provides a method for automatically logging off Hive data. Figure 1 A flow chart of the Hive data automatic offline method provided by the embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes steps S110 to S140.

[0034] S110: Obtain Hive table data and task meta information, and obtain data logs corresponding to the Hive table data and task meta information.

[0035] Specifically, obtain Hive table data from the Hive database to be analyzed.

[0036] Obtain task meta-information from the task scheduling system; wherein, task meta-information includes at least import task meta-information, processing task meta-information, and export task meta-information. It should be noted that the first type of import task is used to import business data (MySQL data) or other external data (such as Kafka, etc.) into the Hive database. This type of task records the data source and target table information; the second type of processing task is used to process one or more Hive table data and finally write the results to another Hive table; the third type of export task is used to export Hive table data to an external database (such as MySQL, HBase, etc.) for business use. This type of task also records the data source and target table information. All of the above tasks may depend on one or more tables, and may also depend on other tasks to produce or update table data.

[0037] Obtain data logs from the HDFS system; data logs include at least read and write logs and user behavior logs.

[0038] Figure 2 This is a schematic diagram of the relationship between data, tasks and various logs provided in the embodiment of the present invention. Figure 2 As an example, we retrieve all Hive table data D from the Hive database to be analyzed (i.e., the Hive MetaStore), including the Hive table name, owner information, and management information. We also retrieve metadata for all tasks T from the task scheduling system, including the task name, task type, owner information, and configuration information. Finally, we retrieve data logs from the HDFS system, including read and write logs, task execution logs, and user behavior logs. These data sets are interconnected.

[0039] It should be noted that Hive MetaStore is the metadata management database that comes with the Hive system. All Hive table data can be collected from this database. Table types can be divided into: common tables, external tables, and view tables. Figure 3 For data that does not need to be updated regularly, it is manually marked as a static table. In addition, through naming, the business domain to which a table belongs can be automatically determined.

[0040] S120. Determine a data flow chain diagram based on the association relationship between the data log, Hive table data and task meta information, and determine a data status table based on the data flow chain diagram.

[0041] Preferably, the upstream and downstream relationship between Hive table data and task meta-information is obtained; the data log is parsed to obtain the usage and output information of the Hive table in the Hive table data; then, based on the upstream and downstream relationship between the Hive table data and task meta-information, and the usage and output information of the Hive table in the Hive table data, the data flow chain diagram is determined; and based on the data flow chain diagram and the usage and output information of the Hive table, the data status table is determined.

[0042] Specifically, the upstream and downstream relationship between Hive table data and task meta-information is obtained, which includes: determining the dependent output relationship between Hive table data and task meta-information based on the task configuration information and task code information in the task meta-information, and using the dependent output relationship as the upstream and downstream relationship between the Hive table data and the task meta-information.

[0043] Next, the data logs are parsed to obtain Hive table usage and output information within the Hive table data. The parsing steps include: obtaining the corresponding file information from each ClientId through read and write logs; obtaining the corresponding task through ClientId association, and obtaining the corresponding Hive table through file path association; obtaining the SQL used to execute the query through user behavior logs, and extracting the Hive tables used and written to through the SQL; from this, the usage information corresponding to each data point in the Hive table data can be determined.

[0044] Then, the data flow chain diagram is determined based on the upstream and downstream relationships between Hive table data and task meta-information, as well as the usage and output information of the Hive table in the Hive table data. Figure 3 A schematic diagram of a data flow link provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, task T0 produces data D1 and D2; then, task T1 uses data D1 and D2 to produce data D3, which is used by developers, such as data analysis; or, task T2 depends on task T1 and uses data D4 to produce data D5, where data D4 is queried and used by the system and developers; or, task T3 depends on tasks T1 and T2, which is equivalent to using data D3 and D5 produced by tasks T1 and T2 to produce data D6. Figure 3 The data flow link diagram shown can easily find the upstream and downstream relationships between data and tasks, that is, which task produces a certain data and which tasks depend on it. It can also directly obtain the specific usage of the data.

[0045] Finally, based on the data flow chain diagram and Hive table usage and output information, the data status table is determined. This includes: determining the output tasks, creation time, consecutive failure days, downstream quantity, and consecutive days without queries corresponding to each data table based on the data flow chain diagram and Hive table usage and output information; and combining the output tasks, creation time, consecutive failure days, downstream quantity, and consecutive days without queries corresponding to each data table to obtain the data status table. Figure 4 The data status provided in the embodiment of the present invention is shown as follows: Figure 4 As shown, for example, according to Figure 3 The data flow chain diagram and Hive table usage and output information shown in the figure can determine that the creation time of Hive table data D1 is 2023-01-01, the corresponding output task is T0, the number of consecutive failure days is 0, the number of downstream data is 1, and the number of consecutive days without query is 365. Combining the output tasks, creation time, consecutive failure days, number of downstream data, and consecutive days without query for all data tables, we can get Figure 4 The data status table is shown.

[0046] S130: Obtain a preset data offline policy, and determine candidate offline data based on the data offline policy and the data status table.

[0047] Figure 5 A schematic diagram of a process for determining candidate offline data provided by an embodiment of the present invention, such as Figure 5 As shown, it specifically includes steps S131 to S132.

[0048] S131, based on the data quality detection sub-strategy in the data offline strategy, the data status table is detected to obtain a quality inspection result.

[0049] Specifically, before selecting candidate offline data, a quality check must be performed on the collected basic data. Specific quality check criteria include: the amount of collected basic data and tasks cannot fluctuate significantly, the amount of data without downstream resources cannot increase dramatically, and there cannot be a large number of Hive table behavior logs that cannot be parsed to reveal usage and output information. If the basic data fails to meet the preset conditions of the quality check sub-strategy, the current task will be interrupted and the administrator will intervene to manually judge and handle the problem. The task will then be restarted after confirmation.

[0050] S132: If the quality inspection result is determined to be normal, candidate offline data is determined based on the screening sub-strategy in the data offline strategy and the data status table.

[0051] After the quality inspection results are determined to be normal, candidate offline data is determined based on the screening sub-strategy in the data offline strategy and the data status table. The specific process is as follows:

[0052] Preset offline conditions determine whether the data and tasks in the data status table meet the conditions. If they do, they are placed in the offline candidate pool. Specific offline conditions include: 1. Tasks that have failed for a long time. The term "long-term" can be determined based on the task scheduling cycle. For example, if the cycle is "days," 15 consecutive days of failure; or if the cycle is "weeks," 4 consecutive weeks of failure. 2. Data tables without downstream tasks, that is, tables not directly used by other tasks.

[0053] Preset filtering conditions, and use these filtering conditions to filter the data and tasks in the offline candidate pool. The filtered data and tasks enter the pre-offline pool. Specific filtering conditions include: 1. The table or task was created recently, and the corresponding business has not yet been launched and needs to be retained. The "recent" here can be set manually according to actual conditions. For example, tables created within one month can be considered recent. 2. Some tables that require special protection, such as manually maintained static tables. 3. Tables and tasks that have been used recently by developers or systems and may be used for statistical and analytical scenarios. The "recent" here can be set manually according to actual conditions. 4. Need to be retained for other reasons, for example, tables or tasks manually added to the whitelist.

[0054] According to the blood relationship between data and tasks in the data status table, the parent table of offline data and tasks is expanded into the pre-offline pool. Figure 3 In the example, data D3 has tasks T2 and T3 downstream. When the system finds that data D5 (corresponding to task T2) and data D6 (corresponding to task T3) can be taken offline, that is, the downstream of D3 can be taken offline, then the parent table D3 of data D5 and data D6 can also be taken offline.

[0055] When data or tasks enter the pre-offline pool for a preset number of consecutive days, they enter the available offline pool. Finally, all data that enters the available offline pool is determined as candidate offline data.

[0056] In this embodiment, the data to be taken offline is screened in three rounds: a candidate pool, a pre-offline pool, and a possible offline pool. This can more accurately analyze the status of useless data, facilitate adjustment of the offline strategy, and prevent erroneous offline operations.

[0057] S140: Perform corresponding data offline operation on the candidate offline data according to a preset data offline operation strategy.

[0058] Figure 6 A flow chart of the data offline operation provided by an embodiment of the present invention is shown as follows: Figure 6 As shown, it includes steps S141 to S145.

[0059] S141: Obtain a notice period corresponding to the data offline operation policy. Generate offline notice information corresponding to the candidate offline data within the notice period and send it to the corresponding receiving terminal. The corresponding receiving terminals include: the task owner, creator, business domain owner, upstream owner, and system administrator who produced the Hive table data. These receiving terminals can be prioritized as follows: task owner > creator > business domain owner > upstream owner > system administrator.

[0060] In addition, it should be noted that the time interval of the forecast period in this embodiment includes but is not limited to the time of one forecast, and there may be multiple forecast times during the period.

[0061] S142: If a data retention request sent by the receiving terminal is detected within the notice period, the target data corresponding to the data retention request is removed from the candidate offline data to update the candidate offline data, and the target data is saved in the whitelist database.

[0062] S143, obtaining the execution time interval corresponding to the data offline operation policy, deleting the candidate offline data and saving it to the backup library. If the candidate offline data has a corresponding scheduling task within the execution time interval, suspending the task scheduling at the same time.

[0063] S144, obtain the data recovery time interval corresponding to the data offline operation strategy. If a data recovery request sent by the receiving terminal is detected within the data recovery time interval, obtain the target recovery data corresponding to the data recovery request for data rollback recovery, and remove the target recovery data from the backup library to update the backup library.

[0064] S145: If it is detected that the current system time exceeds the end time point of the data recovery time interval, the data in the backup library is deleted.

[0065] In one embodiment, in order to ensure security and reduce the adverse impact of incorrect offline operations on the business, some security designs will be performed before the offline operation is implemented on the candidate offline data, including: setting a whitelist to save the data and tasks that need to be retained; setting the backup and reconstruction table SQL. Through the backup and reconstruction table SQL, even if the data is cleared, a Hive table of the same format can be quickly established, and then the cleared data can be restored by re-running the corresponding task; setting a rollback to restore the data and tasks retained in the backup library through rollback; setting the operation switch and the effective range. The operation switch is used to shut down the entire system during special periods, and the effective range is used to set the range of offline data; and setting a date reminder. If it is a holiday, the offline operation will be stopped to prevent important messages from being ignored or abnormalities from being handled in time. If it is a weekday, the daily reminder and the number of offline operations will be set to minimize the potential impact and the complexity of problem handling.

[0066] Figure 7 A schematic diagram of a specific process of data offline operation provided by an embodiment of the present invention. Figure 7 The following example illustrates the process of data offline operation:

[0067] S1. Check the switch status and date. Only when the switch is turned on and it is not a holiday, do the offline operation;

[0068] S2. Obtain a list of data to be taken offline, excluding data that is on the whitelist or in rollback status and needs to be retained. Then sort the filtered data by the priority of the person in charge;

[0069] S3. For each person in charge, process the data under their name in turn. Once the processing is completed, or the number reaches the set limit, the detailed results of this execution will be sent to the relevant person in charge;

[0070] S4. If the number has not reached the upper limit, a offline determination is performed for each data that can be offlined. Specifically, the process includes: determining the current status of the data; if the number of offline notifications for the data is less than the preset number, continuing to issue offline notifications; if the data has been successfully offlined or has been reminded, skipping the process; if the number of offline notifications for the data is greater than or equal to the preset number, or if the previous offline operation failed, performing the offline operation. If there is a task, the task will be paused first, then the table creation SQL will be backed up, and finally the table will be moved to the backup database.

[0071] S5. The final result of the offline, including details and status, will be sent to the administrator. The administrator can be aware of any abnormalities.

[0072] Through the above steps, the embodiment of the present invention parses the operation log and HDFS read and write messages to obtain the usage and output relationship between tasks and data, and builds a complete data flow link, thereby more accurately mining useless data based on the upstream and downstream relationships of the data in the data flow link and the set screening rules; finally, the useless data is automatically offlined to release storage and computing resources, which greatly reduces the risk of manual error operation, improves the efficiency of data cleaning, and improves the efficiency and accuracy of Hive data offline.

[0073] The embodiment of the present invention also provides a Hive data automatic offline device, Figure 8 A schematic block diagram of a Hive data automatic offline device provided by an embodiment of the present invention, such as Figure 8 As shown, the system includes an acquisition module 810 , an association module 820 , an extraction module 830 and an offline module 840 .

[0074] The acquisition module 810 is used to acquire Hive table data and task meta information, and acquire data logs corresponding to the Hive table data and task meta information.

[0075] Specifically, obtain Hive table data from the Hive database to be analyzed.

[0076] Obtain task meta-information from the task scheduling system; wherein, task meta-information includes at least import task meta-information, processing task meta-information, and export task meta-information. It should be noted that the first type of import task is used to import business data (MySQL data) or other external data (such as Kafka, etc.) into the Hive database. This type of task records the data source and target table information; the second type of processing task is used to process one or more Hive table data and finally write the results to another Hive table; the third type of export task is used to export Hive table data to an external database (such as MySQL, HBase, etc.) for business use. This type of task also records the data source and target table information. All of the above tasks may depend on one or more tables, and may also depend on other tasks to produce or update table data.

[0077] Obtain data logs from the HDFS system; data logs include at least read and write logs and user behavior logs.

[0078] Figure 2 This is a schematic diagram of the relationship between data, tasks and various logs provided in the embodiment of the present invention. Figure 2 As an example, we retrieve all Hive table data D from the Hive database to be analyzed (i.e., the Hive MetaStore), including the Hive table name, owner information, and management information. We also retrieve metadata for all tasks T from the task scheduling system, including the task name, task type, owner information, and configuration information. Finally, we retrieve data logs from the HDFS system, including read and write logs, task execution logs, and user behavior logs. These data sets are interconnected.

[0079] It should be noted that Hive MetaStore is the metadata management database that comes with the Hive system. All Hive table data can be collected from this database. Table types can be divided into: common tables, external tables, and view tables. Figure 3 For data that does not need to be updated regularly, it is manually marked as a static table. In addition, through naming, the business domain to which a table belongs can be automatically determined.

[0080] The association module 820 is used to determine the data flow chain diagram based on the association relationship between the data log, Hive table data and task meta information, and determine the data status table based on the data flow chain diagram.

[0081] Preferably, the upstream and downstream relationship between Hive table data and task meta-information is obtained; the data log is parsed to obtain the usage and output information of the Hive table in the Hive table data; then, based on the upstream and downstream relationship between the Hive table data and task meta-information, and the usage and output information of the Hive table in the Hive table data, the data flow chain diagram is determined; and based on the data flow chain diagram and the usage and output information of the Hive table, the data status table is determined.

[0082] Specifically, the upstream and downstream relationship between Hive table data and task meta-information is obtained, which includes: determining the dependent output relationship between Hive table data and task meta-information based on the task configuration information and task code information in the task meta-information, and using the dependent output relationship as the upstream and downstream relationship between the Hive table data and the task meta-information.

[0083] Next, the data logs are parsed to obtain Hive table usage and output information within the Hive table data. The parsing steps include: obtaining the corresponding file information from each ClientId through read and write logs; obtaining the corresponding task through ClientId association, and obtaining the corresponding Hive table through file path association; obtaining the SQL used to execute the query through user behavior logs, and extracting the Hive tables used and written to through the SQL; from this, the usage information corresponding to each data point in the Hive table data can be determined.

[0084] Then, the data flow chain diagram is determined based on the upstream and downstream relationships between Hive table data and task meta-information, as well as the usage and output information of the Hive table in the Hive table data. Figure 3 A schematic diagram of a data flow link provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, task T0 produces data D1 and D2; then, task T1 uses data D1 and D2 to produce data D3, which is used by developers, such as data analysis; or, task T2 depends on task T1 and uses data D4 to produce data D5, where data D4 is queried and used by the system and developers; or, task T3 depends on tasks T1 and T2, which is equivalent to using data D3 and D5 produced by tasks T1 and T2 to produce data D6. Figure 3 The data flow link diagram shown can easily find the upstream and downstream relationships between data and tasks, that is, which task produces a certain data and which tasks depend on it. It can also directly obtain the specific usage of the data.

[0085] Finally, based on the data flow chain diagram and Hive table usage and output information, the data status table is determined. This includes: determining the output tasks, creation time, consecutive failure days, downstream quantity, and consecutive days without queries corresponding to each data table based on the data flow chain diagram and Hive table usage and output information; and combining the output tasks, creation time, consecutive failure days, downstream quantity, and consecutive days without queries corresponding to each data table to obtain the data status table. Figure 4 The data status provided in the embodiment of the present invention is shown as follows: Figure 4 As shown, for example, according to Figure 3 The data flow chain diagram and Hive table usage and output information shown in the figure can determine that the creation time of Hive table data D1 is 2023-01-01, the corresponding output task is T0, the number of consecutive failure days is 0, the number of downstream data is 1, and the number of consecutive days without query is 365. Combining the output tasks, creation time, consecutive failure days, number of downstream data, and consecutive days without query for all data tables, we can get Figure 4 The data status table is shown.

[0086] The extraction module 830 is configured to obtain a preset data offline policy and determine candidate offline data based on the data offline policy and the data status table.

[0087] Preferably, the extraction module 830 is used in S131 and S132, specifically as follows:

[0088] S131. Based on the data quality detection sub-strategy in the data offline strategy, the data status table is tested to obtain a quality inspection result. The specific process includes:

[0089] Before selecting candidate offline data, the collected basic data must be quality-checked. Specific quality check criteria include: the amount of collected basic data and tasks cannot fluctuate significantly, the amount of data without downstream resources cannot increase dramatically, and there cannot be a large number of Hive table behavior logs that cannot be parsed to reveal usage and output information. If the basic data fails to meet the preset conditions of the quality check sub-strategy, the current task will be interrupted and the administrator will intervene to manually judge and handle the problem. The task will then be restarted after confirmation.

[0090] S132: If the quality inspection result is determined to be normal, candidate offline data is determined based on the screening sub-strategy in the data offline strategy and the data status table. The specific process includes:

[0091] Preset offline conditions determine whether the data and tasks in the data status table meet the conditions. If they do, they are placed in the offline candidate pool. Specific offline conditions include: 1. Tasks that have failed for a long time. The term "long-term" can be determined based on the task scheduling cycle. For example, if the cycle is "days," 15 consecutive days of failure; or if the cycle is "weeks," 4 consecutive weeks of failure. 2. Data tables without downstream tasks, that is, tables not directly used by other tasks.

[0092] Preset filtering conditions, and use these filtering conditions to filter the data and tasks in the offline candidate pool. The filtered data and tasks enter the pre-offline pool. Specific filtering conditions include: 1. The table or task was created recently, and the corresponding business has not yet been launched and needs to be retained. The "recent" here can be set manually according to actual conditions. For example, tables created within one month can be considered recent. 2. Some tables that require special protection, such as manually maintained static tables. 3. Tables and tasks that have been used recently by developers or systems and may be used for statistical and analytical scenarios. The "recent" here can be set manually according to actual conditions. 4. Need to be retained for other reasons, for example, tables or tasks manually added to the whitelist.

[0093] According to the blood relationship between data and tasks in the data status table, the parent table of offline data and tasks is expanded into the pre-offline pool. Figure 3 In the example, data D3 has tasks T2 and T3 downstream. When the system finds that data D5 (corresponding to task T2) and data D6 (corresponding to task T3) can be taken offline, that is, the downstream of D3 can be taken offline, then the parent table D3 of data D5 and data D6 can also be taken offline.

[0094] When data or tasks enter the pre-offline pool for a preset number of consecutive days, they enter the available offline pool. Finally, all data that enters the available offline pool is determined as candidate offline data.

[0095] In this embodiment, the data to be taken offline is screened in three rounds: a candidate pool, a pre-offline pool, and a possible offline pool. This can more accurately analyze the status of useless data, facilitate adjustment of the offline strategy, and prevent erroneous offline operations.

[0096] The offline module 840 is configured to perform a corresponding data offline operation on the candidate offline data according to a preset data offline operation strategy.

[0097] Preferably, the offline module 840 is used for S141 to S145, specifically as follows:

[0098] S141: Obtain a notice period corresponding to the data offline operation policy. Generate offline notice information corresponding to the candidate offline data within the notice period and send it to the corresponding receiving terminal. The corresponding receiving terminals include: the task owner, creator, business domain owner, upstream owner, and system administrator who produced the Hive table data. These receiving terminals can be prioritized as follows: task owner > creator > business domain owner > upstream owner > system administrator.

[0099] In addition, it should be noted that the time interval of the forecast period in this embodiment includes but is not limited to the time of one forecast, and there may be multiple forecast times during the period.

[0100] S142: If a data retention request sent by the receiving terminal is detected within the notice period, the target data corresponding to the data retention request is removed from the candidate offline data to update the candidate offline data, and the target data is saved in the whitelist database.

[0101] S143, obtaining the execution time interval corresponding to the data offline operation policy, deleting the candidate offline data and saving it to the backup library. If the candidate offline data has a corresponding scheduling task within the execution time interval, suspending the task scheduling at the same time.

[0102] S144, obtain the data recovery time interval corresponding to the data offline operation strategy. If a data recovery request sent by the receiving terminal is detected within the data recovery time interval, obtain the target recovery data corresponding to the data recovery request for data rollback recovery, and remove the target recovery data from the backup library to update the backup library.

[0103] S145: If it is detected that the current system time exceeds the end time point of the data recovery time interval, the data in the backup library is deleted.

[0104] In one embodiment, in order to ensure security and reduce the adverse impact of incorrect offline operations on the business, some security designs will be performed before the offline operation is implemented on the candidate offline data, including: setting a whitelist to save the data and tasks that need to be retained; setting the backup and reconstruction table SQL. Through the backup and reconstruction table SQL, even if the data is cleared, a Hive table of the same format can be quickly established, and then the cleared data can be restored by re-running the corresponding task; setting a rollback to restore the data and tasks retained in the backup library through rollback; setting the operation switch and the effective range. The operation switch is used to shut down the entire system during special periods, and the effective range is used to set the range of offline data; and setting a date reminder. If it is a holiday, the offline operation will be stopped to prevent important messages from being ignored or abnormalities from being handled in time. If it is a weekday, the daily reminder and the number of offline operations will be set to minimize the potential impact and the complexity of problem handling.

[0105] Figure 7 A schematic diagram of a specific process of data offline operation provided by an embodiment of the present invention. Figure 7 Taking the figure as an example, the process of data offline operation is described in detail. The steps are as follows:

[0106] S1. Check the switch status and date. Only when the switch is turned on and it is not a holiday, do the offline operation;

[0107] S2. Obtain a list of data to be taken offline, excluding data that is on the whitelist or in rollback status and needs to be retained. Then sort the filtered data by the priority of the person in charge;

[0108] S3. For each person in charge, process the data under their name in turn. Once the processing is completed, or the number reaches the set limit, the detailed results of this execution will be sent to the relevant person in charge;

[0109] S4. If the number has not reached the upper limit, a offline determination is performed for each data that can be offlined. Specifically, the process includes: determining the current status of the data; if the number of offline notifications for the data is less than the preset number, continuing to issue offline notifications; if the data has been successfully offlined or has been reminded, skipping the process; if the number of offline notifications for the data is greater than or equal to the preset number, or if the previous offline operation failed, performing the offline operation. If there is a task, the task will be paused first, then the table creation SQL will be backed up, and finally the table will be moved to the backup database.

[0110] S5. The final result of the offline, including details and status, will be sent to the administrator. The administrator can be aware of any abnormalities.

[0111] Through the above-mentioned device, the embodiment of the present invention parses the operation log and HDFS read and write messages to obtain the usage and output relationship between tasks and data, and builds a complete data flow link, so as to more accurately mine useless data according to the upstream and downstream relationships of the data in the data flow link and the set screening rules; finally, the useless data is automatically offlined to release storage and computing resources, which greatly reduces the risk of manual error operation, improves the efficiency of data cleaning, and improves the efficiency and accuracy of Hive data offline.

[0112] The above-mentioned Hive data automatic offline method can be implemented in the form of a computer program. The computer program can be used in the following ways: Figure 9 Runs on the computer equipment shown.

[0113] See also Figure 9 , Figure 9 The figure is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device may be used to execute a method for automatically logging off Hive data.

[0114] See Figure 9 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0115] The storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the Hive data automatic offline method, wherein the storage medium 503 can be a volatile storage medium or a non-volatile storage medium.

[0116] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0117] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the Hive data automatic offline method.

[0118] The network interface 505 is used for network communication, such as providing data information transmission. Those skilled in the art will understand that Figure 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0119] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the corresponding functions of the aforementioned Hive data automatic offline method.

[0120] Those skilled in the art will understand that Figure 9 The embodiment of the computer device shown in the figure does not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and processor are the same as those in the figure. Figure 9 The embodiments shown are consistent and will not be described again here.

[0121] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0122] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be volatile or non-volatile. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps included in the aforementioned method for automatically logging off Hive data.

[0123] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, or units with the same function may be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.

[0125] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the objectives of the embodiments of the present invention.

[0126] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a computer-readable storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for automatically logging off Hive data, characterized in that: The method comprises: Obtain Hive table data and task meta information, and obtain data logs corresponding to the Hive table data and the task meta information; Determine a data flow chain diagram based on the association relationship between the data log, the Hive table data, and the task meta information, and determine a data status table based on the data flow chain diagram; Obtaining a preset data offline strategy, and determining candidate offline data based on the data offline strategy and the data status table; According to the preset data offline operation strategy, the corresponding data offline operation is performed on the candidate offline data; The obtaining of a preset data offline strategy and determining candidate offline data based on the data offline strategy and the data status table includes: Detecting the data status table based on the data quality detection sub-strategy in the data offline strategy to obtain a quality inspection result; If the quality inspection result is determined to be normal, determining the candidate offline data based on the screening sub-strategy in the data offline strategy and the data status table; The step of performing a corresponding data offline operation on the candidate offline data according to a preset data offline operation strategy includes: Acquire a notice period time interval corresponding to the data offline operation policy, generate offline notice notification information corresponding to the candidate offline data within the notice period time interval, and send the offline notice notification information to the corresponding receiving terminal; If a data retention request sent by a receiving terminal is detected within the notice period, the target data corresponding to the data retention request is removed from the candidate offline data to update the candidate offline data, and the target data is saved in a whitelist database; Obtaining an execution time interval corresponding to the data offline operation policy, deleting the candidate offline data and saving it to a backup repository; if the candidate offline data has a corresponding scheduled task within the execution time interval, pausing the task simultaneously; Obtaining a data recovery time interval corresponding to the data offline operation policy; if a data recovery request sent by a receiving terminal is detected within the data recovery time interval, obtaining target recovery data corresponding to the data recovery request for data rollback recovery, and removing the target recovery data from the backup repository to update the backup repository; If it is detected that the current system time exceeds the end time point of the data recovery time interval, the data in the backup library will be deleted.

2. The method according to claim 1, characterized in that The obtaining of Hive table data and task meta information, and obtaining data logs corresponding to the Hive table data and the task meta information, includes: Obtain the Hive table data from the Hive database to be analyzed; Acquire the task meta-information from the task scheduling system; wherein the task meta-information includes at least import task meta-information, processing task meta-information and export task meta-information; The data log is obtained from the HDFS system; wherein the data log includes at least a read / write log and a user behavior log.

3. The method according to claim 1, characterized in that The determining of a data flow chain diagram based on the association relationship among the data log, the Hive table data, and the task meta-information, and determining a data status table based on the data flow chain diagram, includes: Obtaining the upstream and downstream relationship between the Hive table data and the task meta information; Parsing the data log to obtain usage and output information of the Hive table in the Hive table data; Determine the data flow chain diagram according to the upstream and downstream relationship between the Hive table data and the task meta information, and the usage and output information of the Hive table in the Hive table data; The data status table is determined according to the data flow chain diagram and the usage and output information of the Hive table.

4. The method according to claim 3, characterized in that The obtaining of the upstream and downstream relationship between the Hive table data and the task meta information includes: According to the task configuration information and task code information in the task meta-information, a dependency output relationship between the Hive table data and the task meta-information is determined, and the dependency output relationship is used as the upstream and downstream relationship between the Hive table data and the task meta-information.

5. The method according to claim 3, characterized in that Determining the data status table according to the data flow chain diagram and the usage and output information of the Hive table includes: Determine the output task, creation time, consecutive failure days, downstream quantity, and consecutive no-query days corresponding to each data table based on the data flow chain diagram and the usage and output information of the Hive table; The output tasks, creation time, consecutive failure days, downstream quantity and consecutive no-query days corresponding to each data table are combined to obtain the data status table.

6. A Hive data automatic offline device, characterized in that: include: An acquisition module is used to obtain Hive table data and task meta information, and obtain data logs corresponding to the Hive table data and the task meta information; An association module, configured to determine a data flow chain diagram based on the association relationship among the data log, the Hive table data, and the task meta-information, and determine a data status table based on the data flow chain diagram; An extraction module, configured to obtain a preset data offline strategy, and determine candidate offline data based on the data offline strategy and the data status table; An offline module is used to perform a corresponding data offline operation on the candidate offline data according to a preset data offline operation strategy; The extraction module is specifically used for: The data status table is tested based on the data quality detection sub-strategy in the data offline strategy to obtain a quality inspection result; if the quality inspection result is determined to be normal, the candidate offline data is determined based on the screening sub-strategy in the data offline strategy and the data status table; The offline module is specifically used for: Acquire a notice period time interval corresponding to the data offline operation policy, generate offline notice notification information corresponding to the candidate offline data within the notice period time interval, and send the offline notice notification information to the corresponding receiving terminal; If a data retention request sent by a receiving terminal is detected within the notice period, the target data corresponding to the data retention request is removed from the candidate offline data to update the candidate offline data, and the target data is saved to the whitelist library; the execution time interval corresponding to the data offline operation policy is obtained, the candidate offline data is deleted and saved to the backup library, and if the candidate offline data has a corresponding scheduled task within the execution time interval, the task is suspended at the same time; the data recovery time interval corresponding to the data offline operation policy is obtained, and if a data recovery request sent by a receiving terminal is detected within the data recovery time interval, the target recovery data corresponding to the data recovery request is obtained for data rollback recovery, and the target recovery data is removed from the backup library to update the backup library; If it is detected that the current system time exceeds the end time point of the data recovery time interval, the data in the backup library will be deleted.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images