Data warehouse cleaning method and device, computer device and storage medium
By acquiring and parsing the task execution logs of the data warehouse, multi-dimensional task evaluation indicators are generated to identify tasks to be cleaned, thus solving the problems of inaccurate data warehouse cleanup and resource waste, and achieving precise and effective cleanup.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA PING AN PROPERTY INSURANCE CO LTD
- Filing Date
- 2022-11-07
- Publication Date
- 2026-05-26
AI Technical Summary
In existing technologies, data warehouse cleaning mainly relies on manual labor, leading to resource waste and inaccurate cleaning.
By acquiring the task execution logs of the data warehouse, parsing multi-dimensional MapReduce task attributes, generating multi-dimensional task evaluation metrics, and identifying and cleaning up tasks based on these metrics.
It enables precise and effective cleaning of the data warehouse, reduces resource waste, and improves the accuracy of cleaning.
Smart Images
Figure CN115757373B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a data warehouse cleaning method, apparatus, computer equipment, and storage medium. Background Technology
[0002] With the development of big data technology, various organizations are increasingly relying on it for their production and operations. For example, in the financial and insurance sector, business activities exist as tasks on big data platforms, and related data is stored in data warehouses. Over time and with the increase in business volume, these tasks consume significant storage and computing resources. However, current data warehouse cleanup is mostly done manually. Manual data warehouse cleanup often fails to effectively clean the data warehouse due to inaccurate task assessments, leading to a waste of data warehouse resources. Summary of the Invention
[0003] The purpose of this application is to provide a data warehouse cleaning method, apparatus, computer equipment, and storage medium to reduce resource waste in the data warehouse.
[0004] To address the aforementioned technical problems, this application provides a data warehouse cleanup method, employing the following technical solution:
[0005] Retrieve task execution logs from the data warehouse;
[0006] The task execution logs are parsed to obtain multi-dimensional MR task attributes;
[0007] Based on the multi-dimensional MR task attributes, multi-dimensional task evaluation indicators are generated for each existing task.
[0008] Based on the multi-dimensional task evaluation indicators, tasks to be cleaned up are identified among the existing tasks.
[0009] Clean up the tasks to be cleaned to complete the data warehouse cleanup.
[0010] Furthermore, prior to the step of obtaining the task execution logs of the data warehouse, the following steps are also included:
[0011] Get existing tasks;
[0012] Generate an MR task for the existing task, wherein the MR task is a mapreduce task;
[0013] Run the MR task and monitor its execution process to obtain a task execution log.
[0014] Furthermore, the step of parsing the task execution log to obtain multi-dimensional MR task attributes includes:
[0015] The task execution log is parsed using HQL to extract structured information from multiple preset dimensions, resulting in multi-dimensional MR task attributes.
[0016] Furthermore, the step of generating multi-dimensional task evaluation metrics for each existing task based on the multi-dimensional MR task attributes includes:
[0017] ETL processing is performed on the multi-dimensional MR task attributes corresponding to each existing task to obtain the multi-dimensional task evaluation index for each existing task.
[0018] Furthermore, the step of determining the tasks to be cleaned up among the existing tasks based on the multi-dimensional task evaluation indicators includes:
[0019] Based on the multi-dimensional task evaluation indicators, calculate the resource consumption evaluation value for each existing task.
[0020] Existing tasks whose resource consumption assessment values meet the preset resource consumption conditions are selected to obtain high-consumption tasks;
[0021] Based on the multi-dimensional task evaluation indicators, existing tasks that meet the preset value conditions are selected from the existing tasks to obtain low-value tasks.
[0022] The high-consumption tasks and the low-value tasks are identified as tasks to be cleaned up.
[0023] Furthermore, the step of calculating the resource consumption assessment value of each existing task based on the multi-dimensional task evaluation indicators includes:
[0024] The indicators in the multi-dimensional task evaluation index are standardized.
[0025] The weights of each indicator after standardization are determined by a preset weighting algorithm.
[0026] The obtained indicator weights and the standardized indicators are weighted and calculated to obtain the resource consumption assessment value of each existing task.
[0027] To address the aforementioned technical problems, this application also provides a data warehouse cleanup device, which employs the following technical solution:
[0028] The log acquisition module is used to acquire task execution logs from the data warehouse.
[0029] The log parsing module is used to parse the task execution log to obtain multi-dimensional MR task attributes;
[0030] The indicator generation module is used to generate multi-dimensional task evaluation indicators for each existing task based on the multi-dimensional MR task attributes.
[0031] The task determination module is used to determine the tasks to be cleaned up from the existing tasks based on the multi-dimensional task evaluation indicators.
[0032] The task cleanup module is used to clean up the tasks to be cleaned up, thereby completing the data warehouse cleanup.
[0033] Furthermore, the task determination module also includes:
[0034] The resource consumption calculation submodule is used to calculate the resource consumption evaluation value of each existing task according to the multi-dimensional task evaluation index.
[0035] The first selection submodule is used to select existing tasks whose resource consumption assessment values meet the preset resource consumption conditions, thus obtaining high-consumption tasks.
[0036] The second selection submodule is used to select existing tasks that meet preset value conditions from the existing tasks according to the multi-dimensional task evaluation indicators, and obtain low-value tasks.
[0037] The task determination submodule is used to identify the high-consumption tasks and the low-value tasks as tasks to be cleaned up.
[0038] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:
[0039] Retrieve task execution logs from the data warehouse;
[0040] The task execution logs are parsed to obtain multi-dimensional MR task attributes;
[0041] Based on the multi-dimensional MR task attributes, multi-dimensional task evaluation indicators are generated for each existing task.
[0042] Based on the multi-dimensional task evaluation indicators, tasks to be cleaned up are identified among the existing tasks.
[0043] Clean up the tasks to be cleaned to complete the data warehouse cleanup.
[0044] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:
[0045] Retrieve task execution logs from the data warehouse;
[0046] The task execution logs are parsed to obtain multi-dimensional MR task attributes;
[0047] Based on the multi-dimensional MR task attributes, multi-dimensional task evaluation indicators are generated for each existing task.
[0048] Based on the multi-dimensional task evaluation indicators, tasks to be cleaned up are identified among the existing tasks.
[0049] Clean up the tasks to be cleaned to complete the data warehouse cleanup.
[0050] Compared with existing technologies, the embodiments of this application have the following main advantages: The task operation logs of the data warehouse are obtained; the big data platform runs existing tasks in the form of MapReduce tasks, and the operation process is recorded in the task operation logs; the task operation logs are parsed to obtain multi-dimensional MapReduce task attributes, which describe the attribute information of the MapReduce task in multiple dimensions; based on the multi-dimensional MapReduce task attributes, multi-dimensional task evaluation indicators are generated for each existing task; the multi-dimensional task evaluation indicators describe existing tasks from multiple dimensions, and based on the multi-dimensional task evaluation indicators, the resource consumption and task value of existing tasks can be comprehensively evaluated, thereby accurately identifying and cleaning tasks to be cleaned, achieving precise and effective cleaning of the data warehouse, and reducing the waste of data warehouse resources. Attached Figure Description
[0051] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;
[0053] Figure 2 This is a flowchart of an embodiment of the data warehouse cleanup method according to this application;
[0054] Figure 3 This is a schematic diagram of a structure of an embodiment of the data warehouse cleanup apparatus according to this application;
[0055] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation
[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.
[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0059] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.
[0061] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.
[0062] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.
[0063] It should be noted that the data warehouse cleanup method provided in this application embodiment is generally executed by a server, and correspondingly, the data warehouse cleanup device is generally located in the server.
[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0065] Continue to refer to Figure 2 A flowchart of an embodiment of the data warehouse cleanup method according to this application is shown. The data warehouse cleanup method includes the following steps:
[0066] Step S201: Obtain the task execution log of the data warehouse.
[0067] In this embodiment, the data warehouse cleanup method runs on an electronic device (e.g., Figure 1 The server shown can communicate with the terminal via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.
[0068] Specifically, a data warehouse contains multiple tasks, which may be related to specific business operations. Task execution logs, or historical records of task execution, document the process of each task.
[0069] It should be emphasized that, to further ensure the privacy and security of the above task execution logs, the above task execution logs can also be stored in a blockchain node.
[0070] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.
[0071] Furthermore, before step S201 above, the process may include: obtaining existing tasks; generating MR tasks for existing tasks, wherein the MR tasks are MapReduce tasks; running the MR tasks and monitoring the running process of the MR tasks to obtain task running logs.
[0072] The existing tasks can be those currently existing in the data warehouse. In one embodiment, the existing tasks can be T+1 offline tasks (T is a time unit, and T+1 offline tasks can be tasks that generate daily reports or daily metrics). These tasks account for a significant proportion of big data business and consume a lot of resources.
[0073] Specifically, the process involves acquiring existing tasks from the data warehouse and submitting them to a big data cluster, which can be a Hadoop (Big Data Software System Framework) cluster. These existing tasks need to be transformed into MapReduce tasks, which are then allocated computing resources by the Hadoop cluster and scheduled by Yarn, the component within Hadoop responsible for resource scheduling and task management.
[0074] The execution process of MR tasks is monitored, and the generated historical records are stored in the jobhistory HDFS file. The jobhistory can be a server that stores the historical records of MR task execution.
[0075] In this embodiment, existing tasks are transformed into MapReduce tasks, which are then run on a big data platform. The running process of the MapReduce tasks is monitored to obtain task running logs, which can then be used to clean up the data warehouse.
[0076] Step S202: Analyze the task execution log to obtain multi-dimensional MR task attributes.
[0077] Specifically, the task execution logs are parsed to generate multi-dimensional MR task attributes. These multi-dimensional MR task attributes can include MR task attributes from multiple dimensions. MR task attributes refer to the attribute information of MR tasks, which can quantitatively describe MR tasks.
[0078] Furthermore, step S202 above may include: parsing the task execution log through HQL to extract structured information from multiple preset dimensions and obtain multi-dimensional MR task attributes.
[0079] Specifically, the task execution log can be a nested JSON structure. The JSON structure can be parsed using HQL (Hibernate Query Language, an object-oriented query language similar to SQL, but it is object-oriented and focuses on their properties). During the parsing process, the attribute information in the task execution log is identified, and information is extracted from the task execution log from multiple preset attribute dimensions. The extracted information is then written into a structured Hive table to obtain multi-dimensional MapReduce task attributes.
[0080] In one embodiment, a multi-dimensional MR attribute task may include task submission time, task completion time, number of map tasks, number of reduce tasks, etc.
[0081] In this embodiment, task execution logs are parsed using HQL, thereby extracting MR task attributes from multiple dimensions, which prepares the data for data warehouse cleanup based on task execution logs.
[0082] Step S203: Based on the multi-dimensional MR task attributes, generate multi-dimensional task evaluation indicators for each existing task.
[0083] Specifically, multi-dimensional MR task attributes are descriptions of the MR tasks corresponding to existing tasks. Multi-dimensional MR task attributes can be statistically analyzed and calculated according to preset calculation methods to obtain multi-dimensional task evaluation indicators that describe existing tasks.
[0084] Furthermore, step S203 may include: performing ETL processing on the multi-dimensional MR task attributes corresponding to each existing task to obtain multi-dimensional task evaluation indicators for each existing task.
[0085] Specifically, each existing task has multi-dimensional MR task attributes, which can be processed by ETL (Extract, Transform, Load) to obtain the corresponding multi-dimensional task evaluation indicators for each existing task.
[0086] In one embodiment, multi-dimensional task evaluation metrics may include metrics such as the CPU (central processing unit) resources consumed by an existing task within a preset time, task execution time, number of MapReduce (MR) operations, amount of HDFS data read, and number of job stages in task startup.
[0087] In this embodiment, the multi-dimensional MR task attributes are processed using ETL to accurately extract the required multi-dimensional task evaluation indicators.
[0088] Step S204: Based on the multi-dimensional task evaluation indicators, identify the tasks to be cleaned up among the existing tasks.
[0089] Specifically, based on multi-dimensional task evaluation metrics, existing tasks in the data warehouse can be analyzed and classified to identify tasks with unreasonable resource consumption and mark them as tasks to be cleaned up. The multi-dimensional task evaluation metrics include multiple dimensions of task evaluation indicators. When one of the task evaluation indicators fails to meet the preset evaluation conditions, the existing task can be identified as a task to be cleaned up; alternatively, existing tasks can be comprehensively evaluated based on multiple dimensions of task evaluation indicators, and if they do not meet the preset evaluation conditions, they can be identified as tasks to be cleaned up.
[0090] Step S205: Clean up the tasks to be cleaned up and complete the data warehouse cleanup.
[0091] Specifically, the identified tasks to be cleaned are cleaned up, for example, by deleting the tasks, sending notification messages to the terminals logged in by the preset accounts so that developers can delete the tasks according to the notification messages, or modifying the tasks to reduce their resource consumption, thereby completing the cleanup of the data warehouse.
[0092] In this embodiment, task execution logs of the data warehouse are obtained. The big data platform runs existing tasks in the form of MapReduce tasks, and the execution process is recorded in the task execution logs. The task execution logs are parsed to obtain multi-dimensional MapReduce task attributes, which describe the attribute information of the MapReduce task in multiple dimensions. Based on the multi-dimensional MapReduce task attributes, multi-dimensional task evaluation indicators are generated for each existing task. The multi-dimensional task evaluation indicators describe the existing tasks from multiple dimensions. According to the multi-dimensional task evaluation indicators, the resource consumption and task value of the existing tasks can be comprehensively evaluated, so as to accurately find the tasks to be cleaned and clean them up, thereby achieving precise and effective cleanup of the data warehouse and reducing the waste of data warehouse resources.
[0093] Furthermore, step S204 may include: calculating the resource consumption assessment value of each existing task according to the multi-dimensional task evaluation index; selecting existing tasks whose resource consumption assessment values meet the preset resource consumption conditions to obtain high-consumption tasks; selecting existing tasks that meet the preset value conditions from among the existing tasks according to the multi-dimensional task evaluation index to obtain low-value tasks; and identifying high-consumption tasks and low-value tasks as tasks to be cleaned up.
[0094] Specifically, tasks to be cleaned up can be categorized into two types: high-consumption tasks and low-value tasks. High-consumption tasks are those that consume significant computing and storage resources during execution. Existing tasks can be evaluated using multi-dimensional task assessment metrics to calculate their resource consumption assessment values. These values measure the resource consumption of existing tasks numerically; a higher value indicates greater resource consumption.
[0095] Resource consumption assessment values can be calculated using a single-dimensional or multi-dimensional approach. Single-dimensional calculation refers to determining whether a single task assessment indicator (which can then be used as the resource consumption assessment value) meets preset resource consumption conditions. If a single task assessment indicator meets the preset resource consumption conditions, the existing task is identified as a high-consumption task. For example, if the CPU resources consumed within a preset time exceed the preset resource consumption condition of 5000C, or the task runtime exceeds the preset resource consumption condition of 300 minutes, or the amount of HDFS data read by the task exceeds the preset resource consumption condition of 2TB, or the number of MapReduce (MR) runs by the task exceeds the preset resource consumption condition of 10,000, or the number of job stages run by the task exceeds the preset resource consumption condition of 100, then the existing task is identified as a high-consumption task.
[0096] Multi-dimensional calculation refers to the comprehensive calculation of resource consumption from multiple task evaluation indicators. If the total resource consumption evaluation value reaches the preset resource consumption condition, the existing task is identified as a high-consumption task. In one embodiment, the multi-dimensional task evaluation indicators can be input into a pre-trained task evaluation model. The task evaluation model can be built based on a neural network. The task evaluation model outputs a resource consumption evaluation value. Based on the magnitude of the resource consumption evaluation value, it is determined whether the existing task meets the preset resource consumption condition.
[0097] Low-value tasks can be those with low value, such as tasks with dormant data tables. A dormant data table can be one where the task's data table has no corresponding downstream data table, and the table's storage and data volume have not changed within a preset time period; or all upstream data tables corresponding to the task's data tables are offline (e.g., the task containing the upstream data table is offline, or the upstream data table exists in isolation from the task). Low-value tasks have low business and practical value and can be cleaned up.
[0098] In this embodiment, tasks are evaluated from the perspectives of resource consumption and value, thereby accurately identifying various existing tasks that need to be cleaned up.
[0099] Furthermore, the steps described above for calculating the resource consumption assessment value of each existing task based on multi-dimensional task evaluation indicators may include: standardizing each indicator in the multi-dimensional task evaluation indicators; determining the indicator weight of each standardized indicator through a preset weighting algorithm; and performing a weighted calculation on the obtained indicator weights and the standardized indicators to obtain the resource consumption assessment value of each existing task.
[0100] Specifically, the multi-dimensional task evaluation index includes a variety of task evaluation indicators, and their numerical ranges may vary significantly. Each indicator can be standardized according to a pre-defined standardization processing strategy.
[0101] Each indicator can also have its own weight, and the sum of the weights of all indicators is 1. The weight of each indicator can be determined by a preset weighting algorithm, such as the CRITIC algorithm, the analytic hierarchy process (AHP), or the Relief algorithm. In one embodiment, the indicator weights can be preset and added to each indicator after querying.
[0102] Based on the obtained indicator weights, the standardized indicators are weighted and calculated, specifically by weighted summation, to obtain the resource consumption assessment value for each existing task.
[0103] In this embodiment, each indicator is standardized for unified calculation, and indicator weights are added to each indicator to distinguish and differentiate them, thereby ensuring the accuracy of the final calculated resource consumption assessment value.
[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).
[0105] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0106] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a data warehouse cleanup device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.
[0107] like Figure 3 As shown, the data warehouse cleanup device 300 described in this embodiment includes: a log acquisition module 301, a log parsing module 302, an indicator generation module 303, a task determination module 304, and a task cleanup module 305, wherein:
[0108] The log acquisition module 301 is used to acquire the task execution logs of the data warehouse.
[0109] The log parsing module 302 is used to parse the task execution logs to obtain multi-dimensional MR task attributes.
[0110] The indicator generation module 303 is used to generate multi-dimensional task evaluation indicators for each existing task based on multi-dimensional MR task attributes.
[0111] The task determination module 304 is used to determine the tasks to be cleaned up from among the existing tasks based on multi-dimensional task evaluation indicators.
[0112] The task cleanup module 305 is used to clean up tasks to be cleaned up and complete the data warehouse cleanup.
[0113] In this embodiment, task execution logs of the data warehouse are obtained. The big data platform runs existing tasks in the form of MapReduce tasks, and the execution process is recorded in the task execution logs. The task execution logs are parsed to obtain multi-dimensional MapReduce task attributes, which describe the attribute information of the MapReduce task in multiple dimensions. Based on the multi-dimensional MapReduce task attributes, multi-dimensional task evaluation indicators are generated for each existing task. The multi-dimensional task evaluation indicators describe the existing tasks from multiple dimensions. According to the multi-dimensional task evaluation indicators, the resource consumption and task value of the existing tasks can be comprehensively evaluated, so as to accurately find the tasks to be cleaned and clean them up, thereby achieving precise and effective cleanup of the data warehouse and reducing the waste of data warehouse resources.
[0114] In some optional implementations of this embodiment, the data warehouse cleanup device 300 may further include: a task acquisition module, a task generation module, and a runtime monitoring module, wherein:
[0115] The task acquisition module is used to acquire existing tasks.
[0116] The task generation module is used to generate MapReduce tasks for existing tasks, where MapReduce tasks are MapReduce tasks.
[0117] The operation monitoring module is used to run MR tasks and monitor the operation process of MR tasks to obtain task operation logs.
[0118] In this embodiment, existing tasks are transformed into MapReduce tasks, which are then run on a big data platform. The running process of the MapReduce tasks is monitored to obtain task running logs, which can then be used to clean up the data warehouse.
[0119] In some optional implementations of this embodiment, the log parsing module 302 is also used to parse the task running log through HQL to extract structured information from multiple preset dimensions and obtain multi-dimensional MR task attributes.
[0120] In this embodiment, task execution logs are parsed using HQL, thereby extracting MR task attributes from multiple dimensions, which prepares the data for data warehouse cleanup based on task execution logs.
[0121] In some optional implementations of this embodiment, the index generation module 303 is also used to perform ETL processing on the multi-dimensional MR task attributes corresponding to each existing task to obtain the multi-dimensional task evaluation index of each existing task.
[0122] In this embodiment, the multi-dimensional MR task attributes are processed using ETL to accurately extract the required multi-dimensional task evaluation indicators.
[0123] In some optional implementations of this embodiment, the task determination module 304 may include: a consumption calculation submodule, a first selection submodule, a second selection submodule, and a task determination submodule, wherein:
[0124] The resource consumption calculation submodule is used to calculate the resource consumption assessment value of each existing task based on multi-dimensional task evaluation indicators.
[0125] The first selection submodule is used to select existing tasks whose resource consumption assessment values meet the preset resource consumption conditions, thus obtaining high-consumption tasks.
[0126] The second selection submodule is used to select existing tasks that meet preset value conditions from among the existing tasks based on multi-dimensional task evaluation indicators, thus obtaining low-value tasks.
[0127] The task identification submodule is used to identify high-consumption tasks and low-value tasks as tasks to be cleaned up.
[0128] In this embodiment, tasks are evaluated from the perspectives of resource consumption and value, thereby accurately identifying various existing tasks that need to be cleaned up.
[0129] In some optional implementations of this embodiment, the consumption calculation submodule may include: a standard processing unit, a weight determination unit, and an index calculation unit, wherein:
[0130] The standard processing unit is used to standardize the various indicators in the multi-dimensional task evaluation metrics.
[0131] The weight determination unit is used to determine the weight of each indicator after standardization by using a preset weight algorithm.
[0132] The indicator calculation unit is used to perform weighted calculations on the obtained indicator weights and the standardized indicators to obtain the resource consumption assessment value of each existing task.
[0133] In this embodiment, each indicator is standardized for unified calculation, and indicator weights are added to each indicator to distinguish and differentiate them, thereby ensuring the accuracy of the final calculated resource consumption assessment value.
[0134] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0135] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.
[0136] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.
[0137] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for data warehouse cleaning methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.
[0138] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, such as executing computer-readable instructions for the data warehouse cleanup method.
[0139] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.
[0140] The computer device provided in this embodiment can execute the data warehouse cleanup method described above. The data warehouse cleanup method here can be any of the data warehouse cleanup methods described in the various embodiments above.
[0141] In this embodiment, task execution logs of the data warehouse are obtained. The big data platform runs existing tasks in the form of MapReduce tasks, and the execution process is recorded in the task execution logs. The task execution logs are parsed to obtain multi-dimensional MapReduce task attributes, which describe the attribute information of the MapReduce task in multiple dimensions. Based on the multi-dimensional MapReduce task attributes, multi-dimensional task evaluation indicators are generated for each existing task. The multi-dimensional task evaluation indicators describe the existing tasks from multiple dimensions. According to the multi-dimensional task evaluation indicators, the resource consumption and task value of the existing tasks can be comprehensively evaluated, so as to accurately find the tasks to be cleaned and clean them up, thereby achieving precise and effective cleanup of the data warehouse and reducing the waste of data warehouse resources.
[0142] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the data warehouse cleanup method described above.
[0143] In this embodiment, task execution logs of the data warehouse are obtained. The big data platform runs existing tasks in the form of MapReduce tasks, and the execution process is recorded in the task execution logs. The task execution logs are parsed to obtain multi-dimensional MapReduce task attributes, which describe the attribute information of the MapReduce task in multiple dimensions. Based on the multi-dimensional MapReduce task attributes, multi-dimensional task evaluation indicators are generated for each existing task. The multi-dimensional task evaluation indicators describe the existing tasks from multiple dimensions. According to the multi-dimensional task evaluation indicators, the resource consumption and task value of the existing tasks can be comprehensively evaluated, so as to accurately find the tasks to be cleaned and clean them up, thereby achieving precise and effective cleanup of the data warehouse and reducing the waste of data warehouse resources.
[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0145] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.
Claims
1. A data warehouse cleanup method, characterized in that, Includes the following steps: Retrieve task execution logs from the data warehouse; The task execution logs are parsed to obtain multi-dimensional MR task attributes; Based on the multi-dimensional MR task attributes, multi-dimensional task evaluation indicators are generated for each existing task. Based on the multi-dimensional task evaluation indicators, tasks to be cleaned up are identified among the existing tasks. Clean up the tasks to be cleaned to complete the data warehouse cleanup; The step of determining the tasks to be cleaned up from the existing tasks based on the multi-dimensional task evaluation indicators specifically includes: Based on the multi-dimensional task evaluation indicators, calculate the resource consumption evaluation value for each existing task. Existing tasks whose resource consumption assessment values meet the preset resource consumption conditions are selected to obtain high-consumption tasks; Based on the multi-dimensional task evaluation indicators, existing tasks that meet the preset value conditions are selected from the existing tasks to obtain low-value tasks. The high-consumption tasks and the low-value tasks are identified as tasks to be cleaned up. The step of calculating the resource consumption assessment value of each existing task based on the multi-dimensional task evaluation indicators specifically includes: Any single indicator from the multi-dimensional task evaluation metrics can be used as the resource consumption evaluation value for the corresponding existing task, where the value of the single indicator is the specific numerical value of the resource consumption evaluation value; or, The indicators in the multi-dimensional task evaluation index are standardized. The weights of each indicator after standardization are determined by a preset weighting algorithm. The obtained indicator weights and the standardized indicators are weighted and calculated to obtain the resource consumption assessment value for each existing task; or, Each indicator in the multi-dimensional task evaluation index is input into a pre-trained task evaluation model, which directly outputs the resource consumption evaluation value of each existing task. The task evaluation model is built based on a neural network.
2. The data warehouse cleanup method according to claim 1, characterized in that, Before the step of obtaining the task execution logs of the data warehouse, the following is also included: Get existing tasks; Generate an MR task for the existing task, wherein the MR task is a mapreduce task; Run the MR task and monitor its execution process to obtain a task execution log.
3. The data warehouse cleanup method according to claim 1, characterized in that, The step of parsing the task execution log to obtain multi-dimensional MR task attributes includes: The task execution log is parsed using HQL to extract structured information from multiple preset dimensions, resulting in multi-dimensional MR task attributes.
4. The data warehouse cleanup method according to claim 1, characterized in that, The step of generating multi-dimensional task evaluation metrics for each existing task based on the multi-dimensional MR task attributes includes: ETL processing is performed on the multi-dimensional MR task attributes corresponding to each existing task to obtain the multi-dimensional task evaluation index for each existing task.
5. A data warehouse cleanup device, characterized in that, include: The log acquisition module is used to acquire task execution logs from the data warehouse. The log parsing module is used to parse the task execution log to obtain multi-dimensional MR task attributes; The indicator generation module is used to generate multi-dimensional task evaluation indicators for each existing task based on the multi-dimensional MR task attributes. The task determination module is used to determine the tasks to be cleaned up from the existing tasks based on the multi-dimensional task evaluation indicators. The task determination module is further configured to calculate the resource consumption assessment value of each existing task according to the multi-dimensional task evaluation index; select existing tasks whose resource consumption assessment value meets the preset resource consumption conditions to obtain high-consumption tasks; select existing tasks that meet the preset value conditions from the existing tasks according to the multi-dimensional task evaluation index to obtain low-value tasks; and determine the high-consumption tasks and the low-value tasks as tasks to be cleaned up. The task determination module is further configured to: use any single indicator from the multi-dimensional task evaluation indicators as the resource consumption evaluation value of the corresponding existing task, wherein the value of the single indicator is the specific value of the resource consumption evaluation value; or, standardize each indicator from the multi-dimensional task evaluation indicators; determine the indicator weight of each indicator after standardization using a preset weighting algorithm; perform a weighted calculation on the obtained indicator weight and the standardized indicators to obtain the resource consumption evaluation value of each existing task; or, input each indicator from the multi-dimensional task evaluation indicators into a pre-trained task evaluation model, wherein the task evaluation model directly outputs the resource consumption evaluation value of each existing task, wherein the task evaluation model is built based on a neural network. The task cleanup module is used to clean up the tasks to be cleaned up, thereby completing the data warehouse cleanup.
6. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the data warehouse cleanup method as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data warehouse cleanup method as described in any one of claims 1 to 4.