Hadoop cluster redundancy calculation optimization and treatment method and system

Abstract

The invention discloses a Hadoop cluster redundancy calculation optimization and treatment method and system, and the method comprises the steps: introducing an automatic analysis tool, carrying out the deep analysis of a job configuration item, a data blood relationship and a job processing logic in a Hadoop cluster, and recognizing a redundancy calculation job; aiming at the identified redundant calculation operation, adopting optimization measures of combining similar operation and sharing calculation results; and in combination with the operation use frequency, the service requirements and the operation dependence, the existing operation and process are regularly examined, and historical operation or outdated processing processes which are not needed any more are identified and cleaned. According to the method and the system, redundant operations are accurately positioned by using an automatic tool, unnecessary resource consumption is reduced by optimizing and integrating similar calculation operations, and the overall efficiency and the stability of a cluster are improved by regularly examining and cleaning historical legacy operations.

Description

Technical Field

[0001] The present invention relates to the field of Hadoop cluster redundant computing, and in particular to a method and system for optimizing and managing Hadoop cluster redundant computing. Background Art

[0002] Sometimes, in order to quickly launch a business, a unified plan is not made, resulting in different business needs of different teams, which may require operations on the same data. If these operations are not effectively planned and managed, the same data will be repeatedly processed by different jobs, causing multiple problems:

[0003] 1. The number of jobs is huge, the configuration is complex, and it is difficult to locate redundant calculations.

[0004] In a Hadoop cluster, tens of thousands of jobs are usually running. Each job has many configuration items, including input directories, output directories, job parameters, etc. These jobs may involve multiple data sets and have complex processing logic and dependencies. This complexity makes it extremely difficult to locate which jobs are performing redundant calculations. For example, both Team A and Team B may need to perform summary calculations on a certain data set, but if there is no coordination, the two teams may develop jobs independently. For example, if the jobs of the two teams need to perform the same summary calculations on the data in the same directory, and the calculation results are stored in their respective directories, this will lead to redundant calculations.

[0005] 2. Huge resource consumption affects system performance.

[0006] When multiple redundant jobs are running in a cluster, cluster resources are repeatedly consumed by these jobs, especially during peak hours, which can lead to resource contention and affect the execution of key jobs. For example, in a high-concurrency environment, reading the same large data set multiple times for the same processing will occupy a large amount of I / O bandwidth, causing other jobs to respond more slowly. For example, if multiple jobs read and process hundreds of GB of data in the same directory during a certain period of time, the cluster's I / O resources may be exhausted, causing the execution time of other jobs to increase from minutes to hours.

[0007] 3. Historical problems have accumulated and are difficult to clear up.

[0008] As business needs change, many legacy processes may no longer meet current needs, but they are still running, which leads to the accumulation of redundant computing problems. Cleaning up these historical problems requires a deep understanding of business and technology, which is difficult. For example, an old log analysis job may have been developed several years ago to generate a certain report, but now this report is no longer used, but the job is still running automatically every day, occupying cluster resources. Summary of the invention

[0009] In order to solve the problems of resource waste and system performance degradation caused by repeated data processing, the present invention provides a method and system for optimizing and managing redundant computing in a Hadoop cluster, which uses automated tools to accurately locate redundant jobs, reduces unnecessary resource consumption by optimizing and integrating similar computing jobs, and improves the overall efficiency and stability of the cluster by regularly reviewing and cleaning up historical legacy jobs.

[0010] To achieve the above purpose, the present invention adopts the following technical solutions:

[0011] In one embodiment of the present invention, a method for optimizing and managing redundant computing of a Hadoop cluster is proposed, the method comprising:

[0012] Introduce automated analysis tools to conduct in-depth analysis of job configuration items, data lineage relationships, and job processing logic in the Hadoop cluster to identify redundant computing jobs;

[0013] For the identified redundant computing jobs, optimization measures are taken to merge similar jobs and share computing results.

[0014] Based on job usage frequency, business needs, and job dependencies, regularly review existing jobs and processes to identify and clean up historical jobs or outdated processing processes that are no longer needed.

[0015] Further, the job configuration items are analyzed as follows:

[0016] Extract configuration information for each job from the Hadoop cluster's job scheduler and job management system, including input and output directories, job parameters, and scheduling strategies;

[0017] Store the collected configuration information in the metadata warehouse of the automated analysis tool; create a unique index for the configuration information of each job;

[0018] Analyze the hierarchy and naming conventions of the input and output directories, and calculate the similarity of the directory paths; perform semantic analysis on the job parameters, and use a distance-based algorithm to calculate the similarity of the two job parameters; analyze the scheduling strategy to determine whether the jobs are executed in similar time windows, and compare their execution frequencies to calculate the scheduling similarity.

[0019] Furthermore, the data lineage analysis is as follows:

[0020] By parsing the execution log, configuration file, and job code of each job, a data lineage map is automatically generated; the data lineage map shows the entire path of data from the source to the final output;

[0021] Store the data lineage map in the graph database; locate all upstream or downstream jobs of a data set and the dependencies between jobs by querying the graph database;

[0022] By comparing the data lineage maps of different operations, the similarities and redundancies of data flow paths can be identified.

[0023] Further, the job processing logic is analyzed as follows:

[0024] Parse different types of job codes to extract core processing logic features. These features are stored in a standardized manner to form feature vectors of job processing logic.

[0025] Compare the feature vectors of the job processing logic and calculate the logical similarity between the jobs;

[0026] When it is found that the logic of multiple jobs is highly similar, these jobs are marked as candidates for redundant computing; combined with the results of configuration similarity and data lineage analysis, it is further verified whether these jobs are indeed redundant.

[0027] In one embodiment of the present invention, a system for optimizing and managing redundant computing of a Hadoop cluster is also proposed. The system includes an automated analysis tool. The automated analysis tool includes:

[0028] Redundant computing identification module, which is used to conduct in-depth analysis of job configuration items, data lineage relationships, and job processing logic in the Hadoop cluster to identify redundant computing jobs;

[0029] The job optimization and integration module is used to take optimization measures such as merging similar jobs and sharing computing results for the identified redundant computing jobs;

[0030] The historical legacy job cleanup module is used to regularly review existing jobs and processes based on job usage frequency, business requirements, and job dependencies, and to identify and clean up no longer needed historical jobs or obsolete processing processes.

[0031] Further, the job configuration items are analyzed as follows:

[0032] Extract configuration information for each job from the Hadoop cluster's job scheduler and job management system, including input and output directories, job parameters, and scheduling strategies;

[0033] Store the collected configuration information in the metadata warehouse of the automated analysis tool; create a unique index for the configuration information of each job;

[0034] Analyze the hierarchy and naming conventions of the input and output directories, and calculate the similarity of the directory paths; perform semantic analysis on the job parameters, and use a distance-based algorithm to calculate the similarity of the two job parameters; analyze the scheduling strategy to determine whether the jobs are executed in similar time windows, and compare their execution frequencies to calculate the scheduling similarity.

[0035] Furthermore, the data lineage analysis is as follows:

[0036] By parsing the execution log, configuration file, and job code of each job, a data lineage map is automatically generated; the data lineage map shows the entire path of data from the source to the final output;

[0037] Store the data lineage map in the graph database; locate all upstream or downstream jobs of a data set and the dependencies between jobs by querying the graph database;

[0038] By comparing the data lineage maps of different operations, the similarities and redundancies of data flow paths can be identified.

[0039] Further, the job processing logic is analyzed as follows:

[0040] Parse different types of job codes to extract core processing logic features. These features are stored in a standardized manner to form feature vectors of job processing logic.

[0041] Compare the feature vectors of the job processing logic and calculate the logical similarity between the jobs;

[0042] When it is found that the logic of multiple jobs is highly similar, these jobs are marked as candidates for redundant computing; combined with the results of configuration similarity and data lineage analysis, it is further verified whether these jobs are indeed redundant.

[0043] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method for optimizing and managing redundant computing of the Hadoop cluster is designed.

[0044] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing a method for optimizing and managing redundant computing of a Hadoop cluster.

[0045] Beneficial effects:

[0046] 1. The present invention uses automated analysis tools to deeply analyze job configuration items, data lineage relationships, and job processing logic, accurately locate redundant computing jobs, and use data lineage maps to clearly display data flow paths, helping to scientifically optimize jobs.

[0047] 2. The present invention combines similar jobs and shares intermediate results, reduces repeated calculations, optimizes resource utilization, and further reduces computing resource consumption by caching intermediate results for use by other jobs.

[0048] 3. The replicability and scalability of the solution of the present invention have wide applicability and can be applied to other big data platforms with good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a method for optimizing and managing redundant computing of a Hadoop cluster according to an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of the system structure of the Hadoop cluster redundant computing optimization and management of the present invention;

[0051] Figure 3 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0052] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a system, system, device, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0054] According to the implementation of the present invention, a method for optimizing and managing redundant computing in a Hadoop cluster is proposed. By identifying and eliminating redundant computing in a Hadoop cluster, the problem of resource waste and system performance degradation caused by duplicate data processing is solved. Redundant jobs are accurately located using automated tools, and unnecessary resource consumption is reduced by optimizing and integrating similar computing jobs. The overall efficiency and stability of the cluster are improved by regularly reviewing and cleaning up historical legacy jobs.

[0055] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0056] The method for optimizing and managing redundant computing of Hadoop clusters of the present invention comprises:

[0057] 1. Identify redundant calculations.

[0058] Introduce automated analysis tools to perform a comprehensive scan of all jobs in the Hadoop cluster. Through these tools, in-depth analysis of job configuration items, data lineage relationships, and job processing logic is performed to identify redundant computing jobs. Implement data lineage analysis to track the flow path of data from source to output. This enables a clear understanding of the input and output relationship of each job and rapid location of redundant computing jobs.

[0059] 2. Optimize and integrate operations.

[0060] For the identified redundant computing jobs, optimization measures are taken to merge similar jobs and share computing results. For example, multiple jobs processing the same data are merged into a unified job, or computing results are shared through a cache mechanism to avoid repeated computing, reduce unnecessary data loading and storage operations, and optimize resource scheduling strategies to improve cluster resource utilization efficiency and overall performance.

[0061] 3. Clean up historical tasks.

[0062] Combined with job usage frequency, business requirements, and job dependencies, regularly review existing jobs and processes to identify historical jobs or outdated processing flows that are no longer needed. Clean up or optimize these legacy jobs, gradually eliminate redundant processing flows to reduce resource waste, and ensure that job configuration items in the Hadoop cluster are consistent with current business requirements.

[0063] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0064] In order to explain the above-mentioned method for optimizing and managing redundant computing of Hadoop cluster more clearly, a specific embodiment is used for illustration below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0065] Example:

[0066] like Figure 1 As shown, the specific implementation steps are as follows:

[0067] 1. Identifying Redundant Calculations

[0068] The core function of the automated analysis tool is to automatically scan all jobs in the Hadoop cluster to identify similar or repeated job configuration items, data lineage relationships, and business processing logic.

[0069] Tool implementation principle:

[0070] 1. Job configuration item analysis

[0071] (1) Configuration item data collection:

[0072] Extract the configuration information of each job from the task scheduler (such as YARN) and job management system (such as Oozie) of the Hadoop cluster. This configuration information includes:

[0073] Input directory and output directory: data path, file format, etc.

[0074] Job parameters: such as the number of MapReduce (a computing model for parallel processing of big data), memory allocation, partitioning strategy, etc.

[0075] Scheduling strategy: such as job execution frequency, time window, etc.

[0076] (2) Configuration item data storage and indexing:

[0077] The collected configuration information is stored in the tool's metadata warehouse, and efficient storage and query capabilities are built based on distributed databases such as HBase. To facilitate subsequent analysis, the tool creates a unique index for each job's configuration information, which can include directory paths, hash values ​​of job parameters, etc., for fast query and comparison.

[0078] (3) Configuration similarity calculation:

[0079] Calculate similarity of job configuration items, including:

[0080] Path similarity: Analyze the hierarchy and naming conventions of the input and output directories and calculate the similarity of the directory paths. For example, if the input directories of two jobs only have the last level directory name different, the similarity is high.

[0081] Parameter similarity: Perform semantic analysis on job parameters and use distance-based algorithms (such as cosine similarity) to calculate the similarity of two job parameters. For example, if the memory allocation and partitioning strategies of two jobs are the same or similar, the similarity is high.

[0082] Scheduling similarity: Analyze the scheduling strategies of jobs, determine whether jobs are executed in similar time windows, and compare their execution frequencies.

[0083] Example data:

[0084] Job A configuration:

[0085] Enter directory: / data / original / sales

[0086] Output directory: / data / result / sales_summary_A

[0087] Job parameters: mapTasks=10, reduceTasks=5

[0088] Job B configuration:

[0089] Enter directory: / data / original / sales

[0090] Output directory: / data / result / sales_summary_B

[0091] Job parameters: mapTasks=10, reduceTasks=5

[0092] The tool calculated that the path similarity and parameter similarity of the two jobs were both 100%, and determined that they were highly similar in configuration.

[0093] 2. Data lineage analysis

[0094] (1) Data lineage analysis

[0095] The tool automatically generates a data lineage map by parsing the execution log, configuration file, and job code of each job. The data lineage map shows the entire path of data from the source to the final output. The specific implementation includes:

[0096] Log parsing: Extract job input and output files from HDFS, YARN and other logs to build a preliminary path for data flow.

[0097] Configuration file parsing: Parse the XML, JSON and other configuration files of the job to obtain the configuration information of the data source, such as the name of the input table, field mapping relationship, etc.

[0098] (2) Data lineage graph storage and query

[0099] The tool stores the data lineage graph in a graph database and uses the advantages of the graph database to achieve fast lineage query and relationship reasoning.

[0100] The lineage information of each job is stored as nodes and edges. Nodes represent data sets or tables, and edges represent read and write operations of jobs. Through queries, you can quickly locate all upstream or downstream jobs of a data set, as well as the dependencies between jobs.

[0101] (3) Redundant path detection

[0102] The tool compares the data lineage graphs of different jobs to identify the similarities and redundancies of data flow paths. For example, if multiple jobs read the same data source and perform similar processing, the lineage paths of these jobs will overlap to a high degree, and the tool will mark these jobs as candidates for redundant calculations.

[0103] To detect redundant paths more accurately, the tool uses graph matching algorithms (such as subgraph isomorphism) to identify the similarity of path structures and determines the degree of redundancy based on the specific read and write operations of the data.

[0104] Example data:

[0105] In the Hadoop cluster, jobs A, B, and C have the following data dependencies:

[0106] Assignment A:

[0107] Input: Read original sales data from / data / original / sales / 2023 / Q1 /

[0108] Output: Generates intermediate results, stored in / data / temp / sales_cleaned / 2023 / Q1 / , which contains the cleaned and standardized sales data

[0109] Assignment B:

[0110] Input: Read the cleaned sales data from the output path / data / temp / sales_cleaned / 2023 / Q1 / of job A.

[0111] Output: The data is summarized by "Region" and "Product Type", and the results are stored in / data / result / sales_summary_A / 2023 / Q1 / .

[0112] Assignment C:

[0113] Input: Read the original sales data directly from / data / original / sales / 2023 / Q1 / (without the cleaning process of job A)

[0114] Output: Summarized by "sales channel" and "product type", the results are stored in / data / result / sales_summary_B / 2023 / Q1 / .

[0115] The data lineage map generated by the automated analysis tool shows the flow path of data from the original input to the final output. The map clearly shows the following key points:

[0116] Job A is responsible for cleaning and standardizing the raw sales data, and its output data is relied upon by Job B.

[0117] Job B depends on the output of Job A to perform further summary calculations.

[0118] Job C directly reads the same original data source as Job A, but does not use the cleaned data of Job A. Instead, it performs independent summary calculations.

[0119] The data lineage graph reveals the possibility of redundant computation in job C, because job C repeatedly processes the original data without using the intermediate cleansing results generated by job A. This redundancy not only wastes computing resources, but also increases the complexity of the data processing process.

[0120] 3. Job processing logic analysis

[0121] (1) Code parsing and feature extraction

[0122] The tool supports static analysis of different types of job codes (such as MapReduce, Spark, and Hive). By parsing the job code, the core processing logic features are extracted. These features are stored in a standardized manner to form a feature vector of the job processing logic, including:

[0123] MapReduce job: parse the implementation of map() and reduce() functions, and extract features such as data partitioning and aggregation logic.

[0124] Spark job: parses RDD or DataFrame transformation operations, such as map(), reduceByKey(), join(), etc., and extracts transformation chain features.

[0125] Hive job: parses SQL queries, extracts operators such as SELECT, JOIN, and GROUP BY, and their execution order.

[0126] (2) Logical similarity calculation

[0127] The tool compares the feature vectors of the processing logic and calculates the logical similarity between jobs.

[0128] Logical pattern matching: Use a pattern matching-based algorithm to compare the logical patterns in the job code. For example, if two MapReduce jobs implement the same aggregation logic, their logical similarity is very high.

[0129] Operation chain similarity: For Spark and Hive jobs, the tool compares the structural similarity of the operation chains. For example, if two Spark jobs both contain the chain structure of map->reduceByKey->join, the tool will determine that their logical similarity is high.

[0130] (3) Redundant logic identification

[0131] When the tool finds that the processing logic of multiple jobs is highly similar, it marks these jobs as candidates for redundant computing. The tool combines the results of configuration similarity and data lineage analysis to further verify whether these jobs are indeed redundant.

[0132] The tool also generates a detailed analysis report, points out possible redundant logic parts, and suggests how to optimize them. For example, the tool may suggest merging similar MapReduce jobs or reusing intermediate results of Spark jobs.

[0133] Example data:

[0134] In the Hadoop cluster, Job A and Job B execute different business logics respectively. The tool conducts in-depth analysis of the processing logic of the two jobs through static code analysis and feature extraction.

[0135] Assignment A:

[0136] Job type: MapReduce job

[0137] Processing logic:

[0138] Map phase:

[0139] Read the original sales data from the HDFS path / data / original / sales / 2023 / Q1 / .

[0140] Label each record by Product Type and Region.

[0141] Generate a key-value pair <product type_region,sales amount>.

[0142] Reduce phase:

[0143] Aggregate all records by key, which is a combination of product type and region.

[0144] Calculate the total sales for each product type in different regions.

[0145] Output:

[0146] Output the results to / data / result / sales_summary_A / 2023 / Q1 / .

[0147] Assignment B:

[0148] Job type: Spark job

[0149] Processing logic:

[0150] Data reading:

[0151] Read the original sales data from the HDFS path / data / original / sales / 2023 / Q1 / and generate a DataFrame.

[0152] Data conversion:

[0153] Group the data by Product Type and Sales Channel.

[0154] Calculate the total sales for the grouped data.

[0155] Data filtering:

[0156] Filter out records with sales exceeding a certain threshold.

[0157] Output:

[0158] Save the results to the HDFS path: / data / result / sales_summary_B / 2023 / Q1 /

[0159] Processing logic analysis:

[0160] Feature extraction:

[0161] The tool statically analyzes the MapReduce code of job A and the Spark code of job B and extracts the following core features:

[0162] Job A: The characteristics of the Map phase are data tagging by "product type" and "region", and the characteristics of the Reduce phase are aggregation calculations by composite keys.

[0163] Job B: The characteristics of the Spark job are grouping by "product type" and "sales channel", calculating the total sales, and filtering the data.

[0164] Logical similarity calculation:

[0165] The tool compares the Reduce phase of job A with the groupBy (grouping) + agg (statistics or aggregation) operations of job B and finds that both logically group and aggregate sales data, but with different aggregation dimensions (job A by "product type_region" and job B by "product type_sales channel")

[0166] Although Job B has an additional data filtering step, the overall logic is still similar to Job A, especially the aggregation calculation when processing the same data set.

[0167] Redundant judgment:

[0168] The tool found that the processing logic of Job A and Job B has about 80% similarity in core aggregation calculation, although their aggregation dimensions are different (although the aggregation dimensions of Job A and Job B are different, because their processing logic (both grouping and aggregation) is similar, the tool will think that their core calculation logic has a high degree of overlap). This high similarity indicates that there is a possibility of repeated calculation when the two jobs process the same data set. The tool recommends further optimization to reduce this logical redundancy.

[0169] 2. Optimization and Integration Operations

[0170] For the identified redundant computing jobs, by merging similar jobs and sharing computing results, unnecessary data processing operations can be reduced, resource utilization can be optimized, and the overall efficiency of the Hadoop cluster can be improved.

[0171] 1. Merge assignments.

[0172] (1) Pre-merger analysis:

[0173] Input analysis: Through automated analysis tools, it is identified that the input paths of Job A and Job B are the same, both / data / original / sales / 2023 / Q1 / , which indicates that they process the same batch of original data.

[0174] Logical analysis: The analysis tool further pointed out that Job A and Job B have a high degree of similarity in processing logic, especially in data aggregation calculation. Job A aggregates by "product type_region", and Job B aggregates by "product type_sales channel".

[0175] (2) Merger plan design:

[0176] Merge goal: Merge jobs A and B into a new job C. Job C can simultaneously complete the summary calculation of the two dimensions "product type_region" and "product type_sales channel", reducing repeated reading and processing of the same data set.

[0177] Logical expansion: In the merged job C, the data labeling logic of the Map phase is expanded to include both the "product type_region" and "product type_sales channel" dimensions. The aggregation logic of the Reduce phase is also expanded to calculate the total sales for each dimension combination.

[0178] Output design: To facilitate use by different teams, Job C stores the summary results in two directories: / data / result / sales_summary_by_region / 2023 / Q1 / (summary by region and product type) and / data / result / sales_summary_by_channel / 2023 / Q1 / (summary by sales channel and product type).

[0179] (3)Job scheduling and testing.

[0180] Scheduling design: The newly merged job C needs to be configured in the scheduler. Since job C integrates the functions of the previous two jobs, its execution time may need to be appropriately extended, or the scheduling time window may need to be adjusted to avoid resource contention.

[0181] Test execution: Run Job C in the test environment to ensure that the merged logic can correctly process the data and generate the required output. Focus on checking the completeness and correctness of the output data, especially the summary results in different dimensions.

[0182] Example data:

[0183] enter:

[0184] Read the sales data for the first quarter of 2023 from / data / original / sales / 2023 / Q1 / .

[0185] Intermediate results:

[0186] Cache path: / data / temp / sales_summary_cache / 2023 / Q1 /

[0187] Content: Contains intermediate summary results by region and sales channel.

[0188] Output:

[0189] The results summarized by region and product type are stored in / data / result / sales_summary_by_region / 2023 / Q1 / .

[0190] The results are summarized by sales channel and product type in

[0191] / data / result / sales_summary_by_channel / 2023 / Q1 /

[0192] 2. Share calculation results.

[0193] (1) Intermediate result cache design:

[0194] Cache location: During the merging process of job C, some intermediate calculation results (such as regional summary data) can be cached in the shared path / data / temp / sales_summary_cache / 2023 / Q1 / of HDFS for direct reading by other jobs.

[0195] Storage format: Intermediate results can be stored in Parquet or ORC format for efficient reading by subsequent jobs. Choosing an efficient storage format can reduce I / O operations and improve the execution efficiency of subsequent jobs.

[0196] (2) Use of follow-up operations:

[0197] Reading shared data: Other jobs (such as job D) can directly read the cached summary data from the / data / temp / sales_summary_cache / 2023 / Q1 / path without recalculating the summary.

[0198] (3) Result verification and adjustment:

[0199] Verify shared results: When using shared intermediate results for the first time, perform data verification to ensure the accuracy and consistency of the results. If there are errors or inconsistencies in the shared results, adjust the cache logic or storage path.

[0200] Adjustment plan: Based on the feedback from monitoring data, decide whether the cache mechanism needs to be further optimized. For example, you can adjust the storage time of cached data according to the frequency of use, or adjust the granularity of cached data according to the needs of different jobs.

[0201] Example data:

[0202] Read path: / data / temp / sales_summary_cache / 2023 / Q1 /

[0203] Output:

[0204] The further summarized quarterly sales data is stored in / data / result / quarterly_sales_summary / 2023 /

[0205] 3. Cleaning up historical legacy work

[0206] Identify and clean up historical jobs and obsolete processing flows that are no longer needed in the Hadoop cluster, free up cluster resources, simplify job management, and improve overall efficiency.

[0207] 1. Review of historical homework.

[0208] (1) Regular review plan: Every quarter or every six months, organize the technical team and the business team to conduct a joint review of all jobs in the current Hadoop cluster. Develop a detailed review plan, including a list of jobs to be reviewed, responsible team members, and a specific timetable for the review.

[0209] (2) Job usage frequency analysis: Use automated analysis tools to extract the execution frequency and last run time of each job. The tool will generate a report listing the execution frequency data of all jobs. This includes:

[0210] The number of executions per day, week, or month, the time of the last successful execution, and the execution frequency trend in the last 6 months or 1 year. Mark jobs with extremely low execution frequency (such as not running more than once in half a year) as candidates that may need to be cleaned up.

[0211] (3) Business demand confirmation: Confirm the list of low-frequency jobs with the business team and ask whether these jobs still have business demand. The business team can determine whether these jobs are still necessary based on changes in business processes.

[0212] (4) Job dependency analysis: The tool performs data lineage analysis on low-frequency jobs and historical jobs to check whether these jobs are dependent on other jobs. The graph shows the input, processing, and output process of the job, as well as its dependencies with other jobs. If a job is dependent on other important jobs, further confirmation is required to determine whether it can be safely deactivated. For jobs with multiple dependencies, it may be necessary to adjust the dependent jobs before cleaning up.

[0213] (5) Decision-making and Execution:

[0214] Decision Meeting: After all analysis and confirmation work is completed, a decision meeting is held to decide which historical jobs will be deactivated or deleted. This decision should include all relevant parties to ensure that there is no negative impact on existing business processes.

[0215] Job deactivation: For jobs that you decide to deactivate, first disable the job in the scheduler, then wait for a period of time (such as a month) to ensure that there are no unexpected effects. If confirmed, delete the job's configuration and script.

[0216] Job deletion: After confirming that it will not affect other jobs or business processes, the job configuration files, scripts, and related data are completely deleted to release storage resources.

[0217] Example data:

[0218] Execution frequency: Only executed once in the past 6 months, and the last execution was 3 months ago.

[0219] Execution purpose: Used to generate a quarterly report. The business team confirms that the report has been replaced by a new report and is no longer used.

[0220] Marking result: Marked as a low-frequency operation, and it is recommended to clean it up.

[0221] Based on the same inventive concept, the present invention also proposes a system for optimizing and managing redundant computing of Hadoop clusters. The implementation of the system can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the system described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0222] Figure 2 This is a schematic diagram of the system structure of the Hadoop cluster redundant computing optimization and management of the present invention. Figure 2 As shown, the system includes an automated analysis tool, which includes:

[0223] The redundant computing identification module 101 is used to perform in-depth analysis on job configuration items, data lineage relationships, and job processing logic in the Hadoop cluster to identify redundant computing jobs.

[0224] The job configuration items are analyzed as follows:

[0225] Extract configuration information for each job from the Hadoop cluster's job scheduler and job management system, including input and output directories, job parameters, and scheduling strategies;

[0226] Store the collected configuration information in the metadata warehouse of the automated analysis tool; create a unique index for the configuration information of each job;

[0227] Analyze the hierarchy and naming conventions of the input and output directories, and calculate the similarity of the directory paths; perform semantic analysis on the job parameters, and use a distance-based algorithm to calculate the similarity of the two job parameters; analyze the scheduling strategy to determine whether the jobs are executed in similar time windows, and compare their execution frequencies to calculate the scheduling similarity.

[0228] The data lineage analysis is as follows:

[0229] By parsing the execution log, configuration file, and job code of each job, a data lineage map is automatically generated; the data lineage map shows the entire path of data from the source to the final output;

[0230] Store the data lineage map in the graph database; locate all upstream or downstream jobs of a data set and the dependencies between jobs by querying the graph database;

[0231] By comparing the data lineage maps of different operations, the similarities and redundancies of data flow paths can be identified.

[0232] The job processing logic is analyzed as follows:

[0233] Parse different types of job codes to extract core processing logic features. These features are stored in a standardized manner to form feature vectors of job processing logic.

[0234] Compare the feature vectors of the job processing logic and calculate the logical similarity between the jobs;

[0235] When it is found that the logic of multiple jobs is highly similar, these jobs are marked as candidates for redundant computing; combined with the results of configuration similarity and data lineage analysis, it is further verified whether these jobs are indeed redundant.

[0236] The job optimization and integration module 102 is used to take optimization measures such as merging similar jobs and sharing computing results for the identified redundant computing jobs;

[0237] The historical legacy job cleaning module 103 is used to regularly review existing jobs and processes based on job usage frequency, business requirements, and job dependencies, and to identify and clean up no longer needed historical jobs or obsolete processing processes.

[0238] It should be noted that although several modules of the system for optimizing and managing redundant computing of Hadoop clusters are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment.

[0239] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned method for optimizing and managing redundant computing of the Hadoop cluster is designed.

[0240] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned method for optimizing and managing redundant computing of a Hadoop cluster.

[0241] The method and system for optimizing and managing redundant computing of Hadoop clusters proposed in the present invention have the following highlights:

[0242] 1. Through automated analysis tools, in-depth analysis of job configuration items, data lineage relationships, and job processing logic can be performed to accurately locate redundant jobs. Data lineage maps can be used to clearly display data flow paths to help scientifically optimize jobs.

[0243] 2. Merge similar jobs and share intermediate results to reduce repeated calculations, optimize resource utilization, and further reduce computing resource consumption by caching intermediate results for use by other jobs.

[0244] 3. The solution is replicable and scalable, has wide applicability, can be applied to other big data platforms, and has good versatility.

[0245] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0246] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A method for optimizing and managing redundant computing of a Hadoop cluster, characterized in that: The method includes: Introduce automated analysis tools to conduct in-depth analysis of job configuration items, data lineage relationships, and job processing logic in the Hadoop cluster to identify redundant computing jobs; For the identified redundant computing jobs, optimization measures are taken to merge similar jobs and share computing results. Based on job usage frequency, business needs, and job dependencies, regularly review existing jobs and processes to identify and clean up historical jobs or outdated processing processes that are no longer needed.

2. The method for optimizing and managing redundant computing of Hadoop cluster according to claim 1, characterized in that: The job configuration items are analyzed as follows: Extract configuration information for each job from the Hadoop cluster's job scheduler and job management system, including input and output directories, job parameters, and scheduling strategies; Store the collected configuration information in the metadata repository of the automated analysis tool; Create a unique index for each job's configuration information; Analyze the hierarchical structure and naming conventions of input and output directories, and calculate the similarity of directory paths; perform semantic analysis on job parameters, and use a distance-based algorithm to calculate the similarity of two job parameters; Analyze the scheduling strategy to determine whether the jobs are executed in similar time windows, compare their execution frequencies, and calculate the scheduling similarity.

3. The method for optimizing and managing redundant computing of Hadoop cluster according to claim 1, characterized in that: The data kinship analysis is as follows: By parsing the execution log, configuration file, and job code of each job, a data lineage map is automatically generated; the data lineage map shows the entire path of data from the source to the final output; Store the data lineage map in the graph database; locate all upstream or downstream jobs of a data set and the dependencies between jobs by querying the graph database; By comparing the data lineage maps of different operations, the similarities and redundancies of data flow paths can be identified.

4. The method for optimizing and managing redundant computing of Hadoop cluster according to claim 1, characterized in that: The job processing logic is analyzed as follows: Parse different types of job codes to extract core processing logic features. These features are stored in a standardized manner to form feature vectors of job processing logic. Compare the feature vectors of the job processing logic and calculate the logical similarity between jobs; When it is found that the logic of multiple jobs is highly similar, these jobs are marked as candidates for redundant computing; combined with the results of configuration similarity and data lineage analysis, it is further verified whether these jobs are indeed redundant.

5. A system for optimizing and managing redundant computing of Hadoop clusters, characterized in that: The system includes an automated analysis tool, which includes: Redundant computing identification module, which is used to conduct in-depth analysis of job configuration items, data lineage relationships, and job processing logic in the Hadoop cluster to identify redundant computing jobs; The job optimization and integration module is used to take optimization measures such as merging similar jobs and sharing computing results for the identified redundant computing jobs; The historical legacy job cleanup module is used to regularly review existing jobs and processes based on job usage frequency, business requirements, and job dependencies, and to identify and clean up no longer needed historical jobs or obsolete processing processes.

6. The device for optimizing and managing redundant computing of Hadoop cluster according to claim 5, characterized in that: The job configuration items are analyzed as follows: Extract configuration information for each job from the Hadoop cluster's job scheduler and job management system, including input and output directories, job parameters, and scheduling strategies; Store the collected configuration information in the metadata repository of the automated analysis tool; Create a unique index for each job's configuration information; Analyze the hierarchical structure and naming conventions of input and output directories, and calculate the similarity of directory paths; perform semantic analysis on job parameters, and use a distance-based algorithm to calculate the similarity of two job parameters; Analyze the scheduling strategy to determine whether the jobs are executed in similar time windows, compare their execution frequencies, and calculate the scheduling similarity.

7. The device for optimizing and managing redundant computing of Hadoop cluster according to claim 5, characterized in that: The data kinship analysis is as follows: By parsing the execution log, configuration file, and job code of each job, a data lineage map is automatically generated; the data lineage map shows the entire path of data from the source to the final output; Store the data lineage map in the graph database; locate all upstream or downstream jobs of a data set and the dependencies between jobs by querying the graph database; By comparing the data lineage maps of different operations, the similarities and redundancies of data flow paths can be identified.

8. The device for optimizing and managing redundant computing of Hadoop cluster according to claim 5, characterized in that: The job processing logic is analyzed as follows: Parse different types of job codes to extract core processing logic features. These features are stored in a standardized manner to form feature vectors of job processing logic. Compare the feature vectors of the job processing logic and calculate the logical similarity between jobs; When it is found that the logic of multiple jobs is highly similar, these jobs are marked as candidates for redundant computing; combined with the results of configuration similarity and data lineage analysis, it is further verified whether these jobs are indeed redundant.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 4.

Images