Big data resource optimization system and method, storage medium and computer equipment

Abstract

The invention discloses a big data resource optimization system and method, a storage medium and computer equipment, relates to the technical field of data processing, is suitable for the technical field of finance and medical treatment, and mainly aims to solve the problems of high optimization cost and low efficiency of resource optimization of a big data platform in the prior art. The system comprises a data source layer, a middle processing layer and an application layer, the data source layer is used for acquiring related information of a big data cluster from a plurality of data sources; the intermediate processing layer is used for carrying out secondary processing on the related information acquired from the data source layer to obtain secondary processing data convenient for statistical processing; and the application layer is used for performing statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feeding back a statistical result to a related person in charge, so that the related person in charge performs resource optimization processing based on the statistical result.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, applicable to the fields of finance and medical technology, and in particular to a system and method for optimizing big data resources, a storage medium, and a computer device. Background Art

[0002] With the vigorous development of digitalization in the financial and medical fields, the storage pressure of the corresponding big data platform is increasing. A large number of silent tables and worthless financial or medical tasks are running, which adds unnecessary resource waste to the big data cluster. However, most companies in the financial and medical fields often ignore the problem of big data environment resource optimization when building big data clusters. Over time, the big data environment becomes larger and larger, and there are more and more silent financial and medical historical data. The processing tasks continue to accumulate, and the corresponding storage costs and computing costs continue to increase.

[0003] At present, when many companies in the financial and medical fields are optimizing resources, developers manually analyze data and run data cleaning scripts manually at regular intervals. Since big data environment resources need to be cleaned and optimized regularly, the existing manual cleaning and optimization methods inevitably require developers to follow up continuously, resulting in high labor costs for resource optimization and low optimization efficiency. Summary of the invention

[0004] In view of this, the present invention provides a big data resource optimization system and method, storage medium, and computer equipment, the main purpose of which is to solve the problems of high optimization cost and poor effect of existing resource optimization on big data platforms.

[0005] According to one aspect of the present invention, there is provided a system for optimizing big data resources, comprising: a data source layer, an intermediate processing layer, and an application layer;

[0006] The data source layer is used to obtain relevant information of the big data cluster from multiple data sources;

[0007] The intermediate processing layer is used to perform secondary processing on the relevant information obtained in the data source layer to obtain secondary processed data that is convenient for statistical processing;

[0008] The application layer is used to perform statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feed back the statistical results to relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

[0009] Furthermore, the data source layer includes a running log acquisition module, a task scheduling information acquisition module, a task execution information acquisition module, a file system log acquisition module and a permission information acquisition module;

[0010] The operation log acquisition module is used to acquire the operation log of the big data cluster; and parse the operation log to obtain task operation information;

[0011] The task scheduling information acquisition module is used to obtain the task scheduling information of the big data cluster from the scheduling platform;

[0012] The task execution information acquisition module is used to obtain execution related information from the task execution platform;

[0013] The file system log acquisition module is used to acquire the file system information of the big data cluster; and transfer the file system information into a file system data table with table as the primary key;

[0014] The permission information acquisition module is used to acquire the permission information corresponding to the big data cluster from the permission management database.

[0015] Furthermore, the log content acquisition module parses the operation log to obtain task operation information, including:

[0016] Parsing the log folders in the operation log one by one to obtain parsing results corresponding to each log folder;

[0017] The analysis results are formatted and stored in a partition table to obtain the task running information corresponding to each task.

[0018] Furthermore, the log content acquisition module is also used to perform data cleaning processing on the task running information obtained by parsing based on a preset data cleaning cycle to release the storage space of the partition table.

[0019] Further, the log content acquisition module includes a table association relationship determination unit;

[0020] The table association relationship determination unit is used to parse the data query statements in the operation log to obtain target data tables corresponding to the data query statements and to obtain table association relationships between the target data tables.

[0021] Furthermore, the intermediate processing layer includes a cluster association analysis module, a node analysis module, a field association analysis module, and a system and authority information analysis module;

[0022] The cluster association analysis module is used to analyze all the table association relationships in the big data cluster to obtain the cluster association relationship corresponding to the big data cluster;

[0023] The node parsing module is used to parse the out-degree and in-degree of all data tables in the big data cluster, and determine the node type of the data table based on the out-degree and in-degree obtained by parsing; the node type includes a root node and a leaf node;

[0024] The field association analysis module is used to parse the fields in all data tables in the big data cluster to obtain field association relationships;

[0025] The system and permission information parsing module is used to parse the file system data table to obtain the table size, number of table files, creation time, update time, last access time, associated tasks and whether it is a partition table; and determine the person responsible for the table based on the permission information.

[0026] Furthermore, the application layer includes a field-level statistics module, a partition-level statistics module, a table-level statistics module, and a task-level statistics module;

[0027] The field-level statistics module is used to perform statistical processing on the number of times fields are frequently associated, and identify key fields based on the statistical results;

[0028] The partition-level statistics module is used to partition and save the basic information of the table and archive it, and perform statistical processing based on the partition results to obtain partition summary information;

[0029] The table-level statistical module is used to perform statistical processing on the resource consumption of the table based on the statistical indicators, and send the information to be optimized that does not meet the indicator threshold to the relevant responsible person for resource optimization processing;

[0030] The task-level statistical module is used to perform statistical processing on the task value of the task to be analyzed based on the task name and all table information associated with the task, and send the information to be optimized that does not meet the task value threshold to the relevant responsible person for resource optimization processing.

[0031] According to another aspect of the present invention, a method for optimizing big data resources is provided, comprising:

[0032] The data source layer obtains relevant information of the big data cluster from multiple data sources according to the monitoring time period; and sends the relevant information to the intermediate processing layer;

[0033] The intermediate processing layer performs secondary processing on the relevant information to obtain secondary processing data that is convenient for statistical processing;

[0034] Based on the time interval of the statistical cycle, the application layer performs statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

[0035] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for optimizing big data resources.

[0036] According to another aspect of the present invention, there is provided a computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned big data resource optimization method.

[0038] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0039] The present invention provides a system and method for optimizing big data resources, a storage medium, and a computer device. Compared with the prior art, the present invention realizes the all-round information collection of the resource utilization of the big data cluster by acquiring the relevant information of the big data cluster from multiple data sources; secondly, the present invention performs secondary processing on the relevant information of the big data cluster collected from multiple data sources through the intermediate processing layer to obtain secondary processed data that is convenient for statistical processing, thereby realizing the technical effect of multi-source data fusion and facilitating the comprehensive statistics of table-related data resources and task-related data resources. The present invention also performs statistical processing on the secondary processed data obtained by the intermediate processing layer according to different statistical rules through the application layer, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results. Not only can the analysis of the resource utilization of the table level in the financial and medical technology fields be realized, but also the analysis of the data utilization of the task level in the financial and medical technology fields can be realized, so that the analysis results of the big data resources are more comprehensive and thorough, which is convenient for the relevant responsible persons to make more effective optimization decisions.

[0040] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0042] Figure 1 A schematic diagram of the structure of a big data resource optimization system provided by an embodiment of the present invention is shown;

[0043] Figure 2 A schematic diagram of a process flow of a method for optimizing big data resources provided by an embodiment of the present invention is shown;

[0044] Figure 3 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] The embodiment of the present invention provides a system for optimizing big data resources, such as Figure 1 As shown, the system includes: a data source layer, an intermediate processing layer and an application layer;

[0047] The data source layer is used to obtain relevant information of the big data cluster from multiple data sources;

[0048] In the embodiment of the present invention, the optimization system of big data resources is an independent system different from the big data platform, task scheduling platform, and task execution platform, and is applicable to multiple technical fields such as the financial field and the medical field that require optimization of big data resources. The data source layer of the system obtains relevant information of the big data cluster from multiple data sources, wherein the data source can be a big data platform, task scheduling platform, task execution platform, etc. in the financial field, the medical field, etc., which are related platforms for managing, storing, transmitting, and applying big data, and the embodiment of the present invention does not make specific limitations. Among them, the relevant information of the big data cluster may include the operation log information and file system log information of the big data platform in the financial field, the medical field, etc., the scheduling information of the big data cluster by the task scheduling platform, the application information of the big data by the task execution platform, etc., and the embodiment of the present invention does not make specific limitations.

[0049] The intermediate processing layer is used to perform secondary processing on the relevant information obtained in the data source layer to obtain secondary processed data that is convenient for statistical processing;

[0050] In the embodiment of the present invention, the middle processing layer of the optimization system of big data resources is used to fuse the data collected from multiple sources, that is, to perform secondary processing on the relevant information obtained in the data source layer to obtain secondary processed data that is convenient for statistical processing, such as parsing the task scheduling information of fund investment business in the financial field, obtaining information such as task name, scheduling frequency, responsible person, responsible department, etc., which is not specifically limited in the embodiment of the present invention. Through the secondary processing in the embodiment of the present invention, the data is processed and integrated to produce data indicators that fully reflect the cluster storage resources and calculation fields. For example, statistics on the access popularity of the Chinese patent medicine data table in the medical field, and determine the associated tasks of the Chinese patent medicine data table, the task scheduling frequency of the associated tasks, the execution success rate of the associated tasks, etc., which are not specifically limited in the embodiment of the present invention.

[0051] The application layer is used to perform statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feed back the statistical results to relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

[0052] In an embodiment of the present invention, the application layer of the optimization system of big data resources is used to perform statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules. Among them, the statistical rules may include statistics on the number of frequent associations of fields, statistics on the number of times table data is scheduled, statistics on the number of times tasks are scheduled, etc., which are not specifically limited in the embodiment of the present invention. After statistical processing by the application layer, the statistical results are obtained, and then the statistical results are sent to the relevant responsible persons based on the information such as the responsible person and the responsible department obtained by parsing the secondary processing data, so that the relevant responsible persons can perform resource optimization processing based on the statistical results. For example, by summarizing the statistical indicators, silent data tables or resource-consuming data tables are found. For example: the storage capacity of the table is large, but no one has accessed the table in the past three years. Then the table can notify the developer to locate whether it can be offline. If the data table is associated with a task, it is necessary to locate whether the task can also be offline. If it can be offline, it can not only reduce storage resources but also reduce the computing resources consumed by processing.

[0053] For another example, you can associate the table and task information through the task name, summarize and count all the table information associated with the task, and use the associated table information to count whether the task operation is valuable. For example: the task-related table has no downstream table, the task has no downstream associated tasks, etc. For such tasks, you can work with the developers to determine whether the task needs to be taken offline. If it can be taken offline, the computing resource consumption can be reduced. The task mainly counts the task scheduling cycle, the name of the task, the number of days without changes in the task-related table, the number of mr consumed by the task operation, whether all the upstream tables of the task are static tables, whether the sql execution data has data skew, the number of silent days of the daily task, and other labels. It fully displays the resource consumption of the task operation, whether the table generated by the task is a valuable table, whether there are similar tasks, etc., reflecting whether the task can be governed and optimized.

[0054] For example, in the financial investment scenario, the statistical information of subscription tables of different business units such as the financial investment sector, energy investment sector, pharmaceutical investment sector, consumer investment sector, and technology investment sector are collected on a monthly basis, as well as the distribution graph of the table's most recent access time, the distribution graph of the table's most recent update time, the distribution graph of table access heat, the table size distribution graph, the table CPU time consumption distribution graph, etc., to fully display the overall situation of tables under different tenants and guide further resource optimization.

[0055] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to analyze and process the utilization of big data resources more comprehensively, another big data resource optimization system is provided, such as Figure 1 As shown, the data source layer includes a running log acquisition module, a task scheduling information acquisition module, a task execution information acquisition module, a file system log acquisition module and a permission information acquisition module;

[0056] The operation log acquisition module is used to acquire the operation log of the big data cluster; and parse the operation log to obtain task operation information;

[0057] In an embodiment of the present invention, the operation log acquisition module can obtain the yarn log of the big data cluster. The yarn log is a log record of a resource management and job scheduling system. Usually, Yarn will retain the log file after the application is completed, failed, or killed. These log files are usually stored on HDFS, and users can view these logs through the web interface of YARNResourceManager or use command line tools. After obtaining the operation log, the system also needs to parse and process the operation log to obtain the task operation information. Among them, the task operation information includes task id, task name, task related sql statement, task start time, task end time, executor, execution virtual account, queue information, task consumption cpu time, task read and write hdfs file size, task consumption map\reduce number, etc., which is not specifically limited in the embodiment of the present invention.

[0058] The task scheduling information acquisition module is used to obtain the task scheduling information of the big data cluster from the scheduling platform;

[0059] In the embodiment of the present invention, the task scheduling information acquisition module can directly acquire the task scheduling information of the big data cluster from the scheduling platform through the data interface of the scheduling platform.

[0060] The task execution information acquisition module is used to obtain execution related information from the task execution platform;

[0061] In the embodiment of the present invention, the task execution information acquisition module can directly acquire execution related information from each task execution platform through the data interface opened by the task execution platform, and can also associate the acquired execution related information with the task scheduling information obtained by the task scheduling information acquisition module.

[0062] The file system log acquisition module is used to acquire the file system information of the big data cluster; and transfer the file system information into a file system data table with table as the primary key;

[0063] In an embodiment of the present invention, the file system log acquisition module can acquire the Image log of the big data cluster and obtain the file system information, such as obtaining the hdfs path information of the big data cluster recorded in the Image log. It should be noted that after obtaining the file system information, the file system information needs to be transferred to a file system data table with the table as the primary key for subsequent query and statistical processing.

[0064] The permission information acquisition module is used to acquire the permission information corresponding to the big data cluster from the permission management database.

[0065] In the embodiment of the present invention, the authority information acquisition module can directly obtain relevant information of the authority management database hms through the interface, including cluster authority information, database responsible person and responsible department information, etc., which is not specifically limited in the embodiment of the present invention.

[0066] It should be noted that the information obtained from multiple data sources includes the bottom-level YARN log information, which integrates permission information, task scheduling information, cluster metadata and other information. It helps to monitor the resource consumption of data tables and tasks in an all-round and root-cause manner, and provides users with comprehensive reference value.

[0067] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to decompose and process the data in the huge big data cluster operation log, another big data resource optimization system is provided, and the log content acquisition module parses and processes the operation log to obtain task operation information, including:

[0068] Parsing the log folders in the operation log one by one to obtain parsing results corresponding to each log folder;

[0069] In the embodiment of the present invention, the log content acquisition module of the system is classified by log folder, traverses all log folders, parses the log folders one by one, and obtains the parsing results corresponding to each log folder. Among them, each time a log folder is parsed, the parsing result is saved to the database (each folder is parsed separately, mainly to solve the problem of too much log volume and insufficient memory).

[0070] The analysis results are formatted and stored in a partition table to obtain the task running information corresponding to each task.

[0071] In the embodiment of the present invention, when saving, the parsing results need to be formatted first to ensure that the parsed data has a standardized data format, which is convenient for later data sorting and statistics. After the data is formatted, according to statistical requirements, the formatted parsing results are stored in the hive partition table, such as partitioning according to the task ID, using the task ID as the partition key to divide the data into different folders, so as to obtain the task running information corresponding to each task. In this way, only the data of a specific partition can be scanned during the query, thereby improving the query efficiency.

[0072] It should be noted that the log content acquisition module in the embodiment of the present invention is also used to perform data cleaning on the task operation information obtained by parsing based on a preset data cleaning cycle to release the storage space of the above partition table. For example, if the data cleaning cycle is set to 30 days, the parsed log data only saves the data of the latest 30 days. This automatic cleaning method can alleviate the pressure of data storage.

[0073] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to obtain the association relationship between data tables from the operation log to facilitate the subsequent data statistical analysis, another big data resource optimization system is provided, wherein the log content acquisition module includes a table association relationship determination unit;

[0074] The table association relationship determination unit is used to parse the data query statements in the operation log to obtain target data tables corresponding to the data query statements and to obtain table association relationships between the target data tables.

[0075] In an embodiment of the present invention, a table association relationship determination unit is used to parse and process data query statements in a running log, that is, to parse SQL statements, obtain target data tables corresponding to SQL statements, and simultaneously obtain table association relationships between target data tables. Among them, table association relationships include one-to-one association relationships (One-to-One Relationship), one-to-many association relationships (One-to-Many Relationship), many-to-one association relationships (Many-to-OneRelationship) and many-to-many association relationships (Many-to-Many Relationship), which are not specifically limited in the embodiment of the present invention. In a relational database, the association relationship between data tables is established by sharing the values ​​of the same columns (usually primary keys and foreign keys). This association relationship allows users to obtain related data across multiple data tables.

[0076] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to integrate the data obtained from multiple data sources, facilitate data statistics and analysis from multiple angles in the later stage, and fully reflect the data indicators of cluster storage resources and calculation fields, another big data resource optimization system is provided, such as Figure 1 As shown, the intermediate processing layer includes a cluster association analysis module, a node analysis module, a field association analysis module, and a system and authority information analysis module;

[0077] The cluster association analysis module is used to analyze all the table association relationships in the big data cluster to obtain the cluster association relationship corresponding to the big data cluster;

[0078] The node parsing module is used to parse the out-degree and in-degree of all data tables in the big data cluster, and determine the node type of the data table based on the out-degree and in-degree obtained by parsing; the node type includes a root node and a leaf node;

[0079] The field association analysis module is used to parse the fields in all data tables in the big data cluster to obtain field association relationships;

[0080] The system and permission information parsing module is used to parse the file system data table to obtain the table size, number of table files, creation time, update time, last access time, associated tasks and whether it is a partition table; and determine the person responsible for the table based on the permission information.

[0081] In the embodiment of the present invention, the intermediate processing layer is mainly used for secondary parsing of the operation log parsing results obtained by the data source layer, the association relationship between data tables and other intermediate result tables. It includes parsing all the table association relationships in the big data cluster through the cluster association parsing module, obtaining the cluster association relationship corresponding to the big data cluster, and saving the cluster association relationship in the relational database. The node parsing module parses the in-degree and out-degree of all data tables in the big data cluster, and determines the node type of each data table based on the parsed out-degree and in-degree; wherein the node type includes the root node and the leaf node. The field association parsing module parses the fields in all data tables in the big data cluster to obtain the field association relationship. The file system data table obtained by the data source layer is parsed by the system and permission information parsing module to obtain the table size, the number of table files, the creation time, the update time, the last access time, the associated tasks and whether it is a partition table; and the person in charge of the table is determined based on the permission information, etc., which is not specifically limited in the embodiment of the present invention.

[0082] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to analyze the utilization of data resources in a more hierarchical manner, another big data resource optimization system is provided, such as Figure 1 As shown, the application layer includes a field-level statistics module, a partition-level statistics module, a table-level statistics module, and a task-level statistics module;

[0083] The field-level statistics module is used to perform statistical processing on the number of times fields are frequently associated, and identify key fields based on the statistical results;

[0084] The partition-level statistics module is used to partition and save the basic information of the table and archive it, and perform statistical processing based on the partition results to obtain partition summary information;

[0085] The table-level statistical module is used to perform statistical processing on the resource consumption of the table based on the statistical indicators, and send the information to be optimized that does not meet the indicator threshold to the relevant responsible person for resource optimization processing;

[0086] The task-level statistical module is used to perform statistical processing on the task value of the task to be analyzed based on the task name and all table information associated with the task, and send the information to be optimized that does not meet the task value threshold to the relevant responsible person for resource optimization processing.

[0087] In the embodiment of the present invention, the application layer is mainly a data processing layer for customers, and can perform statistical analysis on big data resources at multiple levels according to customer needs.

[0088] Among them, the field-level statistical module is used to perform statistical processing on the number of times the fields are frequently associated, and identify key fields based on the statistical results; for example, the number of times the "company market value" field in the financial investment scenario is frequently associated is statistically processed to determine whether the field is a key field; the number of times the "drug ingredient" field in the medical scenario is frequently associated is statistically processed to determine whether the field is a key field, etc., and the embodiments of the present invention do not make specific limitations.

[0089] Among them, the partition-level statistical module is used to partition and save the basic information of the table and archive it, and perform statistical processing according to the partition results to obtain partition summary information; for example, the information in each data table is partitioned and saved and archived according to the tenant information, the corresponding tenant data is generated, and the data file is exported to the relational database using the sqoop method. The relational database uses partition tables to store different tenant data. Then, all the table information under the tenant is obtained according to the partition results for statistical processing to obtain the big data resource utilization corresponding to the tenant, and then the statistical information of the subscription tables of different business units is counted on a monthly basis, as well as the distribution diagram of the table's recent access time, the distribution diagram of the table's recent update time, the distribution diagram of the table's access heat, the distribution diagram of the table's size, the distribution diagram of the table's CPU consumption time, etc., to fully display the overall situation of the tables under different tenants, guide further resource optimization, etc., and the embodiments of the present invention are not specifically limited.

[0090] Among them, the table-level statistical module is used to perform statistical processing on the resource consumption of the table based on statistical indicators, and send the information to be optimized that does not meet the indicator threshold to the relevant responsible person for resource optimization processing; for example, the storage capacity and access volume of the table are used as statistical indicators for statistical processing. When the storage capacity of the table is greater than the set storage capacity threshold and the access volume is less than the set access volume threshold, it is considered that the data table does not meet the indicator threshold, and the relevant information of the data table needs to be sent to the relevant responsible person, who will then evaluate whether to keep the data table, etc. The embodiment of the present invention does not make specific limitations.

[0091] Among them, the task-level statistics module is used to perform statistical processing on the task value of the task to be analyzed based on the task name and all table information associated with the task, and send the information to be optimized that does not meet the task value threshold to the relevant responsible person for resource optimization processing; for example, the task association table has no downstream table, the task has no downstream associated tasks, etc. For such tasks, developers can determine whether the task needs to be taken offline. If it can be taken offline, the computing resource consumption can be reduced. The task mainly counts the scheduling cycle of the task, the name of the task, the number of days without change in the task association table, the number of mr consumed by the task operation, whether all the upstream tables of the task are static tables, whether the sql execution data has data skew, the number of silent days of the daily task, and other labels. Comprehensively display the resource consumption of the task operation, whether the table generated by the task is a valuable table, whether there are similar tasks, etc., to reflect whether the task can be governed and optimized.

[0092] The embodiment of the present invention provides an optimization system for big data resources. Compared with the prior art, the present invention realizes the all-round information collection of the resource utilization of the big data cluster by obtaining the relevant information of the big data cluster from multiple data sources; secondly, the present invention performs secondary processing on the relevant information of the big data cluster collected from multiple data sources through the intermediate processing layer to obtain secondary processed data that is convenient for statistical processing, thereby realizing the technical effect of multi-source data fusion and facilitating the comprehensive statistics of table-related data resources and task-related data resources. The present invention also performs statistical processing on the secondary processed data obtained by the intermediate processing layer according to different statistical rules through the application layer, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results. Not only can the analysis of the resource utilization at the table level in the field of finance and medical technology be realized, but also the analysis of the data utilization at the task level in the field of finance and medical technology can be realized, so that the analysis results of the big data resources are more comprehensive and thorough, which is convenient for the relevant responsible persons to make more effective optimization decisions.

[0093] As the above Figure 1 The system shown in the figure is implemented in operation. The embodiment of the present invention provides a method for optimizing big data resources, such as Figure 2 As shown, the method includes:

[0094] 101. The data source layer obtains relevant information of the big data cluster from multiple data sources according to the monitoring time period; and sends the relevant information to the intermediate processing layer;

[0095] In an embodiment of the present invention, the data source layer obtains relevant information of the big data cluster from multiple data sources according to the monitoring time period. Among them, the monitoring time period is a time period for obtaining data set as needed, which can be specifically set to 12 hours, 24 hours, etc., and the embodiment of the present invention does not make specific limitations. Among them, the data source can be a big data platform, task scheduling platform, task execution platform, etc. in the financial field, medical field, etc., which is used to manage, store, transmit, and apply big data. The embodiment of the present invention does not make specific limitations. Among them, the relevant information of the big data cluster may include operation log information and file system log information of big data platforms in the financial field, medical field, etc., scheduling information of the big data cluster by the task scheduling platform, and application information of the big data by the task execution platform, etc., which are not specifically limited in the embodiment of the present invention.

[0096] 102. The intermediate processing layer performs secondary processing on the relevant information to obtain secondary processing data that is convenient for statistical processing;

[0097] In the embodiment of the present invention, the middle processing layer of the optimization system of big data resources performs fusion processing on the data collected from multiple sources, that is, performs secondary processing on the relevant information obtained in the data source layer to obtain secondary processed data that is convenient for statistical processing, such as parsing the task scheduling information of fund investment business in the financial field, obtaining information such as task name, scheduling frequency, responsible person, responsible department, etc., which is not specifically limited in the embodiment of the present invention. Through the secondary processing in the embodiment of the present invention, the data is processed and integrated to produce data indicators that fully reflect the cluster storage resources and calculation fields. For example, statistics on the access popularity of the Chinese patent medicine data table in the medical field, and determine the associated tasks of the Chinese patent medicine data table, the task scheduling frequency of the associated tasks, the execution success rate of the associated tasks, etc., which are not specifically limited in the embodiment of the present invention.

[0098] 103. Based on the time interval of the statistical cycle, the application layer performs statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feeds back the statistical results to relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

[0099] In an embodiment of the present invention, the application layer of the optimization system of big data resources is used to perform statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules. Among them, the statistical rules may include statistics on the number of frequent associations of fields, statistics on the number of times table data is scheduled, statistics on the number of times tasks are scheduled, etc., which are not specifically limited in the embodiment of the present invention. After statistical processing by the application layer, the statistical results are obtained, and then the statistical results are sent to the relevant responsible persons based on the information such as the responsible person and the responsible department obtained by parsing the secondary processing data, so that the relevant responsible persons can perform resource optimization processing based on the statistical results. For example, by summarizing the statistical indicators, silent data tables or resource-consuming data tables are found. For example: the storage capacity of the table is large, but no one has accessed the table in the past three years. Then the table can notify the developer to locate whether it can be offline. If the data table is associated with a task, it is necessary to locate whether the task can also be offline. If it can be offline, it can not only reduce storage resources but also reduce the computing resources consumed by processing.

[0100] It should be noted that the method in the embodiment of the present invention integrates multiple data sources, automatically realizes timed parsing data and statistical results, and imports data into a relational database for display and secondary statistics. Moreover, the method can connect the web page with the big data environment, and can count specific report information according to different tenant information, so that users can optimize and monitor their own data. The big data side obtains tenant information and table information under the tenant by calling the interface of the web service, generates corresponding tenant data according to the table contained in the tenant, and exports the data file to the relational database in the sqoop way. The relational database uses a partition table to store different tenant data to avoid problems such as memory overflow of the web project due to too much data. The web service mainly sets the optimization threshold according to the user, automatically screens out targeted problem tables and problem tasks, and the selected problem data will be sent to the corresponding person in charge by email for processing. The method of the present invention allows users to easily monitor the situation of tasks and tables belonging to tenants, and the system can automatically schedule parsing, analysis, statistics, and derivatives according to the cycle. It can fully monitor the execution and basic situation of tasks and tables, without the need for professional developers to manually summarize statistics, and visualize the entire monitoring result page.

[0101] The embodiment of the present invention provides a method for optimizing big data resources. Compared with the prior art, the present invention realizes the all-round information collection of the resource utilization of the big data cluster by acquiring the relevant information of the big data cluster from multiple data sources; secondly, the present invention performs secondary processing on the relevant information of the big data cluster collected from multiple data sources through the intermediate processing layer to obtain secondary processed data that is convenient for statistical processing, thereby realizing the technical effect of multi-source data fusion and facilitating the comprehensive statistics of table-related data resources and task-related data resources. The present invention also performs statistical processing on the secondary processed data obtained by the intermediate processing layer according to different statistical rules through the application layer, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results. Not only can the analysis of the resource utilization at the table level in the field of finance and medical technology be realized, but also the analysis of the data utilization at the task level in the field of finance and medical technology can be realized, so that the analysis results of the big data resources are more comprehensive and thorough, which is convenient for the relevant responsible persons to make more effective optimization decisions.

[0102] According to one embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method for optimizing big data resources in any of the above method embodiments.

[0103] Figure 3 A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0104] like Figure 3 As shown, the computer device may include: a processor (processor) 202 , a communication interface (Communications Interface) 204 , a memory (memory) 206 , and a communication bus 208 .

[0105] The processor 202 , the communication interface 204 , and the memory 206 communicate with each other via the communication bus 208 .

[0106] The communication interface 204 is used to communicate with other devices such as clients or other servers.

[0107] The processor 202 is used to execute the program 210, and specifically can execute the relevant steps of the above-mentioned big data resource optimization method.

[0108] Specifically, the program 210 may include program codes, which include computer operation instructions.

[0109] The processor 202 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0110] The memory 206 is used to store the program 210. The memory 206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0111] The program 210 may be specifically configured to enable the processor 202 to perform the following operations:

[0112] The data source layer obtains relevant information of the big data cluster from multiple data sources according to the monitoring time period; and sends the relevant information to the intermediate processing layer;

[0113] The intermediate processing layer performs secondary processing on the relevant information to obtain secondary processing data that is convenient for statistical processing;

[0114] Based on the time interval of the statistical cycle, the application layer performs statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

[0115] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The non-Company software tools or components appearing in the embodiments of the present invention are only examples and do not represent actual use.

Claims

1. A big data resource optimization system, characterized in that: include: Data source layer, intermediate processing layer and application layer; The data source layer is used to obtain relevant information of the big data cluster from multiple data sources; The intermediate processing layer is used to perform secondary processing on the relevant information obtained in the data source layer to obtain secondary processed data that is convenient for statistical processing; The application layer is used to perform statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feed back the statistical results to relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

2. The system according to claim 1, characterized in that The data source layer includes a running log acquisition module, a task scheduling information acquisition module, a task execution information acquisition module, a file system log acquisition module and a permission information acquisition module; The operation log acquisition module is used to acquire the operation log of the big data cluster; and parse the operation log to obtain task operation information; The task scheduling information acquisition module is used to obtain the task scheduling information of the big data cluster from the scheduling platform; The task execution information acquisition module is used to obtain execution related information from the task execution platform; The file system log acquisition module is used to acquire the file system information of the big data cluster; and transfer the file system information into a file system data table with table as the primary key; The permission information acquisition module is used to acquire the permission information corresponding to the big data cluster from the permission management database.

3. The system according to claim 2, characterized in that The log content acquisition module parses the operation log to obtain task operation information, including: Parsing the log folders in the operation log one by one to obtain parsing results corresponding to each log folder; The analysis results are formatted and stored in a partition table to obtain the task running information corresponding to each task.

4. The system according to claim 3, characterized in that The log content acquisition module is also used to perform data cleaning processing on the task running information obtained through analysis based on a preset data cleaning cycle to release the storage space of the partition table.

5. The system according to claim 2, characterized in that The log content acquisition module includes a table association relationship determination unit; The table association relationship determination unit is used to parse the data query statements in the operation log to obtain target data tables corresponding to the data query statements and to obtain table association relationships between the target data tables.

6. The system according to claim 5, characterized in that The intermediate processing layer includes a cluster association analysis module, a node analysis module, a field association analysis module, and a system and authority information analysis module; The cluster association analysis module is used to analyze all the table association relationships in the big data cluster to obtain the cluster association relationship corresponding to the big data cluster; The node parsing module is used to parse the out-degree and in-degree of all data tables in the big data cluster, and determine the node type of the data table based on the out-degree and in-degree obtained by parsing; the node type includes a root node and a leaf node; The field association analysis module is used to parse the fields in all data tables in the big data cluster to obtain field association relationships; The system and permission information parsing module is used to parse the file system data table to obtain the table size, number of table files, creation time, update time, last access time, associated tasks and whether it is a partition table; and determine the person responsible for the table based on the permission information.

7. The system according to any one of claims 1 to 6, characterized in that: The application layer includes a field-level statistics module, a partition-level statistics module, a table-level statistics module, and a task-level statistics module; The field-level statistics module is used to perform statistical processing on the number of times fields are frequently associated, and identify key fields based on the statistical results; The partition-level statistics module is used to partition and save the basic information of the table and archive it, and perform statistical processing based on the partition results to obtain partition summary information; The table-level statistical module is used to perform statistical processing on the resource consumption of the table based on the statistical indicators, and send the information to be optimized that does not meet the indicator threshold to the relevant responsible person for resource optimization processing; The task-level statistical module is used to perform statistical processing on the task value of the task to be analyzed based on the task name and all table information associated with the task, and send the information to be optimized that does not meet the task value threshold to the relevant responsible person for resource optimization processing.

8. A method for optimizing big data resources, characterized in that: include: The data source layer obtains relevant information of the big data cluster from multiple data sources according to the monitoring time period; and sending the relevant information to the intermediate processing layer; The intermediate processing layer performs secondary processing on the relevant information to obtain secondary processing data that is convenient for statistical processing; Based on the time interval of the statistical cycle, the application layer performs statistical processing on the secondary processing data obtained by the intermediate processing layer according to different statistical rules, and feeds back the statistical results to the relevant responsible persons, so that the relevant responsible persons can perform resource optimization processing based on the statistical results.

9. A storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction executes an operation corresponding to the big data resource optimization method as described in claim 8.

10. A computer device, comprising a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the big data resource optimization method as described in claim 8.

Images