Resource digital operation management method and system
The application information is obtained through the interfaces of the resource manager and the SQL query engine, and data statistics and optimization are carried out, which solves the problem of unbalanced resource use in the enterprise, and the rational allocation and efficient utilization of resources are achieved, ensuring that resource use is consistent with business goals.
Patent Information
- Application Number
- CN202410983048.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
Smart Images

Figure CN118916404A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of resource management, and in particular to a resource digital operation management method and system. Background Art
[0002] Currently, digital resources are purchased in a unified manner in enterprises or organizations, and resource consumption statistics are not collected for each user, task, or service. This means that enterprises and organizations do not have a clear understanding of resource usage. In addition, there may be an imbalance in resource allocation during actual use, resulting in some users or tasks occupying too many resources while other users or tasks have insufficient resources.
[0003] In the actual operation of enterprises, due to the lack of a clear calculation and allocation mechanism for resource usage costs, the allocation of resource usage fees may be unfair or opaque; resource allocation may not be optimized according to business needs and priorities, resulting in inconsistency between resource usage and business goals. Therefore, designing an efficient resource digital management solution has become a technical problem that technicians in this field need to solve urgently. Summary of the invention
[0004] In view of the above-mentioned defects, an embodiment of the present invention discloses a resource digital operation and management method, which can improve the resource utilization efficiency within the enterprise.
[0005] The first aspect of the embodiment of the present invention discloses a resource digital operation management method, including:
[0006] Obtain corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time;
[0007] According to the set statistical logic, statistics are collected on each data information in the application information to determine the resource consumption status of each user within a set time period;
[0008] Data fluctuation detection is performed on the resource consumption status within the set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, an early warning operation is performed on the data within the corresponding set time period; and data on the resource consumption status of each user within the set time period is displayed.
[0009] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the operation management method further includes:
[0010] Obtain resource consumption information of completed queries through the SQL query engine's application interface;
[0011] Collect data statistics on the number of queries submitted by each user and resource consumption information according to the set time interval;
[0012] Displays the number of queries and query ratios for each user's resource consumption information.
[0013] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the resource consumption information includes CPU time, memory usage information and I / O operation information; after the data statistics of the number of queries submitted by each user and the resource consumption situation are collected according to the set time interval, it also includes:
[0014] Determine whether the corresponding query is a slow query or a non-slow query according to the resource consumption information. If the corresponding query is a slow query, transmit the data details of the slow query SQL to the corresponding user or display the slow query data.
[0015] The data details of the slow query SQL are matched with a preset resource usage pattern to determine a corresponding query scheduling strategy, and the corresponding slow query is adjusted to a corresponding execution time according to the query scheduling strategy.
[0016] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the step of performing statistics on each data information in the application information according to the set statistical logic to determine the resource consumption status of each user within the set time period includes:
[0017] Identify the user ID of each application from the obtained application information to ensure that each task can be associated with the corresponding user;
[0018] Classify tasks into offline tasks and real-time tasks according to attributes or configurations in application information;
[0019] Calculate resource consumption data of each task during execution, wherein the resource consumption data includes CPU time, memory usage, and storage I / O;
[0020] Summarize the resource consumption data of each user and calculate the total resource consumption of each user;
[0021] Compare the resource consumption of each user with the total resource consumption of the cluster and calculate the resource share of each user;
[0022] The data obtained by the above statistics are divided into different time windows to determine the resource consumption status in different time periods.
[0023] As an optional implementation manner, in the first aspect of the embodiment of the present invention, after determining the resource consumption status in different time periods, the method further includes:
[0024] Analyze resource consumption status in different time periods to determine resource usage trends, and determine peak and trough periods of resource usage and resource usage patterns based on the resource usage trends;
[0025] After comparing the resource consumption of each user with the total resource consumption of the cluster and calculating the resource proportion of each user, the following steps are also included:
[0026] Determine the department information corresponding to each user based on the attribute information of each user;
[0027] Aggregating user data based on the department information to determine total resource consumption for each department;
[0028] Compare the resource consumption of each department with the total resource consumption of the cluster and calculate the resource share of each department.
[0029] As an optional implementation manner, in the first aspect of the embodiment of the present invention, after comparing the resource consumption of each department with the total resource consumption of the cluster and calculating the resource proportion of each department, the method further includes:
[0030] Determine the resource usage costs of each department in the corresponding period based on the total resource usage costs and the resource proportion of each department, and display the resource usage costs of each department;
[0031] The application interface of the resource manager includes the API of YARN ResourceManager; the SQL query engine includes the Presto REST API.
[0032] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the operation management method further includes:
[0033] Obtain the usage status information of each wide table in the corresponding data warehouse within the set period;
[0034] Determine wide table statistics information according to usage status information of each wide table, wherein the wide table statistics information includes the number of wide table queries and the number of field queries;
[0035] According to the wide table statistics, data is matched with the pre-built wide table screening strategy. If the set requirements are met, the wide table in the data warehouse is not modified; if the set requirements are not met, the next step is executed;
[0036] Determine the content to be optimized of the corresponding wide table in the data warehouse, and optimize the corresponding wide table according to the content to be optimized to implement content update of the data warehouse.
[0037] A second aspect of an embodiment of the present invention discloses a resource digital operation management system, including:
[0038] Acquisition module: used to acquire corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time;
[0039] Statistics module: used to collect statistics of various data information in the application information according to the set statistical logic to determine the resource consumption status of each user within the set time period;
[0040] Early warning module: used to detect data fluctuations in the resource consumption status within a set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, an early warning operation is performed on the data within the corresponding set time period; and data on the resource consumption status of each user within the set time period is displayed.
[0041] The third aspect of an embodiment of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the resource digital operation and management method disclosed in the first aspect of the embodiment of the present invention.
[0042] A fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the resource digital operation and management method disclosed in the first aspect of an embodiment of the present invention.
[0043] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0044] The resource digital operation management method in the embodiment of the present invention counts the various data in the application information according to the set statistical logic, and can accurately calculate the resource consumption status of each user within the set time period. This accurate statistics can not only help managers quickly identify hot spots and bottlenecks in resource use, but also provide a scientific basis for resource allocation and optimization, and promote the rational allocation and efficient use of resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1 It is a flowchart of a resource digital operation management method disclosed in an embodiment of the present invention;
[0047] Figure 2 It is a schematic diagram of the process of querying data display disclosed in an embodiment of the present invention;
[0048] Figure 3 It is a schematic diagram of a specific process of resource consumption statistics disclosed in an embodiment of the present invention;
[0049] Figure 4 is a schematic diagram of image display disclosed in an embodiment of the present invention;
[0050] Figure 5 It is a structural diagram of a resource digital operation management system provided by an embodiment of the present invention;
[0051] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] It should be noted that the terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0054] At present, digital resources are purchased uniformly in enterprises or organizations, and resource consumption statistics are not performed for each user, task, and service; this makes it impossible for enterprise organizations to have a clear understanding of resource usage; and there may also be problems of unbalanced resource allocation in the actual use process, resulting in some users or tasks occupying too many resources, while other users or tasks have insufficient resources. In the actual operation process of the enterprise, due to the lack of a clear calculation and allocation mechanism for resource usage costs, the allocation of resource usage fees may be unfair or opaque; resource allocation may not be optimized according to business needs and priorities, resulting in inconsistent resource usage with business goals. Based on this, the embodiment of the present invention discloses a resource digital operation management method, system, electronic device, and storage medium, which counts various data in application information according to the set statistical logic, and can accurately calculate the resource consumption status of each user within a set time period. This precise statistics can not only help managers quickly identify hot spots and bottlenecks in resource usage, but also provide a scientific basis for resource allocation and optimization, and promote the rational allocation and efficient use of resources.
[0055] Embodiment 1
[0056] See also Figure 1 , Figure 1 It is a flow chart of the resource digital operation and management method disclosed in the embodiment of the present invention. Among them, the execution subject of the method described in the embodiment of the present invention is an execution subject composed of software and / or hardware, and the execution subject can receive relevant information by wired or / and wireless means, and can send certain instructions. Of course, it can also have certain processing functions and storage functions. The execution subject can control multiple devices, such as a remote physical server or cloud server and related software, or it can be a local host or server and related software that performs related operations on a device placed somewhere. In some scenarios, multiple storage devices can also be controlled, and the storage devices can be placed in the same place or different places as the devices. For example Figure 1 As shown, the resource-based digital operation management method includes the following steps:
[0057] S101: Acquire corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time;
[0058] S102: Counting various data information in the application information according to a set statistical logic to determine the resource consumption status of each user within a set time period;
[0059] S103: Perform data fluctuation detection on the resource consumption status within the set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, perform early warning operations on the data within the corresponding set time period; and display data on the resource consumption status of each user within the set time period.
[0060] The embodiment of the present invention obtains detailed application information (including ID, user information, name, queue status, tracking URL, start and completion time, etc.) through the application interface of the resource manager, which provides a solid foundation for subsequent statistical analysis. This comprehensive information capture capability ensures the integrity and accuracy of the data and helps to understand resource usage more accurately.
[0061] The embodiment of the present invention counts various data in the application information according to the set statistical logic, and can accurately calculate the resource consumption status of each user in the set time period. This accurate statistics can not only help managers quickly identify hot spots and bottlenecks in resource usage, but also provide a scientific basis for resource allocation and optimization.
[0062] The embodiment of the present invention detects data fluctuations in resource consumption within a set time period and automatically determines whether the fluctuation parameters exceed the set requirements, thereby realizing instant discovery of abnormal resource consumption. Once an abnormality is detected, an early warning operation is immediately performed, which helps to respond quickly and solve the problem, prevent excessive consumption or waste of resources, and ensure stable operation of the system.
[0063] The embodiment of the present invention displays the resource consumption status of each user within a set time period in an intuitive manner, so that both managers and users can clearly understand the resource usage. This transparent display method helps to improve management efficiency, and also helps to enhance users' awareness of resource usage and promote the rational allocation and efficient use of resources.
[0064] Based on the above comprehensive information management and accurate statistical analysis, managers can formulate resource allocation strategies more scientifically and optimize resource allocation. At the same time, these data also provide strong support for decision-making, so that managers can have a basis to rely on when facing decisions such as resource allocation and system upgrades, and make more reasonable and efficient decisions.
[0065] More preferably, Figure 2 As shown, the operation management method further includes:
[0066] S104: Obtaining resource consumption information of the completed query through the application interface of the SQL query engine;
[0067] S105: Collecting data statistics on the number of queries and resource consumption information submitted by each user according to a set time interval;
[0068] S106: Displaying the number of queries and the query proportion data for each user's resource consumption information.
[0069] The embodiment of the present invention obtains the resource consumption information of the completed query through the application interface of the SQL query engine, so that the system administrator or database administrator can clearly understand the occupation of system resources by each query operation. This transparency helps to identify queries with high resource consumption, so as to take corresponding measures to optimize and avoid resource bottlenecks.
[0070] The embodiment of the present invention collects data statistics on the number of queries submitted by each user and the resource consumption information according to the set time interval, which can help the system understand the distribution of resource usage in different time periods. Based on these data, the system can adjust the resource allocation strategy more intelligently to ensure that there are enough resources to meet user needs during high-load periods, and reasonably release resources during low-load periods to save costs.
[0071] The embodiment of the present invention displays the number of queries and the proportion of queries for each user's resource consumption information, which not only allows administrators to understand the query behavior patterns of users, but also identifies potential abuse or abnormal behaviors. This is crucial for maintaining the security and stability of the system, because administrators can promptly discover and handle user behaviors that may have an adverse impact on the system.
[0072] By displaying the number of queries and the resource consumption ratio, the embodiment of the present invention allows users to intuitively see the impact of their query operations on system resources, thereby more actively optimizing their query statements or query logic and reducing unnecessary resource consumption. At the same time, administrators can also guide users to optimize performance based on these data to improve overall system performance and user experience.
[0073] These statistics and display results can also serve as a basis for decision-making. For example, when evaluating whether to upgrade hardware or expand system capacity, these data can provide strong support. In addition, they can also help managers set various indicators more scientifically and reasonably when formulating SLA (service level agreement) or resource usage policies.
[0074] More preferably, the resource consumption information includes CPU time, memory usage information and I / O operation information; after the data statistics of the number of queries submitted by each user and the resource consumption situation according to the set time interval are collected, it also includes:
[0075] Determine whether the corresponding query is a slow query or a non-slow query according to the resource consumption information. If the corresponding query is a slow query, transmit the data details of the slow query SQL to the corresponding user or display the slow query data.
[0076] The data details of the slow query SQL are matched with a preset resource usage pattern to determine a corresponding query scheduling strategy, and the corresponding slow query is adjusted to a corresponding execution time according to the query scheduling strategy.
[0077] The embodiment of the present invention can more accurately understand the specific impact of query operations on system resources by refining resource consumption information into CPU time, memory usage information and I / O operation information. This refined analysis helps to quickly locate performance bottlenecks and then take targeted optimization measures.
[0078] The embodiment of the present invention determines whether a query is a slow query or a non-slow query based on resource consumption information, and timely transmits the data details of the slow query SQL to the corresponding user or displays the data, which can significantly improve the user's understanding of their own query performance. The user can adjust the query logic and optimize the query statement based on this information, thereby improving the query efficiency. At the same time, this also helps to reduce excessive consumption of system resources due to user ignorance.
[0079] The embodiment of the present invention matches the data details of the slow query SQL with the preset resource usage mode to determine the corresponding query scheduling strategy, and adjusts the execution time of the slow query according to the strategy, which is an intelligent resource management method. Through this method, the system can arrange the execution of slow queries during low-load periods, thereby reducing the impact on the normal business of the system. This intelligent scheduling not only improves the utilization rate of system resources, but also ensures the stability and response speed of the system during high-load periods.
[0080] The embodiments of the present invention can significantly improve user experience and satisfaction by timely notifying users of slow queries and providing optimization suggestions, and reducing user waiting time through intelligent query scheduling. Users will feel the system's attention and optimization of their query operations, thereby enhancing their trust and dependence on the system.
[0081] More preferably, Figure 3 As shown, the process of collecting statistics of various data information in the application information according to the set statistical logic to determine the resource consumption status of each user within the set time period includes:
[0082] S1021: Identify a user ID of each application from the acquired application information to ensure that each task can be associated with a corresponding user;
[0083] S1022: Classify the task into an offline task and a real-time task according to the attribute or configuration in the application information;
[0084] S1023: Calculate resource consumption data of each task during execution, where the resource consumption data includes CPU time, memory usage, and storage I / O;
[0085] S1024: Summarize the resource consumption data of each user and calculate the total resource consumption of each user;
[0086] S1025: Compare the resource consumption of each user with the total resource consumption of the cluster, and calculate the resource proportion of each user;
[0087] S1026: Divide the above statistically obtained data into different time windows to determine the resource consumption status in different time periods.
[0088] By identifying the user ID of each application from the application information (S1021), the system can ensure that each task can be accurately associated with the corresponding user. This accurate user association mechanism enables the system to track the resource usage of each user, providing strong support for subsequent resource allocation and scheduling.
[0089] The embodiment of the present invention classifies tasks into offline tasks and real-time tasks (S1022), which helps the system to assign different resource priorities according to the nature of the tasks. Real-time tasks usually require faster response time and higher resource guarantees, while offline tasks can be performed when system resources are relatively loose. This classification management helps to optimize the utilization efficiency of system resources and improve overall performance.
[0090] The embodiment of the present invention summarizes the resource consumption data of each user (S1024) and calculates the total resource consumption of each user (S1025), so that the system can clearly grasp the resource occupation of each user. This user-level resource consumption analysis helps to identify users with high resource consumption, and then take corresponding measures to allocate or optimize resources.
[0091] The embodiment of the present invention compares the resource consumption of each user with the total resource consumption of the cluster (S1025), calculates the resource proportion of each user, and helps evaluate the relative position of the user in the cluster resources. At the same time, the statistical data is divided according to different time windows (S1026) to determine the resource consumption status of different time periods, which helps the system administrator understand the trend and pattern of resource usage and provides a reference for future resource planning and scheduling.
[0092] like Figure 4As shown in the figure, it is the corresponding data statistics result; by calling the API of YARN ResourceManager, the completed application information is obtained, including application ID, user, name, queue, status, final status, progress, tracking URL, start time and completion time. Statistics are collected on the resource usage of each user per hour, and the resource consumption of each user in each time period is displayed. The task and resource history change chart for the past 30 days is displayed. When drastic changes occur, they can be discovered, analyzed and processed in time, and statistics are collected on each user, the number of offline tasks, the number of real-time tasks, and the proportion of total resources, etc., to provide data support for resource payment.
[0093] By accessing the Presto REST API, we can incrementally crawl the resource consumption of completed queries and count the number of queries submitted by each user and the resource consumption. We can display all the queries of the day by time period and arrange the query requests of users reasonably.
[0094] According to the above statistical results, a charging plan for resource usage fees is formulated. The fee is determined based on the resource usage of each user and department, achieving a more fair and refined cost allocation.
[0095] Monitor the usage of the entire cluster resources, reasonably allocate resources between components and coordinate the number of tasks in each time period to prevent all resources from being used intensively or being idle.
[0096] More preferably, after determining the resource consumption status in different time periods, the method further includes:
[0097] Analyze resource consumption status in different time periods to determine resource usage trends, and determine peak and trough periods of resource usage and resource usage patterns based on the resource usage trends;
[0098] After comparing the resource consumption of each user with the total resource consumption of the cluster and calculating the resource proportion of each user, the following steps are also included:
[0099] Determine the department information corresponding to each user based on the attribute information of each user;
[0100] Aggregating user data based on the department information to determine total resource consumption for each department;
[0101] Compare the resource consumption of each department with the total resource consumption of the cluster and calculate the resource share of each department.
[0102] By analyzing the resource consumption status in different time periods, the embodiment of the present invention can identify the trend of resource usage, including peak and trough periods and specific resource usage patterns. This trend analysis is crucial for formulating resource allocation strategies, optimizing resource scheduling, and predicting future resource needs. Based on resource usage trends, the system can take more proactive management measures, such as expanding resources in advance during peak periods and recycling or maintaining resources during trough periods, to ensure that the system always remains in the best operating state.
[0103] By determining the department information corresponding to the user based on the user attribute information, the embodiment of the present invention can further aggregate the resource consumption data from the user level to the department level. This department-level data aggregation helps the management understand the resource usage from a more macro perspective and evaluate the resource requirements and efficiency of different departments. Calculating the resource share of each department and comparing it with the total resource consumption of the cluster allows the management to clearly see the contribution and burden of each department in terms of resource usage. This helps to formulate reasonable resource allocation policies, avoid excessive concentration or waste of resources, and promote balanced use of resources among departments.
[0104] The combination of the above technical effects provides management with more comprehensive and in-depth resource usage data and analysis results. These data and analysis results can not only serve as the basis for formulating resource allocation, scheduling and expansion strategies, but also provide strong support for business decision-making, cost control and strategic planning.
[0105] More preferably, after comparing the resource consumption of each department with the total resource consumption of the cluster and calculating the resource proportion of each department, the method further includes:
[0106] Determine the resource usage costs of each department in the corresponding period based on the total resource usage costs and the resource proportion of each department, and display the resource usage costs of each department;
[0107] The application interface of the resource manager includes the API of YARN ResourceManager; the SQL query engine includes the Presto REST API.
[0108] The embodiment of the present invention combines the total resource usage cost with the resource proportion of each department, calculates the resource usage cost of each department in the corresponding period, and displays it, so that the system can provide users with a more refined cost allocation view. This cost allocation method not only helps each department understand its own financial burden in resource usage, but also promotes cost awareness and resource utilization efficiency among departments.
[0109] This increased transparency in cost sharing and expense display will help enhance communication and cooperation within the organization and promote the rational allocation and efficient use of resources.
[0110] The embodiment of the present invention introduces the calculation and display of resource usage costs, so that the management can more accurately grasp the resource usage costs of each department, providing strong support for financial control and management. The management can formulate corresponding budgets and cost control strategies based on the resource usage costs of each department to ensure the realization of the overall financial goals of the organization.
[0111] The embodiment of the present invention clearly points out that the resource manager application interface includes the API of YARN ResourceManager, and the SQL query engine includes the Presto REST API, which reflects the flexibility and compatibility of the system design. As a key component in the Hadoop ecosystem, YARN ResourceManager provides powerful resource management and scheduling capabilities; while Presto REST API provides users with an efficient and convenient SQL query interface. This choice of technical implementation enables the system to fully utilize the advantages of the existing technology ecosystem to achieve automation and intelligence of resource management.
[0112] By providing a detailed display and query interface for resource usage fees, users can more intuitively understand their own cost and efficiency in resource usage. This improvement in information transparency helps to enhance users' trust and satisfaction with the system and promote good interaction and cooperation between users and the system.
[0113] More preferably, the operation management method further includes:
[0114] Obtain the usage status information of each wide table in the corresponding data warehouse within the set period;
[0115] Determine wide table statistics information according to usage status information of each wide table, wherein the wide table statistics information includes the number of wide table queries and the number of field queries;
[0116] According to the wide table statistics, data is matched with the pre-built wide table screening strategy. If the set requirements are met, the wide table in the data warehouse is not modified; if the set requirements are not met, the next step is executed;
[0117] Determine the content to be optimized of the corresponding wide table in the data warehouse, and optimize the corresponding wide table according to the content to be optimized to implement content update of the data warehouse.
[0118] The wide table screening strategy here plays a key role in the operation of the data warehouse; it determines which wide tables need to be optimized and the optimization priority; the specific wide table screening strategy can be matched by the following parameters: query frequency and magnitude, field usage frequency, table size and growth rate, data redundancy, resource consumption, business importance, historical data and expired data, related tables and indexes, data quality, etc.
[0119] Specifically, you can set filtering strategies based on the query frequency and query volume of wide tables. For example, wide tables with high query frequency or large query volume may require more attention and optimization. Analyze the usage frequency of each field in the wide table. For fields with low usage frequency, consider archiving or deleting them to reduce the width of the table. Set a threshold. When the size or growth rate of the wide table exceeds the threshold, trigger the optimization process. This helps to manage the storage space of the data warehouse in a timely manner. Analyze the redundancy of data in the wide table. For tables with a lot of redundant data, consider data cleaning or reconstruction. Monitor the system resources consumed when querying wide tables (such as CPU, memory, I / O, etc.), and focus on and optimize tables with excessive resource consumption. Set filtering strategies based on the importance of the business carried by the wide table. For core business data, more frequent monitoring and optimization may be required. Set strategies to identify and process historical data and expired data. For example, for data that exceeds a certain retention period, consider archiving or deleting it. Analyze the relationship between the wide table and other tables, as well as the use of indexes. For tables with complex relationships or improper index use, redesign or optimization may be required. Set data quality indicators (such as accuracy, completeness, consistency, etc.), and focus on and optimize wide tables with poor data quality. Custom strategies can be flexibly set in the wide table screening of the data warehouse according to specific business needs, the characteristics of the data warehouse, and the goals of the organization. The following are some possible custom strategies: Business priority strategy: Set different wide table screening priorities according to the needs and importance of different businesses for data. For example, for core business data, you may need to set a higher screening frequency and stricter optimization standards. Cost-effectiveness strategy: Consider the cost required to optimize the wide table (such as time, manpower, hardware resources, etc.) and the expected benefits (such as performance improvement, storage space saving, etc.), set a reasonable cost-effectiveness ratio threshold, and only when the optimization benefit is greater than the cost, optimize the wide table. Data growth prediction strategy: Based on the historical data growth, predict the data growth trend of the wide table in the future. For wide tables that are expected to grow faster, you may need to expand or optimize in advance to avoid performance bottlenecks. Data usage pattern strategy: Analyze the data usage pattern of the wide table, such as query type, query time distribution, query result set size, etc., and set the corresponding screening strategy. For example, for wide tables that frequently perform complex queries, you may need to pay more attention to index and query optimization strategies. Data quality monitoring strategy: Set data quality monitoring indicators, such as data accuracy, completeness, and consistency, and perform regular data quality checks on wide tables. For wide tables with poor data quality, more frequent screening and optimization may be required. User feedback strategy: Set corresponding wide table screening strategies based on user feedback and complaints. For example, for wide tables where users report poor query performance, performance optimization may need to be prioritized.Technology trends and best practices strategies: Pay attention to technology trends and best practices in the database and big data fields, and apply new technologies and methods to wide table screening. For example, you can consider using machine learning algorithms to predict the growth trend and performance bottlenecks of wide tables. Security strategy: Consider the security requirements of the data and set corresponding wide table screening strategies. For example, for wide tables that store sensitive information, stricter access control and encryption measures may be required. The above custom strategies can be combined and adjusted according to the specific needs of the organization to achieve the best wide table screening effect. At the same time, it is also necessary to evaluate and update the strategy regularly to adapt to the development and changes of business and technology.
[0120] The resource digital operation management method in the embodiment of the present invention counts the various data in the application information according to the set statistical logic, and can accurately calculate the resource consumption status of each user within the set time period. This accurate statistics can not only help managers quickly identify hot spots and bottlenecks in resource use, but also provide a scientific basis for resource allocation and optimization, and promote the rational allocation and efficient use of resources.
[0121] Embodiment 2
[0122] See also Figure 5 , Figure 5 Schematic diagram of the structure of the resource digital operation management system disclosed in the embodiment of the present invention. Figure 5 As shown, the resource digital operation management system may include:
[0123] Acquisition module 21: used to acquire corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time;
[0124] Statistics module 22: used to collect statistics of various data information in the application information according to the set statistics logic to determine the resource consumption status of each user within a set time period;
[0125] Early warning module 23: used to perform data fluctuation detection on the resource consumption status within the set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, an early warning operation is performed on the data within the corresponding set time period; and data display of the resource consumption status of each user within the set time period is performed.
[0126] The resource digital operation management method in the embodiment of the present invention counts the various data in the application information according to the set statistical logic, and can accurately calculate the resource consumption status of each user within the set time period. This accurate statistics can not only help managers quickly identify hot spots and bottlenecks in resource use, but also provide a scientific basis for resource allocation and optimization, and promote the rational allocation and efficient use of resources.
[0127] Embodiment 3
[0128] See also Figure 6 , Figure 6 Schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may be a computer, a server, etc. Of course, in certain circumstances, it may also be a smart device such as a mobile phone, a tablet computer, a monitoring terminal, and an image acquisition device with processing functions. Figure 6 As shown, the electronic device may include:
[0129] A memory 510 storing executable program codes;
[0130] a processor 520 coupled to the memory 510;
[0131] The processor 520 calls the executable program code stored in the memory 510 to execute part or all of the steps in the resource digital operation management method in the first embodiment.
[0132] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute part or all of the steps in the resource digital operation management method in the first embodiment.
[0133] The embodiment of the present invention further discloses a computer program product, wherein when the computer program product is run on a computer, the computer is enabled to execute part or all of the steps in the resource digital operation management method in the first embodiment.
[0134] An embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps in the resource digital operation management method in embodiment one.
[0135] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0138] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the method described in each embodiment of the present invention.
[0139] In the embodiments provided by the present invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.
[0140] A person of ordinary skill in the art can understand that some or all of the steps in the various methods of the embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.
[0141] The resource digital operation and management method, system, electronic device and storage medium disclosed in the embodiments of the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A resource digital operation management method, characterized in that: include: Obtain corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time; According to the set statistical logic, statistics are collected on each data information in the application information to determine the resource consumption status of each user within a set time period; Data fluctuation detection is performed on the resource consumption status within the set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, an early warning operation is performed on the data within the corresponding set time period; and data on the resource consumption status of each user within the set time period is displayed.
2. The resource digital operation management method according to claim 1, characterized in that: The operation management method further includes: Obtain resource consumption information of completed queries through the SQL query engine's application interface; Collect data statistics on the number of queries submitted by each user and resource consumption information according to the set time interval; Displays the number of queries and query ratios for each user's resource consumption information.
3. The resource digital operation management method according to claim 2, characterized in that: The resource consumption information includes CPU time, memory usage information and I / O operation information; after the data statistics of the number of queries submitted by each user and the resource consumption situation are collected according to the set time interval, it also includes: Determine whether the corresponding query is a slow query or a non-slow query according to the resource consumption information. If the corresponding query is a slow query, transmit the data details of the slow query SQL to the corresponding user or display the slow query data. The data details of the slow query SQL are matched with a preset resource usage pattern to determine a corresponding query scheduling strategy, and the corresponding slow query is adjusted to a corresponding execution time according to the query scheduling strategy.
4. The resource digital operation management method according to claim 1, characterized in that: The step of collecting statistics of each data information in the application information according to the set statistical logic to determine the resource consumption status of each user within the set time period includes: Identify the user ID of each application from the obtained application information to ensure that each task can be associated with the corresponding user; Classify tasks into offline tasks and real-time tasks according to attributes or configurations in application information; Calculate resource consumption data of each task during execution, wherein the resource consumption data includes CPU time, memory usage, and storage I / O; Summarize the resource consumption data of each user and calculate the total resource consumption of each user; Compare the resource consumption of each user with the total resource consumption of the cluster and calculate the resource share of each user; The data obtained by the above statistics are divided into different time windows to determine the resource consumption status in different time periods.
5. The resource digital operation management method according to claim 4, characterized in that: After determining the resource consumption status in different time periods, the method further includes: Analyze resource consumption status in different time periods to determine resource usage trends, and determine peak and trough periods of resource usage and resource usage patterns based on the resource usage trends; After comparing the resource consumption of each user with the total resource consumption of the cluster and calculating the resource proportion of each user, the following steps are also included: Determine the department information corresponding to each user based on the attribute information of each user; Aggregating user data based on the department information to determine total resource consumption for each department; Compare the resource consumption of each department with the total resource consumption of the cluster and calculate the resource share of each department.
6. The resource digital operation management method according to claim 5, characterized in that: After comparing the resource consumption of each department with the total resource consumption of the cluster and calculating the resource proportion of each department, the following is also included: Determine the resource usage costs of each department in the corresponding period based on the total resource usage costs and the resource proportion of each department, and display the resource usage costs of each department; The application interface of the resource manager includes the API of YARN ResourceManager; the SQL query engine includes the Presto REST API.
7. The resource digital operation management method according to claim 1, characterized in that: The operation management method further includes: Obtain the usage status information of each wide table in the corresponding data warehouse within the set period; Determine wide table statistics information according to usage status information of each wide table, wherein the wide table statistics information includes the number of wide table queries and the number of field queries; According to the wide table statistics, data is matched with the pre-built wide table screening strategy. If the set requirements are met, the wide table in the data warehouse is not modified; if the set requirements are not met, the next step is executed; Determine the content to be optimized of the corresponding wide table in the data warehouse, and optimize the corresponding wide table according to the content to be optimized to implement content update of the data warehouse.
8. A resource digital operation management system, characterized in that: include: Acquisition module: used to acquire corresponding application information through the application interface of the resource manager, wherein the application information includes application ID, user information, name, queue, status information, tracking URL, start time and completion time; Statistics module: used to collect statistics of various data information in the application information according to the set statistical logic to determine the resource consumption status of each user within the set time period; Early warning module: used to detect data fluctuations in the resource consumption status within a set time period to determine whether the corresponding fluctuation parameters exceed the set requirements. If so, an early warning operation is performed on the data within the corresponding set time period; and data on the resource consumption status of each user within the set time period is displayed.
9. An electronic device, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the resource digital operation management method described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the resource digital operation management method according to any one of claims 1 to 7.
