Resource consumption statistical method, storage medium and electronic device

Abstract

The invention discloses a resource consumption statistical method, a storage medium and an electronic device, and relates to the technical field of smart home / smart home, and the resource consumption statistical method comprises the steps: obtaining execution metadata information in an offline data bin; analyzing and processing the execution metadata information to obtain a task execution identifier; calling resource consumption information corresponding to the task execution identifier; according to the method, all the task data in the offline data bin can be obtained, the resource consumption information of the task to be counted is called according to the task execution identifier, the demand information is completely collected, and the consumption statistical result is obtained. And the collected data can be calculated and counted according to the customization requirements of statistical personnel, so that powerful support is provided for optimizing the stability of off-line counting, improving the expansibility, evaluating the value of data tasks, carrying out data management work and the like.

Description

Technical Field

[0001] This application relates to the field of smart home / intelligent home technology, and more specifically, to a statistical method for resource consumption, a storage medium, and an electronic device. Background Technology

[0002] As a core component for data storage and processing, the performance, efficiency, and resource consumption of offline data warehouses directly affect the effectiveness of data analysis and decision support. Therefore, statistical analysis of the resource consumption of offline data warehouses is a crucial step in optimizing data management and improving decision-making quality.

[0003] In the early stages of development, traditional offline data warehouses, based on developers' familiarity with the business and the limited number of tasks and data, allow developers to effectively manage data warehouse tasks and data using memory or document records. However, as business needs continue to iterate, the amount of data and tasks in the data warehouse increases, making it increasingly difficult to optimize and iterate the data warehouse. The increasing amount of documentation makes it impossible for developers to have a complete grasp of all task information through memory and document records.

[0004] The native offline data warehouse open-source components lack resource consumption statistics functions, or can only query individual tasks or the entire dataset, and do not provide developers with functions for statistical analysis and calculation of resource consumption data. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for statistical analysis of resource consumption, in order to solve the problem that the resource consumption statistical function of offline data warehouse open source components is missing, or can only be queried for a single task, or can only be queried as a whole, and does not provide developers with the function of statistical analysis and calculation of resource consumption data.

[0006] Firstly, this application provides a statistical method for resource consumption, including:

[0007] Retrieve execution metadata information from the offline data warehouse;

[0008] The execution metadata information is analyzed and extracted to obtain the task execution identifier;

[0009] Retrieve the resource consumption information corresponding to the task execution identifier, wherein the resource consumption information is a portion of the execution metadata information;

[0010] The resource consumption information is converted and processed according to preset requirements to obtain consumption statistics.

[0011] Optionally, the execution metadata information includes: data hierarchy relationships and log information; obtaining the execution metadata information from the offline data warehouse includes:

[0012] Determine the data collection time range and query statement for the task to be statistically analyzed, wherein the query statement is generated based on the data collection time range;

[0013] Based on the query statement, the execution metadata information of multiple tasks in the offline data warehouse is determined, and a query result table is generated based on the execution metadata information. The multiple tasks include the task to be counted, and the query result table is used to display the data hierarchy relationship of each task.

[0014] Based on the lowest level information of the data hierarchy, determine the log information for each task.

[0015] Optionally, the step of analyzing and extracting the execution metadata information to obtain the task execution identifier includes:

[0016] Based on the log information and the task to be counted, a task execution identifier template is determined. The task execution identifier template includes a structural pattern of character information and time information of the task to be counted.

[0017] Based on the task execution identifier template, the execution metadata information is traversed to determine the task execution identifier that matches the task execution identifier template.

[0018] Optionally, retrieving the resource consumption information corresponding to the task execution identifier includes:

[0019] A retrieval request is generated based on the task execution identifier; the retrieval request is used to obtain all information about the task corresponding to the task execution identifier;

[0020] Based on the retrieval request, the execution metadata information of the task to be statistically analyzed is obtained;

[0021] The execution metadata information of the task to be statistically analyzed is stripped to obtain the resource consumption information.

[0022] Optionally, the step of converting the resource consumption information according to preset requirements to obtain consumption statistics results includes:

[0023] The resource consumption information is saved to the local server and then converted to a new format to obtain the first consumption information.

[0024] The first consumption information is classified and processed according to a preset category to obtain the second consumption information;

[0025] Based on the information category corresponding to the preset requirements, the second consumption information is extracted and arithmetic operations are performed to obtain the consumption statistics result.

[0026] Optionally, after obtaining the consumption statistics, the method further includes:

[0027] The consumption statistics are parsed and the results are stored in a preset file.

[0028] The preset file is sent to the statistician, and / or the preset file is pushed to the display tool, which is used to display the specific content of the preset file.

[0029] Secondly, this application provides a resource consumption statistics device, comprising:

[0030] The acquisition module is used to retrieve execution metadata information from the offline data warehouse;

[0031] The processing module is used to analyze and extract the execution metadata information to obtain the task execution identifier;

[0032] The retrieval module is used to retrieve the resource consumption information corresponding to the task execution identifier, wherein the resource consumption information is a portion of the execution metadata information;

[0033] The processing module is also used to perform conversion processing on the resource consumption information according to preset requirements to obtain consumption statistics results.

[0034] Optionally, the device further includes: a determining module;

[0035] The determining module is used to determine the collection time range and query statement of the task to be statistically analyzed, wherein the query statement is generated based on the collection time range.

[0036] The determining module is further configured to determine the execution metadata information of multiple tasks in the offline data warehouse according to the query statement, and generate a query result table according to the execution metadata information. The multiple tasks include the task to be counted, and the query result table is used to display the data hierarchy relationship of each task.

[0037] The determining module is further configured to determine the log information of each task based on the minimum level information of the data hierarchy.

[0038] Optionally, the determining module is further configured to determine a task execution identifier template based on the log information and the task to be counted, wherein the task execution identifier template includes a structural pattern of character information and time information of the task to be counted;

[0039] The determining module is further configured to traverse the execution metadata information according to the task execution identifier template, and determine the task execution identifier that matches the task execution identifier template.

[0040] Optionally, the apparatus further includes: a generation module;

[0041] The generation module is further configured to generate a retrieval request based on the task execution identifier; the retrieval request is used to obtain all information of the task corresponding to the task execution identifier.

[0042] The determining module is further configured to obtain the execution metadata information of the task to be counted based on the retrieval request;

[0043] The processing module is also used to strip the execution metadata information of the task to be statistically analyzed to obtain the resource consumption information.

[0044] Optionally, the device further includes: a storage module;

[0045] The storage module is also used to save the resource consumption information to a local server and perform format conversion processing to obtain the first consumption information;

[0046] The processing module is further configured to classify the first consumption information according to a preset category to obtain the second consumption information;

[0047] The processing module is further configured to extract the second consumption information according to the information category corresponding to the preset requirements and perform arithmetic operations to obtain the consumption statistics result.

[0048] Optionally, the device further includes: a transmitting module;

[0049] The processing module is also used to parse the consumption statistics results and store the parsed results in a preset file;

[0050] The sending module is also used to send the preset file to the statistician, and / or to push the preset file to the display tool, which is used to display the specific content of the preset file.

[0051] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0052] The memory stores computer-executed instructions;

[0053] The processor executes computer execution instructions stored in the memory to implement the resource consumption statistics method as described in the first aspect and various possible implementations of the first aspect.

[0054] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, are used to implement the resource consumption statistical method as described in the first aspect and various possible implementations of the first aspect.

[0055] Fifthly, this application provides a program product, including a computer program, which, when executed by a processor, implements the resource consumption statistics method described above.

[0056] This application provides a method, apparatus, storage medium, and electronic device for statistical analysis of resource consumption. The method obtains execution metadata information from an offline data warehouse, then parses and processes this information to identify the unique identifier of the task execution. Subsequently, based on these task execution identifiers, the system retrieves and extracts the corresponding resource consumption information. According to preset demand standards, the system performs necessary conversions and statistical processing on this resource consumption information, ultimately generating consumption statistics results. This method can comprehensively cover all task data in the offline data warehouse, accurately retrieving the required task resource consumption information through task execution identifiers, ensuring comprehensive information collection. Furthermore, it can perform precise calculations and statistical analysis on the collected data according to the personalized customization needs of statistical personnel, providing strong support for optimizing the stability of offline data warehouses, enhancing their scalability, evaluating the value of data tasks, and promoting the effective implementation of data governance. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the hardware environment for a resource consumption statistics method according to an embodiment of this application;

[0060] Figure 2 A flowchart illustrating a statistical method for resource consumption provided in this application. Figure 1 ;

[0061] Figure 3 A flowchart illustrating a statistical method for resource consumption provided in this application. Figure 2 ;

[0062] Figure 4 A flowchart illustrating a statistical method for resource consumption provided in this application. Figure 3 ;

[0063] Figure 5A schematic diagram of the structure of a resource consumption statistics device provided in this application;

[0064] Figure 6 A schematic diagram of the structure of a resource consumption statistical device provided in this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0066] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0067] According to one aspect of the embodiments of this application, a method for statistically analyzing resource consumption is provided. This method is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, in this embodiment, the above-mentioned method for statistically analyzing resource consumption can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0068] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0069] As a core component for data storage and processing, the performance, efficiency, and resource consumption of offline data warehouses directly affect the effectiveness of data analysis and decision support. Therefore, statistical analysis of the resource consumption of offline data warehouses is a crucial step in optimizing data management and improving decision-making quality.

[0070] In the early stages of traditional offline data warehouse use, developers can effectively manage tasks and data based on memory or documentation, given their familiarity with the business and the limited amount of tasks and data. However, as business needs evolve, the amount of data and tasks in the data warehouse increases, making optimization and iteration more difficult. The growing volume of documentation also makes it impossible for developers to have a comprehensive understanding of all tasks from memory alone. Ultimately, this leads to serious problems such as redundant development, resource abuse, and inconsistent data definitions, severely impacting the stability, professionalism, and scalability of the offline data warehouse.

[0071] Traditional offline data warehouses essentially perform various data processing operations, which consume computing and storage resources. Proper resource allocation planning greatly helps improve the stability and data processing capabilities of offline data warehouses. However, common open-source data resource statistics components are either too fragmented, making it impossible to statistically analyze the overall task, a specific type of task, or a specific time period of offline tasks, or too centralized, only showing the overall cluster usage but not pinpointing a specific task or SQL task.

[0072] In the daily work of data development, quantifying the resource consumption of offline tasks and optimizing tasks and offline data warehouses based on the quantification results is a crucial aspect. Therefore, if we can systematically and flexibly statistically analyze offline resource consumption and adjust the statistical criteria or display results according to needs, it will greatly improve aspects of data development work such as architecture optimization, data governance, and task value calculation.

[0073] Existing offline data warehouse resource consumption information statistics and collection methods have the following drawbacks:

[0074] The inability to automatically summarize the resource consumption of specific offline tasks results in each task being in a separate state. Data developers spend a lot of time manually compiling statistics when organizing information, which is prone to errors and consumes a lot of manpower, resulting in a very low return on investment. Furthermore, as the workload increases, this information cannot be directly used for offline resource statistics in the data warehouse.

[0075] The inability to break down overall resource consumption information into specific tasks results in only seeing the overall data warehouse usage, leading to significant errors in assessing cluster health. It also prevents the identification of problematic tasks, the development of targeted optimization solutions, and the implementation of data governance initiatives.

[0076] To address the aforementioned issues, this application provides a method for statistical analysis of resource consumption. Based on the shortcomings and limitations of traditional resource information display pages, it systematically designs a technical architecture for statistical analysis of resource consumption in offline data warehouses. It details and standardizes the data acquisition layer, data calculation layer, and data display layer, enabling flexible resource statistics based on needs. It accurately calculates resource consumption and execution time records at the task granularity of offline data warehouses. The statistical results can be used for targeted optimization of offline data warehouse task execution and can also calculate resource scarcity at various times throughout the day. This effectively enables automated analysis of offline data warehouses, providing strong support for data governance.

[0077] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0078] Figure 2 A flowchart illustrating a method for statistical analysis of resource consumption provided in this application embodiment. Figure 1 .like Figure 2 As shown, the resource consumption statistics method provided in this embodiment includes:

[0079] S101: Retrieve execution metadata information from the offline data warehouse.

[0080] Among them, execution metadata information refers to detailed information about a series of data generated during task execution, such as attributes, structure, source, and relationships.

[0081] Understandably, after a task is completed, the data generated during the task execution is usually stored in an offline data warehouse. The data in the offline data warehouse can be retrieved using a scheduling tool. For example, the Azkaban workflow scheduler can be used as a scheduling tool.

[0082] S102: Analyze and extract the execution metadata information to obtain the task execution identifier.

[0083] Understandably, the execution metadata obtained is all the data in the offline data warehouse, including various types of data that are not needed by the statisticians. Therefore, the task execution identifier (ID) of the task to be tested is usually determined based on the data requirements of the task to be statistically analyzed, as well as the structure of the execution metadata, execution date, etc.

[0084] S103: Retrieve the resource consumption information corresponding to the task execution identifier. The resource consumption information is part of the execution metadata information.

[0085] Understandably, the task execution identifier only corresponds to the execution metadata of the task to be statistically analyzed that the statistician needs. The execution metadata of the task to be statistically analyzed can be obtained based on the task execution identifier. Most of the data statistical operations of the statistician are focused on data resource consumption. Therefore, the resource manager can be used to retrieve the task resource consumption information of the task to be tested, so that the resource consumption can be calculated and statistically analyzed based on the execution metadata and resource consumption information of the task to be tested.

[0086] The resource manager (YARN, Yet Another Resource Negotiator) is essentially a component that manages and schedules resources within the cluster in a unified manner. It supports multi-tenant environments and can provide different resource quotas and priorities to different statistical personnel. This enables offline data warehouses to process multiple tasks simultaneously and ensures that each task receives sufficient resources to guarantee its execution efficiency.

[0087] S104: Convert and process the resource consumption information according to the preset requirements to obtain the consumption statistics.

[0088] Understandably, different statisticians may have different statistical needs for the same task. For example, a supervisor might need information on the overall resource consumption of the task and its proportion within the total resource consumption. Conversely, a tester might need information on the resource distribution of the task throughout the day. The resource consumption information can be calculated according to the specific needs of the statisticians to obtain the statistical results.

[0089] This embodiment provides a method for statistical analysis of resource consumption. The method acquires execution metadata information from an offline data warehouse; analyzes and processes the execution metadata information to obtain task execution identifiers; retrieves resource consumption information corresponding to the task execution identifiers (the resource consumption information is a portion of the execution metadata information); and performs conversion processing on the resource consumption information according to preset requirements to obtain consumption statistics. This method can acquire all task data in the offline data warehouse and retrieve the resource consumption information of the task to be statistically analyzed based on the task execution identifier, thus fully collecting the required information. Furthermore, it can perform calculations and statistics on the collected data according to the customized needs of statistical personnel, providing strong support for optimizing the stability of the offline data warehouse, improving scalability, evaluating the value of data tasks, and carrying out data governance work.

[0090] Figure 3 A flowchart illustrating a method for statistical analysis of resource consumption provided in this application embodiment. Figure 2 .like Figure 3 As shown, in Figure 2 Based on the embodiments, a possible implementation of determining the task execution identifier is described in detail. The resource consumption statistics method shown in this embodiment includes:

[0091] S201: Determine the data collection time range and query statement for the task to be statistically analyzed.

[0092] The query statement is generated based on the time range of the data collection.

[0093] Understandably, when retrieving data from an offline data warehouse, setting a collection time to collect task execution metadata can effectively avoid inaccurate statistical results due to data collected too early. The collection time range can be in days or hours; the specific collection time depends on the needs of the statisticians. Based on the collection time range, a corresponding Structured Query Language (SQL) should be designed.

[0094] S202: Based on the query statement, determine the execution metadata information of multiple tasks in the offline data warehouse, and generate a query result table based on the execution metadata information.

[0095] Among them, multiple tasks contain tasks to be statistically analyzed, and the query result table is used to display the data hierarchy relationship of each task.

[0096] Understandably, the query statement can retrieve the execution metadata of all tasks within the collection time range, and store the collected execution metadata in the query result table according to the task and the hierarchical relationship between the data. Tasks in offline data warehouses often have complex calling logic, and these calling relationships, i.e., the hierarchical relationships between data, are displayed in relation to each other using the query result table, providing dimensional information to help with subsequent statistical summaries of resource consumption.

[0097] The hierarchical relationships between data are presented in the form of a hierarchical list, which typically includes: projects, execution flows, and jobs. Projects are at the top level, execution flows are below projects, and jobs are below execution flows.

[0098] Project: A project typically refers to a set of well-defined execution flows or data pipelines. These flows and pipelines are used to automate various steps in data processing, from data extraction and transformation to loading, as well as tasks such as training and predicting machine learning models.

[0099] Execution flow: Execution flow is used to organize and manage a set of related jobs. An execution flow usually represents a specific workflow and can contain multiple jobs that are executed in a certain order and with certain dependencies. Execution flow allows the definition of dependencies between jobs, parameter passing, etc., making complex workflows easier to manage and improving maintainability and reusability.

[0100] A job is a specific node in the execution flow, and the smallest unit of execution in the workflow. Each job can have its own parameters and configurations, allowing each job to be customized according to its specific needs. Jobs can be various types of execution tasks.

[0101] S203: Determine the log information for each task based on the lowest level information of the data hierarchy.

[0102] Understandably, jobs are associated with log information, which is generated during job execution. The scheduling tool stores the task execution logs in a database (such as MySQL), and the logs for the corresponding task can be retrieved later based on information such as the task's execution ID.

[0103] S204: Determine the task execution identifier template based on the log information and the tasks to be statistically analyzed.

[0104] The task execution identifier template includes a structural pattern of character and time information of the task to be counted, which may include static parts (such as prefixes and separators) and dynamic parts (such as dates and serial numbers).

[0105] Understandably, the log information includes the date and time of each job's execution, and each job has its own unique log information. The job execution time information is located at a fixed position in the log information. For example, a regular expression can be used to represent the task execution identifier template. An expression is a powerful text processing tool that can match and extract strings that conform to a specific pattern; this string is the task execution ID. First, define the regular expression, that is, based on the format and content of the log data, define a regular expression that can match the task execution ID. For example, if the task execution ID starts with a specific character followed by a combination of numbers or letters, then a regular expression can be defined to match this pattern.

[0106] S205: Based on the task execution identifier template, traverse the execution metadata information to determine the task execution identifier that matches the task execution identifier template.

[0107] Understandably, after defining the task execution identifier template, a programming language is used to apply the template, traversing the stored execution metadata information source (i.e., the query result table) to retrieve each execution metadata record. From each execution metadata record, the task execution identifier field is extracted, and the extracted task execution identifier is matched against the task execution identifier template. This may involve string matching, regular expression matching, date format matching, etc. Successfully matched task execution identifiers are recorded and saved for later use when retrieving resource consumption information.

[0108] This embodiment provides a method for statistical analysis of resource consumption. The method involves determining the data collection time range and query statement for the task to be analyzed; based on the query statement, determining the execution metadata information of multiple tasks in an offline data warehouse, and generating a query result table based on the execution metadata information; determining the log information of each task based on the lowest level of the data hierarchy; determining a task execution identifier template based on the log information and the task to be analyzed; and traversing the execution metadata information according to the task execution identifier template to determine the task execution identifier that matches the template. This method can acquire all task data in the offline data warehouse. Statisticians can determine the task execution identifier based on the tasks they need to analyze, so that the resource consumption of the task under test can be obtained through the task execution identifier, facilitating data governance and improving the accuracy of task data statistics.

[0109] Figure 4 A flowchart illustrating a method for statistical analysis of resource consumption provided in this application embodiment. Figure 3 .like Figure 4 As shown, in Figure 2Based on the embodiments, a possible implementation method for obtaining consumption statistics results is described in detail. The resource consumption statistics method shown in this embodiment includes:

[0110] S301: Generate a retrieval request based on the task execution identifier.

[0111] Among them, the retrieval request is used to obtain all information about the task corresponding to the task execution identifier;

[0112] Understandably, offline data warehouses can use components like YARN to retrieve resource consumption information. The YARN component's web interface can display the specific resource consumption information for a task execution ID. To automate the retrieval, a shell script is used to construct a request that includes the task execution ID and connect to the YARN component's interface. The request is then sent through the interface to obtain the task information corresponding to the task execution identifier.

[0113] S302: Based on the retrieval request, obtain the execution metadata information of the task to be statistically analyzed.

[0114] Understandably, when using tools in a shell script to send a retrieval request to the receiving endpoint of the YARN component, the response returned by the YARN component is usually a data packet in JavaScript Object Notation (JSON) or Extensible Markup Language (XML) format, which contains the execution metadata information of the task to be analyzed.

[0115] S303: Extract the execution metadata information of the statistical task to obtain resource consumption information.

[0116] Understandably, after parsing the response returned by the YARN component, we need to extract information related to resource consumption. This information may include task ID, memory usage, and disk interface, etc. This information is usually in the form of key-value pairs and can be extracted using text processing tools or JSON / XML parsing tools in shell scripts.

[0117] S304: Save the resource consumption information to the local server and perform format conversion to obtain the first consumption information.

[0118] Understandably, after parsing the resource consumption information, the information is first saved to the local server and then cleaned and converted to save time and resources in subsequent calculations. Saving the data to the local server also ensures data security. In addition, if there is network instability or connection problems, saving the collected data to the local server can prevent data transmission interruption or loss.

[0119] S305: Classify the first consumption information according to the preset category to obtain the second consumption information.

[0120] The preset categories are also the storage structure of the HIVE data warehouse tool.

[0121] Understandably, after storing the data on the local server, it needs to be transferred to a HIVE table for computation. First, a corresponding HIVE table is created. HIVE tables can store and manage data, mapping structured data files to a database table. After categorizing and storing the first consumption information in the HIVE table, the resource consumption information in the table becomes the second consumption information, facilitating subsequent on-demand data retrieval and computation.

[0122] S306: Based on the information category corresponding to the preset requirements, extract the second consumption information and perform arithmetic operations to obtain the consumption statistics results.

[0123] Understandably, different statisticians will have different statistical needs, resulting in different statistical results. Therefore, during calculations, the corresponding information can be queried from the table according to the needs, the retrieved data can be processed, and units of resources can be converted to obtain the statistical results. Preset requirements can be customized by the statisticians, such as resource distribution during the entire day's task execution time, summary of resource consumption at the task granularity, ranking of long-running low-resource-consumption tasks, and ranking of high-resource-consumption tasks.

[0124] For example, statisticians need to calculate the memory usage percentage and total execution time of a task, which can be divided into two parts. Memory usage percentage calculation: Select the item and execution flow from the table. Calculate the total memory usage for each item and execution flow combination and convert it to a percentage relative to the total cluster memory. The total cluster memory is the total memory used by the task being analyzed. Save the results grouped by item and execution flow. Total execution time calculation: Select the item and execution flow from the table. Calculate the total execution time for each item and execution flow, and save the results grouped by item and execution flow. Finally, concatenate the two calculation results based on the item and execution flow, with the concatenation condition being a match between the item name and the execution flow name.

[0125] The above statistics include the name of the scheduled execution flow, task execution duration, and memory usage percentage. Alternatively, execution time can be segmented into intervals to identify peak resource usage periods, or task execution time / resource consumption can be calculated to determine the rationality of task resource allocation.

[0126] Optionally, the consumption statistics results are parsed and the parsed results are stored in a preset file;

[0127] Send the preset file to the statistician, and / or push the preset file to the display tool, which is used to display the specific content of the preset file.

[0128] Understandably, after calculating the consumption statistics, these results are processed and stored in a pre-defined file convenient for statisticians to view. For example, the consumption results can be stored using comma-separated values ​​(CSV). CSV is a simple file format used to store tabular data. That is, the data is stored in plain text, with each line representing a record and each field separated by commas. After storage, the file can be sent to statisticians via email, communication tools, etc. Alternatively, the data in the file can be directly pushed to the statisticians' dashboard tools (e.g., Grafana) in tabular form. The tabular format allows for a more intuitive and clear presentation of the data, making the statistical results more transparent.

[0129] This embodiment provides a method for statistical analysis of resource consumption. The method generates a retrieval request based on a task execution identifier. The retrieval request is used to obtain all information about the task corresponding to the task execution identifier. Based on the retrieval request, the execution metadata information of the task to be analyzed is obtained. The execution metadata information of the task to be analyzed is stripped to obtain resource consumption information. The resource consumption information is saved to a local server and undergoes format conversion to obtain first consumption information. The first consumption information is then categorized according to preset categories to obtain second consumption information. Based on the information category corresponding to preset requirements, the second consumption information is extracted and arithmetic operations are performed to obtain the consumption statistics result. This method obtains the resource consumption information of the task to be tested through a scheduling component and performs calculations on the resource consumption information to obtain statistical results, helping data developers reduce manual calculation workload and improve work efficiency.

[0130] Figure 5 A schematic diagram of the structure of a resource consumption statistics device provided in this application. Figure 5 As shown, this application provides a resource consumption statistics device 500, which includes:

[0131] Module 501 is used to obtain execution metadata information from the offline data warehouse;

[0132] Processing module 502 is used to analyze and extract the execution metadata information to obtain the task execution identifier;

[0133] The retrieval module 503 is used to retrieve the resource consumption information corresponding to the task execution identifier, wherein the resource consumption information is part of the execution metadata information;

[0134] The processing module 502 is also used to perform conversion processing on the resource consumption information according to preset requirements to obtain consumption statistics results.

[0135] Optionally, the device further includes: a determining module 504;

[0136] The determining module 504 is used to determine the collection time range and query statement of the task to be statistically analyzed, wherein the query statement is generated based on the collection time range.

[0137] The determining module 504 is further configured to determine the execution metadata information of multiple tasks in the offline data warehouse according to the query statement, and generate a query result table according to the execution metadata information. The multiple tasks include the task to be counted, and the query result table is used to display the data hierarchy relationship of each task.

[0138] The determining module 504 is further configured to determine the log information of each task based on the minimum level information of the data hierarchy relationship.

[0139] Optionally, the determining module 504 is further configured to determine a task execution identifier template based on the log information and the task to be counted, wherein the task execution identifier template includes a structural pattern of character information and time information of the task to be counted;

[0140] The determining module 504 is further configured to traverse the execution metadata information according to the task execution identifier template and determine the task execution identifier that matches the task execution identifier template.

[0141] Optionally, the apparatus further includes: a generation module 505;

[0142] The generation module 505 is further configured to generate a retrieval request based on the task execution identifier; the retrieval request is used to obtain all information of the task corresponding to the task execution identifier.

[0143] The determining module 504 is further configured to obtain the execution metadata information of the task to be counted based on the retrieval request;

[0144] The processing module 502 is further configured to strip the execution metadata information of the task to be statistically analyzed to obtain the resource consumption information.

[0145] Optionally, the device further includes: a storage module 506;

[0146] The storage module 506 is also used to save the resource consumption information to a local server and perform format conversion processing to obtain the first consumption information;

[0147] The processing module 502 is further configured to classify the first consumption information according to a preset category to obtain the second consumption information;

[0148] The processing module 502 is further configured to extract the second consumption information and perform arithmetic operations according to the information category corresponding to the preset requirements to obtain the consumption statistics result.

[0149] Optionally, the device further includes: a transmitting module 507;

[0150] The processing module 502 is further configured to parse the consumption statistics results and store the parsed results in a preset file;

[0151] The sending module 507 is further configured to send the preset file to the statistician, and / or push the preset file to the display tool, the display tool being used to display the specific content of the preset file.

[0152] The resource consumption statistics device provided in this application embodiment is similar in principle and technical effect to the implementation of each part of the aforementioned resource consumption statistics method, and will not be described again here.

[0153] Figure 6 A schematic diagram of the structure of a resource consumption statistics device provided in this application. Figure 6 As shown, this application provides a resource consumption statistics device 600, which includes: a receiver 601, a transmitter 602, a processor 603, and a memory 604.

[0154] Receiver 601 is used to receive instructions and data;

[0155] Transmitter 602 is used to send commands and data;

[0156] Memory 604 is used to store instructions executed by the computer;

[0157] Processor 603 is configured to execute computer execution instructions stored in memory 604 to implement the various steps of the resource consumption statistics method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the resource consumption statistics method.

[0158] Optionally, the memory 604 can be either standalone or integrated with the processor 603.

[0159] When the memory 604 is set up independently, the electronic device also includes a bus for connecting the memory 604 and the processor 603.

[0160] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0161] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in any of the foregoing embodiments.

[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the foregoing embodiments.

[0163] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application.

[0164] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disc, etc.

[0165] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0166] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0167] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0168] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A statistical method for resource consumption, characterized in that, include: Retrieve execution metadata information from the offline data warehouse; The execution metadata information is analyzed and extracted to obtain the task execution identifier; Retrieve the resource consumption information corresponding to the task execution identifier; wherein, the resource consumption information is the information in the execution metadata information indicating the use and consumption of resources; The resource consumption information is converted and processed according to preset requirements to obtain consumption statistics.

2. The method according to claim 1, characterized in that, The execution metadata information includes: data hierarchy and log information; The process of obtaining execution metadata information from the offline data warehouse includes: Determine the data collection time range and query statement for the task to be statistically analyzed, wherein the query statement is generated based on the data collection time range; Based on the query statement, the execution metadata information of multiple tasks in the offline data warehouse is determined, and a query result table is generated based on the execution metadata information. The multiple tasks include the task to be counted, and the query result table is used to display the data hierarchy relationship of each task. Based on the lowest level information of the data hierarchy, determine the log information for each task.

3. The method according to claim 2, characterized in that, The step of analyzing and extracting the execution metadata information to obtain the task execution identifier includes: Based on the log information and the task to be counted, a task execution identifier template is determined. The task execution identifier template includes a structural pattern of character information and time information of the task to be counted. Based on the task execution identifier template, the execution metadata information is traversed to determine the task execution identifier that matches the task execution identifier template.

4. The method according to claim 1, characterized in that, The step of retrieving the resource consumption information corresponding to the task execution identifier includes: A retrieval request is generated based on the task execution identifier; the retrieval request is used to obtain all information about the task corresponding to the task execution identifier; Based on the retrieval request, the execution metadata information of the task to be statistically analyzed is obtained; The execution metadata information of the task to be statistically analyzed is stripped to obtain the resource consumption information.

5. The method according to claim 1, characterized in that, The step of converting the resource consumption information according to preset requirements to obtain consumption statistics results includes: The resource consumption information is saved to the local server and then converted to a new format to obtain the first consumption information. The first consumption information is classified and processed according to a preset category to obtain the second consumption information; Based on the information category corresponding to the preset requirements, the second consumption information is extracted and arithmetic operations are performed to obtain the consumption statistics result.

6. The method according to claim 1, characterized in that, After obtaining the consumption statistics, the method further includes: The consumption statistics are parsed and the results are stored in a preset file. The preset file is sent to the statistician, and / or the preset file is pushed to the display tool, which is used to display the specific content of the preset file.

7. A resource consumption statistics device, characterized in that, include: The acquisition module is used to retrieve execution metadata information from the offline data warehouse; The processing module is used to analyze and extract the execution metadata information to obtain the task execution identifier; The retrieval module is used to retrieve the resource consumption information corresponding to the task execution identifier, wherein the resource consumption information is a portion of the execution metadata information; The processing module is also used to perform conversion processing on the resource consumption information according to preset requirements to obtain consumption statistics results.

8. The apparatus according to claim 7, characterized in that, The device further includes: a determining module; The determining module is used to determine the collection time range and query statement of the task to be statistically analyzed, wherein the query statement is generated based on the collection time range. The determining module is further configured to determine the execution metadata information of multiple tasks in the offline data warehouse according to the query statement, and generate a query result table according to the execution metadata information. The multiple tasks include the task to be counted, and the query result table is used to display the data hierarchy relationship of each task. The determining module is further configured to determine the log information of each task based on the minimum level information of the data hierarchy.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 6.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 6 through the computer program.

Images