A cloud computing-based cross-platform big data resource optimization method and system

Abstract

This invention discloses a cross-platform big data resource optimization method and system based on cloud computing, relating to the fields of cloud computing and big data technology. It includes: S1, collecting raw indicator data from different cloud service providers, different virtualization technologies, and different big data frameworks, and simultaneously extracting descriptive information corresponding to each raw indicator data. The descriptive information includes the indicator name, resource type, statistical caliber, aggregation method, and collection source. This invention constructs a unified resource indicator semantic library by collecting raw indicators and descriptive information from multiple sources to achieve indicator semantic parsing and mapping, eliminating semantic ambiguity between indicators on different platforms; it compares and synchronously calibrates the time of each monitoring source by marking and calibrating the time base; it unifies the indicator time step through sampling feature normalization processing, and then constructs accurate task-level resource profiles with coverage, consistency, and integrity labels according to task instances and stages.

Description

Technical Field

[0001] This invention relates to the field of cloud computing and big data technology, specifically to a cross-platform big data resource optimization method and system based on cloud computing. Background Technology

[0002] Cloud computing provides elastically scalable computing and storage capabilities for big data processing. Big data tasks can be deployed across different cloud service providers, virtualization technologies, and big data frameworks, enabling cross-platform collaborative operation. To ensure these tasks run efficiently and stably in the cloud environment, operations and maintenance systems typically collect various operational data, including fundamental metrics such as CPU utilization, memory usage, disk read / write speeds, and network traffic, as well as business-side information such as job start time, stage duration, and number of failures. Theoretically, by fully utilizing this operational data, intelligent analysis and prediction of cross-platform big data tasks can be performed, providing a basis for resource allocation, elastic scaling, and fault avoidance, thereby achieving true big data resource optimization.

[0003] A prominent technical problem in existing cloud-based cross-platform big data resource management technologies is the difficulty in forming a unified, accurate, and directly applicable resource profile and time-series view for optimization decisions from cross-platform operational data. Different platforms and big data frameworks use different naming conventions, statistical periods, sampling granularities, and time bases for monitoring indicators. Even the meaning of the same term can differ across systems, making it impossible to directly align and compare the operational curves of the same big data task at different nodes. Current practices often use simple field mapping or coarse aggregation to merge multi-source monitoring data, ignoring the impact of semantic differences and time offsets in indicators. This results in distorted and incomplete resource profiles that fail to accurately reflect changes in resource requirements at each stage of the task. Resource optimization decisions based on this distorted operational data often lack specificity and stability, leading to inaccurate resource scaling thresholds in cross-platform environments and unreasonable timing for resource reservation and reclamation, thus weakening the actual effectiveness of big data resource optimization in cloud computing environments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a cloud computing-based cross-platform big data resource optimization method and system, which solves the problems in existing technologies such as the difficulty in forming a unified and accurate resource profile and time-series view of cross-platform big data operation data, the distortion of profiles caused by differences in indicator semantics and time benchmarks, and the lack of targeted resource optimization decisions.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a cross-platform big data resource optimization method based on cloud computing, comprising: S1. Collect raw indicator data from different cloud service providers, different virtualization technologies and different big data frameworks, and simultaneously extract the descriptive information corresponding to each raw indicator data. The descriptive information includes the indicator name, resource type, statistical scope, aggregation method and collection source. S2. Construct a unified resource indicator semantic library, perform semantic parsing and mapping on the original indicator data. The unified resource indicator semantic library contains standard indicator entries and their associated unified semantic tags and statistical definitions. Semantic parsing and mapping maps each original indicator data to a standard indicator entry based on the description information and keywords in the indicator name, and generates the corresponding numerical conversion strategy. S3. Perform time base calibration on the timestamps of the raw indicator data from different monitoring sources. The time base calibration estimates the time offset of each monitoring source relative to the unified time base by comparing the synchronization marks related to the same task or the same event in each monitoring source, and corrects the timestamps of the raw indicator data according to the time offset. S4. Perform sampling feature normalization on the index sequence after semantic mapping and time calibration to construct multidimensional resource time series data with a unified time step. S5. Based on multi-dimensional resource time-series data, aggregate resource usage according to task instances and task stages to construct a task-level resource profile; S6. Provide task-level resource profiles to the resource optimization decision module as input for formulating elastic scaling strategies, cross-platform load balancing strategies, or resource reservation and reclamation strategies.

[0006] Preferably, the construction of a unified resource indicator semantic library includes: The metrics related to computing resources, storage resources, network resources, and task execution status are abstracted into a set of standard metric items; Each standard indicator entry is associated with a unified semantic label and statistical definition; Initial mapping rules are preset for typical raw indicators of different cloud platforms and big data frameworks. The initial mapping rules specify the correspondence between typical raw indicators and standard indicator items.

[0007] Preferably, the semantic parsing and mapping of the original indicator data includes: Retrieve standard indicator entries from the unified resource indicator semantic library that match the descriptive information of the original indicator data; When multiple matching standard indicator entries exist, they are prioritized according to the data source and keywords in the indicator name, and the best matching standard indicator entry is selected as the target. Based on the differences between the statistical definitions of the target standard indicator items and the statistical caliber of the original indicator data, a linear or non-linear numerical transformation strategy is generated.

[0008] Preferably, the synchronization marker includes the task start log time, the task phase completion log time, the node heartbeat time, or a global event identifier generated by the distributed tracing system.

[0009] Preferably, the step of performing sampling feature normalization on the index sequence after semantic mapping and time calibration includes: Set a uniform target sampling period; For high-frequency index sequences with a sampling period shorter than the target sampling period, a sliding window aggregation method is used to compress them to the target sampling period. For low-frequency index sequences with a sampling period longer than the target sampling period, linear interpolation or compensation based on historical patterns is used to extend them to the target sampling period.

[0010] Preferably, the aggregation function used in the sliding window aggregation method is the mean function, and its calculation formula is as follows: ; in, These are the aggregated index values ​​within the target sampling period. For the first in the sliding window The index values ​​of each original sampling point This represents the number of original sampling points contained within the sliding window.

[0011] Preferably, the calculation formula for the linear interpolation method is as follows: ; in, The time to be interpolated The index value, and They are respectively The index values ​​of two known sampling points before and after time step. and These are the timestamps for the two known sampling points, respectively.

[0012] Preferably, the construction of the task-level resource profile includes: Receive task execution metadata, which includes task instance identifier and task phase division information; Multidimensional resource time-series data belonging to the same task instance identifier are segmented according to task stage information. The multidimensional resource time-series data within each task phase are aggregated to form the resource usage vector for that task phase. Combine the resource usage vectors of all task phases in chronological order to form a task-level resource profile object with a time dimension.

[0013] Preferably, the method further includes: Calculate and label coverage, consistency, and completeness tags for task-level resource profile objects. Coverage tags represent the proportion of task stages covered by the profile, consistency tags represent the similarity of profiles of the same task on different platform nodes, and completeness tags represent the missing data of each resource dimension in the profile.

[0014] This invention also provides a cloud computing-based cross-platform big data resource optimization system, comprising: The cross-platform resource data acquisition module is used to collect raw indicator data from different cloud service providers, different virtualization technologies, and different big data frameworks, and simultaneously extract the descriptive information corresponding to each raw indicator data. The indicator semantic parsing and mapping module is connected to the cross-platform resource data acquisition module and the unified resource indicator semantic library. It is used to perform semantic parsing and mapping on the raw indicator data and output unified indicator data with standard semantic tags. The time base calibration module is connected to the output of the indicator semantic parsing and mapping module. It is used to estimate and correct the time offset of each monitoring source and output a unified indicator sequence on the time axis. The sampling feature normalization module, connected to the time base calibration module, is used to perform sampling feature normalization on the unified index sequence of the time axis and output multi-dimensional resource time series data with a unified time step. The task-level resource profile building module, connected to the sampling feature normalization module, is used to aggregate resource usage according to task instances and task stages to build task-level resource profiles. The resource optimization decision interface module is connected to the task-level resource profile building module and is used to provide task-level resource profiles to the external resource optimization decision module.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention collects original indicators and descriptive information from multiple sources to construct a unified resource indicator semantic library, enabling semantic parsing and mapping of indicators and eliminating semantic ambiguity between different platforms. It compares and calibrates the time of each monitoring source by comparing and synchronizing the markers, thus unifying the time benchmark. It then unifies the time step of indicators through sampling feature normalization processing, and constructs accurate task-level resource profiles with coverage, consistency, and completeness labels according to task instances and stages. These profiles are used as decision inputs to formulate strategies such as elastic scaling, cross-platform load balancing, and resource reservation and reclamation, thereby improving cross-platform resource utilization efficiency and task operation stability, and ensuring the scientific and targeted nature of resource optimization decisions. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the semantic parsing and mapping process for indicators in this invention; Figure 3 This is a flowchart illustrating the process of constructing a task-level resource profile for this invention. Figure 4 This is a system structure diagram of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1

[0019] Please see Figure 1-3 This embodiment provides a cross-platform big data resource optimization method for cloud computing, and provides a detailed description in conjunction with a multi-cloud collaborative big data analysis business scenario. The specific implementation includes the following steps: The first step is S1, which involves collecting raw metric data from different cloud service providers, virtualization technologies, and big data frameworks. Simultaneously, descriptive information is extracted for each raw metric, including the metric name, resource type, statistical definition, aggregation method, and data source. In real-world multi-cloud scenarios, enterprises typically connect to multiple cloud service providers such as Alibaba Cloud, Amazon AWS, and Huawei Cloud, employing different virtualization technologies like KVM, Docker, and container cloud within each platform. Big data processing involves various frameworks such as Hadoop, Spark, and Flink. To achieve cross-platform resource optimization, comprehensive collection of raw metric data from these heterogeneous environments is necessary.

[0020] During the data acquisition phase, a lightweight acquisition agent can be deployed to achieve unified retrieval of data from multiple sources. For Alibaba Cloud, its cloud monitoring API can be called to obtain basic computing, storage, and network metrics such as CPU utilization, memory usage, disk IOPS, and network inbound / outbound bandwidth of ECS instances. For Amazon AWS, relevant resource metrics of EC2 instances are collected through the CloudWatch interface. For Huawei Cloud, metrics are collected using its CES monitoring service. At the virtualization technology level, for KVM virtualization environments, the libvirt tool is used to collect resource usage data of the host machine and virtual machines. For Docker container environments, the Docker API is used to obtain the CPU quota usage, memory limit thresholds, and actual usage of containers. At the big data framework level, for the Hadoop framework, the metadata operation latency of the NameNode and the data block read / write speed of the DataNode are collected. For the Spark framework, the task execution time of the Executor and the amount of Shuffle data are collected. For the Flink framework, the throughput of the TaskManager and the Checkpoint completion time are collected. While collecting raw metric data, it is necessary to simultaneously extract the descriptive information corresponding to each metric. For example, when a "CPU utilization" metric is collected, its metric name in the descriptive information is "CPU utilization", its resource type is "computing resource", its statistical caliber is "average occupancy percentage calculated over a 5-minute period", its aggregation method is "mean aggregation", and its collection source is "Alibaba Cloud ECS Cloud Monitoring". This method can ensure that each raw metric has a clear attribute identifier, laying the foundation for subsequent semantic mapping.

[0021] Next, step S2 is executed, which involves constructing a unified resource indicator semantic library. Semantic parsing and mapping are performed on the raw indicator data. The unified resource indicator semantic library contains standard indicator entries and their associated unified semantic tags and statistical definitions. Semantic parsing and mapping, based on the descriptive information and keywords in the indicator name, maps each raw indicator data to a standard indicator entry and generates a corresponding numerical conversion strategy. When constructing the unified resource indicator semantic library, various resource indicators need to be abstracted and categorized first. Indicators related to computing resources, storage resources, network resources, and task running status are abstracted into a set of standard indicator entries. Standard indicator entries for computing resources may include "CPU core utilization," "actual memory utilization," and "GPU computing power utilization," etc.; standard indicator entries for storage resources may include "disk space utilization," "disk read / write throughput," and "object storage access latency," etc.; standard indicator entries for network resources may include "network bandwidth utilization," "network packet loss rate," and "network transmission latency," etc.; and standard indicator entries for task running status may include "task startup success rate," "task phase execution duration," and "task failure retries," etc.

[0022] After abstracting the standard indicator items, a unified semantic tag and statistical definition are associated with each standard indicator item. For example, the standard indicator item "CPU core utilization" is associated with the semantic tag "computing resources - core computing power utilization", and its statistical definition is "the ratio of the number of CPU cores actually used to the total number of cores within the statistical period, with the statistical period uniformly set at 5 minutes and the aggregation method being average aggregation". The standard indicator item "disk space utilization" is associated with the semantic tag "storage resources - space utilization", and its statistical definition is "the ratio of used disk space to total disk space at the statistical moment, with the statistical period set at 5 minutes and the aggregation method being instantaneous value sampling". Meanwhile, initial mapping rules are preset for typical raw metrics of different cloud platforms and big data frameworks. The initial mapping rules specify the correspondence between typical raw metrics and standard metric entries. For example, the raw metric "CPU utilization" of Alibaba Cloud ECS is preset to correspond to the standard metric "CPU core utilization" in the semantic library. The raw metric "CPU usage" of Amazon AWS EC2 also corresponds to the standard metric. The raw metric "disk utilization" of Hadoop DataNode corresponds to the standard metric "disk space utilization". The initial mapping rules can be stored in the semantic library in the form of configuration files, supporting subsequent dynamic updates and expansions.

[0023] After completing the construction of the unified resource indicator semantic library, semantic parsing and mapping operations are performed on the original indicator data. First, standard indicator entries that match the description information of the original indicator data are retrieved from the unified resource indicator semantic library. This can be achieved through keyword matching algorithms. For example, for an original indicator from a Docker container, the indicator name in its description information is "container CPU quota utilization" and the resource type is "computing resource". Then, by retrieving standard indicator entries of the "computing resource" category in the semantic library, entries containing keywords such as "CPU", "occupancy", and "utilization rate" are selected. When multiple matching standard indicator entries exist, they are prioritized based on the data collection source and keywords in the indicator name. The best-matching standard indicator entry is selected as the target. The priority ranking weight can be set according to the business relevance of the data collection source and the keyword matching degree. For example, for indicators from big data task execution nodes, the priority weight of standard indicators of the "task running status" category is set to 0.6, "computing resources" category to 0.3, and "storage resources" category to 0.1. The keyword matching degree is calculated based on the number of matched keywords and the matching position. Keywords that completely match the indicator name have a higher weight than keywords that match the description information. The comprehensive score of each candidate standard indicator entry is obtained through weighted calculation, and the one with the highest score is selected as the target entry.

[0024] After identifying the target standard metric items, a linear or non-linear numerical transformation strategy is generated based on the differences between the statistical definitions of the target standard metric items and the statistical caliber of the original metric data. For example, if an original metric comes from AWS EC2 and its "CPUUsage" is peak data collected over a 1-minute period, while the target standard metric "CPU core utilization" is defined as a 5-minute average, a numerical transformation strategy needs to be generated to convert the five consecutive 1-minute peak data points into a 5-minute average. If the original metric's statistical unit is "percentage" while the standard metric requires a "decimal" unit, a linear transformation strategy is generated, dividing the original value by 100 to obtain the standard metric value. For some non-linear statistical caliber differences, such as the original metric being a logarithmic resource utilization value while the standard metric is a linear value, a non-linear transformation strategy is generated to convert logarithmic to linear, ensuring that the transformed metric value conforms to the statistical definition of the standard metric. Through this semantic parsing and mapping step, semantic ambiguity of metrics across different platforms and frameworks can be effectively eliminated, achieving standardization of metric data and solving the problem of data misalignment caused by discrepancies in the meanings of metrics across different systems in the background technology.

[0025] The S3 step then proceeds, which involves time-base calibration of the timestamps of the raw metric data from different monitoring sources. Time-base calibration estimates the time offset of each monitoring source relative to a unified time base by comparing synchronization markers related to the same task or event across different monitoring sources, and corrects the timestamps of the raw metric data accordingly. In multi-source monitoring environments, time bases from different monitoring sources often deviate. For example, if the time server of Alibaba Cloud Monitoring is not synchronized with the time server of the enterprise's self-built monitoring system, it can cause a timestamp offset of several seconds or even tens of seconds for the same task's metric data. Without calibration, this can lead to deviations in subsequent time-series data analysis.

[0026] During the time base calibration process, it is first necessary to determine a unified time base. The standard time of the National Time Service Center can be used as the unified time base. Then, the synchronization markers in each monitoring source are screened. Synchronization markers include task start log time, task phase completion log time, node heartbeat time, or global event identifiers generated by the distributed tracing system. For example, a cross-platform big data task will record the start time in the monitoring logs of Alibaba Cloud, AWS, and enterprise private cloud when it starts. This start time is the synchronization marker. The global event trigger time corresponding to the TraceID generated by the distributed tracing system can also be used as a synchronization marker across monitoring sources.

[0027] After obtaining the synchronization marker, the time offset of each monitoring source is estimated by comparing the timestamps of the same synchronization marker in different monitoring sources with the unified time base. For example, if the timestamp of a task startup log in monitoring source A is T1, and the timestamp of the synchronization marker in the unified time base is T0, then the time offset of monitoring source A is Δt = T1 - T0. If Δt is positive, it means that the time of monitoring source A is faster than the unified time base; if it is negative, it means that it is slower than the unified time base. After obtaining the time offset of each monitoring source, the timestamps of all original indicator data within the monitoring source are corrected based on the offset. The correction formula is Tcorrection = TOriginal - Δt. Through this time base calibration step, the time axis of all multi-source indicator data can be unified, solving the problem of incomparable running curves caused by inconsistent time bases of different monitoring sources in the background technology.

[0028] After completing the time base calibration, step S4 is executed, which involves normalizing the sampling features of the indicator sequences after semantic mapping and time calibration to construct multi-dimensional resource time-series data with a unified time step. The sampling periods for indicators from different monitoring sources vary; for example, some high-frequency indicators have a sampling period of 1 minute, while some low-frequency indicators have a sampling period of 10 minutes. Direct time-series analysis would result in data dimension mismatch due to the inconsistent time steps, thus requiring sampling feature normalization.

[0029] First, a unified target sampling period is set. This target sampling period can be determined based on business needs and the sensitivity of the metrics to change. For example, for computing resource metrics with high real-time requirements, the target sampling period can be set to 5 minutes. This value is based on the fact that the execution interval of big data tasks is usually more than 5 minutes. A 5-minute sampling period ensures data real-time performance while avoiding data redundancy caused by overly frequent sampling. For high-frequency metric sequences with sampling periods shorter than the target sampling period, a sliding window aggregation method is used to compress them to the target sampling period. The aggregation function used in the sliding window aggregation method is the mean function, and its calculation formula is as follows: ; in, These are the aggregated index values ​​within the target sampling period. For the first in the sliding window The index values ​​of each original sampling point This represents the number of original sampling points included within the sliding window. For example, if the sampling period for a high-frequency indicator is 1 minute and the target sampling period is 5 minutes, then the size of the sliding window is N=5. Five consecutive 1-minute sampling points are aggregated into one 5-minute indicator value using the mean function. If there are missing sampling points within a certain 5-minute period, the value of the previous sampling point can be used to fill in the gaps. The basis for filling in the gaps is that the value fluctuation of the high-frequency indicator is small within a short period of time, and filling in the previous value will not have a significant impact on the aggregation result.

[0030] For low-frequency index sequences with a sampling period longer than the target sampling period, linear interpolation or historical pattern-based compensation is used to extend them to the target sampling period. The calculation formula for linear interpolation is as follows: ; in, The time to be interpolated The index value, and They are respectively The index values ​​of two known sampling points before and after time step. and These are the timestamps of the two known sampling points. For example, if the sampling period of a low-frequency indicator is 10 minutes and the target sampling period is 5 minutes, the indicator value at time point t0=0 minutes is X0=20%, and the indicator value at time point t1=10 minutes is X1=30%. Then, at time point t=5 minutes, X(5)=20%+(30%-20%) / (10-0)*(5-0)=25% can be obtained by linear interpolation. For indicators with obvious historical change patterns, a compensation method based on historical patterns can be adopted. For example, if the historical data of a certain disk space occupancy rate indicator shows a uniform growth pattern, the indicator value at the interpolation time can be calculated based on the historical growth slope to ensure that the extended indicator sequence conforms to its change pattern. Through sampling feature normalization processing, multi-dimensional resource time series data with unified time steps can be constructed, providing a standardized data foundation for the subsequent construction of task-level resource profiles.

[0031] After obtaining the multidimensional resource time-series data, step S5 is executed. This involves aggregating resource usage based on the multidimensional resource time-series data, according to task instances and task stages, to construct a task-level resource profile. First, task execution metadata needs to be received. This metadata includes the task instance identifier and task stage division information. The task execution metadata dataset is provided by the big data task scheduling system. For example, the instance identifier of a big data analysis task might be "Task-20250520-001," and its task stages might be divided into "Data Acquisition Stage," "Data Cleaning Stage," "Data Modeling Stage," and "Result Output Stage." The start and end times of each stage are recorded by the scheduling system and synchronized to the resource profile construction module.

[0032] After obtaining the task execution metadata, the multidimensional resource time-series data belonging to the same task instance identifier are segmented according to the task stage division information. For example, the CPU, memory, network and other resource time-series data of the "Task-20250520-001" instance in the data collection stage (00:00-00:30) are divided into dedicated data for this stage, and the resource time-series data in the data cleaning stage (00:30-01:10) are divided into corresponding stage data, and so on to complete the data segmentation of each stage. Subsequently, the multidimensional resource time-series data within each task phase are aggregated to form the resource usage vector for that task phase. The aggregation method can be determined according to the indicator type. For computational resource indicators, mean aggregation is used, and for task status indicators, maximum value or cumulative value aggregation is used. For example, the resource usage vector for the data acquisition phase can be represented as [CPU core utilization average 0.65, actual memory utilization average 0.72, network bandwidth utilization average 0.81, cumulative task data acquisition volume 1024GB], where each vector dimension corresponds to the aggregation result of a type of standard indicator.

[0033] Finally, the resource usage vectors for all task stages are combined chronologically to form a task-level resource profile object with a time dimension. For example, the task-level resource profile of "Task-20250520-001" contains resource usage vectors for four stages, arranged in chronological order of data collection, data cleaning, data modeling, and result output. Each vector is associated with the start and end times of its corresponding stage, thus forming a complete profile with both time and resource dimensions. Furthermore, the method includes calculating and labeling the task-level resource profile object with coverage, consistency, and completeness tags. The coverage tag represents the proportion of task stages covered by the profile, the consistency tag represents the similarity of profiles for the same task on different platform nodes, and the completeness tag represents the missing data in each resource dimension of the profile.

[0034] The coverage label is calculated as the ratio of the number of covered task stages to the total number of task stages. For example, if a task has four stages and the profile only covers the first three stages, the coverage label value is 0.75, and the label can be divided into three levels: "High Coverage (≥0.8)", "Medium Coverage (0.5-0.8)", and "Low Coverage (<0.5)". The consistency label is calculated using the cosine similarity algorithm, which calculates the cosine similarity of resource usage vectors for the same task on different platform nodes. The closer the similarity value is to 1, the higher the consistency. The label can be divided into "High Consistency (≥0.9)", "Medium Consistency (0.7-0.9)", and "Low Consistency (<0.7)". The completeness label is calculated as the ratio of the number of resource dimensions without missing data to the total number of resource dimensions. For example, if a profile contains eight resource dimensions and two dimensions have missing data, the completeness label value is 0.75, and the label level is the same as the coverage label. By labeling profiles with these three types of labels, the quality of the profile can be quickly assessed, providing a reference for subsequent optimization decisions.

[0035] Finally, step S6 is executed, which provides the task-level resource profile to the resource optimization decision-making module as input for formulating elastic scaling strategies, cross-platform load balancing strategies, or resource reservation and reclamation strategies. The task-level resource profile accurately reflects the resource requirements of each task instance at different stages. For example, CPU and memory requirements are much higher during the data modeling stage than in other stages, while network bandwidth requirements are highest during the data acquisition stage. Based on this profile, the resource optimization decision-making module can formulate targeted elastic scaling strategies, expanding CPU and memory resources in advance before the data modeling stage and scaling them down promptly after the stage ends. For cross-platform load balancing strategies, tasks can be scheduled to nodes with more stable and consistent resource usage based on the resource usage consistency tags of the same task on different platform nodes. For resource reservation and reclamation strategies, sufficient resources can be reserved for high-priority tasks based on the coverage and integrity tags of the profile, while idle resources can be reclaimed promptly for low-priority tasks with low resource utilization. Through this step, standardized and accurate resource profiles are transformed into effective resource optimization decisions, solving the problem of inaccurate resource allocation caused by decisions based on distorted data in the background technology, and improving the utilization efficiency of cross-platform big data resources and the stability of task operation.

[0036] Example 2

[0037] Please see Figure 4 This embodiment also provides a cloud computing-based cross-platform big data resource optimization system to implement the cloud computing-based cross-platform big data resource optimization method in Embodiment 1. The system includes: The cross-platform resource data acquisition module serves as the data input entry point for the entire system. Its core function is to collect raw indicator data from different cloud service providers, virtualization technologies, and big data frameworks, and simultaneously extract descriptive information corresponding to each raw indicator. During operation, this module initiates data requests to the monitoring interfaces of various cloud service providers, the management interfaces of various virtualization technologies, and the monitoring components of major data frameworks according to preset collection strategies. For example, it periodically calls the Alibaba Cloud Monitoring API, AWS CloudWatch interface, and Huawei Cloud CES interface. Simultaneously, it obtains virtualization layer data through libvirt and Docker API, and big data framework data through Hadoop YARN monitoring, Spark monitoring, and FlinkDashboard. After collecting the raw indicator data, the module's built-in descriptive information extraction component automatically parses the attributes of each indicator, extracting descriptive information such as indicator name, resource type, statistical scope, aggregation method, and collection source. It then associates and stores the raw indicator data with the descriptive information, providing a complete data carrier for subsequent module processing.

[0038] The indicator semantic parsing and mapping module connects to the cross-platform resource data acquisition module and the unified resource indicator semantic library. Its core function is to perform semantic parsing and mapping on the raw indicator data, outputting unified indicator data with standard semantic tags. During runtime, this module first obtains the raw indicator data with associated descriptive information from the cross-platform resource data acquisition module. Then, it retrieves the standard indicator entries, semantic tags, and initial mapping rules from the unified resource indicator semantic library. Through built-in keyword matching and priority sorting algorithms, it completes the matching from raw indicators to standard indicator entries. Simultaneously, it generates a numerical conversion strategy based on statistical differences to standardize the raw indicator values, ultimately outputting standard indicator data with unified semantic tags. This achieves semantic unification of heterogeneous indicators, laying the foundation for subsequent time calibration.

[0039] The time base calibration module, connected to the output of the indicator semantic parsing and mapping module, has the core function of estimating and correcting the time offset of each monitoring source, and outputting a unified indicator sequence with a consistent timeline. During operation, this module first acquires the standard indicator data output by the indicator semantic parsing and mapping module. Simultaneously, it filters synchronization markers such as task start log time and task phase completion log time from each monitoring source. The timestamps of these synchronization markers are compared with the unified time base provided by the National Time Service Center to calculate the time offset of each monitoring source. Then, based on this offset, the timestamps of all standard indicator data under each monitoring source are corrected to ensure that the timeline of all data remains consistent, eliminating time discrepancies between multi-source data.

[0040] The sampling feature normalization module is connected to the time base calibration module. Its core function is to perform sampling feature normalization on the time-axis unified indicator sequence, outputting multi-dimensional resource time-series data with a unified time step. During operation, this module first reads the preset unified target sampling period, then classifies the time-calibrated indicator sequence. For high-frequency indicators with sampling periods shorter than the target period, the mean aggregation function is called for sliding window compression. For low-frequency indicators with sampling periods longer than the target period, linear interpolation or historical pattern compensation is used for expansion. Finally, the indicator sequences with different sampling periods are converted into multi-dimensional resource time-series data with a unified time step, providing standardized time-series data for resource profiling.

[0041] The task-level resource profiling module is connected to the sampling feature normalization module. Its core function is to aggregate resource usage based on task instances and task stages to construct task-level resource profiles. During runtime, this module first receives task execution metadata from the big data task scheduling system. Then, it segments and aggregates the normalized multi-dimensional resource time-series data according to task instance identifiers and stage division information, generating resource usage vectors for each stage. These vectors are then combined chronologically to form task-level resource profile objects. Simultaneously, it calculates and labels coverage, consistency, and completeness, completing the entire process of resource profile construction.

[0042] The resource optimization decision interface module connects to the task-level resource profiling module. Its core function is to provide task-level resource profiles to the external resource optimization decision module. During runtime, this module standardizes and encapsulates the completed task-level resource profiles and tag information, and synchronizes them to the external resource optimization decision module in real time via a RESTful API or message queue. This provides accurate data input for formulating strategies such as elastic scaling, load balancing, and resource reservation and reclamation, achieving efficient integration between system modules and the decision module, and ensuring the scientific and targeted nature of resource optimization decisions.

[0043] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cross-platform big data resource optimization method based on cloud computing, characterized in that, include: S1. Collect raw indicator data from different cloud service providers, different virtualization technologies and different big data frameworks, and simultaneously extract the descriptive information corresponding to each raw indicator data. The descriptive information includes the indicator name, resource type, statistical scope, aggregation method and collection source. S2. Construct a unified resource indicator semantic library, perform semantic parsing and mapping on the original indicator data. The unified resource indicator semantic library contains standard indicator entries and their associated unified semantic tags and statistical definitions. Semantic parsing and mapping maps each original indicator data to a standard indicator entry based on the description information and keywords in the indicator name, and generates the corresponding numerical conversion strategy. S3. Perform time base calibration on the timestamps of the raw indicator data from different monitoring sources. The time base calibration estimates the time offset of each monitoring source relative to the unified time base by comparing the synchronization marks related to the same task or the same event in each monitoring source, and corrects the timestamps of the raw indicator data according to the time offset. S4. Perform sampling feature normalization on the index sequence after semantic mapping and time calibration to construct multidimensional resource time series data with a unified time step. S5. Based on multi-dimensional resource time-series data, aggregate resource usage according to task instances and task stages to construct a task-level resource profile; S6. Provide task-level resource profiles to the resource optimization decision module as input for formulating elastic scaling strategies, cross-platform load balancing strategies, or resource reservation and reclamation strategies.

2. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The construction of a unified resource indicator semantic library includes: The metrics related to computing resources, storage resources, network resources, and task execution status are abstracted into a set of standard metric items; Each standard indicator entry is associated with a unified semantic label and statistical definition; Initial mapping rules are preset for typical raw indicators of different cloud platforms and big data frameworks. The initial mapping rules specify the correspondence between typical raw indicators and standard indicator items.

3. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The semantic parsing and mapping of the original indicator data includes: Retrieve standard indicator entries from the unified resource indicator semantic library that match the descriptive information of the original indicator data; When multiple matching standard indicator entries exist, they are prioritized according to the data source and keywords in the indicator name, and the best matching standard indicator entry is selected as the target. Based on the differences between the statistical definitions of the target standard indicator items and the statistical caliber of the original indicator data, a linear or non-linear numerical transformation strategy is generated.

4. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The synchronization markers include task start log time, task phase completion log time, node heartbeat time, or global event identifiers generated by the distributed tracing system.

5. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The step of performing sampled feature normalization on the index sequence after semantic mapping and time calibration includes: Set a uniform target sampling period; For high-frequency index sequences with a sampling period shorter than the target sampling period, a sliding window aggregation method is used to compress them to the target sampling period. For low-frequency index sequences with a sampling period longer than the target sampling period, linear interpolation or compensation based on historical patterns is used to extend them to the target sampling period.

6. The cross-platform big data resource optimization method based on cloud computing according to claim 5, characterized in that, The aggregation function used in the sliding window aggregation method is the mean function, and its calculation formula is as follows: ； in, These are the aggregated index values ​​within the target sampling period. For the first in the sliding window The index values ​​of each original sampling point This represents the number of original sampling points contained within the sliding window.

7. The method for optimizing cross-platform big data resources based on cloud computing according to claim 5, characterized in that, The calculation formula for the linear interpolation method is as follows: ； in, The time to be interpolated The index value, and They are respectively The index values ​​of two known sampling points before and after time step. and These are the timestamps for the two known sampling points, respectively.

8. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The construction of the task-level resource profile includes: Receive task execution metadata, which includes task instance identifier and task phase division information; Multidimensional resource time-series data belonging to the same task instance identifier are segmented according to task stage information. The multidimensional resource time-series data within each task phase are aggregated to form the resource usage vector for that task phase. Combine the resource usage vectors of all task phases in chronological order to form a task-level resource profile object with a time dimension.

9. The method for optimizing cross-platform big data resources based on cloud computing according to claim 1, characterized in that, The method further includes: Calculate and label coverage, consistency, and completeness tags for task-level resource profile objects. Coverage tags represent the proportion of task stages covered by the profile, consistency tags represent the similarity of profiles of the same task on different platform nodes, and completeness tags represent the missing data of each resource dimension in the profile.

10. A cloud computing-based cross-platform big data resource optimization system, applied to the cloud computing-based cross-platform big data resource optimization method according to any one of claims 1-9, characterized in that, include: The cross-platform resource data acquisition module is used to collect raw indicator data from different cloud service providers, different virtualization technologies, and different big data frameworks, and simultaneously extract the descriptive information corresponding to each raw indicator data. The indicator semantic parsing and mapping module is connected to the cross-platform resource data acquisition module and the unified resource indicator semantic library. It is used to perform semantic parsing and mapping on the raw indicator data and output unified indicator data with standard semantic tags. The time base calibration module is connected to the output of the indicator semantic parsing and mapping module. It is used to estimate and correct the time offset of each monitoring source and output a unified indicator sequence on the time axis. The sampling feature normalization module, connected to the time base calibration module, is used to perform sampling feature normalization on the unified index sequence of the time axis and output multi-dimensional resource time series data with a unified time step. The task-level resource profile building module, connected to the sampling feature normalization module, is used to aggregate resource usage according to task instances and task stages to build task-level resource profiles. The resource optimization decision interface module is connected to the task-level resource profile building module and is used to provide task-level resource profiles to the external resource optimization decision module.

Images