Big data-based computing power resource dynamic analysis system and method

Abstract

The invention discloses a computing power resource dynamic analysis system and method based on big data, and relates to the technical field of data analys.The method comprises the steps that operation records in a training period are collected, user scores of the operation records are calculated, and user levels corresponding to the operation records are determined based on the range of the user levels; calculating a task score according to a task scoring model corresponding to the user level, calculating a predicted value of a platform index through a platform index prediction model, and generating a regulation and control strategy model in combination with the task score, the platform index and a platform regulation and control strategy; newly uploaded task content is collected in real time, predicted values of real-time task scores and platform indexes are calculated according to a task scoring model and a platform index prediction model, a real-time platform regulation and control strategy is obtained through generation of a regulation and control strategy model and calculation, the platform is adjusted, and the computing power resource utilization rate and the task execution efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis, and particularly relates to a computing power resource dynamic analysis system and method based on big data. BACKGROUND

[0002] With the rapid development of big data computing technology, the number of digital infrastructure services based on big data and cloud computing is increasing. Such services realize unified management and efficient scheduling of computing power resources by integrating multi-source computing resources and combining user computing power demand. The existing computing power resource management methods mainly include resource allocation based on artificial experience and task scheduling based on fixed strategy. Artificial scheduling is difficult to realize accurate allocation in scenarios with large fluctuations in computing power demand, which may cause part of GPU resources to be idle or tasks to be congested, and the overall computing power utilization rate is low. The scheduling mechanism based on fixed rules lacks comprehensive analysis of historical operation data and real-time load information, and is difficult to adapt to complex environments with multi-type task parallel execution and dynamic load changes. Therefore, how to based on big data technology and intelligent algorithm, deep mining of computing power resource use rule and accurate prediction of future trend, and on this basis combining real-time task load, priority and GPU running state, constructing intelligent scheduling mechanism that can be dynamically adjusted, become the technical problem needed to be solved in the field of computing power management and optimization, so as to realize accurate prediction of computing power demand and adaptive allocation of GPU tasks, and significantly improve the computing power resource utilization rate and task execution efficiency. SUMMARY

[0003] The present application aims to provide a computing power resource dynamic analysis system and method based on big data to solve the problems in the prior art.

[0004] In order to solve the above technical problems, the present application provides the following technical scheme: a computing power resource dynamic analysis method based on big data, the method comprising: Step S100: extracting user parameters, calculating user scores, determining the range of each user level, and summarizing the historical operation records of each user level through the historical operation records of the computing power infrastructure service platform; Step S200: analyzing the historical operation records in the historical operation record set of each user level, and respectively constructing a task score model and a platform index prediction model; Step S300: Set a training period, collect running records within the training period, calculate the user score of the running records, determine the user level corresponding to the running records based on the range of user levels, calculate the task score according to the task scoring model corresponding to the user level, calculate the predicted value of the platform index through the platform index prediction model, and train and generate a control strategy model by combining the task score, platform index and platform control strategy. Step S400: Collect newly uploaded task content in real time, calculate the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculate the real-time platform control strategy by generating a control strategy model, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform.

[0005] Furthermore, step S100 includes: Step S101: In the computing infrastructure service platform, an environmental monitoring device is deployed in the environment where the platform device is located and communicates with the platform device to monitor the environmental parameters of the platform device during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the operating parameters of the platform device when the task starts to execute, associate the environmental parameters, task parameters and operating parameters according to the time series alignment method, generate a time-tagged running data group, record the execution time of the task, and statistically summarize all running data groups within the time range to generate an operation record, and upload the operation record to the computing infrastructure service platform; Step S102: Obtain user information from historical operation records, extract user parameters, including company size, industry type, and job level, assign values ​​to user parameters, preset the weight of each user parameter, and calculate the user score by weighted summation of user parameters. Step S103: Calculate all user ratings corresponding to historical operation records, calculate the variance of user ratings as a, preset b user levels, and generate the range of each user level as: {[A1, A1+1×a), [A1+1×a, A1+2×a), ..., [A1+(b-1)×a, A2]}, where A1 and A2 represent the minimum and maximum values ​​of user ratings, respectively; Step S104: Based on the range of each user level, determine the user level in the historical operation records, filter the historical operation records of the same user level, and generate a set of historical operation records for each user level; By deploying environmental monitoring equipment in the computing infrastructure service platform, environmental parameters during the operation of the platform equipment can be collected in real time and correlated with task parameters and operation parameters over time to generate operation data groups with time tags, thereby realizing full-cycle and full-element recording of operation data and improving data integrity and accuracy. By using time series correlation to statistically summarize all operational data within the task execution time range, it can effectively reflect the changes in the operating status of computing equipment during task execution, which facilitates subsequent performance analysis and fault tracing, and improves the level of intelligent operation and maintenance management. By extracting and parameterizing user information from historical operation records, and then performing weighted summation based on preset weights, user scores are obtained. This achieves a quantitative representation of user behavior and operational data, providing a quantifiable basis for user hierarchical management and service optimization. By calculating the variance of user ratings and dividing them into multiple user level ranges, historical operation records can be classified and aggregated according to user level to form a set of historical operation records for different user levels. This enables user-level statistics and group analysis, which helps the platform provide differentiated computing power scheduling and resource allocation strategies for different types of users.

[0006] Furthermore, step S200 includes: Step S201: Collect task parameters from the historical operation records of a certain user level, analyze the task parameters, and construct a task scoring model. Step S202: In the historical operation record set of a certain user level, collect the operation parameters and environmental parameters in the historical operation records, analyze the operation parameters and environmental parameters, and build a platform indicator prediction model; Furthermore, step S201 includes: Step S201-1: Obtain task content, extract the main content of the task content through natural language processing, calculate the similarity of the main content between each historical running record, preset a similarity threshold, and summarize the historical running records that exceed the similarity threshold to obtain a set of historical running records with the same main content. Step S201-2: In the historical running record set of a certain main content, collect task parameters, including task type and data scale, and assign values ​​to the task parameters to obtain the execution time corresponding to the historical running record, and normalize the execution time. Step S201-3: Combine each task parameter and the normalized execution time to generate a task training set. Use the value of each task parameter as input and the normalized execution time as output. Use a linear regression model to train the model and generate task rating models for different user levels when facing different main content. By extracting the main content of the task content through natural language processing methods and clustering and summarizing historical operation records based on similarity calculation, it can automatically identify the semantic relationship between different tasks and group tasks with similar content into the same category, thereby realizing the structured management of task-level data and avoiding the inefficient classification method based on keyword matching in the traditional way. In the historical operation record set with the same main content, uniformly collecting task parameters and normalizing the execution time can effectively eliminate the interference caused by differences in task scale and execution environment, making the subsequent model training data more standardized and more comparable, and improving the modeling accuracy and stability. By combining task parameters with normalized execution time and training with a linear regression model, it is possible to generate scoring models for different user levels under different main tasks, thereby enabling quantitative analysis of computing power utilization efficiency and task completion performance of users at different levels, and supporting the platform to carry out differentiated resource scheduling and performance optimization. By introducing natural language processing and linear regression modeling techniques, the computing infrastructure platform is equipped with the ability to automatically learn task characteristics and user behavior patterns from historical operational data, thereby enabling task feature mining, performance trend analysis, and adaptive optimization, and improving the system's intelligent management level.

[0007] Furthermore, step S202 includes: Step S202-1: Summarize the task scores and calculate the variance of the task scores as c. Preset k task levels and generate the range of each task level as: {[B1, B1+1×c), [B1+1×c, B1+2×c), ..., [B1+(k-1)×c, B2]}, where B1 and B2 represent the minimum and maximum values ​​of the task scores, respectively. Step S202-2: Filter historical operation records of the same task level, preset time windows, divide the historical operation records according to the time windows, calculate the throughput, average latency and failure rate in each time window and perform normalization calculation, preset the weights of throughput, average latency and failure rate, and perform weighted summation of the normalized throughput, average latency and failure rate to calculate the platform performance index of each time window. Step S202-3: Obtain the running parameters and environmental parameters for each time window in the historical running records and perform normalization calculations. Establish a mapping relationship between the normalized running parameters and environmental parameters within the same time window and the platform performance indicators. Use the Pearson correlation coefficient method to calculate the correlation between each running parameter and environmental parameter and the platform performance indicators. Set a preset correlation threshold and mark the running parameters and environmental parameters that exceed the threshold as highly correlated running parameters and highly correlated environmental parameters, respectively. Step S202-4: Count the number of times each operating parameter and environmental parameter is marked as a highly correlated operating parameter and a highly correlated environmental parameter in the entire time window, calculate the occurrence frequency of each operating parameter and environmental parameter, preset an occurrence frequency threshold, and mark the operating parameters and environmental parameters that exceed the occurrence frequency threshold as characteristic operating parameters and characteristic environmental parameters, respectively. Step S202-5: Using the normalized feature running parameters and feature environment parameters as inputs, and the platform performance index of the next time window as output, train the random forest model to generate a platform index prediction model corresponding to each task level. By calculating the variance of task scores and classifying task levels, refined hierarchical management of tasks is achieved. Based on the range of different task levels, historical operation records are filtered and classified, so that task data is reasonably layered, providing a clear task distribution map for subsequent performance analysis and resource allocation. By normalizing and weighting the platform performance metrics such as throughput, average latency, and failure rate for different task levels within a time window, the platform's performance under different task levels can be comprehensively quantified, providing a more accurate basis for dynamic scheduling and resource optimization, and improving the platform's resource utilization efficiency and service quality. By analyzing the correlation between operating parameters, environmental parameters, and platform performance indicators using the Pearson correlation coefficient method, we can identify operating and environmental parameters that are significantly related to platform performance, providing a basis for subsequent optimization decisions. This enables the platform to adaptively adjust under different task execution environments, ensuring stable system operation. By statistically analyzing the frequency of occurrence and threshold filtering of operating and environmental parameters, we can effectively identify characteristic parameters that have a high impact on platform performance during multiple task executions. This provides a precise basis for subsequent task scheduling and resource allocation, and lays the foundation for long-term optimization of platform performance. By normalizing the characteristic operating parameters and environmental parameters and training them with a random forest model, a platform performance prediction model corresponding to each task level can be generated. This model can predict the performance of future tasks based on historical data, providing the platform with powerful intelligent scheduling capabilities, reducing resource waste, and improving overall operating efficiency. By introducing a performance prediction model based on machine learning, the platform can automatically learn task characteristics and performance patterns from historical operational data, and adaptively optimize resource scheduling and performance configuration according to task type and operating environment, thereby achieving continuous self-optimization and enhancing the platform's intelligent management capabilities.

[0008] Furthermore, step S300 includes: Step S301: Set a training period of several consecutive days, collect the running records within the training period, extract the user information from the running records, calculate the user score, and determine the user level corresponding to the running record based on the user level range. Step S302: Obtain the main content of the task content in the running record, match the task scoring model of the user level in the face of the main content, extract the task parameters of the running record, and input the normalized task parameters into the task scoring model to calculate the task score; Step S303: Based on the range of task levels, determine the task level of the running record, obtain the platform indicator prediction model corresponding to the task level, collect the characteristic running parameters and characteristic environment parameters in the time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S304: Obtain the platform control strategy corresponding to each time window. Take the task score, platform index and predicted value of platform index of the current time window as input, and control strategy of the next time window as output. Use the strategy gradient model for training, and perform iterative updates of the strategy by setting the optimization target of platform performance index to generate the control strategy model. By collecting running records and calculating user scores within a preset training period, combined with a user level classification mechanism, the running characteristics and behavior patterns of users at different levels can be accurately identified. This enables the system to perform differentiated processing based on user characteristics during subsequent task execution and performance prediction, thereby improving the personalization and accuracy of the model. By matching the task scoring model corresponding to the user level and inputting the task parameters into the model to calculate the task score, the system can automatically complete the performance evaluation of different tasks without human intervention, effectively improving the real-time and objectivity of task scoring and providing a reliable quantitative basis for platform task scheduling and performance optimization. By utilizing platform metric prediction models corresponding to task levels, performance metrics for future time windows can be predicted, enabling early detection of platform operation trends and potential performance fluctuations. This provides forward-looking support for proactive system scheduling and resource optimization, significantly improving platform stability and resource utilization efficiency. By adopting a strategy gradient model, the current task score, platform metrics, and future performance predictions are used as inputs to drive the iterative update of the optimization target. This enables the platform control strategy to be continuously optimized under dynamically changing task loads and operating environments, giving the system self-learning and adaptive control capabilities, thereby achieving continuous optimization of performance targets. Through the interconnected process of user rating modeling, task performance evaluation, platform metric prediction, and strategy iteration optimization, a closed-loop self-optimization system covering users, tasks, and the platform is formed. The system continuously updates model parameters during the training cycle, achieving dynamic feedback and continuous improvement from data collection to strategy application, and significantly improving the platform's intelligence level and overall operating efficiency. By automating the data collection, modeling, and strategy optimization processes, intelligent management of the platform's operational status throughout the entire process is achieved. This reduces the complexity and uncertainty of traditional manual regulation, thereby enabling the platform to operate efficiently, stably, and intelligently.

[0009] Furthermore, step S400 includes: Step S401: Collect newly uploaded task content in the computing infrastructure service platform in real time, obtain user information of the user who uploaded the task content, calculate the real-time user score, and determine the real-time user level based on the user level range; Step S402: Extract the real-time main content, match the real user level to obtain the task scoring model in the face of the real-time main content, extract the real-time task parameters, and input the normalized real-time task parameters into the task scoring model to calculate the real-time task score. Step S403: Based on the range of task levels, determine the real-time task level, obtain the platform indicator prediction model corresponding to the real-time task level, collect the characteristic running parameters and characteristic environment parameters in the current time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S404: Input the predicted values ​​of the real-time task score, real-time platform index, and platform index for the next time window into the control strategy model to generate a real-time platform control strategy, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform; By acquiring platform metric prediction models corresponding to task levels in real time and combining them with the operating parameters and environmental parameters of the current time window, predictions can be made in advance of future performance fluctuations, providing a forward-looking basis for platform scheduling and resource allocation. The real-time prediction mechanism enhances the platform's emergency response capabilities and stability, avoiding potential resource bottlenecks and performance degradation. By inputting real-time task scores, real-time platform metrics, and future time window predictions into the control strategy model, accurate real-time control strategies can be generated. This allows for real-time adjustments to the platform's resource allocation and task scheduling, effectively avoiding resource waste and performance bottlenecks, and ensuring that the platform maintains optimal operating conditions under constantly changing loads and environments. By introducing real-time data streams and intelligent generation of dynamic control strategies, this invention enables the platform to adaptively adjust scheduling strategies according to real-time task requirements, user levels, and operating environments, greatly improving the system's intelligence level, reducing manual intervention, enhancing the system's automated management capabilities, and ensuring continuous optimization of the platform under varying workloads. By collecting, analyzing, and predicting the platform's operating status in real time, the allocation of platform resources can be precisely adjusted to ensure the efficient execution of different tasks. The task scheduling and resource allocation mechanism not only improves the overall efficiency of the platform but also avoids problems such as over-allocation or insufficient resources, thereby greatly enhancing the user experience and platform stability.

[0010] To better implement the above methods, a dynamic analysis system for computing resources based on big data is also proposed. The system includes a user level module, a training model module, a control strategy model module, and a real-time control module. User Level Module: Extracts user parameters from the historical operation records of the computing infrastructure service platform, calculates user scores, determines the range of each user level, and summarizes the historical operation records of each user level; Training Model Module: Analyze the historical operation records in the set of historical operation records for each user level, and build task scoring models and platform indicator prediction models respectively; Regulation strategy model module: Set a training period, collect running records within the training period, calculate the user score of the running records, determine the user level corresponding to the running records based on the range of user levels, calculate the task score according to the task score model corresponding to the user level, calculate the predicted value of the platform index through the platform index prediction model, and train and generate the regulation strategy model by combining the task score, platform index and platform regulation strategy. Real-time control module: Collects newly uploaded task content in real time, calculates the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculates the real-time platform control strategy by generating a control strategy model, and uploads the real-time platform control strategy to the computing infrastructure service platform for platform adjustment.

[0011] Furthermore, the user level module includes a unit for generating operation records and a unit for determining user levels: The runtime record generation unit: In the computing infrastructure service platform, environmental monitoring equipment is deployed in the environment where the platform equipment is located and communicates with the platform equipment to monitor the environmental parameters of the platform equipment during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the running parameters of the platform equipment when the task starts to execute, and associate the environmental parameters, task parameters and running parameters according to the time series alignment method to generate a time-tagged running data group, record the execution time of the task, and statistically summarize all running data groups within the time range to generate a runtime record; Determine user level units: Obtain user information from historical operation records, extract user parameters, including company size, industry type, and job level, assign values ​​to user parameters, preset the weight of each user parameter, perform weighted summation of user parameters to calculate user scores, count all user scores corresponding to historical operation records, calculate the variance of user scores as 'a', preset 'b' user levels, and generate the range of each user level.

[0012] Furthermore, the training model module includes a task scoring model unit and a platform metric prediction model unit: Task scoring model unit: Collect task parameters from the historical execution records of a certain user level, analyze the task parameters, and construct a task scoring model; Platform metric prediction model unit: Collect operating parameters and environmental parameters from the historical operation record set of a certain user level, analyze the operating parameters and environmental parameters, and construct a platform metric prediction model.

[0013] Compared with the prior art, the beneficial effects of the present invention are: by constructing a task scoring model and a platform indicator prediction model, and combining real-time collected task and operation data, the platform control strategy is dynamically generated, realizing intelligent allocation and dynamic optimization of computing resources, avoiding resource idleness or overload, and significantly improving the overall utilization and operating efficiency of the computing platform. By analyzing user behavior and calculating user ratings from historical operation records, users are divided into different levels, and differentiated models are built for users of different levels, thereby achieving personalized and hierarchical control of computing resources and improving the service experience of high-value users. By comprehensively considering three types of data—environmental parameters, task parameters, and operational parameters—and through time series alignment and normalization, a multi-dimensional data association model is established to improve the accuracy of task scoring and platform performance indicator prediction. By evaluating task complexity and resource requirements through a task scoring model and then combining it with a platform indicator prediction model to assess future performance changes, the system can predict bottlenecks and adjust resources in advance before task execution, thus realizing a closed-loop control system from data collection to modeling and analysis, strategy optimization, and real-time regulation. By predicting platform performance indicators and formulating control strategies in advance, the system can schedule resources and adjust parameters before task execution, reducing execution latency and failure rate, and ensuring stable operation of the platform under high load conditions. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a dynamic analysis method for computing resources based on big data according to the present invention. Figure 2 This is a schematic diagram of the structure of a big data-based dynamic analysis system for computing resources according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] Please see Figure 1 and Figure 2 This invention provides a technical solution: a method for dynamic analysis of computing resources based on big data, the method comprising: Step S100: Extract user parameters from the historical operation records of the computing infrastructure service platform, calculate user scores, determine the range of each user level, and summarize the historical operation records of each user level; Step S100 includes: Step S101: In the computing infrastructure service platform, an environmental monitoring device is deployed in the environment where the platform device is located and communicates with the platform device to monitor the environmental parameters of the platform device during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the operating parameters of the platform device when the task starts to execute, associate the environmental parameters, task parameters and operating parameters according to the time series alignment method, generate a time-tagged running data group, record the execution time of the task, and statistically summarize all running data groups within the time range to generate an operation record, and upload the operation record to the computing infrastructure service platform; Step S102: Obtain user information from historical operation records, extract user parameters, including company size, industry type, and job level, assign values ​​to user parameters, preset the weight of each user parameter, and calculate the user score by weighted summation of user parameters. Step S103: Calculate all user ratings corresponding to historical operation records, calculate the variance of user ratings as a, preset b user levels, and generate the range of each user level as: {[A1, A1+1×a), [A1+1×a, A1+2×a), ..., [A1+(b-1)×a, A2]}, where A1 and A2 represent the minimum and maximum values ​​of user ratings, respectively; Step S104: Based on the range of each user level, determine the user level in the historical operation records, filter the historical operation records of the same user level, and generate a set of historical operation records for each user level; For example, a sample of user information in historical operation records: User U1001 is a medium-sized enterprise in the smart manufacturing industry and holds the position of technical manager. User U1002 works for a large enterprise, is in the fintech industry, and holds the position of R&D Director. User U1003's company is a small business, the industry type is education, and the job level is engineer; User U1004's company is a medium-sized enterprise, the industry type is medical AI, and the job level is technical director; Assignment and weight setting: Example of company size values: Small = 1, Medium = 2, Large = 3, with a weight of 0.4; Example of values ​​corresponding to industry type: Manufacturing = 2, Finance = 3, Education = 1, Healthcare = 2, with a weight of 0.3; Example of values ​​corresponding to job levels: Engineer = 1, Manager = 2, Director = 3, with a weight of 0.3; The calculated user rating for user U1001 is 2; User U1002's user rating is 3; User U1003's user rating is 1; User U1004's user rating is 2.3.

[0017] Step S200: Analyze the historical operation records in the historical operation record set for each user level, and construct a task scoring model and a platform indicator prediction model respectively; Step S200 includes: Step S201: Collect task parameters from the historical operation records of a certain user level, analyze the task parameters, and construct a task scoring model. Step S201 includes: Step S201-1: Obtain task content, extract the main content of the task content through natural language processing, calculate the similarity of the main content between each historical running record, preset a similarity threshold, and summarize the historical running records that exceed the similarity threshold to obtain a set of historical running records with the same main content. Step S201-2: In the historical running record set of a certain main content, collect task parameters, including task type and data scale, and assign values ​​to the task parameters to obtain the execution time corresponding to the historical running record, and normalize the execution time. Step S201-3: Combine each task parameter and the normalized execution time to generate a task training set. Use the value of each task parameter as input and the normalized execution time as output, and train the linear regression model to generate task rating models for different user levels when facing different main content.

[0018] Step S202: In the historical operation record set of a certain user level, collect the operation parameters and environmental parameters in the historical operation records, analyze the operation parameters and environmental parameters, and build a platform indicator prediction model; Step S202 includes: Step S202-1: Summarize the task scores and calculate the variance of the task scores as c. Preset k task levels and generate the range of each task level as: {[B1, B1+1×c), [B1+1×c, B1+2×c), ..., [B1+(k-1)×c, B2]}, where B1 and B2 represent the minimum and maximum values ​​of the task scores, respectively. Step S202-2: Filter historical operation records of the same task level, preset time windows, divide the historical operation records according to the time windows, calculate the throughput, average latency and failure rate in each time window and perform normalization calculation, preset the weights of throughput, average latency and failure rate, and perform weighted summation of the normalized throughput, average latency and failure rate to calculate the platform performance index of each time window. Step S202-3: Obtain the running parameters and environmental parameters for each time window in the historical running records and perform normalization calculations. Establish a mapping relationship between the normalized running parameters and environmental parameters within the same time window and the platform performance indicators. Use the Pearson correlation coefficient method to calculate the correlation between each running parameter and environmental parameter and the platform performance indicators. Set a preset correlation threshold and mark the running parameters and environmental parameters that exceed the threshold as highly correlated running parameters and highly correlated environmental parameters, respectively. Step S202-4: Count the number of times each operating parameter and environmental parameter is marked as a highly correlated operating parameter and a highly correlated environmental parameter in the entire time window, calculate the occurrence frequency of each operating parameter and environmental parameter, preset an occurrence frequency threshold, and mark the operating parameters and environmental parameters that exceed the occurrence frequency threshold as characteristic operating parameters and characteristic environmental parameters, respectively. Step S202-5: Using the normalized feature running parameters and feature environment parameters as inputs, and the platform performance index of the next time window as output, train the random forest model to generate a platform index prediction model corresponding to each task level. For example, the system collects platform operation data every 30 minutes as a time window. The throughput for time window W1 is 80, the average latency is 4.2, and the failure rate is 2.0. The throughput of time window W2 is 100, the average latency is 3.8, and the failure rate is 1.5. The throughput of time window W3 is 120, the average latency is 5.0, and the failure rate is 3.5. The throughput for time window W4 was 110, the average latency was 4.0, and the failure rate was 1.0. Computing platform performance metrics: The performance metric for the W1 platform during the time window is 0.36. The performance metric for the W2 platform during the time window is 0.74; The performance metric for the W3 platform during the time window is 0.50. The performance metric for the W4 platform during the time window is 0.87; Suppose we collect the following parameters in each time window: During time window W1, CPU utilization was 60%, memory usage was 8%, network bandwidth was 100%, and temperature was 45%. During time window W2, CPU utilization was 70%, memory usage was 10%, network bandwidth was 120%, and temperature was 47°C. During time window W3, CPU utilization was 85%, memory usage was 12%, network bandwidth was 130%, and temperature was 55%. During time window W4, CPU utilization was 75%, memory usage was 11%, network bandwidth was 125%, and temperature was 50%. Calculate the Pearson correlation coefficient: CPU utilization was 0.82; Memory usage is 0.78; The network bandwidth is 0.65; The temperature is 0.72. If the correlation threshold is set to 0.7, then highly correlated parameters include CPU utilization, memory usage, and temperature. Assume the calculation is repeated across 10 time windows: CPU utilization was marked as high with 9 correlations and a frequency of 0.9. The memory usage was marked as high with 8 associations and a frequency of 0.8. The network bandwidth was marked as high with 4 occurrences and a frequency of 0.4. Temperature was marked as highly correlated 7 times and occurred 0.7 times. The frequency threshold is set to 0.6, and the characteristic operating parameters are: CPU utilization and memory usage; the characteristic environmental parameter is: temperature.

[0019] Step S300: Set a training period, collect running records within the training period, calculate the user score of the running records, determine the user level corresponding to the running records based on the range of user levels, calculate the task score according to the task scoring model corresponding to the user level, calculate the predicted value of the platform index through the platform index prediction model, and train and generate a control strategy model by combining the task score, platform index and platform control strategy. Step S300 includes: Step S301: Set a training period of several consecutive days, collect the running records within the training period, extract the user information from the running records, calculate the user score, and determine the user level corresponding to the running record based on the user level range. Step S302: Obtain the main content of the task content in the running record, match the task scoring model of the user level in the face of the main content, extract the task parameters of the running record, and input the normalized task parameters into the task scoring model to calculate the task score; Step S303: Based on the range of task levels, determine the task level of the running record, obtain the platform indicator prediction model corresponding to the task level, collect the characteristic running parameters and characteristic environment parameters in the time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S304: Obtain the platform control strategy corresponding to each time window. Take the task score, platform index and predicted value of platform index of the current time window as input, and control strategy of the next time window as output. Use the strategy gradient model for training. Iterate and update the strategy by setting the optimization target of platform performance index to generate control strategy model.

[0020] Step S400: Collect newly uploaded task content in real time, calculate the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculate the real-time platform control strategy by generating a control strategy model, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform. Step S400 includes: Step S401: Collect newly uploaded task content in the computing infrastructure service platform in real time, obtain user information of the user who uploaded the task content, calculate the real-time user score, and determine the real-time user level based on the user level range; Step S402: Extract the real-time main content, match the real user level to obtain the task scoring model in the face of the real-time main content, extract the real-time task parameters, and input the normalized real-time task parameters into the task scoring model to calculate the real-time task score. Step S403: Based on the range of task levels, determine the real-time task level, obtain the platform indicator prediction model corresponding to the real-time task level, collect the characteristic running parameters and characteristic environment parameters in the current time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S404: Input the predicted values ​​of the real-time task score, real-time platform index, and platform index for the next time window into the control strategy model to generate a real-time platform control strategy, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform.

[0021] To better implement the above methods, a dynamic analysis system for computing resources based on big data is also proposed. The system includes a user level module, a training model module, a control strategy model module, and a real-time control module. User Level Module: Extracts user parameters from the historical operation records of the computing infrastructure service platform, calculates user scores, determines the range of each user level, and summarizes the historical operation records of each user level; The user level module includes a unit for generating operation records and a unit for determining user levels. The runtime record generation unit: In the computing infrastructure service platform, environmental monitoring equipment is deployed in the environment where the platform equipment is located and communicates with the platform equipment to monitor the environmental parameters of the platform equipment during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the running parameters of the platform equipment when the task starts to execute, and associate the environmental parameters, task parameters and running parameters according to the time series alignment method to generate a time-tagged running data group, record the execution time of the task, and statistically summarize all running data groups within the time range to generate a runtime record; Determine user level units: Obtain user information from historical operation records, extract user parameters, including company size, industry type, and job level, assign values ​​to user parameters, preset the weight of each user parameter, perform weighted summation of user parameters to calculate user scores, count all user scores corresponding to historical operation records, calculate the variance of user scores as 'a', preset 'b' user levels, and generate the range of each user level.

[0022] Training Model Module: Analyze the historical operation records in the set of historical operation records for each user level, and build task scoring models and platform indicator prediction models respectively; The training model module includes a task scoring model unit and a platform metric prediction model unit: Task scoring model unit: Collect task parameters from the historical execution records of a certain user level, analyze the task parameters, and construct a task scoring model; Platform metric prediction model unit: Collect operating parameters and environmental parameters from the historical operation record set of a certain user level, analyze the operating parameters and environmental parameters, and construct a platform metric prediction model.

[0023] Regulation strategy model module: Set a training period, collect running records within the training period, calculate the user score of the running records, determine the user level corresponding to the running records based on the range of user levels, calculate the task score according to the task score model corresponding to the user level, calculate the predicted value of the platform index through the platform index prediction model, and train and generate the regulation strategy model by combining the task score, platform index and platform regulation strategy. Real-time control module: Collects newly uploaded task content in real time, calculates the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculates the real-time platform control strategy by generating a control strategy model, and uploads the real-time platform control strategy to the computing infrastructure service platform for platform adjustment.

[0024] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for dynamic analysis of computing resources based on big data, characterized in that, The methods include: Step S100: Extract user parameters from the historical operation records of the computing infrastructure service platform, calculate user scores, determine the range of each user level, and summarize the historical operation records of each user level; Step S200: Analyze the historical operation records in the historical operation record set for each user level, and construct a task scoring model and a platform indicator prediction model respectively; Step S300: Set a training period, collect running records within the training period, calculate the user score of the running records, determine the user level corresponding to the running records based on the range of user levels, calculate the task score according to the task scoring model corresponding to the user level, calculate the predicted value of the platform index through the platform index prediction model, and train and generate a control strategy model by combining the task score, platform index and platform control strategy. Step S400: Collect newly uploaded task content in real time, calculate the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculate the real-time platform control strategy by generating a control strategy model, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform.

2. The method for dynamic analysis of computing resources based on big data according to claim 1, characterized in that, Step S100 includes the following steps: Step S101: In the computing infrastructure service platform, an environmental monitoring device is deployed in the environment where the platform device is located and communicates with the platform device to monitor the environmental parameters of the platform device during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the operating parameters of the platform device when the task starts to execute, associate the environmental parameters, task parameters and operating parameters according to the time series alignment method, generate a time-tagged running data group, record the execution time of the task, and statistically summarize all running data groups within the time range to generate an operation record, and upload the operation record to the computing infrastructure service platform; Step S102: Obtain user information from historical operation records, extract user parameters, including company size, industry type, and job level, assign values ​​to user parameters, preset the weight of each user parameter, and calculate the user score by weighted summation of user parameters. Step S103: Calculate all user ratings corresponding to historical operation records, calculate the variance of user ratings as a, preset b user levels, and generate the range of each user level as: {[A1, A1+1×a), [A1+1×a, A1+2×a), ..., [A1+(b-1)×a, A2]}, where A1 and A2 represent the minimum and maximum values ​​of user ratings, respectively; Step S104: Based on the range of each user level, determine the user level in the historical operation records, filter the historical operation records of the same user level, and generate a set of historical operation records for each user level.

3. The method for dynamic analysis of computing resources based on big data according to claim 2, characterized in that, Step S200 includes the following steps: Step S201: Collect task parameters from the historical operation records of a certain user level, analyze the task parameters, and construct a task scoring model. Step S202: Collect the running parameters and environmental parameters from the historical running record set of a certain user level, analyze the running parameters and environmental parameters, and build a platform indicator prediction model.

4. The method for dynamic analysis of computing resources based on big data according to claim 3, characterized in that, The step S201 of constructing the task scoring model includes the following steps: Step S201-1: Obtain task content, extract the main content of the task content through natural language processing, calculate the similarity of the main content between each historical running record, preset a similarity threshold, and summarize the historical running records that exceed the similarity threshold to obtain a set of historical running records with the same main content. Step S201-2: In the historical running record set of a certain main content, collect task parameters, including task type and data scale, and assign values ​​to the task parameters to obtain the execution time corresponding to the historical running record, and normalize the execution time. Step S201-3: Combine each task parameter and the normalized execution time to generate a task training set. Use the value of each task parameter as input and the normalized execution time as output, and train the linear regression model to generate task rating models for different user levels when facing different main content.

5. The method for dynamic analysis of computing resources based on big data according to claim 3, characterized in that, The construction of the platform indicator prediction model in step S202 includes the following steps: Step S202-1: Summarize the task scores and calculate the variance of the task scores as c. Preset k task levels and generate the range of each task level as: {[B1, B1+1×c), [B1+1×c, B1+2×c), ..., [B1+(k-1)×c, B2]}, where B1 and B2 represent the minimum and maximum values ​​of the task scores, respectively. Step S202-2: Filter historical operation records of the same task level, preset time windows, divide the historical operation records according to the time windows, calculate the throughput, average latency and failure rate in each time window and perform normalization calculation, preset the weights of throughput, average latency and failure rate, and perform weighted summation of the normalized throughput, average latency and failure rate to calculate the platform performance index of each time window. Step S202-3: Obtain the running parameters and environmental parameters for each time window in the historical running records and perform normalization calculations. Establish a mapping relationship between the normalized running parameters and environmental parameters within the same time window and the platform performance indicators. Use the Pearson correlation coefficient method to calculate the correlation between each running parameter and environmental parameter and the platform performance indicators. Set a preset correlation threshold and mark the running parameters and environmental parameters that exceed the threshold as highly correlated running parameters and highly correlated environmental parameters, respectively. Step S202-4: Count the number of times each operating parameter and environmental parameter is marked as a highly correlated operating parameter and a highly correlated environmental parameter in the entire time window, calculate the occurrence frequency of each operating parameter and environmental parameter, preset an occurrence frequency threshold, and mark the operating parameters and environmental parameters that exceed the occurrence frequency threshold as characteristic operating parameters and characteristic environmental parameters, respectively. Step S202-5: Using the normalized feature running parameters and feature environment parameters as inputs, and the platform performance index of the next time window as output, train the random forest model to generate a platform index prediction model corresponding to each task level.

6. The method for dynamic analysis of computing resources based on big data according to claim 5, characterized in that, Step S300 includes the following steps: Step S301: Set a training period of several consecutive days, collect the running records within the training period, extract the user information from the running records, calculate the user score, and determine the user level corresponding to the running record based on the user level range. Step S302: Obtain the main content of the task content in the running record, match the task scoring model of the user level in the face of the main content, extract the task parameters of the running record, and input the normalized task parameters into the task scoring model to calculate the task score; Step S303: Based on the range of task levels, determine the task level of the running record, obtain the platform indicator prediction model corresponding to the task level, collect the characteristic running parameters and characteristic environment parameters in the time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S304: Obtain the platform control strategy corresponding to each time window. Take the task score, platform index and predicted value of platform index of the current time window as input, and control strategy of the next time window as output. Use the strategy gradient model for training. Iterate and update the strategy by setting the optimization target of platform performance index to generate control strategy model.

7. The method for dynamic analysis of computing resources based on big data according to claim 6, characterized in that, Step S400 includes the following steps: Step S401: Collect newly uploaded task content in the computing infrastructure service platform in real time, obtain user information of the user who uploaded the task content, calculate the real-time user score, and determine the real-time user level based on the user level range; Step S402: Extract the real-time main content, match the real user level to obtain the task scoring model in the face of the real-time main content, extract the real-time task parameters, and input the normalized real-time task parameters into the task scoring model to calculate the real-time task score. Step S403: Based on the range of task levels, determine the real-time task level, obtain the platform indicator prediction model corresponding to the real-time task level, collect the characteristic running parameters and characteristic environment parameters in the current time window, and input them into the platform indicator prediction model to calculate the predicted value of the platform indicator in the next time window. Step S404: Input the predicted values ​​of the real-time task score, real-time platform index, and platform index for the next time window into the control strategy model to generate a real-time platform control strategy, and upload the real-time platform control strategy to the computing power infrastructure service platform to adjust the platform.

8. A big data-based dynamic analysis system for computing resources, used to implement the big data-based dynamic analysis method for computing resources as described in any one of claims 1-7, characterized in that, The system includes a user level module, a training model module, a control strategy model module, and a real-time control module. The user level module extracts user parameters, calculates user scores, determines the range of each user level, and summarizes the historical operation records of each user level by using the historical operation records of the computing infrastructure service platform. The training model module analyzes the historical operation records in the historical operation record set for each user level to construct a task scoring model and a platform indicator prediction model respectively. The regulation strategy model module: sets a training period, collects running records within the training period, calculates the user score of the running records, determines the user level corresponding to the running records based on the range of user levels, calculates the task score according to the task score model corresponding to the user level, calculates the predicted value of the platform index through the platform index prediction model, and trains and generates the regulation strategy model by combining the task score, platform index and platform regulation strategy. The real-time control module: collects newly uploaded task content in real time, calculates the predicted values ​​of real-time task scores and platform indicators based on the task scoring model and platform indicator prediction model, calculates the real-time platform control strategy by generating a control strategy model, and uploads the real-time platform control strategy to the computing infrastructure service platform for platform adjustment.

9. A dynamic analysis system for computing resources based on big data according to claim 8, characterized in that, The user level module includes a unit for generating operation records and a unit for determining user level: The unit that generates the operation record: In the computing infrastructure service platform, an environmental monitoring device is deployed in the environment where the platform device is located and communicates with the platform device to monitor the environmental parameters of the platform device during operation, obtain the task content uploaded by the user, parse the task content and extract the task parameters, collect the operation parameters of the platform device when the task starts to execute, associate the environmental parameters, task parameters and operation parameters according to the time series alignment method, generate operation data groups with time tags, record the execution time of the task, and statistically summarize all operation data groups within the time range to generate an operation record; The user level determination unit: obtains user information from historical operation records, extracts user parameters, including company size, industry type, and job level, assigns values ​​to the user parameters, presets the weight of each user parameter, performs weighted summation of the user parameters to calculate the user score, counts all user scores corresponding to historical operation records, calculates the variance of the user scores as 'a', presets 'b' user levels, and generates the range of each user level.

10. A dynamic analysis system for computing resources based on big data according to claim 8, characterized in that, The training model module includes a task scoring model unit and a platform metric prediction model unit: The task scoring model unit: collects task parameters from the historical operation records in a set of historical operation records for a certain user level, analyzes the task parameters, and constructs a task scoring model; The platform metric prediction model unit: collects the operating parameters and environmental parameters from the historical operating records of a certain user level, analyzes the operating parameters and environmental parameters, and constructs a platform metric prediction model.

Images