A computer resource allocation management system and method based on data analysis

Abstract

The present invention discloses a computer resource allocation management system and method based on data analysis, which relates to the technical field of computer resource management. The method comprises: real-time data collection and historical data collection; building a spatiotemporal data warehouse based on the real-time data and historical data; extracting static and dynamic features of tasks, and generating a high-dimensional task feature vector by combining the static and dynamic features; building a resource semantic knowledge graph to define computing resource entities, relationships, and constraint rules; building a demand forecasting model by combining the spatiotemporal data warehouse and the resource semantic knowledge graph, and outputting a probability distribution forecast of future resource demand; and calculating resource utilization efficiency and load balancing indicators in real time by combining real-time data, and dynamically adjusting resource allocation decisions. The present invention proactively allocates resources based on the demand forecasting model built by the spatiotemporal data warehouse and the resource semantic knowledge graph, avoiding resource shortages or idleness, and improving resource utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of computer resource management, and in particular to a computer resource allocation management system and method based on data analysis. Background Art

[0002] During the operation of a computer system, the proper allocation of resources is crucial to system performance and task execution efficiency. Traditional computer resource allocation management methods rely on pre-set static policies or manual experience, making them difficult to adapt to complex and changing task loads and resource usage. As computer application scenarios become increasingly complex, the resource demands of tasks are becoming more dynamic and diverse. Static allocation policies often lead to irrational resource allocation, resulting in wasted or insufficient resources, and failing to fully utilize the effectiveness of computer resources.

[0003] Furthermore, existing technologies for resource allocation lack comprehensive analysis and in-depth mining of resource usage data. Decisions based solely on a few indicators, such as resource utilization, fail to uncover underlying patterns and implicit dependencies in resource usage, making it difficult to accurately predict task resource requirements. Consequently, resource allocation lacks scientific and forward-looking principles, failing to meet the demands of efficient modern computer systems. Summary of the Invention

[0004] The object of the present invention is to provide a computer resource allocation management system and method based on data analysis to solve the problems raised in the prior art.

[0005] To achieve the above object, the present invention provides the following technical solution: a computer resource allocation management method based on data analysis, the method comprising:

[0006] Collect computer resource usage metrics, task metadata, and runtime event data in real time, and collect historical data of the same type;

[0007] Based on real-time data and historical data of the same type, a spatiotemporal data warehouse is constructed; the spatiotemporal data warehouse stores data in layers and establishes a lineage tracking record for each data point;

[0008] Extracting static features and dynamic features of the task, and combining the static features and dynamic features to generate a high-dimensional task feature vector;

[0009] Taking the high-dimensional task feature vector as input, a resource semantic knowledge graph is constructed to define computing resource entities, relationships, and constraint rules. Through automated knowledge extraction, based on the spatiotemporal data warehouse, co-occurrence patterns and potential regularities of resource usage are discovered, and implicit dependencies are derived using a rule engine.

[0010] Combining the spatiotemporal data warehouse and resource semantic knowledge graph, we build a demand forecasting model, divide the task lifecycle into stages, align data at different time scales, integrate multi-time scale data to perform demand deduction, and output a probability distribution forecast of future resource demand.

[0011] Analyze the prediction results of the demand forecasting model, combine with real-time data, calculate resource utilization efficiency and load balancing indicators in real time, and dynamically adjust resource allocation decisions.

[0012] According to the above scheme, the computer resource usage indicator data includes CPU indicators, memory indicators, read and write delays and the number of TCP connections; the CPU indicators include core utilization and context switching frequency; the memory indicators include resident set size and page fault rate; the core utilization reflects the busyness of the CPU in a certain period of time, and the context switching frequency reflects the consumption of CPU resources when switching tasks; the resident set size reflects the actual amount of physical memory occupied by the process, and the page fault rate reflects the frequency of memory access anomalies;

[0013] The task metadata includes process and container information and runtime configuration; the process and container information includes process ID, container name and label, and resource request and limit values; the runtime configuration includes scheduling policy and cgroup parameters; the process ID, container name and label are used to uniquely identify the task entity, and the resource request and limit values ​​​​demonstrate the resource demand boundaries of the task; the scheduling policy reflects the execution order of tasks in the system, and the cgroup parameters are used to fine-tune the control of task resources;

[0014] The runtime event data includes task lifecycle and abnormal events; the task lifecycle includes process start or termination timestamps; by recording process start or termination timestamps, the complete lifecycle of the task is tracked; by recording abnormal events, faults and abnormal conditions during task operation are discovered;

[0015] Collect historical resource usage records, event logs, and performance indicators of the same type and store them aligned by timestamp.

[0016] According to the above solution, the tiered storage includes a three-tier storage architecture for hot, warm, and cold data. The hot data storage layer stores original precision monitoring data in a columnar format; the warm data storage layer downsamples the data in the time dimension and stores it, optimizing storage efficiency while retaining key features; the cold data storage layer archives and stores historical data of the same type collected; an automated data migration mechanism is established between the storage layers to complete data lifecycle management according to preset policies;

[0017] The lineage tracking record includes metadata identification and conversion operations; the metadata identification includes the data source node, the precise acquisition timestamp and the data format version number; the conversion operation includes data downsampling, unit standardization ETL processing process and operation execution time;

[0018] Establish a time series index in the time dimension and a multi-level spatial index in the space dimension, and implement efficient multi-dimensional query through the joint time and space index.

[0019] According to the above solution, the static features reflect the resource configuration requirements and operation constraints of the task; the static features include, based on the task metadata, obtaining the attribute characteristics of the process and container; the process includes a unique identifier consisting of the process ID and container label, the declared resource request and limit values, the configured scheduling policy type, and the cgroup control group parameters;

[0020] The dynamic features reflect the behavioral characteristics and resource usage patterns of the task during actual operation; the dynamic features include obtaining the behavioral pattern characteristics of the task based on the runtime event data; the behavioral pattern characteristics include the complete life cycle length of the process from startup to termination, the triggering frequency of abnormal events, and the time series fluctuation pattern of resource usage;

[0021] The static features are standardized, categorical variables are converted into one-hot encoded vectors, and numerical parameters are normalized and scaled. The dynamic features are subjected to time series feature engineering, which includes logarithmic transformation of life cycle duration and time distribution statistics of abnormal events. The processed static features and dynamic features are spliced ​​according to predetermined dimensional specifications to generate a high-dimensional task feature vector.

[0022] According to the above solution, the computing resource entities include computing node entities, storage entities, network entities and task entities; the relationships include demand relationships, competition relationships and collaboration relationships; the constraint rules include hardware configuration constraints, performance matching constraints and policy compliance constraints;

[0023] Extracting time series data of task resource usage based on the spatiotemporal data warehouse to identify high-frequency co-occurring resource combination patterns; analyzing spatiotemporal local characteristics of resource access based on the timestamp sequence of task execution trajectories; calculating statistical correlation coefficients between different computer resource usage indicator data, constructing a multidimensional resource usage association matrix, and quantifying the usage dependency intensity between different computer resource usage indicator data;

[0024] Based on a predefined set of domain rules, the logical consistency of the identified resource patterns is verified; the implicit dependency relationships between entities are derived, including resource competition chains and task collaboration chains; a weighted directed graph model is constructed, and the confidence of each dependency is quantified through a probabilistic method to form a dynamically adjusted resource semantic knowledge graph network.

[0025] According to the above solution, phased data divided according to the task life cycle is extracted from the spatiotemporal data warehouse, and the phased data includes resource usage records of the initialization phase, the calculation phase and the closing phase;

[0026] The resource constraint rules and entity relationships defined in the resource semantic knowledge graph are used as prior knowledge input, and time axis alignment processing is performed on real-time data of different time granularities;

[0027] Extract sliding window statistical features from high-frequency data to capture the instantaneous fluctuation pattern of resource demand; calculate statistics in the aggregated data phase to identify the trend of resource usage; cross-modally splice the semantic relationship features derived from the resource semantic knowledge graph with the temporal features to form fusion features;

[0028] Based on the fusion features, the probability distribution of computer resources in the future time window is predicted, and a multi-dimensional demand prediction result is output. The multi-dimensional demand prediction result includes the predicted computer resource demand, expected value and confidence interval, and the prediction time granularity is dynamically adjusted according to the characteristics of the task life cycle stage;

[0029] The deviation between the predicted resource demand and the actual resource consumption is monitored in real time; when the prediction error exceeds a threshold for three consecutive cycles, the online adjustment of the parameters of the demand forecasting model is triggered.

[0030] According to the above solution, the dynamic adjustment of resource allocation decision includes:

[0031] Based on the computer resource usage indicator data collected in real time and combined with the constraint rules of the resource semantic knowledge graph, the weighted comprehensive efficiency value of the current CPU utilization and memory utilization is calculated, and the load balancing status of the network and storage subsystems is evaluated based on the number of TCP connections and read and write latency indicators. When calculating the weighted comprehensive efficiency value, different weights are assigned to CPU utilization and memory utilization based on actual needs to reflect the relative importance of the two in system performance. By analyzing the changing trend of the number of TCP connections and the distribution of read and write latency, it is determined whether there is a load imbalance problem in the network and storage subsystems.

[0032] Comparing the forecast results output by the demand forecast model with the real-time resource occupancy, calculating the resource supply and demand gap of each computing node, the resource supply and demand gap being obtained by comparing the predicted resource demand with the current actual resource occupancy; generating a decision parameter matrix, the decision parameter matrix including resource allocation direction and priority;

[0033] When the predicted demand exceeds the currently configured tasks, backup resources are allocated in order of priority. When resources are continuously inefficiently utilized, a recycling and reallocation process is initiated, and the resource quota limits of the container are adjusted in real time based on the task metadata, and the feedback is fed back to the demand forecast model.

[0034] When allocating backup resources, resources are allocated to tasks in sequence according to the priority order of the decision parameter matrix. In the recovery and reallocation process, for tasks whose resource utilization rate is long-term below the threshold, some or all of their resources are recovered and reallocated to the tasks in need. The adjustment results are fed back to the demand forecasting model to help the demand forecasting model optimize the forecast results based on the actual resource allocation situation.

[0035] A computer resource allocation and management system based on data analysis, the system includes: a database module, a knowledge graph module, a demand forecast module, a resource decision module and a coordination module;

[0036] The database module includes a data acquisition module and a data warehouse module; the data acquisition module collects computer resource usage indicator data, task metadata and runtime event data in real time, and simultaneously collects historical data of the same type; the data warehouse module builds a spatiotemporal data warehouse, stores data in layers, and performs data lineage tracking;

[0037] The knowledge graph module includes a feature extraction module and a graph construction module; the feature extraction module extracts static features and dynamic features from the database module to generate a high-dimensional task feature vector; the graph construction module constructs a resource semantic knowledge graph, maintains computing resource entity relationships and constraint rules, and realizes automated knowledge extraction;

[0038] The demand forecasting module includes a feature fusion module, a model prediction module, and a model optimization module; the feature fusion module aligns multi-time scale data and fuses features extracted from the database module and the knowledge graph module; the model prediction module predicts the probability distribution of resource requirements according to the task life cycle stage and dynamically adjusts the prediction time granularity; the model optimization module monitors prediction errors and triggers online parameter adjustment;

[0039] The resource decision module includes an efficiency evaluation module, a policy generation module, and an execution control module; the efficiency evaluation module is used to evaluate resource utilization efficiency and load balancing indicators; the policy generation module generates a decision parameter matrix containing allocation direction and priority; and the execution control module implements resource allocation or recycling operations and adjusts container quotas in real time.

[0040] The coordination module includes a task scheduling module, an exception handling module and a feedback control module; the task scheduling module manages the data flow and task triggering between modules; the exception handling module detects resource competition conflicts and implements emergency measures; the feedback control module feeds back the execution results to the demand forecasting module and the knowledge graph module.

[0041] According to the above solution, the exception handling module monitors the computing resource entities, relationships and constraint rules defined in the resource semantic knowledge graph in the knowledge graph module in real time, and detects resource competition conflicts by combining the time series data of resource usage and task execution trajectory in the database module;

[0042] When a conflict is detected, emergency measures are immediately initiated; the conflict includes situations where the same computing resource is requested by multiple tasks with high priority at the same time or when the resource usage exceeds the constraint threshold during task execution. The emergency measures include suspending resource allocation for low-priority tasks first and releasing resources to ensure the operation of critical tasks.

[0043] According to the above scheme, after the resource decision module completes the resource allocation or recycling operation and adjusts the container quota in real time, the feedback control module receives the computer resource usage indicator data, task metadata and runtime event data collected in real time by the data acquisition module, and feeds the data back to the demand prediction module and the knowledge graph module for optimizing and updating the demand prediction model and resource semantic knowledge graph.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. This invention collects multi-dimensional data in real time and builds a spatiotemporal data warehouse. It combines historical data for in-depth analysis, extracts static and dynamic features of tasks, and constructs a resource semantic knowledge graph. This allows for a comprehensive and accurate understanding of resource usage patterns and task requirements, improving decision-making accuracy.

[0046] 2. The demand forecasting model constructed based on the spatiotemporal data warehouse and resource semantic knowledge graph outputs a probability distribution forecast of future resource demand, proactively allocates resources, effectively avoids resource shortages or idleness, and improves resource utilization;

[0047] 3. The present invention dynamically adjusts resource allocation decisions based on prediction results and real-time data, and promptly initiates resource allocation, recovery or reallocation processes to ensure load balancing of each computing node and improve the overall operating efficiency and stability of the computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of the steps of a computer resource allocation management method based on data analysis of the present invention;

[0049] Figure 2 This is a structural diagram of a computer resource allocation and management system based on data analysis according to the present invention;

[0050] Figure 3 A schematic diagram of the execution strategy structure of a computer resource allocation and management system based on data analysis according to the present invention;

[0051] Figure 4 The diagram is a schematic diagram of the feedback control structure of a computer resource allocation management system and method based on data analysis of the present invention. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0053] Example: Figures 1-4 As shown, the present invention provides a technical solution, a computer resource allocation management method based on data analysis, the method comprising:

[0054] S1. Collect computer resource usage indicator data, task metadata, and runtime event data in real time, and collect historical data of the same type;

[0055] Specifically, computer resource usage indicator data includes CPU indicators, memory indicators, read and write latency, and the number of TCP connections; CPU indicators include core utilization and context switch frequency; memory indicators include resident set size and page fault rate; core utilization reflects the CPU's busyness within a certain period of time, and context switch frequency reflects the CPU resource consumption when switching tasks; resident set size reflects the actual amount of physical memory occupied by the process, and page fault rate reflects the frequency of memory access anomalies; for example: during operation, the CPU core utilization averages 70%, the context switch frequency is 100 times per second; the resident set size is 500MB, the page fault rate is 0.1%; the read and write latency averages 10ms, and the number of TCP connections is stable at 20. This is just an example and is not a limitation;

[0056] Specifically, task metadata includes process and container information and runtime configuration; process and container information includes process ID, container name and label, and resource request and limit values; runtime configuration includes scheduling policy and cgroup parameters; process ID, container name and label are used to uniquely identify the task entity, and resource request and limit values ​​show the resource demand boundary of the task; scheduling policy reflects the execution order of tasks in the system, and cgroup parameters are used to fine-tune task resources; for example: the process ID of a data processing task is 1234, the container name is A, and the label is X; the resource request is 2 CPU cores and 1GB memory, and the resource limit is 4 CPU cores and 2GB memory; the scheduling policy is first-come, first-served, and the cgrou parameter is set to limit the CPU usage to no more than 80%. This is just an example and no limitation is imposed;

[0057] Specifically, runtime event data includes task lifecycle and exception events. Task lifecycle includes process start or termination timestamps. By recording process start or termination timestamps, the complete lifecycle of a task can be tracked. By recording exception events, faults and exceptions during task execution can be discovered. For example, a task starts at 9:00 AM and terminates at 10:30 AM. Two exception events occurred during the execution: a network connection interruption and a disk I / O error. This is just an example and is not intended to be limiting.

[0058] Specifically, historical resource usage records, event logs, and performance indicators of the same type are collected and stored aligned by timestamp.

[0059] S2. Build a spatiotemporal data warehouse based on real-time data and historical data of the same type. The spatiotemporal data warehouse stores data in layers and establishes a lineage tracking record for each data point.

[0060] Specifically, tiered storage includes a three-tier storage architecture for hot, warm, and cold data. The hot data storage layer stores original precision monitoring data in a columnar format; the warm data storage layer downsamples the data in the time dimension and stores it, optimizing storage efficiency while retaining key features; the cold data storage layer archives the collected historical data of the same type; for example, the warm data storage layer merges the data per minute into the data per five minutes and stores it, while the cold data storage layer archives historical data older than one month. An automated data migration mechanism is established between the storage layers to complete data lifecycle management according to preset policies.

[0061] Furthermore, the lineage tracking record includes metadata identification and conversion operations; the metadata identification includes the data source node, the precise collection timestamp, and the data format version number; the conversion operation includes data downsampling, unit normalization ETL processing, and the operation execution time. For example, the metadata identification is data source node B, the precise collection timestamp is 9:30 am, the data format version number is V1, and the conversion operation includes the ETL processing process of unit normalization of the in-memory data, and the operation execution time is 9:31 am. This is only an example and is not limiting.

[0062] Furthermore, a time series index is established in the time dimension, and a multi-level spatial index is constructed in the space dimension. Through the joint time and space index, efficient multi-dimensional query of resource usage of data processing tasks at specific time and specific node can be quickly queried.

[0063] S3. Extract the static and dynamic features of the task, and combine them to generate a high-dimensional task feature vector;

[0064] Specifically, static features reflect the resource configuration requirements and operation constraints of the task; static features include, based on task metadata, obtaining the attribute characteristics of the process and container; the process includes a unique identifier consisting of the process ID and the container label, the declared resource request and limit values, the configured scheduling policy type and the cgroup control group parameters; dynamic features reflect the behavioral characteristics and resource usage patterns of the task during actual operation; for example: the unique identifier consisting of the process ID and the container label is 1234-AX, the declared resource request and limit values ​​are CPU 2 cores and 1GB memory and CPU 4 cores and 2GB memory respectively; the configured scheduling policy The type is first-come, first-served; the cgroup control group parameter is to limit CPU usage to no more than 80%; dynamic features include obtaining task behavior pattern characteristics based on runtime event data; behavior pattern characteristics include the complete life cycle of the process from startup to termination, the triggering frequency of abnormal events, and the time series fluctuation pattern of resource usage. For example, the complete life cycle of a process from startup to termination is 90 minutes, the triggering frequency of abnormal events is 2 times / 90 minutes, and the time series fluctuation pattern of resource usage shows that CPU utilization is low at the beginning and end of the task and high in the middle. This is only an example and is not limited.

[0065] Furthermore, static features are standardized, categorical variables are converted into one-hot encoded vectors, and numerical parameters are normalized and scaled. Dynamic features are subjected to time series feature engineering, which includes logarithmic transformation of life cycle duration and time distribution statistics of abnormal events. The processed static and dynamic features are spliced ​​according to predetermined dimensional specifications to generate a high-dimensional task feature vector.

[0066] S4. Take high-dimensional task feature vectors as input to construct a resource semantic knowledge graph, defining computing resource entities, relationships, and constraint rules. Through automated knowledge extraction and based on a spatiotemporal data warehouse, discover co-occurrence patterns and underlying regularities in resource usage, and use a rule engine to derive implicit dependencies.

[0067] Specifically, computing resource entities include computing node entities, storage entities, network entities, and task entities; relationships include demand relationships, competition relationships, and collaboration relationships; and constraint rules include hardware configuration constraints, performance matching constraints, and policy compliance constraints. For example, a demand relationship means that a task requires resources from computing nodes and storage entities; a competition relationship means that multiple tasks may compete for resources from the same computing node; and a collaboration relationship means that a task collaborates with a network entity to complete data transmission. Hardware configuration constraints include the number of CPU cores and memory capacity of the node; performance matching constraints mean that the resource request of a task cannot exceed the performance ceiling of the computing node; and policy compliance constraints mean that the scheduling strategy of a task must comply with the overall strategy of the platform.

[0068] Furthermore, based on the spatiotemporal data warehouse, time series data of task resource usage are extracted to identify high-frequency co-occurring resource combination patterns; based on the timestamp sequence of task execution trajectory, the spatiotemporal local characteristics of resource access are analyzed; the statistical correlation coefficients between different computer resource usage indicator data are calculated, and a multidimensional resource usage association matrix is ​​constructed to quantify the usage dependency intensity between different computer resource usage indicator data. For example, the correlation coefficient between CPU utilization and memory utilization is 0.8, which indicates that there is a strong positive correlation between CPU utilization and memory utilization; based on the predefined domain rule set, the logical consistency of the identified resource patterns is verified; the implicit dependency relationship between entities is derived, and the implicit dependency relationship includes the resource competition chain and the task collaboration chain; a weighted directed graph model is constructed, and the confidence of each dependency relationship is quantified through a probabilistic method to form a dynamically adjusted resource semantic knowledge graph network.

[0069] S5. Combine the spatiotemporal data warehouse and resource semantic knowledge graph to build a demand forecasting model. This model divides the task lifecycle into stages, aligns data at different time scales, integrates data at multiple time scales to perform demand deduction, and outputs a probability distribution forecast of future resource demand.

[0070] Specifically, extract the phased data divided by the task life cycle from the spatiotemporal data warehouse, which includes resource usage records in the initialization phase, calculation phase and closing phase; input the resource constraint rules and entity relationships defined in the resource semantic knowledge graph as prior knowledge, and perform time axis alignment processing on real-time data of different time granularities; extract sliding window statistical features from high-frequency data to capture the instantaneous fluctuation pattern of resource demand; calculate the statistics in the aggregated data phase to identify the trend of resource usage; cross-reference the semantic relationship features derived from the resource semantic knowledge graph with the time series features. Modal splicing forms fusion features. Based on the fusion features, the probability distribution of computer resources in the future time window is predicted, and multi-dimensional demand prediction results are output. The multi-dimensional demand prediction results include predicted computer resource requirements, expected values, and confidence intervals. For example, the predicted computer resource requirements are: CPU 1.5 cores, 800MB of memory, expected values: CPU 1.6 cores, 850MB of memory, and confidence intervals: CPU [1.4, 1.8] cores, memory [750, 900]MB. This is only an example and is not limited. The prediction time granularity is dynamically adjusted according to the characteristics of the task life cycle stage.

[0071] Furthermore, the deviation between the predicted resource demand and the actual resource consumption is monitored in real time; when the prediction error for three consecutive periods exceeds the threshold, the online adjustment of the parameters of the demand forecasting model is triggered. For example, when the prediction error for three consecutive periods exceeds 10%, the online adjustment of the parameters of the demand forecasting model is triggered.

[0072] S6. Analyze the prediction results of the demand forecasting model, combine them with real-time data, calculate resource utilization efficiency and load balancing indicators in real time, and dynamically adjust resource allocation decisions;

[0073] Specifically, based on real-time collected computer resource usage indicator data and combined with the constraint rules of the resource semantic knowledge graph, the weighted comprehensive efficiency value of the current CPU utilization and memory utilization is calculated. Based on the number of TCP connections and read and write latency indicators, the load balancing status of the network and storage subsystems is evaluated. When calculating the weighted comprehensive efficiency value, different weights are assigned to CPU utilization and memory utilization based on actual needs to reflect their relative importance in system performance. For example, if the CPU utilization is assigned a weight of 0.6 and the memory utilization is assigned a weight of 0.4, and the current CPU utilization of the data processing task is 70% and the memory utilization is 60%, then the weighted comprehensive efficiency value is 0.6×70%+0.4×60%=66%. By analyzing the changing trend of the number of TCP connections and the distribution of read and write latency, it is determined whether there is a load imbalance problem in the network and storage subsystems.

[0074] Furthermore, the forecast results output by the demand forecast model are compared with the real-time resource usage to calculate the resource supply and demand gap of each computing node. The resource supply and demand gap is obtained by comparing the predicted resource demand with the current actual resource usage. For example, if the predicted demand is 1.5 CPU cores and 800MB of memory, and the current actual resource usage is 1.2 CPU cores and 600MB of memory, then the CPU supply and demand gap is 0.3 cores and the memory supply and demand gap is 200MB. This is just an example and is not restrictive. A decision parameter matrix is ​​generated, which includes resource allocation direction and priority.

[0075] Furthermore, when the predicted demand exceeds the currently configured task, backup resources are allocated in order of priority. When resources are continuously inefficiently utilized, a reclamation and reallocation process is initiated, and the container's resource quota limits are adjusted in real time based on task metadata, with feedback fed back to the demand forecasting model. For example, if the generated decision parameter matrix indicates that the resource allocation direction requires allocating 0.3 CPU cores and 200MB of memory from other idle computing nodes to this task, and the task has a high priority, then 0.3 CPU cores and 200MB of memory are allocated from the idle computing nodes to this task. This is just an example and is not a limitation.

[0076] Furthermore, when allocating backup resources, resources are allocated to tasks in sequence according to the priority order of the decision parameter matrix; in the recovery and reallocation process, for tasks whose resource utilization is long-term below the threshold, some or all of their resources are recovered and reallocated to the required tasks; the adjustment results are fed back to the demand forecasting model to help the demand forecasting model optimize the forecast results based on the actual resource allocation situation.

[0077] The present invention provides another technical solution, a computer resource allocation and management system based on data analysis, which includes: a database module, a knowledge graph module, a demand forecasting module, a resource decision module and a coordination module;

[0078] The database module includes a data acquisition module and a data warehouse module. The data acquisition module collects computer resource usage indicator data, task metadata, and runtime event data in real time, and simultaneously collects historical data of the same type. The data warehouse module builds a spatiotemporal data warehouse, stores data in layers, and tracks data lineage.

[0079] The knowledge graph module includes a feature extraction module and a graph construction module; the feature extraction module extracts static and dynamic features from the database module and generates high-dimensional task feature vectors; the graph construction module constructs a resource semantic knowledge graph, maintains the entity relationships and constraint rules of computing resources, and realizes automated knowledge extraction;

[0080] The demand forecasting module includes a feature fusion module, a model prediction module, and a model optimization module. The feature fusion module aligns multi-timescale data and integrates features extracted from the database module and the knowledge graph module. The model prediction module predicts the probability distribution of resource requirements according to the task lifecycle stage and dynamically adjusts the prediction time granularity. The model optimization module monitors prediction errors and triggers online parameter adjustments.

[0081] The resource decision module includes an efficiency evaluation module, a policy generation module, and an execution control module. The efficiency evaluation module is used to evaluate resource utilization efficiency and load balancing indicators. The policy generation module generates a decision parameter matrix containing allocation direction and priority. The execution control module implements resource allocation or recycling operations and adjusts container quotas in real time.

[0082] The coordination module includes a task scheduling module, an exception handling module and a feedback control module; the task scheduling module manages the data flow and task triggering between modules; the exception handling module detects resource competition conflicts and implements emergency measures; the feedback control module feeds back the execution results to the demand forecasting module and the knowledge graph module.

[0083] The exception handling module monitors the computing resource entities, relationships, and constraint rules defined in the resource semantic knowledge graph in the knowledge graph module in real time, and detects resource competition conflicts by combining the time series data of resource usage and task execution trajectories in the database module.

[0084] When a conflict is detected, emergency measures are immediately initiated; conflict situations include conflicts where the same computing resource is simultaneously requested by multiple tasks with high priority or resource usage exceeds the constraint threshold during task execution. Emergency measures include suspending resource allocation for low-priority tasks first and releasing resources to ensure the operation of critical tasks.

[0085] The feedback control module receives the computer resource usage indicator data, task metadata, and runtime event data collected in real time by the data acquisition module after the resource decision module completes the resource allocation or recycling operation and adjusts the container quota in real time, and feeds the data back to the demand prediction module and the knowledge graph module for optimizing and updating the demand prediction model and the resource semantic knowledge graph.

[0086] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A computer resource allocation management method based on data analysis, characterized in that: The method includes: Collect computer resource usage metrics, task metadata, and runtime event data in real time, and collect historical data of the same type; Based on real-time data and historical data of the same type, a spatiotemporal data warehouse is constructed; the spatiotemporal data warehouse stores data in layers and establishes a lineage tracking record for each data point; Extracting static features and dynamic features of the task, and combining the static features and dynamic features to generate a high-dimensional task feature vector; Taking the high-dimensional task feature vector as input, a resource semantic knowledge graph is constructed to define computing resource entities, relationships, and constraint rules. Through automated knowledge extraction, based on the spatiotemporal data warehouse, co-occurrence patterns and potential regularities of resource usage are discovered, and implicit dependencies are derived using a rule engine. Combining the spatiotemporal data warehouse and resource semantic knowledge graph, we build a demand forecasting model, divide the task lifecycle into stages, align data at different time scales, integrate multi-time scale data to perform demand deduction, and output a probability distribution forecast of future resource demand. Analyze the prediction results of the demand forecasting model, combine with real-time data, calculate resource utilization efficiency and load balancing indicators in real time, and dynamically adjust resource allocation decisions.

2. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: The computer resource usage indicator data includes CPU indicators, memory indicators, read and write delays and the number of TCP connections; the CPU indicators include core utilization and context switching frequency; the memory indicators include resident set size and page fault rate; The task metadata includes process and container information and runtime configuration; the process and container information includes process ID, container name and label, and resource request and limit values; the runtime configuration includes scheduling policy and cgroup parameters; The runtime event data includes task lifecycle and abnormal events; the task lifecycle includes process start or termination timestamp; Collect historical resource usage records, event logs, and performance indicators of the same type and store them aligned by timestamp.

3. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: The tiered storage includes a three-tier storage architecture for hot, warm, and cold data. The hot data storage layer stores original precision monitoring data in a columnar format; the warm data storage layer downsamples the data in the time dimension to optimize storage efficiency while retaining key features; the cold data storage layer archives and stores historical data of the same type. An automated data migration mechanism is established between the storage layers to complete data lifecycle management according to preset policies. The lineage tracking record includes metadata identification and conversion operations; the metadata identification includes the data source node, the precise acquisition timestamp and the data format version number; the conversion operation includes data downsampling, unit standardization ETL processing process and operation execution time; Establish a time series index in the time dimension and a multi-level spatial index in the space dimension, and implement efficient multi-dimensional query through the joint time and space index.

4. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: The static features reflect the resource configuration requirements and operation constraints of the task; the static features include, based on the task metadata, obtaining the attribute characteristics of the process and container; the process includes a unique identifier consisting of the process ID and container label, declared resource requests and limits, configured scheduling policy type and cgroup control group parameters; The dynamic features reflect the behavioral characteristics and resource usage patterns of the task during actual operation; the dynamic features include obtaining the behavioral pattern characteristics of the task based on the runtime event data; The behavioral pattern characteristics include the complete life cycle length of the process from startup to termination, the triggering frequency of abnormal events, and the time series fluctuation pattern of resource usage; The static features are standardized, categorical variables are converted into one-hot encoded vectors, and numerical parameters are normalized and scaled. The dynamic features are subjected to time series feature engineering, which includes logarithmic transformation of life cycle duration and time distribution statistics of abnormal events. The processed static features and dynamic features are spliced ​​according to predetermined dimensional specifications to generate a high-dimensional task feature vector.

5. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: The computing resource entities include computing node entities, storage entities, network entities and task entities; the relationships include demand relationships, competition relationships and collaboration relationships; the constraint rules include hardware configuration constraints, performance matching constraints and policy compliance constraints; Extracting time series data of task resource usage based on the spatiotemporal data warehouse to identify high-frequency co-occurring resource combination patterns; analyzing spatiotemporal locality characteristics of resource access based on the timestamp sequence of task execution trajectories; Calculate the statistical correlation coefficients between different computer resource usage indicator data, construct a multi-dimensional resource usage correlation matrix, and quantify the usage dependency intensity between different computer resource usage indicator data; Based on a predefined set of domain rules, the logical consistency of the identified resource patterns is verified; the implicit dependency relationships between entities are derived, including resource competition chains and task collaboration chains; a weighted directed graph model is constructed, and the confidence of each dependency is quantified through a probabilistic method to form a dynamically adjusted resource semantic knowledge graph network.

6. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: Extracting phased data divided according to the task life cycle from the spatiotemporal data warehouse, the phased data including resource usage records of the initialization phase, the calculation phase, and the closing phase; The resource constraint rules and entity relationships defined in the resource semantic knowledge graph are used as prior knowledge input, and time axis alignment processing is performed on real-time data of different time granularities; Extract sliding window statistical features from high-frequency data to capture the instantaneous fluctuation pattern of resource demand; calculate statistics in the aggregated data phase to identify the trend of resource usage; cross-modally splice the semantic relationship features derived from the resource semantic knowledge graph with the temporal features to form fusion features; Based on the fusion features, the probability distribution of computer resources in the future time window is predicted, and a multi-dimensional demand prediction result is output. The multi-dimensional demand prediction result includes the predicted computer resource demand, expected value and confidence interval, and the prediction time granularity is dynamically adjusted according to the characteristics of the task life cycle stage; The deviation between the predicted resource demand and the actual resource consumption is monitored in real time; when the prediction error exceeds a threshold for three consecutive cycles, the online adjustment of the parameters of the demand forecasting model is triggered.

7. The computer resource allocation management method based on data analysis according to claim 1, characterized in that: The dynamically adjusting resource allocation decision includes: Based on the computer resource usage indicator data collected in real time and combined with the constraint rules of the resource semantic knowledge graph, the weighted comprehensive efficiency value of the current CPU utilization and memory utilization is calculated, and based on the number of TCP connections and read and write latency indicators, the load balancing status of the network and storage subsystems is evaluated; Comparing the forecast results output by the demand forecast model with the real-time resource occupancy, calculating the resource supply and demand gap of each computing node, and generating a decision parameter matrix, wherein the decision parameter matrix includes resource allocation direction and priority; When the predicted demand exceeds the currently configured tasks, backup resources are allocated in order of priority; when resources are continuously inefficiently utilized, the recycling and reallocation process is initiated, and the resource quota limit of the container is adjusted in real time according to the task metadata, and fed back to the demand forecasting model.

8. A computer resource allocation and management system based on data analysis, applied to a computer resource allocation and management method based on data analysis according to any one of claims 1 to 7, characterized in that: The system includes: database module, knowledge graph module, demand forecast module, resource decision module and coordination module; The database module includes a data acquisition module and a data warehouse module; the data acquisition module collects computer resource usage indicator data, task metadata and runtime event data in real time, and simultaneously collects historical data of the same type; the data warehouse module builds a spatiotemporal data warehouse, stores data in layers, and performs data lineage tracking; The knowledge graph module includes a feature extraction module and a graph construction module; the feature extraction module extracts static features and dynamic features from the database module to generate a high-dimensional task feature vector; the graph construction module constructs a resource semantic knowledge graph, maintains computing resource entity relationships and constraint rules, and realizes automated knowledge extraction; The demand forecasting module includes a feature fusion module, a model prediction module, and a model optimization module; the feature fusion module aligns multi-time scale data and fuses features extracted from the database module and the knowledge graph module; the model prediction module predicts the probability distribution of resource requirements according to the task life cycle stage and dynamically adjusts the prediction time granularity; the model optimization module monitors prediction errors and triggers online parameter adjustment; The resource decision module includes an efficiency evaluation module, a policy generation module, and an execution control module; the efficiency evaluation module is used to evaluate resource utilization efficiency and load balancing indicators; the policy generation module generates a decision parameter matrix containing allocation direction and priority; and the execution control module implements resource allocation or recycling operations and adjusts container quotas in real time. The coordination module includes a task scheduling module, an exception handling module and a feedback control module; the task scheduling module manages the data flow and task triggering between modules; the exception handling module detects resource competition conflicts and implements emergency measures; the feedback control module feeds back the execution results to the demand forecasting module and the knowledge graph module.

9. The computer resource allocation and management system based on data analysis according to claim 8, characterized in that: The exception handling module monitors the computing resource entities, relationships and constraint rules defined in the resource semantic knowledge graph in the knowledge graph module in real time, and detects resource competition conflicts in combination with the time series data of resource usage and task execution trajectory in the database module; When a conflict is detected, emergency measures are immediately initiated; the conflict includes situations where the same computing resource is requested by multiple tasks with high priority at the same time or when the resource usage exceeds the constraint threshold during task execution. The emergency measures include suspending resource allocation for low-priority tasks first and releasing resources to ensure the operation of critical tasks.

10. The computer resource allocation and management system based on data analysis according to claim 8, characterized in that: After the resource decision module completes the resource allocation or recycling operation and adjusts the container quota in real time, the feedback control module receives the computer resource usage indicator data, task metadata and runtime event data collected in real time by the data acquisition module, and feeds the data back to the demand prediction module and the knowledge graph module for optimizing and updating the demand prediction model and resource semantic knowledge graph.

Images