A method and device for generating a K8S-based node resource image

Abstract

The present disclosure provides a method and device for generating a K8S-based node resource portrait, wherein the method comprises: collecting real-time data and historical data of node resources from a k8s cluster and a related system of the k8s cluster; extracting multi-resource dimension features and time dimension features from the real-time data and the historical data; and generating a dynamically updated resource portrait for each node based on the multi-resource dimension features and the time dimension features, thereby solving the problem of lack of comprehensive evaluation of multi-resource dimensions for nodes in the related art.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a method and apparatus for generating node resource profiles based on Kubernetes. Background Technology

[0002] With the rapid development of cloud computing technology, containerization has become the core carrier for business deployment and resource management. Kubernetes (k8s), as a powerful open-source container orchestration system, is widely used in large-scale container cluster management due to its strong automated deployment, scaling, and load balancing capabilities. In containerized mixed deployment scenarios (i.e., deploying containers of various business types, such as online services and offline tasks, core business and non-core business, on the same physical node or virtual machine), the k8s scheduling system bears the critical responsibility of rationally allocating Pods (container instances) to nodes, directly affecting resource utilization and business stability.

[0003] Currently, the core scheduling logic of the native Kubernetes scheduler is based on fundamental rules such as "resource allocability" and "node affinity," and its decision-making is mainly based on the following aspects: 1) Resource Remaining Amount Assessment: By comparing the difference between the node's allocatable field (i.e., the total amount of resources that the node can allocate to Pods) and the allocated resources, it can be determined whether the node has enough CPU, memory, and other resources to accommodate Pods to be scheduled (e.g., check whether the Pod's requests resource requirements are less than the node's remaining allocable resources). 2) Basic Filtering and Scoring: The scheduling process is divided into two stages: "Filtering" and "Score". The filtering stage excludes nodes that do not meet the resource requirements. The scoring stage sorts the nodes based on factors such as the remaining resources of the nodes and the node load (e.g., the LeastRequestedPriority priority strategy tends to select nodes with a higher proportion of remaining resources), and finally selects the node with the highest score. 3) Extended scheduling capabilities: Kubernetes supports extending scheduling logic through custom schedulers or scheduling plugins (such as priority scheduling based on PriorityClass, node affinity scheduling based on NodeSelector or NodeAffinity). However, existing extension solutions still rely on static rules or real-time resource status as the core basis and do not systematically incorporate historical load characteristics analysis.

[0004] Furthermore, most industry optimization solutions for Kubernetes scheduling focus on specific scenarios (such as scheduling of heterogeneous resources like GPUs and high-availability scheduling). Their core ideas still revolve around "real-time resource supply and demand matching." For example, they collect node CPU and memory usage in real time based on Metrics Server and dynamically adjust scheduling thresholds; they monitor node load through custom controllers and trigger Pod rescheduling (such as migrating Pods on overloaded nodes to idle nodes); and they introduce resource overcommitment technology to improve resource utilization. However, these solutions rely on static overcommitment ratio configurations and do not dynamically adjust based on the actual load capacity of the nodes.

[0005] The above technical solutions can basically meet the requirements in single business scenarios or clusters with stable loads. However, in scenarios with multiple services deployed together and frequent load fluctuations, their limitations gradually become apparent. In particular, the lack of comprehensive evaluation across multiple resource dimensions makes it impossible to fully reflect the resource carrying capacity of nodes.

[0006] Currently, no effective solution has been proposed to address the problem of nodes lacking comprehensive evaluation across multiple resource dimensions in related technologies. Summary of the Invention

[0007] The purpose of this disclosure is to address the shortcomings of existing technologies by providing a method and apparatus for generating node resource profiles based on Kubernetes, so as to at least solve the problem of the lack of comprehensive evaluation of nodes across multiple resource dimensions in related technologies.

[0008] According to one aspect of this disclosure, a method for generating a node resource profile based on Kubernetes is provided, comprising: collecting real-time data and historical data of node resources from a Kubernetes cluster and its associated system; extracting multi-resource dimension features and time dimension features from the real-time data and the historical data; and generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features.

[0009] Optionally, real-time and historical data of node resources are collected from the Kubernetes cluster and its associated systems. This includes collecting basic node information, real-time resource usage data, and historical data for various types of node resources. The basic node information includes total resource capacity and allocable resources. The real-time resource usage data includes the actual CPU utilization, actual memory usage, disk IOPS, and network throughput of the node and running Pods. The historical data includes the percentile, mean, maximum, and minimum resource usage over 24 hours, as well as the peak resource usage every 2 hours. Outliers are removed from the collected data using the median absolute deviation method.

[0010] Optionally, extracting multi-resource dimension features and time dimension features from the real-time data and the historical data includes: extracting the multi-resource dimension features from the real-time data and the historical data, and evaluating the risk level of the node in each resource dimension, wherein the multi-resource dimension features include the usage intensity index of the node in multiple resource dimensions; and using an improved multi-model ensemble algorithm to extract the time dimension features from the real-time data and the historical data, wherein the time dimension features include resource usage patterns, periodic fluctuation patterns, and resource balance in the time dimension.

[0011] Optionally, before using the improved multi-model ensemble algorithm to extract the time-dimensional features from the real-time data and the historical data, the method further includes: fusing the Holt-Winters seasonality model, the linear regression model, and the trend-adjusted model to obtain the improved multi-model ensemble algorithm.

[0012] Optionally, based on the multi-resource dimension features and the time dimension features, a dynamically updated resource profile is generated for each node, including: determining information for a node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer based on the multi-resource dimension features and the time dimension features. The basic information layer includes the node name, analysis timestamp, and last update time; the multi-resource status layer includes the current actual utilization rate of CPU, memory, and disk, the percentile, mean, and extreme values ​​of each resource over the past 24 hours, and the predicted value of resource usage trends and average utilization rate for the next 24 hours; the risk assessment layer includes the risk level and risk trigger threshold for each resource dimension; the tidal feature layer includes intraday peak / valley period markers and periodic fluctuation labels; the balance assessment layer includes time dimension balance and multi-resource dimension balance; and the optimization suggestion layer includes scheduling optimization suggestions automatically generated based on the profile data. Using the information from the node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer, a dynamically updated resource profile is generated for the node.

[0013] Optionally, after generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, the method further includes: persisting the node's resource profile to the etcd of the Kubernetes cluster in the form of a custom resource definition CRD, while providing multi-dimensional interfaces for downstream components to call, wherein the multi-dimensional interfaces include the native API interface of the Kubernetes cluster and third-party integration interfaces; and ensuring that the fields in the ResourceInsights CRD are consistent with the real-time calculation results through atomic operations of the Kubernetes cluster's API Server.

[0014] Optionally, after generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, the method further includes: using the node's resource profile to achieve intelligent scheduling decisions and resource optimization: peak-shifting scheduling based on tidal features, forward-looking scaling based on predicted trends, and node filtering based on risk levels.

[0015] According to another aspect of this disclosure, a node resource profile generation apparatus based on Kubernetes is provided, comprising: a collection unit for collecting real-time data and historical data of node resources from a Kubernetes cluster and its associated system; an extraction unit for extracting multi-resource dimension features and time dimension features from the real-time data and the historical data; and a generation unit for generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features.

[0016] Optionally, the acquisition unit is also used to: acquire node basic information, real-time resource usage data, and historical data of various types of resources of the node, wherein the node basic information includes total resource capacity and allocable resource amount, the real-time resource usage data includes actual CPU utilization, actual memory utilization, disk IOPS, and network throughput of the node and running Pods, and the historical data includes the quantile, mean, maximum, and minimum values ​​of resource usage within 24 hours, as well as the peak value of resource usage every 2 hours; and to clean outliers from the acquired data using the median absolute deviation method.

[0017] Optionally, the extraction unit is further configured to: extract the multi-resource dimension features from the real-time data and the historical data, and evaluate the risk level of the node in each resource dimension, wherein the multi-resource dimension features include the usage intensity index of the node in the multi-resource dimension; and use an improved multi-model ensemble algorithm to extract the time dimension features from the real-time data and the historical data, wherein the time dimension features include resource usage patterns, periodic fluctuation patterns, and resource balance in the time dimension.

[0018] Optionally, the extraction unit is further configured to: before extracting the time dimension features from the real-time data and the historical data using the improved multi-model ensemble algorithm, fuse the Holt-Winters seasonality model, the linear regression model, and the trend adjustment model to obtain the improved multi-model ensemble algorithm.

[0019] Optionally, the generation unit is further configured to: determine the information of the node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer based on the multi-resource dimension features and the time dimension features, wherein the information of the basic information layer includes the node name, analysis timestamp, and last update time; the information of the multi-resource status layer includes the current actual utilization rate of CPU, memory, and disk, the quantile, mean, and extreme values ​​of each resource in the past 24 hours, and the predicted value of the resource usage trend and average utilization rate in the next 24 hours; the information of the risk assessment layer includes the risk level and risk trigger threshold of each resource dimension; the information of the tidal feature layer includes intraday peak / valley period markers and periodic fluctuation labels; the information of the balance assessment layer includes time dimension balance and multi-resource dimension balance; the information of the optimization suggestion layer includes scheduling optimization suggestions automatically generated based on the profile data; and generate dynamically updated resource profiles for the node using the information of the node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer.

[0020] Optionally, the generation unit is further configured to: after generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, persist the node's resource profile to the etcd of the k8s cluster in the form of a custom resource definition CRD, and provide multi-dimensional interfaces for downstream components to call, wherein the multi-dimensional interfaces include the native API interface of the k8s cluster and third-party integration interfaces; and ensure that the fields in the ResourceInsights CRD are consistent with the real-time calculation results through atomic operations of the k8s cluster's API Server.

[0021] Optionally, the apparatus of this application further includes: a service unit, configured to generate a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, and then use the node's resource profile to achieve intelligent scheduling decisions and resource optimization: peak-shifting scheduling based on tidal features, forward-looking scaling based on predicted trends, and node filtering based on risk levels.

[0022] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform the Kubernetes-based node resource profile generation method of this disclosure.

[0023] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to execute the Kubernetes-based node resource profile generation method described in this disclosure.

[0024] One or more technical solutions provided in this disclosure collect real-time and historical data of node resources from the k8s cluster and its associated system; extract multi-resource dimension features and time dimension features from the real-time and historical data; and generate dynamically updated resource profiles for each node based on the multi-resource dimension features and the time dimension features, which can solve the problem of lack of comprehensive multi-resource dimension evaluation of nodes in related technologies. Attached Figure Description

[0025] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of an example system in which the various methods described herein may be implemented according to exemplary embodiments of the present disclosure; Figure 2 A flowchart is shown of a method for generating a Kubernetes-based node resource profile according to an exemplary embodiment of the present disclosure; Figure 3 A schematic block diagram of an apparatus for generating a node resource profile based on K8S according to an exemplary embodiment of the present disclosure is shown. Figure 4 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0027] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0028] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0029] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0030] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0031] The present disclosure is described below with reference to the accompanying drawings.

[0032] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows: Node resource profiling refers to the digital and structured description of the resource usage status, historical patterns, future trends, and risk levels of nodes in a Kubernetes cluster, achieved through multi-dimensional data collection and feature extraction. It includes real-time usage of multiple resources such as CPU, memory, and disk, historical statistics (e.g., quantiles, means), time-based tidal characteristics (e.g., peak / off-peak periods), risk assessment results, and balance indicators. It forms the core data foundation for scheduling decisions and resource optimization.

[0033] Mixed containerization deployment scenario: refers to a scenario in which multiple types of containerized services (such as online services and offline tasks, core business and non-core business) are deployed on the same physical node or virtual machine. The goal is to improve resource utilization, but it is prone to overload of some nodes and idleness of some nodes due to uneven scheduling.

[0034] Multi-resource dimension: refers to the coverage of multiple types of node resources, including but not limited to hardware resources such as CPU, memory, disk I / O, and network bandwidth. By collecting the actual usage, fluctuation range, and predicted trends of each resource, it breaks through the limitations of traditional focus on only CPU and memory, and achieves a comprehensive characterization of the node resource status.

[0035] The time-dimensional tidal characteristic refers to the periodic fluctuations in node resource usage over time, such as daily peaks (e.g., 9:00-18:00) and troughs (e.g., 0:00-6:00), and weekly / monthly periodic changes. Extracting this characteristic by analyzing historical data can support "peak-shifting scheduling" and avoid uneven temporal resource usage.

[0036] ResourceInsights CRD: refers to a standardized resource designed based on Kubernetes Custom Resource Definition (CRD). It is used to store structured data of node resource profiles (such as node name, multi-resource prediction value, risk level, balance index, etc.), and follows the insights.siod.io / v1 API version specification. It supports access through Kubernetes native API, kubectl command line, etc., to achieve standardized storage and linkage of profile data.

[0037] Multi-model integrated prediction algorithm: refers to the improved algorithm used for resource trend prediction in this scheme. It integrates the Holt-Winters seasonal model, linear regression model and trend adjustment model, combined with outlier detection (MAD method), seasonal decomposition (7-day cycle identification) and dynamic weight allocation to achieve accurate prediction of the usage trend (rising / stable / falling) and average utilization rate of node resources in the next 24 hours.

[0038] Resource Risk Level: Based on the comparison between predicted resource values ​​and preset thresholds, risk assessment classification (high / medium / low) is performed on each resource dimension (CPU, memory, disk, etc.) of the node. For example, a predicted CPU value ≥80% is "high risk", 56%-80% is "medium risk", and <56% is "low risk", which is used to assist the scheduler in avoiding high-load nodes.

[0039] Off-peak scheduling: This refers to a strategy that uses the time-dimensional tidal characteristics of node resource profiles to stagger the peak load periods of different services. For example, peak nighttime services can be scheduled to nodes that are heavily loaded during the day and idle at night, avoiding the same node from carrying too many peak services at the same time and achieving a time-balanced utilization of resources.

[0040] Dynamic scaling and linkage: This refers to combining the tidal characteristics and predicted trends of node resource profiles with the linkage of the Kubernetes HPA (Horizontal Pod Autoscaler) component to expand the number of Pod replicas in advance before business peaks and automatically shrink during off-peak periods, so that resource allocation dynamically matches actual load demand and improves resource utilization efficiency.

[0041] Quantile statistics refer to statistical methods for analyzing historical resource usage data of nodes, such as P99 (99th quantile, i.e., the resource utilization rate does not exceed this value for 99% of the time), P95, P50 (median), etc., which are used to describe the distribution characteristics of resource usage (such as extreme peaks, central tendency), and can reflect the true pressure of resource load better than the mean.

[0042] Median Absolute Deviation (MAD): A statistical method for detecting outliers in data. By calculating the median of the absolute deviation of the data from the median, it identifies and filters out abnormal data points that exceed a reasonable range (such as sudden peak interference), ensuring the accuracy of resource feature extraction and prediction.

[0043] After analyzing the technical solutions in the background technology, the applicant recognized that these solutions can basically meet the requirements in a single business scenario or a cluster with stable load. However, in scenarios with multiple services deployed together and frequent load fluctuations, their limitations gradually become apparent, mainly in the following aspects: 1) Focusing only on the "remaining allocation" of node resources without considering the "actual usage" of resources (such as the actual CPU utilization and actual memory usage of Pods already running on the node), leads to a disconnect between scheduling decisions and actual load. 2) The analysis did not include the time dimension and ignored the business tidal load characteristics of the nodes (such as daytime business peaks and nighttime load troughs, or periodic fluctuation patterns), resulting in some nodes being overloaded during the day and resources being idle at night, or cross-time period scheduling imbalance. 3) Lack of comprehensive evaluation across multiple resource dimensions (e.g., focusing only on CPU and memory without considering the usage characteristics of resources such as disk I / O and network bandwidth) fails to fully reflect the resource carrying capacity of nodes; 4) The above defects are particularly prominent in containerized mixed deployment scenarios, resulting in low resource utilization, uneven scheduling, and even affecting business stability.

[0044] Dynamic perception, accurate description, and efficient scheduling of node resources have become core technical challenges. To address these issues, this application provides a method embodiment for generating node resource profiles based on Kubernetes (k8s). This invention focuses on building node resource profiles in containerized hybrid environments. By integrating feature analysis across time and multiple resource dimensions, it provides refined data support for k8s scheduling systems, dynamic service scaling components, and resource governance platforms, representing a key technology for intelligent resource management and scheduling in container cloud platforms. In this embodiment, the aforementioned method for generating node resource profiles based on k8s can be applied to applications such as... Figure 1 The hardware environment shown consists of management terminal 101 and Kubernetes cluster 103 (including its associated systems). Figure 1 As shown, the management terminal 101 is connected to the Kubernetes cluster 103 via a network. A database 105 can be set up on or independently of the Kubernetes cluster 103 to provide data storage services for the Kubernetes cluster 103. The aforementioned network includes, but is not limited to, wide area networks (WANs), metropolitan area networks (MANs), or local area networks (LANs). The method for generating node resource profiles based on Kubernetes in this embodiment can be executed by the management terminal 101.

[0045] This exemplary embodiment provides a method for generating a node resource profile based on Kubernetes (K8S), specifically involving intelligent management and scheduling optimization of node resources based on Kubernetes. With the popularization of containerization technology, enterprise data centers generally adopt a multi-service mixed deployment mode, that is, deploying multiple types of container workloads such as online services, offline tasks, and batch processing jobs on the same physical node or virtual machine at the same time to improve resource utilization. Figure 2 A flowchart illustrating a method for generating a Kubernetes-based node resource profile according to an exemplary embodiment of this disclosure is shown, such as... Figure 2 As shown, the method includes the following steps: Step S201: Collect real-time and historical data of node resources from the k8s cluster and its associated system.

[0046] Specifically, basic node information, real-time resource usage data, and historical data of various types of resources of a node can be collected. The basic node information includes total resource capacity and allocable resources. The real-time resource usage data includes the actual CPU utilization, actual memory utilization, disk IOPS, and network throughput of the node and running Pods. The historical data includes the percentile, mean, maximum, and minimum values ​​of resource usage within 24 hours, as well as the peak value of resource usage every 2 hours. Outliers are removed from the collected data using the median absolute deviation method.

[0047] Step S202: Extract multi-resource dimension features and time dimension features from the real-time data and the historical data.

[0048] Specifically, the multi-resource dimension features can be extracted from the real-time data and the historical data, and the risk level of the node in each resource dimension can be evaluated. The multi-resource dimension features include the usage intensity index of the node in each resource dimension. An improved multi-model ensemble algorithm is obtained by integrating the Holt-Winters seasonal model, the linear regression model, and the trend adjustment model. The improved multi-model ensemble algorithm is used to extract the time dimension features from the real-time data and the historical data. The time dimension features include resource usage patterns, periodic fluctuation patterns, and resource balance in the time dimension.

[0049] Step S203: Based on the multi-resource dimension features and the time dimension features, generate a dynamically updated resource profile for each node.

[0050] Specifically, information for a node can be determined based on the multi-resource dimension features and the time dimension features, including a basic information layer, a multi-resource status layer, a risk assessment layer, a tidal characteristic layer, a balance assessment layer, and an optimization suggestion layer. The basic information layer includes the node name, analysis timestamp, and last update time. The multi-resource status layer includes the current actual utilization rate of CPU, memory, and disk; the percentile, mean, and extreme values ​​of each resource over the past 24 hours; and the predicted resource usage trend and average utilization rate for the next 24 hours. The risk assessment layer includes the risk level and risk trigger threshold for each resource dimension. The tidal characteristic layer includes intraday peak / valley period markers and periodic fluctuation labels. The balance assessment layer includes time-dimensional balance and multi-resource-dimensional balance. The optimization suggestion layer includes scheduling optimization suggestions automatically generated based on the node profile data. Using the information from the basic information layer, multi-resource status layer, risk assessment layer, tidal characteristic layer, balance assessment layer, and optimization suggestion layer, a dynamically updated resource profile is generated for the node.

[0051] Subsequently, by defining a custom resource CRD, the node's resource profile is persisted to the etcd of the Kubernetes cluster in the form of a ResourceInsights CRD. At the same time, multi-dimensional interfaces are provided for downstream components to call. These multi-dimensional interfaces include the native API interfaces of the Kubernetes cluster and third-party integration interfaces. Through atomic operations of the Kubernetes cluster's API Server, the fields in the ResourceInsights CRD are ensured to be consistent with the real-time calculation results.

[0052] It can also leverage node resource profiles to achieve intelligent scheduling decisions and resource optimization: peak-shifting scheduling based on tidal characteristics, forward-looking scaling up and down based on predicted trends, and node filtering based on risk levels.

[0053] Through the above steps, real-time and historical data of node resources are collected from the Kubernetes cluster and its associated system; multi-resource dimension features and time dimension features are extracted from the real-time and historical data; based on the multi-resource dimension features and the time dimension features, a dynamically updated resource profile is generated for each node, which can solve the problem of lack of comprehensive multi-resource dimension evaluation of nodes in related technologies.

[0054] This solution constructs a dynamically updated node resource profile, integrating time-based and multi-resource-based features, to provide refined data support for Kubernetes scheduling and resource management. Specifically, it includes the following core modules and processes, which are described and illustrated below through preferred embodiments.

[0055] 1) Multi-dimensional data acquisition module This module is responsible for collecting real-time and historical data on node resources from the Kubernetes cluster and related systems, providing raw input for profile building, specifically including: Data collection dimensions: Covering multiple resource types including CPU, memory, disk I / O, and network bandwidth, while also incorporating the time dimension (real-time data + historical periodic data). It also saves aggregated resource data every 2 hours to describe the daily trend of resource usage.

[0056] Data collected includes: Node basic information: total resource capacity, allocable resources (obtained via Kubernetes API server); Real-time resource usage data: actual CPU utilization, actual memory usage, disk IOPS, network throughput, etc. of nodes and running Pods (collected via Metrics Server and kubelet, updated every 10 seconds); Historical statistics: percentile values ​​of resource usage over 24 hours (e.g., CPU utilization P99, P95, P50), mean, maximum, minimum, and peak resource usage every 2 hours (stored via time-series databases such as Prometheus, retaining data from the last 7 days for trend analysis).

[0057] Data preprocessing: Outliers are detected and cleaned using the median absolute deviation (MAD) method to ensure data accuracy.

[0058] 2) Feature Extraction and Analysis Module Based on the collected raw data, we extract actual usage characteristics from multiple resource dimensions and tidal load characteristics from the time dimension, breaking through the limitations of traditional methods that rely solely on "remaining resource quantity": Multi-resource dimension features: usage intensity indicators of computing nodes on resources such as CPU, memory, and disk, such as 24-hour average utilization, peak utilization (P99 / P95), and fluctuation range (difference between maximum and minimum values); generate resource risk features, and assess the risk level (high / medium / low) of nodes in each resource dimension by comparing predicted values ​​with preset thresholds (CPU≥80%, memory≥85%, disk≥90%).

[0059] Time dimension characteristics: An improved multi-model ensemble algorithm (integrating Holt-Winters seasonality model, linear regression and trend adjustment model) is used to analyze resource usage patterns within a 14-day period, identify daily peak periods (e.g., 9:00-18:00), trough periods (e.g., 0:00-6:00), and weekly / monthly cyclical fluctuation patterns; the resource balance in the time dimension is calculated (measured by the variance of the peak usage rate every 2 hours within 24 hours, the smaller the variance, the more balanced the time distribution).

[0060] 3) Node Resource Profile Construction Module Based on the above characteristics, a dynamically updated resource profile is generated for each node, including the following core content: Basic information layer: node name, analysis timestamp, last update time; Multi-resource state layer: Real-time status: Current actual utilization of CPU, memory, and disk; Historical statistics: Percentile, mean, and extreme values ​​of each resource over the past 24 hours; Forecast trend: Resource usage trend (increasing / stable / decreasing) and predicted average utilization over the next 24 hours; Risk assessment layer: Risk level (high / medium / low) and risk trigger threshold for each resource dimension; Tidal characteristic layer: intraday peak / valley period markers, periodic fluctuation labels (such as "high load during the day" and "idle at night"); Balance assessment layer: Time dimension balance (variance value), multi-resource dimension balance (calculated by Euclidean distance, the closer the value is to 1, the more balanced the resource allocation). Optimization suggestion layer: Scheduling optimization suggestions automatically generated based on profile data (such as "migrate some workloads to low-risk nodes" and "increase the number of replicas before peak hours").

[0061] 4) Portrait Application Interface Module This module standardizes the storage and access of node resource profiles through custom resource definitions (CRDs). The profile results are persisted to etcd in the Kubernetes cluster in the form of ResourceInsights CRDs. It also provides multi-dimensional interfaces for downstream components to call. The specific design is as follows: 4.1) CRD storage structure All feature data of the node resource profile (including multi-resource status, risk assessment, tidal characteristics, balance index, etc.) are encapsulated in the spec.prediction field of ResourceInsights CRD.

[0062] For example, basic identifiers are bound to nodes via metadata.name (in the format ri-{node-name}) and spec.prediction.node_name; multi-resource data includes fields such as cpu_usage_p99 (99th percentile) and mem_predicted_avg (predicted average); time features record peak values ​​every 2 hours using the cpu_usage_max_2hours array, combined with cpu_trend (rising / stable / falling) to describe tidal patterns; risk and balance indicators such as cpu_risk_level (high / medium / low) and cpu_balance_time (time dimension variance) are directly written into the CRD, supporting fast querying.

[0063] 4.2) Standardized access interface Using the native Kubernetes API, downstream components can directly access CRD resources through the Kubernetes API Server. For example, the scheduler can obtain the profile of a specified node through the following interface: GET / apis / insights.siod.io / v1 / namespaces / kube-system / resourceinsights / ri-{node-name}; Batch query node data by filtering by tags (e.g., node.siod.io / name={node name}), and supports filtering by risk level (cpu_risk_level=high) or tidal characteristics (cpu_trend=increasing).

[0064] The kubectl command-line interface provides convenient command-line tool query capabilities.

[0065] For example, `kubectl get ri -n kube-system`: View an overview of the resource profiles of all nodes; `kubectl describe ri ri-worker-01 -n kube-system`: View a detailed profile of a specified node (including quantiles, predicted trends, optimization suggestions, etc.).

[0066] 4.3) Third-party integration interfaces Change notifications are achieved through the status.last_updated field of the CRD, and it supports linkage with monitoring systems (such as Prometheus), scheduling plugins, and automatic scaling components (HPA): the monitoring system listens for changes in the CRD and collects indicators such as cpu_risk_level and mem_balance_multi_resources in real time, triggering alarms (such as high-risk node alarms and resource imbalance alarms). When making decisions, the custom scheduler reads the cpu_predicted_avg (future utilization prediction) and cpu_balance_time (time balance) of the nodes through the CRD interface, and prioritizes low-risk, load-balanced nodes. HPA automatically scales up Pods before peak business periods based on the node's cpu_usage_max_2hours (historical peak hours) and cpu_trend (trend) to avoid node overload.

[0067] 4.4) Data Consistency Guarantee The updating of profile data is achieved through atomic operations of the Kubernetes API Server, ensuring that the spec.prediction field in the CRD is consistent with the real-time calculation results; at the same time, the CRD is recalculated and updated by a scheduled task (every hour) to ensure the timeliness of historical statistical data and predicted trends.

[0068] The technical solution adopted in this application is: 1) A method for constructing node resource profiles based on the fusion of time and multiple resource dimensions It innovatively integrates time-dimensional features (such as intraday tidal patterns and periodic fluctuation patterns) with multi-resource-dimensional features (actual usage percentiles, fluctuation ranges, and predicted trends of CPU, memory, disk, etc.) to construct a dynamic node resource profile, breaking through the limitations of traditional methods that rely solely on real-time remaining resource quantities, and achieving accurate characterization of node resource status.

[0069] 2) Resource feature extraction mechanism combining historical statistics and prediction algorithms An improved multi-model ensemble prediction algorithm (integrating Holt-Winters seasonality model, linear regression, and trend-adjusted model) is adopted to extract tidal load characteristics in the time dimension based on historical resource data (such as 24-hour quantiles and peak values ​​every 2 hours) and predict future resource usage trends. At the same time, the actual usage intensity and balance characteristics of multiple resource dimensions are extracted through quantile calculation, variance analysis, and other methods to form a comprehensive resource feature system.

[0070] 3) Standardized storage and access scheme for node resource profiles based on Custom Resource Definitions (CRDs) Node resource profile data (including multiple resource statuses, risk levels, tidal characteristics, optimization suggestions, etc.) is standardized and stored in the k8s cluster in the form of ResourceInsights CRD. Unified access and linkage of profile data are achieved through k8s native API, kubectl command line and third-party interfaces to ensure data consistency and ecosystem compatibility, and to provide standardized data interfaces for downstream functions such as scheduling and monitoring.

[0071] 4) Dynamic scheduling and resource optimization mechanism based on resource profiles By leveraging the risk level, time balance, and predicted trends in node resource profiles, intelligent scheduling decisions and resource optimization can be achieved. This includes peak-shifting scheduling based on tidal characteristics (such as scheduling peak nighttime services to idle daytime nodes), forward-looking scaling up and down based on predicted trends (such as scaling up in advance before peak hours), and node filtering based on risk levels to solve the problem of uneven resource scheduling in containerized mixed deployment scenarios.

[0072] 5) Quantitative assessment methods for resource risk level and balance An innovative resource risk level assessment mechanism (based on a three-stage classification of predicted values ​​and dynamic thresholds: high / medium / low risk) and a resource balance quantification index (calculated in the time dimension by the 24-hour resource peak variance, and in the multi-resource dimension by the Euclidean distance between the actual usage ratio and the expected ratio) are designed to provide quantifiable decision-making basis for resource scheduling and optimization.

[0073] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0074] This exemplary embodiment also provides a Kubernetes-based node resource profile generation apparatus for implementing the above embodiments and preferred embodiments, which will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0075] Figure 3 A schematic block diagram of an apparatus for generating a Kubernetes-based node resource profile according to an exemplary embodiment of the present disclosure is shown, such as... Figure 3 As shown, the device includes: The acquisition unit 31 is used to acquire real-time and historical data of node resources from the k8s cluster and the associated system of the k8s cluster; Extraction unit 32 is used to extract multi-resource dimension features and time dimension features from the real-time data and the historical data; The generation unit 33 is used to generate a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features.

[0076] The above scheme collects real-time and historical data of node resources from the Kubernetes cluster and its associated system; extracts multi-resource dimension features and time dimension features from the real-time and historical data; and generates a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, which can solve the problem of lack of comprehensive multi-resource dimension evaluation of nodes in related technologies.

[0077] Optionally, the acquisition unit is also used to: acquire node basic information, real-time resource usage data, and historical data of various types of resources of the node, wherein the node basic information includes total resource capacity and allocable resource amount, the real-time resource usage data includes actual CPU utilization, actual memory utilization, disk IOPS, and network throughput of the node and running Pods, and the historical data includes the quantile, mean, maximum, and minimum values ​​of resource usage within 24 hours, as well as the peak value of resource usage every 2 hours; and to clean outliers from the acquired data using the median absolute deviation method.

[0078] Optionally, the extraction unit is further configured to: extract the multi-resource dimension features from the real-time data and the historical data, and evaluate the risk level of the node in each resource dimension, wherein the multi-resource dimension features include the usage intensity index of the node in the multi-resource dimension; and use an improved multi-model ensemble algorithm to extract the time dimension features from the real-time data and the historical data, wherein the time dimension features include resource usage patterns, periodic fluctuation patterns, and resource balance in the time dimension.

[0079] Optionally, the extraction unit is further configured to: before extracting the time dimension features from the real-time data and the historical data using the improved multi-model ensemble algorithm, fuse the Holt-Winters seasonality model, the linear regression model, and the trend adjustment model to obtain the improved multi-model ensemble algorithm.

[0080] Optionally, the generation unit is further configured to: determine the information of the node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer based on the multi-resource dimension features and the time dimension features, wherein the information of the basic information layer includes the node name, analysis timestamp, and last update time; the information of the multi-resource status layer includes the current actual utilization rate of CPU, memory, and disk, the quantile, mean, and extreme values ​​of each resource in the past 24 hours, and the predicted value of the resource usage trend and average utilization rate in the next 24 hours; the information of the risk assessment layer includes the risk level and risk trigger threshold of each resource dimension; the information of the tidal feature layer includes intraday peak / valley period markers and periodic fluctuation labels; the information of the balance assessment layer includes time dimension balance and multi-resource dimension balance; the information of the optimization suggestion layer includes scheduling optimization suggestions automatically generated based on the profile data; and generate dynamically updated resource profiles for the node using the information of the node's basic information layer, multi-resource status layer, risk assessment layer, tidal feature layer, balance assessment layer, and optimization suggestion layer.

[0081] Optionally, the generation unit is further configured to: after generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, persist the node's resource profile to the etcd of the k8s cluster in the form of a custom resource definition CRD, and provide multi-dimensional interfaces for downstream components to call, wherein the multi-dimensional interfaces include the native API interface of the k8s cluster and third-party integration interfaces; and ensure that the fields in the ResourceInsights CRD are consistent with the real-time calculation results through atomic operations of the k8s cluster's API Server.

[0082] Optionally, the apparatus of this application further includes: a service unit, configured to generate a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, and then use the node's resource profile to achieve intelligent scheduling decisions and resource optimization: peak-shifting scheduling based on tidal features, forward-looking scaling based on predicted trends, and node filtering based on risk levels.

[0083] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0084] Exemplary embodiments of this disclosure also provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this disclosure.

[0085] Exemplary embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to embodiments of this disclosure.

[0086] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a processor of a computer, the computer program is used to cause the computer to perform a method according to an embodiment of this disclosure.

[0087] refer to Figure 4 The present invention describes a structural block diagram of an electronic device 400 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0088] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0089] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, output unit 407, storage unit 408, and communication unit 409. Input unit 406 can be any type of device capable of inputting information to electronic device 400. Input unit 406 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 407 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 408 may include, but is not limited to, disks and optical discs. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0090] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the method for generating a Kubernetes-based node resource profile can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. In some embodiments, the computing unit 401 can be configured by any other suitable means (e.g., by means of firmware) to perform the method for generating a Kubernetes-based node resource profile.

[0091] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0093] As used in this disclosure, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0095] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0096] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. A method for generating node resource profiles based on Kubernetes, characterized in that, include: Real-time and historical data of node resources are collected from the Kubernetes cluster and its associated systems. Extract multi-resource dimension features and time dimension features from the real-time data and the historical data; Based on the multi-resource dimension features and the time dimension features, a dynamically updated resource profile is generated for each node.

2. The method for generating node resource profiles as described in claim 1, characterized in that, Real-time and historical data of node resources are collected from the Kubernetes cluster and its associated systems, including: The system collects basic node information, real-time resource usage data, and historical data for various types of resources on the nodes. The basic node information includes total resource capacity and allocable resources. The real-time resource usage data includes the actual CPU utilization, actual memory usage, disk IOPS, and network throughput of the nodes and running Pods. The historical data includes the percentile, mean, maximum, and minimum values ​​of resource usage over 24 hours, as well as the peak resource usage every 2 hours. The median absolute deviation method is used to remove outliers from the collected data.

3. The method for generating node resource profiles as described in claim 1, characterized in that, From the real-time data and the historical data, extract multi-resource dimension features and time dimension features, including: The multi-resource dimension features are extracted from the real-time data and the historical data, and the risk level of the node in each resource dimension is evaluated. The multi-resource dimension features include the usage intensity index of the node in each resource dimension. An improved multi-model ensemble algorithm is used to extract the time-dimensional features from the real-time data and the historical data, wherein the time-dimensional features include resource usage patterns, periodic fluctuation patterns, and resource balance over time.

4. The method for generating node resource profiles as described in claim 3, characterized in that, Before using the improved multi-model ensemble algorithm to extract the time-dimensional features from the real-time data and the historical data, the method further includes: An improved multi-model ensemble algorithm is obtained by integrating the Holt-Winters seasonality model, linear regression model, and trend-adjusted model.

5. The method for generating node resource profiles as described in claim 1, characterized in that, Based on the multi-resource dimension features and the time dimension features, a dynamically updated resource profile is generated for each node, including: Based on the multi-resource dimension features and the time dimension features, the information of the node is determined into a basic information layer, a multi-resource status layer, a risk assessment layer, a tidal feature layer, a balance assessment layer, and an optimization suggestion layer. The basic information layer includes the node name, analysis timestamp, and last update time. The multi-resource status layer includes the current actual utilization rate of CPU, memory, and disk; the percentile, mean, and extreme values ​​of each resource over the past 24 hours; and the predicted resource usage trend and average utilization rate for the next 24 hours. The risk assessment layer includes the risk level and risk trigger threshold for each resource dimension. The tidal feature layer includes intraday peak / valley period markers and periodic fluctuation labels. The balance assessment layer includes time dimension balance and multi-resource dimension balance. The optimization suggestion layer includes scheduling optimization suggestions automatically generated based on the profile data. By utilizing information from the node's basic information layer, multi-resource status layer, risk assessment layer, tidal characteristic layer, balance assessment layer, and optimization suggestion layer, a dynamically updated resource profile is generated for the node.

6. The method for generating node resource profiles as described in claim 1, characterized in that, After generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, the method further includes: By defining custom resource descriptions (CRDs), the resource profiles of nodes are persisted to the etcd of the Kubernetes cluster in the form of ResourceInsights CRDs. At the same time, multi-dimensional interfaces are provided for downstream components to call. These multi-dimensional interfaces include native API interfaces of the Kubernetes cluster and third-party integration interfaces. Atomic operations on the API Server of the Kubernetes cluster ensure that the fields in the ResourceInsights CRD are consistent with the real-time calculation results.

7. The method for generating node resource profiles as described in any one of claims 1 to 6, characterized in that, After generating a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features, the method further includes: Intelligent scheduling decisions and resource optimization are achieved by leveraging node resource profiles: peak-shifting scheduling based on tidal characteristics, forward-looking scaling up and down based on predicted trends, and node filtering based on risk levels.

8. A device for generating node resource profiles based on Kubernetes, characterized in that, include: The acquisition unit is used to collect real-time and historical data of node resources from the Kubernetes cluster and its associated systems. The extraction unit is used to extract multi-resource dimension features and time dimension features from the real-time data and the historical data; The generation unit is used to generate a dynamically updated resource profile for each node based on the multi-resource dimension features and the time dimension features.

9. An electronic device, characterized in that, include: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images