Resource monitoring and data analysis system and method based on cloud platform

Abstract

The present invention discloses a resource monitoring and data analysis system and method based on a cloud platform, and relates to the field of cloud computing technology. The system includes a micro-isolation architecture module, and realizes the micro-isolation of the cloud platform through software-defined network technology and security policies; multiple monitoring agents are distributed in each micro-isolation area, including a sensor unit, a preprocessing unit and a control unit. In the data analysis module, the model selection unit can select an adaptive model from the preset analysis model according to the resource type and business attributes of the micro-isolation area. The analysis execution unit uses the selected model to analyze the resource data, and the parameter update unit dynamically updates the parameters of the analysis model according to the changes in the cloud platform business. The data analysis module adopts an adaptive analysis algorithm, adaptively adjusts the parameters and model complexity during the analysis execution process, and has a feedback-based learning mechanism and predictive parameter update capabilities when updating parameters, thereby improving the accuracy and effectiveness of cloud platform resource monitoring and analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a resource monitoring and data analysis system and method based on a cloud platform. Background Art

[0002] With the rapid development of cloud computing technology, cloud platforms are widely used in various industries. Enterprises and organizations have migrated a large number of businesses to cloud platforms to gain advantages such as cost-effectiveness, scalability, and flexibility. However, the increasing scale and complexity of cloud platforms have brought huge challenges to resource management.

[0003] Some existing cloud platform resource monitoring solutions are not comprehensive enough in data collection. They may only focus on the usage of major computing resources and network resources, but ignore some key indicators of storage resources, such as the health status of storage media, access patterns of stored data, etc. In addition, data collection lacks flexibility. When the resource usage pattern changes, the collection frequency and collection parameters often cannot be adjusted in time, resulting in the inability to accurately capture the dynamic changes of resources in some cases.

[0004] Most existing solutions use fixed data analysis models, which often cannot adapt well to different cloud platform environments and business scenarios. For a cloud platform with both storage-intensive and computing-intensive businesses, a single data analysis model cannot accurately analyze and evaluate the utilization efficiency and potential problems of different types of resources.

[0005] Data analysis models lack dynamic adjustment capabilities. As cloud platform services develop and change, such as application updates and new business launches, resource usage patterns will change, but existing analysis models cannot update parameters or switch to more appropriate analysis methods in a timely manner based on these changes, resulting in inaccurate analysis results and failure to provide effective decision-making basis for resource optimization and configuration.

[0006] Based on the above background, the present invention aims to propose an innovative cloud platform-based resource monitoring and data analysis system and method to overcome the limitations of the prior art. Summary of the invention

[0007] The purpose of the present invention is to provide a cloud platform-based resource monitoring and data analysis system and method to solve the problems raised in the prior art.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The resource monitoring and data analysis system based on the cloud platform includes the following:

[0010] Micro-isolation architecture module, used to divide the cloud platform into multiple micro-isolation areas;

[0011] A monitoring agent module, each monitoring agent is set in a micro-isolation area, and is used to collect resource data in the micro-isolation area, and the resource data includes computing resource data, storage resource data and network resource data. The monitoring agent can dynamically adjust the collection frequency and collection parameters according to resource usage;

[0012] The data analysis module is communicatively connected to the monitoring agent, and is used to receive the resource data and analyze the resource data using an adaptive analysis algorithm. The adaptive analysis algorithm can automatically select a suitable model for analysis based on the resource type and business attributes in the micro-isolation area, and dynamically update the parameters of the analysis model based on changes in the cloud platform business.

[0013] The micro-isolation architecture module is implemented through a software-defined network technology unit and a security policy unit; the software-defined network unit separates the control plane and the data plane of the network through SDN, and uniformly manages the entire cloud platform network. For communication between different micro-isolation areas, the SDN controller can set traffic control strategies to ensure the relative independence of resources in each micro-isolation area at the network level, and as the business of the cloud platform changes, SDN can dynamically adjust the network configuration and reallocate network resources, so that the network resources of each micro-isolation area can be optimized according to its own business needs while maintaining isolation from other areas; through SDN, different micro-isolation areas can have independent virtual network address spaces, and in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources.

[0014] The application of the security policy unit in micro-isolation is that the security policy unit can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocols, port numbers, source addresses and destination addresses; only traffic from specific ports of specific network segments is allowed to enter, and other external or internal unauthorized network connections are blocked to prevent network attacks and illegal access, thereby ensuring the security of resources in the area; the security policy unit also includes intrusion detection and prevention mechanisms, and intrusion detection systems and intrusion prevention systems are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, detect and prevent intrusions; if a large number of abnormal network connection attempts or illegal access to key system files are detected in a micro-isolation area, IDS / IPS can take timely measures, such as blocking connections, issuing alarms, etc., to ensure that resources in the area are not affected by malicious attacks and maintain their security and independence.

[0015] The monitoring agent module includes a sensor unit, a preprocessing unit and a control unit; the sensor unit, in the micro-isolation area, collects information by interacting with the operating system kernel and the virtual machine management program; using the performance counter interface provided by the operating system, it can obtain data such as CPU usage, the busyness of each core, and thread scheduling. At the same time, it can also obtain parameters such as memory usage and memory page exchange frequency, which can reflect the real-time consumption status of computing resources in the micro-isolation area; in terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface; at the same time, the sensor will collect specific resource data in the area according to the resource usage mode and business needs of different micro-isolation areas, so as to provide accurate data closely related to the business of the area for subsequent analysis.

[0016] The preprocessing unit includes data cleaning and format conversion. Data cleaning is used to clean abnormal data appearing in the resource monitoring data of the micro-isolation area. The data cleaning process sets thresholds and rules to identify and remove abnormal data. The format conversion function stores and subsequently analyzes all collected and preprocessed data in a unified format.

[0017] The control unit will continuously monitor the resource load conditions in the micro-isolation area and adjust the collection frequency according to the load conditions in the micro-isolation area. In addition, the control unit will also adjust the collection parameters of the sensor according to the resource fluctuations. Resource fluctuations refer to large fluctuations in packet loss and bandwidth of network resources in the micro-isolation area. For storage resources, it refers to abnormal fluctuations in read and write delays. The above operations help analyze the reasons for storage resource fluctuations, better adapt to the dynamic changes of resources in the micro-isolation area, and optimize the monitoring effect.

[0018] The data analysis module includes a model selection unit, an analysis execution unit and a parameter update unit; the model selection unit is a selection mechanism based on resource types and business attributes, and adopts an adaptive analysis algorithm to dynamically identify resources in the micro-isolation area according to resource types and business attributes, analyze resource usage characteristics, and extract features of various resource data of the micro-isolation area collected by the monitoring agent, including computing resources, storage resources and network resources. According to the extracted features, the classification algorithm in machine learning is used to judge the resource type of the micro-isolation area. For areas with complex business logic, the adaptive analysis algorithm will analyze the dependency pattern of the business process on resources, and then select the analysis model based on the analysis results of resource types and business attributes; through the interface with the business system or log analysis, the key links and operations in the business process are parsed, and for each business link, its dependency pattern on resources is analyzed. In addition, the algorithm can timely analyze the rising trend of computing resource utilization through time series analysis technology, and consider this change when selecting the model. According to business changes, the dependency relationship and demand intensity of the business process on resources are re-evaluated; these change information are fed back to the adaptive analysis algorithm so that the resource demand changes brought about by business changes can be adapted when the model is selected.

[0019] The analysis execution unit, when the model selection unit selects the analysis model, will obtain data from the monitoring agent and input it into the selected model, and the model will process the data according to the internal algorithm and logic; during the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data, and the algorithm will adjust the parameters related to the read and write performance evaluation accordingly, so that the analysis results are more in line with the current actual situation and the accuracy of the diagnosis of storage resource problems is improved;

[0020] First, for the selected analysis model, sensitivity analysis is performed on each parameter in the model. Determine which parameters have a greater impact on the accuracy and effectiveness of the analysis results. According to the sensitivity analysis results, a dynamic adjustment strategy is formulated. When the real-time resource data obtained by the monitoring agent indicates that the read and write speed of the storage resource has changed, if the change exceeds the preset ratio, the read and write speed threshold parameters are adjusted according to the pre-set adjustment rules, so as to more accurately analyze the usage of storage resources;

[0021] At the same time, the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of resource data. In the preset thresholds and rules, for micro-isolation areas where resource usage is relatively stable, the algorithm will simplify the analysis model. On the contrary, when abnormal conditions occur in resources in the area, the algorithm will introduce more analysis dimensions and algorithms, increasing the complexity of the analysis model.

[0022] Establish a resource stability assessment indicator system, comprehensively consider factors such as the volatility of resource data, the frequency of changes, and the degree of deviation from historical data, and establish a mapping relationship with the complexity of the analysis model based on the resource stability assessment results; for micro-isolation areas where resource usage is relatively stable, select a simplified analysis model. The simplified model may ignore some minor resource interaction factors and complex calculation processes to reduce computing overhead; when abnormal conditions occur in resources in the area, select a more complex analysis model and determine the additional analysis dimensions and algorithms that need to be introduced;

[0023] The parameter update unit triggers parameter update by monitoring the changes of cloud platform services. The parameter update obtains information about changes in cloud platform services. When business changes are detected, the parameter update unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter update unit has a feedback-based learning mechanism and can learn from the feedback of analysis results. If the analysis execution unit finds that the resource utilization rate in a micro-isolation area continues to deviate from the expected value, the algorithm will feed this information back to the parameter update unit. The parameter update unit will try to adjust the parameters of the analysis model, and then analyze the resource data again to observe the effect after adjustment. Through multiple such feedback and adjustment processes, the algorithm can automatically find more suitable parameter settings, so that the analysis model is continuously optimized to adapt to the ever-changing environment of the cloud platform. In addition, the adaptive analysis algorithm can also perform predictive parameter updates based on historical business changes and trends in resource data. It uses time series analysis and other technologies in machine learning to predict the potential impact of future business changes on resources and update the parameters of the analysis model in advance.

[0024] To achieve the above object, the present invention provides the following technical solutions:

[0025] The resource monitoring and data analysis method based on the cloud platform includes the following steps:

[0026] Step 1: Use the micro-segmentation architecture to divide the cloud platform into multiple micro-segmentation areas;

[0027] Step 2: A monitoring agent is set in each micro-isolation area, and resource data in the micro-isolation area is collected through the monitoring agent. During the collection process, the collection frequency and collection parameters are dynamically adjusted according to resource usage;

[0028] Step 3: Transmit the collected data to the data analysis module;

[0029] Step 4: The data analysis module uses an adaptive analysis algorithm to analyze the resource data. The adaptive analysis algorithm automatically selects a suitable analysis model according to the resource type and business attributes in the micro-isolation area, and dynamically updates the parameters of the analysis model according to changes in the cloud platform business.

[0030] The use of micro-isolation architecture to divide the cloud platform into multiple micro-isolation areas further includes the following: micro-isolation is achieved through software-defined network technology and security policies. The software-defined network separates the control plane and data plane of the network through SDN, and uniformly manages the entire cloud platform network. For the communication between different micro-isolation areas, the SDN controller can set the flow control policy, and as the business of the cloud platform changes, the SDN can dynamically adjust the network configuration and reallocate network resources; through SDN, different micro-isolation areas can have independent virtual network address spaces, and in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources;

[0031] The security policy is applied in micro-isolation. The security policy can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocol, port number, source address and destination address. The security policy also includes intrusion detection and prevention mechanism. Intrusion detection system and intrusion prevention system are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, detect and prevent intrusion behavior.

[0032] The monitoring agent includes the following:

[0033] First, within the micro-isolation area, the sensor unit collects information by interacting with the operating system kernel and the virtual machine hypervisor. In terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface. At the same time, the sensor will collect specific resource data in the area according to the resource usage patterns and business needs of different micro-isolation areas.

[0034] Then, the preprocessing unit performs data cleaning and format conversion. It uses data cleaning methods to clean abnormal data in the resource monitoring data in the micro-isolation area. The data cleaning process sets thresholds and rules to identify and remove abnormal data. The format conversion function stores all collected and preprocessed data in a unified format for subsequent analysis.

[0035] Finally, the control unit is used to continuously monitor the resource load in the micro-isolation area and adjust the collection frequency according to the load in the micro-isolation area. In addition, the control unit will also adjust the sensor's collection parameters according to resource fluctuations. Resource fluctuations refer to large fluctuations in packet loss and bandwidth in the network resources in the micro-isolation area. For storage resources, it refers to abnormal fluctuations in read and write delays.

[0036] The data analysis module includes the following contents:

[0037] The data analysis module includes a model selection unit, an analysis execution unit and a parameter updating unit;

[0038] The model selection unit is a selection mechanism based on resource types and business attributes. For the cloud platform, the model selection unit uses an adaptive analysis algorithm to dynamically identify resources in the micro-isolation area according to resource types and business attributes, analyze resource usage characteristics, and the adaptive analysis algorithm will deeply understand the business attributes of the micro-isolation area. For areas with complex business logic, the adaptive analysis algorithm will analyze the dependency pattern of business processes on resources, and then select an analysis model based on the analysis results of resource types and business attributes;

[0039] Then, when the model selection unit selects the analysis model, the analysis execution unit will obtain data from the monitoring agent and input it into the selected model. The model will process the data according to the internal algorithm and logic. During the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data. At the same time, the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of the resource data. In the preset thresholds and rules, for micro-isolation areas with relatively stable resource usage, the algorithm will simplify the analysis model. On the contrary, when abnormal conditions occur in the resources in the area, the algorithm will introduce more analysis dimensions and algorithms to increase the complexity of the analysis model.

[0040] Finally, a parameter update unit is used to trigger parameter updates by monitoring changes in cloud platform services. Parameter updates obtain information about changes in cloud platform services. When business changes are detected, the parameter update unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter update unit has a feedback-based learning mechanism and can learn from the feedback of analysis results. In addition, the adaptive analysis algorithm can also perform predictive parameter updates based on historical business changes and trends in resource data.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. Accurate micro-isolation area monitoring: Through the micro-isolation architecture module, the cloud platform is divided into multiple micro-isolation areas, realizing fine-grained differentiation of different business or resource types. A monitoring agent is set up in each micro-isolation area. This architecture can avoid the problem of confusion of resource usage information in the overall monitoring and more accurately capture the changes in resources in each area.

[0043] The monitoring agent can dynamically adjust the collection frequency and collection parameters according to resource usage. When resources in the micro-isolation area fluctuate abnormally or are under high load, such as during business peak hours, the monitoring agent can collect data in a more timely and detailed manner, providing an accurate data basis for analyzing resource problems and ensuring the real-time and accuracy of resource monitoring.

[0044] 2. Adaptive analysis improves analysis accuracy: The data analysis module uses an adaptive analysis algorithm; it can automatically select the appropriate analysis model based on the resource type and business attributes in the micro-isolation area;

[0045] The algorithm can also dynamically update the parameters of the analysis model according to changes in the cloud platform business; as the business develops, such as the launch of new applications or changes in business volume, the parameters of the analysis model can be adjusted in a timely manner to ensure that the analysis results always reflect the true status of current resources, avoid invalidation of analysis results due to business changes, and provide a reliable basis for optimal resource allocation.

[0046] 3. Realize optimal resource configuration: Combining the advantages of the above-mentioned precise monitoring and adaptive analysis, the present invention can timely discover resource bottlenecks and abnormal usage; through in-depth analysis of resource data, such as discovering that storage resources in a micro-isolation area have performance degradation caused by frequent reading and writing, or that computing resources are idle and wasted under specific tasks, the system can provide targeted optimization suggestions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of the system organization architecture of the cloud platform-based resource monitoring and data analysis system of the present invention;

[0048] Figure 2 The present invention is a method flow chart of the resource monitoring and data analysis method based on the cloud platform. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example: Figure 1-Figure 2 As shown, the present invention provides a technical solution, a resource monitoring and data analysis system and method based on a cloud platform;

[0051] Cloud platform-based resource monitoring and data analysis system, such as Figure 1 As shown, including the following:

[0052] Micro-isolation architecture module, used to divide the cloud platform into multiple micro-isolation areas;

[0053] A monitoring agent module, each monitoring agent is set in a micro-isolation area, and is used to collect resource data in the micro-isolation area, and the resource data includes computing resource data, storage resource data and network resource data. The monitoring agent can dynamically adjust the collection frequency and collection parameters according to resource usage;

[0054] The data analysis module is communicatively connected to the monitoring agent, and is used to receive the resource data and analyze the resource data using an adaptive analysis algorithm. The adaptive analysis algorithm can automatically select a suitable model for analysis based on the resource type and business attributes in the micro-isolation area, and dynamically update the parameters of the analysis model based on changes in the cloud platform business.

[0055] The micro-isolation architecture module is implemented through a software-defined network technology unit and a security policy implementation unit; the software-defined network unit separates the control plane from the data plane of the network through SDN, and uniformly manages the entire cloud platform network. For the communication between different micro-isolation areas, the SDN controller can set the flow control strategy, and as the business of the cloud platform changes, the SDN can dynamically adjust the network configuration and reallocate network resources; through SDN, different micro-isolation areas can have independent virtual network address spaces, and at the same time, in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources;

[0056] The security policy unit is applied in micro-isolation. The security policy unit can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocol, port number, source address and destination address. The security policy unit also includes intrusion detection and prevention mechanisms. Intrusion detection systems and intrusion prevention systems are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, and detect and prevent intrusion behavior.

[0057] The monitoring agent module includes a sensor unit, a preprocessing unit and a control unit; the sensor unit, in the micro-isolation area, collects information by interacting with the operating system kernel and the virtual machine hypervisor; in terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface; at the same time, the sensor will collect specific resource data in the area according to the resource usage mode and business needs of different micro-isolation areas;

[0058] The preprocessing unit includes data cleaning and format conversion. Data cleaning is used to clean abnormal data appearing in the resource monitoring data of the micro-isolation area. The data cleaning process sets thresholds and rules to identify and remove abnormal data. The format conversion function stores and subsequently analyzes all collected and preprocessed data in a unified format.

[0059] The data analysis module includes a model selection unit, an analysis execution unit and a parameter update unit; the model selection unit is a selection mechanism based on resource type and business attribute, and adopts an adaptive analysis algorithm to dynamically identify resources in the micro-isolation area according to resource type and business attribute, and analyze the usage characteristics of resources. The adaptive analysis algorithm will deeply understand the business attributes of the micro-isolation area. For areas with complex business logic, the adaptive analysis algorithm will analyze the dependency pattern of business processes on resources, and then select the analysis model based on the analysis results of resource type and business attribute;

[0060] The analysis execution unit, when the model selection unit selects the analysis model, will obtain data from the monitoring agent and input it into the selected model, and the model will process the data according to the internal algorithm and logic; during the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data, and the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of the resource data; in the preset thresholds and rules, for the micro-isolation area with relatively stable resource usage, the algorithm will simplify the analysis model, on the contrary, when the resources in the area are abnormal, the algorithm will introduce more analysis dimensions and algorithms to increase the complexity of the analysis model;

[0061] The parameter update unit triggers parameter update by monitoring changes in cloud platform services. The parameter update obtains information about changes in cloud platform services. When business changes are detected, the parameter update unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter update unit has a feedback-based learning mechanism and can learn from feedback on analysis results. In addition, the adaptive analysis algorithm can also perform predictive parameter updates based on historical business changes and trends in resource data.

[0062] Cloud platform-based resource monitoring and data analysis methods, such as Figure 2 As shown, the following steps are included:

[0063] Step 1: Use the micro-segmentation architecture to divide the cloud platform into multiple micro-segmentation areas;

[0064] Step 2: A monitoring agent is set in each micro-isolation area, and resource data in the micro-isolation area is collected through the monitoring agent. During the collection process, the collection frequency and collection parameters are dynamically adjusted according to resource usage;

[0065] Step 3: Transmit the collected data to the data analysis module;

[0066] Step 4: The data analysis module uses an adaptive analysis algorithm to analyze the resource data. The adaptive analysis algorithm automatically selects a suitable analysis model according to the resource type and business attributes in the micro-isolation area, and dynamically updates the parameters of the analysis model according to changes in the cloud platform business.

[0067] The use of micro-isolation architecture to divide the cloud platform into multiple micro-isolation areas further includes the following: micro-isolation is achieved through software-defined network technology and security policies. The software-defined network separates the control plane and data plane of the network through SDN, and uniformly manages the entire cloud platform network. For the communication between different micro-isolation areas, the SDN controller can set the flow control policy, and as the business of the cloud platform changes, the SDN can dynamically adjust the network configuration and reallocate network resources; through SDN, different micro-isolation areas can have independent virtual network address spaces, and in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources;

[0068] The security policy is applied in micro-isolation. The security policy can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocol, port number, source address and destination address. The security policy also includes intrusion detection and prevention mechanism. Intrusion detection system and intrusion prevention system are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, detect and prevent intrusion behavior.

[0069] The monitoring agent includes the following:

[0070] First, within the micro-isolation area, the sensor unit collects information by interacting with the operating system kernel and the virtual machine hypervisor. In terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface. At the same time, the sensor will collect specific resource data in the area according to the resource usage patterns and business needs of different micro-isolation areas.

[0071] Then, the preprocessing unit performs data cleaning and format conversion. It uses data cleaning methods to clean abnormal data in the resource monitoring data in the micro-isolation area. The data cleaning process sets thresholds and rules to identify and remove abnormal data. The format conversion function stores all collected and preprocessed data in a unified format for subsequent analysis.

[0072] Finally, the control unit is used to continuously monitor the resource load in the micro-isolation area and adjust the collection frequency according to the load in the micro-isolation area. In addition, the control unit will also adjust the sensor's collection parameters according to resource fluctuations. Resource fluctuations refer to large fluctuations in packet loss and bandwidth in the network resources in the micro-isolation area. For storage resources, it refers to abnormal fluctuations in read and write delays.

[0073] The data analysis module includes the following contents:

[0074] The data analysis module includes a model selection unit, an analysis execution unit and a parameter updating unit;

[0075] The model selection unit is a selection mechanism based on resource types and business attributes. For the cloud platform, the model selection unit uses an adaptive analysis algorithm to dynamically identify resources in the micro-isolation area according to resource types and business attributes, analyze resource usage characteristics, and the adaptive analysis algorithm will deeply understand the business attributes of the micro-isolation area. For areas with complex business logic, the adaptive analysis algorithm will analyze the dependency pattern of business processes on resources, and then select an analysis model based on the analysis results of resource types and business attributes;

[0076] Then, when the model selection unit selects the analysis model, the analysis execution unit will obtain data from the monitoring agent and input it into the selected model. The model will process the data according to the internal algorithm and logic. During the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data. At the same time, the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of the resource data. In the preset thresholds and rules, for micro-isolation areas with relatively stable resource usage, the algorithm will simplify the analysis model. On the contrary, when abnormal conditions occur in the resources in the area, the algorithm will introduce more analysis dimensions and algorithms to increase the complexity of the analysis model.

[0077] Finally, a parameter update unit is used to trigger parameter updates by monitoring changes in cloud platform services. Parameter updates obtain information about changes in cloud platform services. When business changes are detected, the parameter update unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter update unit has a feedback-based learning mechanism and can learn from the feedback of analysis results. In addition, the adaptive analysis algorithm can also perform predictive parameter updates based on historical business changes and trends in resource data.

[0078] For example, in an enterprise with hundreds of employees, employees access office applications, store and process work files through virtual desktops.

[0079] 1. Application of micro-isolation areas in virtual desktops:

[0080] 1. Division of micro-isolation areas:

[0081] Divide micro-isolation areas according to different user groups or business departments. For example, divide the virtual desktops of the company's finance department into one micro-isolation area, and the R&D department into another area. The virtual desktops in each area have similar resource requirements and security requirements. Isolation between areas is achieved through software-defined networking (SDN) technology and security policies. The SDN controller configures access control rules to allow only specific network traffic to flow between areas. For example, the virtual desktop of the finance department can only communicate with network resources related to the company's internal financial system, while the virtual desktop of the R&D department can interact with related network resources such as code repositories and test environments.

[0082] In each micro-isolation area, virtual desktops are further subdivided according to their functions or application types. For example, in the micro-isolation area of ​​the R&D department, virtual desktops running development tools and virtual desktops running test environments are divided into smaller sub-areas to achieve more refined resource management and security control.

[0083] 2. Resource allocation and security strategy

[0084] Allocate computing resources, such as the number of CPU cores and memory size, to each micro-isolation area. For the virtual desktop of the finance department, you may allocate fewer but stable computing resources because it mainly runs office software and financial applications; while the virtual desktop of the R&D department may require more computing resources, especially for running complex development environments and compilation tasks.

[0085] In terms of storage resources, separate storage volumes or storage quotas are allocated to different areas. The virtual desktop storage of the finance department is mainly used to store financial statements and related documents, and corresponding access rights and storage capacity restrictions are set; the R&D department requires larger storage capacity to store code bases, test data, etc. Security policies ensure that only authorized virtual desktops can access the storage resources of their respective areas to prevent data leakage.

[0086] In terms of network resources, specific network bandwidth and network interfaces are allocated to each micro-isolation area. For example, the virtual desktop of the design department may require a higher network bandwidth to transmit graphic files and design resources, while the virtual desktop of the administrative department has a relatively lower network bandwidth requirement. Through network traffic shaping and priority setting, the network performance of important businesses is guaranteed.

[0087] 2. Deployment of monitoring agents in virtual desktop micro-isolation areas

[0088] 1. Sensor unit data

[0089] Sensor units are deployed on server nodes in the micro-isolation area of ​​each virtual desktop. For computing resources, the sensor unit interacts with virtual machine hypervisors such as VMware vSphere and Hyper-V to collect data such as CPU usage, memory usage, and disk I / O wait time for each virtual desktop instance. For example, in the virtual desktop area of ​​the R&D department, when developers compile large code projects, the sensor unit can capture the peak of CPU usage and memory consumption in real time.

[0090] For storage resources, the sensor unit monitors the read and write operations of the storage volume, available space, allocation of storage blocks, etc. In the virtual desktop area of ​​the finance department, the storage and reading frequency of financial documents and the change trend of storage capacity can be recorded to promptly detect abnormal conditions of storage resources, such as insufficient storage capacity or frequent storage errors.

[0091] In terms of network resources, the sensor unit collects network traffic information between the virtual desktop and the external network or other internal resources, including bandwidth utilization, packet loss rate, network latency, etc. For virtual desktop areas that often use video conferencing or online collaboration tools, the collection of network performance data helps analyze network bottleneck problems.

[0092] 2. Preprocessing unit processes data

[0093] The preprocessing unit processes the raw data collected by the sensor unit. For computing resource data, it converts CPU and memory usage data in different formats into a standard format, such as converting usage data expressed in percentages into values ​​between 0 and 1, to facilitate subsequent analysis and comparison. At the same time, it removes erroneous data caused by temporary sensor failures or temporary abnormalities in virtual machines, such as unreasonable extremely high or extremely low CPU usage values.

[0094] In the storage resource data processing, the preprocessing unit unifies the format of the read and write operation timestamps of the storage blocks and unifies the units of storage capacity (such as converting them all into bytes). For network resource data, it integrates and formats the traffic data collected from different network interfaces to ensure data consistency and accuracy.

[0095] 3. The control unit adjusts the acquisition parameters

[0096] The control unit adjusts the collection parameters according to the resource load and fluctuation in the virtual desktop micro-isolation area. During office hours, when most employees log in to the virtual desktop and start working, the resource load of the virtual desktop increases. The control unit increases the frequency of the sensor unit's collection of computing resource and network resource data in order to understand the changes in resource usage in more detail. For example, the collection frequency of CPU usage may be increased from once every 5 minutes to once every 1 minute.

[0097] When it detects large resource fluctuations in a virtual desktop area, such as large fluctuations in network traffic during large-scale code deployment in the R&D department, the control unit will adjust the depth of network resource data collection and increase the collection of network protocol layer data to better analyze the cause of network performance problems. During low-load periods, such as non-working hours at night, the control unit will appropriately reduce the collection frequency to reduce the occupation of system resources.

[0098] 3. Operation of the Data Analysis Module in Virtual Desktop Scenarios

[0099] 1. Model selection unit adaptation model

[0100] The model selection unit selects a suitable analysis model based on the resource type and business attributes of the virtual desktop micro-isolation area. For the storage-intensive area where the virtual desktops of the finance department are located, the big data storage feature analysis model is selected. This model analyzes the distribution pattern of financial file storage, the relationship between access frequency and storage performance to optimize the configuration of storage resources. For example, if the analysis finds that frequent access to certain financial statements leads to a decrease in storage performance, it can be recommended to migrate these files to a higher-speed storage medium or optimize the storage index.

[0101] In the computing-intensive virtual desktop area of ​​the R&D department, the model selection unit selects the computing task feature analysis model. This model evaluates the efficiency of developers' use of computing resources in the process of compiling code, running test cases, etc. For example, when it is found that the CPU usage rate of a development project is too high for a long time and the compilation time is too long during the compilation process, it is analyzed whether it is a code algorithm problem or a problem of insufficient resource allocation, providing a basis for optimizing computing resource allocation.

[0102] For virtual desktop areas that have both computing and storage requirements, such as the design department (which needs to run graphic design software and store a large number of design files), a comprehensive analysis model is selected to comprehensively consider the performance impact of computing resources on graphics processing and the storage management of design files by storage resources to achieve overall resource optimization.

[0103] 2. Analysis execution unit performs analysis

[0104] The analysis execution unit uses the selected analysis model to analyze resource data. In the virtual desktop environment, for the storage resource analysis of the finance department, the big data storage feature analysis model is used to analyze the time series data of storage access, determine the peak and trough usage periods of storage resources, and the differences in the demand for storage resources by different financial applications. Based on these analysis results, a dynamic allocation strategy for storage resources can be formulated, such as limiting the storage operations of non-critical financial applications during peak hours to ensure the storage performance of core financial business.

[0105] In the computing resource analysis of the R&D department, the analysis execution unit uses the computing task feature analysis model to analyze the CPU and memory usage patterns of different development projects on the virtual desktop. For example, for a large software development project, the analysis of the changes in the demand for computing resources in different development stages (such as coding, debugging, and testing) found that the demand for memory resources is higher in the debugging stage, so more memory resources can be allocated to the virtual desktop in this stage to improve development efficiency.

[0106] 3. Parameter update unit dynamically updates parameters

[0107] The parameter update unit updates the parameters of the analysis model according to changes in the virtual desktop business. When the finance department introduces new financial software or updates the existing software version, the storage format or access method of financial files may change. The parameter update unit will adjust the parameters related to file format and access mode in the big data storage feature analysis model accordingly to ensure that the analysis results still accurately reflect the use of storage resources.

[0108] For example, a new programming language may have different requirements for memory management. The parameter update unit adjusts the model parameters by analyzing the characteristics of the new language and the resource data of the virtual desktop when running the new code to better evaluate and optimize the allocation of computing resources.

[0109] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A cloud-based resource monitoring and data analysis system, characterized by: Includes the following: Micro-isolation architecture module, used to divide the cloud platform into multiple micro-isolation areas; A monitoring agent module, each monitoring agent is set in a micro-isolation area, and is used to collect resource data in the micro-isolation area, and the resource data includes computing resource data, storage resource data and network resource data. The monitoring agent can dynamically adjust the collection frequency and collection parameters according to resource usage; The data analysis module is communicatively connected to the monitoring agent, and is used to receive the resource data and analyze the resource data using an adaptive analysis algorithm. The adaptive analysis algorithm can automatically select a suitable model for analysis based on the resource type and business attributes in the micro-isolation area, and dynamically update the parameters of the analysis model based on changes in the cloud platform business.

2. The cloud platform-based resource monitoring and data analysis system according to claim 1, characterized in that: The micro-isolation architecture module is implemented through a software-defined network technology unit and a security policy implementation unit; the software-defined network unit separates the control plane from the data plane of the network through SDN, and uniformly manages the entire cloud platform network. For the communication between different micro-isolation areas, the SDN controller can set the flow control strategy, and as the business of the cloud platform changes, the SDN can dynamically adjust the network configuration and reallocate network resources; through SDN, different micro-isolation areas can have independent virtual network address spaces, and at the same time, in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources; The security policy unit is applied in micro-isolation. The security policy unit can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocol, port number, source address and destination address. The security policy unit also includes intrusion detection and prevention mechanisms. Intrusion detection systems and intrusion prevention systems are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, and detect and prevent intrusion behavior.

3. The cloud platform-based resource monitoring and data analysis system according to claim 1, characterized in that: The monitoring agent module includes a sensor unit, a preprocessing unit and a control unit; the sensor unit, in the micro-isolation area, collects information by interacting with the operating system kernel and the virtual machine hypervisor; In terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface; at the same time, the sensor will collect specific resource data in different micro-isolation areas according to the resource usage patterns and business needs of different micro-isolation areas; The pre-processing unit includes data cleaning and format conversion, wherein data cleaning is used to clean abnormal data appearing in resource monitoring data in the micro-isolation area, and the data cleaning process sets thresholds and rules to identify and remove abnormal data; The format conversion function will store all collected and pre-processed data in a unified format for subsequent analysis; The control unit will continuously monitor the resource load in the micro-isolation area and adjust the collection frequency according to the load in the micro-isolation area. In addition, the control unit will also adjust the collection parameters of the sensor according to the fluctuation of resources. Resource fluctuation refers to the large fluctuation of packet loss and bandwidth of network resources in the micro-isolation area. For storage resources, it refers to abnormal fluctuation of read and write delays.

4. The cloud platform-based resource monitoring and data analysis system according to claim 1, characterized in that: The data analysis module includes a model selection unit, an analysis execution unit and a parameter updating unit; the model selection unit is a selection mechanism based on resource type and business attribute, and adopts an adaptive analysis algorithm to dynamically identify resources in the micro-isolation area according to resource type and business attribute, and analyze the usage characteristics of resources. For areas with complex business logic, the adaptive analysis algorithm will analyze the dependency pattern of business processes on resources, and then select the analysis model based on the analysis results of resource type and business attribute; The analysis execution unit, when the model selection unit selects the analysis model, will obtain data from the monitoring agent and input it into the selected model, and the model will process the data according to the internal algorithm and logic; during the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data, and the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of the resource data; in the preset thresholds and rules, for the micro-isolation area with relatively stable resource usage, the algorithm will simplify the analysis model, on the contrary, when the resources in the area are abnormal, the algorithm will introduce more analysis dimensions and algorithms to make the model more complicated; The parameter updating unit triggers parameter updating by monitoring changes in cloud platform services. The parameter updating obtains information about changes in cloud platform services. When a service change is detected, the parameter updating unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter updating unit has a feedback-based learning mechanism and can learn from the feedback of the analysis results. In addition, adaptive analysis algorithms can also perform predictive parameter updates based on historical business changes and trends in resource data.

5. A cloud platform-based resource monitoring and data analysis method, characterized in that: The following steps are involved: Step 1: Use the micro-segmentation architecture to divide the cloud platform into multiple micro-segmentation areas; Step 2: A monitoring agent is set in each micro-isolation area, and resource data in the micro-isolation area is collected through the monitoring agent. During the collection process, the collection frequency and collection parameters are dynamically adjusted according to resource usage; Step 3: Transmit the collected data to the data analysis module; Step 4: The data analysis module uses an adaptive analysis algorithm to analyze the resource data. The adaptive analysis algorithm automatically selects a suitable analysis model according to the resource type and business attributes in the micro-isolation area, and dynamically updates the parameters of the analysis model according to changes in the cloud platform business.

6. The cloud platform-based resource monitoring and data analysis method according to claim 5, characterized in that: The method of dividing the cloud platform into multiple micro-isolation areas using the micro-isolation architecture further includes the following contents: Micro-isolation is achieved through software-defined network technology and security policies. The software-defined network separates the control plane from the data plane of the network through SDN, and uniformly manages the entire cloud platform network. For the communication between different micro-isolation areas, the SDN controller can set the flow control policy, and as the business of the cloud platform changes, SDN can dynamically adjust the network configuration and reallocate network resources; through SDN, different micro-isolation areas can have independent virtual network address spaces, and in the process of computing resources accessing storage resources, SDN can control that only authorized computing resources can communicate with specific storage resources; The security policy is applied in micro-isolation. The security policy can set firewall rules at the boundary of each micro-isolation area. These rules can allow or deny network traffic based on network protocol, port number, source address and destination address. The security policy also includes intrusion detection and prevention mechanism. Intrusion detection system and intrusion prevention system are deployed in each micro-isolation area and at the boundary. These systems are used to monitor network traffic and system behavior in real time, detect and prevent intrusion behavior.

7. The cloud platform-based resource monitoring and data analysis method according to claim 5, characterized in that: The monitoring agent includes the following: First, within the micro-isolation area, the sensor unit collects information by interacting with the operating system kernel and the hypervisor; In terms of storage resources, the sensor unit can collect data through the storage device driver interface and the file system related interface; at the same time, the sensor will collect specific resource data in different micro-isolation areas according to the resource usage patterns and business needs of different micro-isolation areas; Then, the preprocessing unit performs data cleaning and format conversion. It uses data cleaning methods to clean abnormal data in the resource monitoring data in the micro-isolation area. The data cleaning process sets thresholds and rules to identify and remove abnormal data. The format conversion function stores all collected and preprocessed data in a unified format for subsequent analysis. Finally, the control unit is used to continuously monitor the resource load in the micro-isolation area and adjust the collection frequency according to the load in the micro-isolation area. In addition, the control unit will also adjust the sensor's collection parameters according to resource fluctuations. Resource fluctuations refer to large fluctuations in packet loss and bandwidth in the network resources in the micro-isolation area. For storage resources, it refers to abnormal fluctuations in read and write delays.

8. The cloud platform-based resource monitoring and data analysis method according to claim 5, characterized in that: The data analysis module includes the following contents: The data analysis module includes a model selection unit, an analysis execution unit and a parameter updating unit; The model selection unit is a selection mechanism based on resource types and business attributes. For cloud platforms, the model selection unit uses an adaptive analysis algorithm to dynamically identify resources in micro-isolation areas based on resource types and business attributes, and analyze resource usage characteristics. For areas with complex business logic, the adaptive analysis algorithm analyzes the dependency pattern of business processes on resources, and then selects an analysis model based on the analysis results of resource types and business attributes. Then, when the model selection unit selects the analysis model, the analysis execution unit will obtain data from the monitoring agent and input it into the selected model. The model will process the data according to the internal algorithm and logic. During the execution process, the adaptive analysis algorithm will adjust the parameters of the analysis model according to the real-time resource data. At the same time, the algorithm can also adjust the complexity of the analysis model according to the complexity and volatility of the resource data. In the preset thresholds and rules, for micro-isolation areas with relatively stable resource usage, the algorithm will simplify the analysis model. On the contrary, when abnormal conditions occur in the resources in the area, the algorithm will introduce more analysis dimensions and algorithms to make the model more complex. Finally, a parameter update unit is used to trigger parameter updates by monitoring changes in cloud platform services. The parameter update obtains information about changes in cloud platform services. When a service change is detected, the parameter update unit adjusts the parameters of the analysis model according to preset rules and algorithms. The adaptive analysis algorithm of the parameter update unit has a feedback-based learning mechanism and can learn from the feedback of the analysis results. In addition, adaptive analysis algorithms can also perform predictive parameter updates based on historical business changes and trends in resource data.

Images