Big data-oriented hybrid cloud security storage management system

Abstract

The invention discloses a big-data-oriented hybrid cloud secure storage management system, and relates to the technical field of hybrid cloud storage management, and the system comprises a hybrid cloud management center which is in communication connection with the following modules: a hybrid cloud data storage module which is used for integrating private cloud and public cloud storage resources, and storing the private cloud and public cloud storage resources; a unified hybrid cloud resource pool is formed, and data is stored and distributed according to service requirements; and the resource monitoring and analyzing module is used for monitoring and analyzing the access flow and the resource use state of the private cloud and the public cloud in real time. By integrating the storage resources of the private cloud and the public cloud, a uniform mixed cloud resource pool is formed, the resource barrier between cloud environments is broken, the efficient integration and uniform allocation of the storage resources are realized, the resource utilization rate is improved, enterprises can more flexibly cope with the change of business requirements, and the business efficiency is improved. And the optimal configuration of the storage resources is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid cloud storage management, and in particular to a hybrid cloud security storage management system for big data. Background Art

[0002] With the advent of the digital age, the amount of data has exploded, and big data technology has become a key force driving the development of various industries. Various fields face many challenges in processing, storing and analyzing massive amounts of data. Traditional single storage architecture, whether it is local storage built by the enterprise or public cloud storage, is difficult to meet the multiple requirements of big data storage such as high capacity, high performance, high reliability and scalability.

[0003] For example, the hybrid cloud security storage management system for big data with Chinese patent publication number: CN109873784A includes: a private cloud module, a public cloud module, a management layer module and a client module; wherein, the client module communicates data and tasks with the private cloud module and the management layer module through a standard S3 interface.

[0004] In the prior art, a hybrid cloud architecture of private cloud and public cloud is used, and a streaming encryption algorithm is used for storage encryption. Inter-cloud migration is transparent to users, and the encryption state is maintained without being destroyed during the migration process, which solves the problem that the existing technical means can no longer restrict the illegal use of user information. However, since private cloud and public cloud resources are prone to contention during high-concurrency access, it will lead to system performance degradation and resource bottlenecks, which in turn affect the normal operation of the business. Therefore, how to comprehensively analyze the traffic pattern, performance and cost of the access process to allocate appropriate priorities and avoid unnecessary expenditures to ensure the normal operation of the business is the problem to be solved by the present invention. To this end, a hybrid cloud security storage management system for big data is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a hybrid cloud security storage management system for big data to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is: A hybrid cloud security storage management system for big data includes a hybrid cloud management center, wherein the hybrid cloud management center is communicatively connected to the following modules, wherein: The hybrid cloud data storage module is used to integrate the storage resources of private and public clouds to form a unified hybrid cloud resource pool. Data is stored in the hybrid cloud resource pool and distributed according to business needs. Various storage devices and spaces are centrally managed and integrated. The resource monitoring and analysis module is used to monitor the access traffic of private clouds and public clouds, and monitor and analyze the resource usage status of private clouds and public clouds to understand the resource usage status and performance; The performance cost evaluation module is used to calculate the cost-performance ratio of each business in private and public cloud storage environments based on the business's access patterns, traffic patterns, and resource usage, and analyze the degree of compatibility between each business and different cloud environments. The priority allocation module is used to assign corresponding priorities to different services based on the degree of compatibility between each service and different cloud environments output by the performance cost evaluation module, and define service resource allocation strategies for different priorities to ensure that key services have sufficient resources when accessed concurrently. The resource allocation and scheduling module is used to dynamically allocate private cloud and public cloud resources based on access traffic, resource usage status, and priority, optimize resource allocation strategies, and balance performance and cost.

[0007] A further improvement of the technical solution of the present invention is that the hybrid cloud data storage module specifically includes: The system automatically scans storage devices and spaces in private and public cloud environments, accesses storage resources in private and public clouds through standardized interfaces, and automatically identifies storage types, capacities, and performance parameters of different cloud platforms. It also categorizes and labels storage resources, distinguishing between high-performance and large-capacity storage resources. Based on the accessed storage resource information, the storage resources of private cloud and public cloud are integrated to build a unified hybrid cloud resource pool; Develop appropriate data storage and distribution strategies based on business needs and data characteristics. Furthermore, further refine storage strategies based on data access frequency and security requirements to ensure data is stored in the appropriate storage location at the appropriate time and in the appropriate manner. According to the established data storage and distribution strategy, business data is stored in the hybrid cloud resource pool, and data is distributed according to business needs.

[0008] A further improvement of the technical solution of the present invention is that: the resource monitoring and analysis module includes an access flow monitoring and analysis unit and a resource usage status perception unit; The access traffic monitoring and analysis unit is used to monitor the access traffic of private clouds and public clouds in real time, analyze traffic patterns, identify high-concurrency access periods and resource contention hotspots, provide real-time traffic data, and help identify potential performance bottlenecks; The resource usage status perception unit is used to dynamically perceive the resource usage status of the private cloud and the public cloud, form a resource usage view, and analyze the resource usage status and performance.

[0009] A further improvement of the technical solution of the present invention is that the access traffic monitoring and analysis unit specifically includes: Deploy traffic monitoring probes (eBPF-based API hooks) in private and public cloud environments to capture real-time traffic data at the network layer (TCP / UDP packet headers), application layer (HTTP / database protocols), and storage layer (block / object storage operations). Simultaneously, using the cloud platform's native monitoring interface (Prometheus), we collect VM / container-level traffic metrics. All data is then standardized to build a comprehensive traffic view covering the entire process. Based on historical traffic data aggregated by time window, a traffic baseline model is constructed using time series analysis to identify periodic patterns. Traffic is then classified using the K-Means clustering algorithm to distinguish between normal business access, burst traffic, and abnormal traffic. Traffic features, including request rate, data volume, and access source IP entropy, are extracted to generate a traffic feature fingerprint library for real-time matching and anomaly detection. Based on real-time traffic data, a sliding window algorithm is used to calculate the number of concurrent requests per unit time. This is combined with preset threshold rules to trigger high-concurrency alerts. Heat map analysis technology is used to aggregate traffic data by storage device, API interface, and user group to identify resource contention hotspots. Furthermore, resource usage data is correlated to locate potential performance bottlenecks. The analyzed traffic patterns and performance bottleneck information are pushed to the hybrid cloud management center via the API, and real-time traffic data is displayed through a visual interface to provide an intuitive understanding of the system operation status.

[0010] A further improvement of the technical solution of the present invention is that the resource usage status sensing unit specifically includes: Through lightweight agents deployed in private and public cloud environments, real-time CPU, memory, storage, and network resource usage data is collected. At the same time, cloud platform native APIs are called to obtain performance data at the virtual machine / container level. Multi-source heterogeneous data is cleaned, fields are aligned (using unified timestamps and naming conventions), and units are converted to form a standardized JSON-formatted resource base dataset. Based on the standardized data of the resource base dataset, we conduct aggregate analysis by cloud platform, business system, and resource type. We generate resource usage views through time series aggregation, including real-time resource utilization heat maps, resource capacity trend curves, and cross-cloud resource distribution maps. Combined with resource usage views, adaptive thresholds based on historical baselines are used to identify abnormal resource status. Correlation analysis technology is used to cross-validate resource performance data with business loads, analyzing performance and locating the root causes of performance bottlenecks. The analyzed resource usage status and performance are pushed to the hybrid cloud management center via the RESTful API, and the resource usage view is displayed through a visual interface to intuitively understand the resource usage and performance status.

[0011] A further improvement of the technical solution of the present invention is that the performance cost evaluation module specifically includes: Through API integration, access resource monitoring and analysis modules collect data on access patterns, traffic patterns, and resource usage status for each business, build business profiles, and annotate key feature tags. At the same time, private and public cloud billing models and cost data, including storage fees, network fees, and computing resource fees, are collected from the cloud platform to form a complete business performance and cost data set. Based on the integrated business performance and cost data sets, the performance of different businesses in private and public cloud storage environments is evaluated. Performance indicators such as response time, throughput, and I / O operation latency are analyzed. Baseline values for each performance indicator are set based on the targets for each in private and public cloud environments. Furthermore, a performance difference matrix is generated based on the business access and traffic patterns, identifying performance bottlenecks in the public cloud caused by network latency. Evaluate the cost expenditures of different businesses in private and public cloud storage environments, analyze the resource usage of businesses in different cloud environments, including storage capacity, network bandwidth, and computing resources. Combined with the cloud platform's billing model, calculate the cost expenditures of businesses in different cloud environments. By comparing cost data from different cloud environments, determine the cost differences between businesses in private and public clouds. Combining the results of performance and cost evaluation, calculate the cost-performance ratio of each business in private cloud and public cloud storage environments. By calculating the cost-performance ratio, analyze the degree of compatibility between each business and different cloud environments.

[0012] A further improvement of the technical solution of the present invention is that: the calculation process of the cost performance of each business in a private cloud storage environment is: For the selected business, traverse all its performance indicators in the private cloud environment, divide each performance indicator value by the corresponding benchmark value to obtain the normalized performance indicator ratio; All normalized performance index ratios are squared and summed, and the square root of the sum is taken to obtain a representative value of the comprehensive performance index. The reciprocal of the representative value of the comprehensive performance index is then taken to obtain the performance measurement index, i.e., the better the performance, the larger the value. Calculate the ratio of the cost of the selected business in the private cloud environment to the average cost of all businesses in the private cloud environment, add 1 to the ratio and take the natural logarithm to obtain the cost measurement index; Multiply the calculated performance metric by the cost metric to get the price / performance ratio of the selected business in the private cloud environment. The calculation process of the cost-performance ratio of each business in the public cloud storage environment is as follows: For the selected business, traverse all its performance indicators in the public cloud environment, divide each performance indicator value by the corresponding benchmark value, and obtain the normalized performance indicator ratio; All normalized performance index ratios are squared and summed, and the square root of the sum is taken to obtain a representative value of the comprehensive performance index. The reciprocal of the representative value of the comprehensive performance index is then taken to obtain the performance measurement index, i.e., the better the performance, the larger the value. Calculate the ratio of the cost of the selected business in the public cloud environment to the average cost of all businesses in the public cloud environment, add 1 to the ratio and take the natural logarithm to obtain the cost measurement index; Multiply the calculated performance metric by the cost metric to obtain the price / performance ratio of the selected business in the public cloud environment.

[0013] A further improvement of the technical solution of the present invention is that the priority allocation module specifically includes: The performance-cost evaluation module receives the price / performance ratio results of each business in the private cloud and public cloud environments, assigns weight coefficients to the price / performance ratio of each business in the private cloud and public cloud environments based on the business type, and then calculates the weighted sum to obtain the matching coefficient. A matching threshold is preset to preliminarily divide the adaptation relationship between the business and the cloud environment and generate a business-cloud environment matching matrix. Based on the matching analysis results, the business is divided into three levels of priority: high priority, medium priority, and low priority; Define resource allocation rules for businesses of different priorities, and automate the execution of rules through a policy engine. High-priority services are allocated to a private cloud-exclusive resource pool, with 30% redundant capacity reserved and QoS enabled. Medium-priority services use a public cloud elastic resource pool, with resource quota limits set and dynamic scheduling implemented. Low-priority services use public cloud on-demand instances or Spot instances, and resource preemption policies are configured.

[0014] A further improvement of the technical solution of the present invention is that the resource allocation and scheduling module specifically includes: Through API interface and system integration, it obtains access traffic data and resource usage status data from the resource monitoring and analysis module, and obtains the priority information of each business from the business priority allocation module, integrating them to form a comprehensive resource allocation decision data set; Based on the integrated data, we analyze the resource requirements of each business. We analyze changes in resource requirements over different time periods based on access traffic data, particularly peak resource requirements during periods of high concurrent access. Combined with resource usage data, we determine current resource usage and remaining capacity, analyze whether there are resource bottlenecks, and determine the urgency of each business's resource needs based on business priorities. Based on the results of resource demand analysis and the formulated resource allocation strategy, for high-priority businesses, private cloud resources are allocated first, and a certain amount of redundant capacity is reserved to ensure that critical businesses can obtain sufficient resource guarantees during high-concurrency access. For medium-priority businesses, public cloud elastic resource pools are used to dynamically adjust resource allocation according to business load, set resource quota limits to avoid excessive resource occupation, and for low-priority businesses, public cloud on-demand instances or Spot instances are used, and resource preemption strategies are configured to prioritize the resource needs of high-priority businesses, dynamically allocate private cloud and public cloud resources, and monitor resource usage and business operation status in real time during the resource allocation process, and dynamically adjust resource allocation according to actual needs.

[0015] Due to the adoption of the above technical solution, the present invention has the following technical advancements compared to the prior art: The present invention provides a hybrid cloud security storage management system for big data. By integrating the storage resources of private clouds and public clouds to form a unified hybrid cloud resource pool, it breaks down the resource barriers between cloud environments and realizes the efficient integration and unified allocation of storage resources. It not only improves resource utilization, but also enables enterprises to respond more flexibly to changes in business needs and ensures the optimal configuration of storage resources.

[0016] The present invention provides a hybrid cloud security storage management system for big data. It combines the access mode, traffic mode and resource usage status of the business to intelligently evaluate the cost-effectiveness of each business in private cloud and public cloud storage environments. Through quantitative analysis, it can clarify the cost-effectiveness of each business in different cloud storage environments, thereby optimizing resource allocation strategies, balancing performance and cost, and achieving maximum resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2 Schematic diagram of the system function modules of the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] Example 1, as Figure 1 、 Figure 2 As shown, the present invention provides a hybrid cloud security storage management system for big data, including a hybrid cloud management center, which is communicatively connected to the following modules, wherein: The hybrid cloud data storage module is used to integrate the storage resources of private cloud and public cloud to form a unified hybrid cloud resource pool, store data in the hybrid cloud resource pool, distribute data according to business needs, centrally manage and integrate various storage devices and spaces, break the resource barriers between cloud environments, realize efficient integration and unified allocation of storage resources, improve resource utilization, and the system automatically scans the storage devices and spaces in private cloud and public cloud environments, and accesses the storage resources of private cloud and public cloud through standardized interfaces, automatically identifies the storage type, capacity and performance parameters of different cloud platforms, and classifies and labels the storage resources to distinguish high-performance storage and large-capacity storage resources. Based on the accessed storage resource information, the storage resources of private cloud and public cloud are integrated to build a unified hybrid cloud resource pool, among which, virtualization technology is used to open Break down the resource barriers between cloud environments, so that the storage resources of different cloud environments form a logical whole, and establish a unified resource management framework to incorporate various storage devices and spaces, realize the centralized management of storage resources, and formulate corresponding data storage and distribution strategies based on business needs and data characteristics. Specifically, business data that requires high-performance processing is preferentially allocated to the high-performance storage resources of the private cloud, and business data with large data volumes but low real-time requirements is allocated to the large-capacity storage resources of the public cloud. At the same time, based on the access frequency and security requirements of the data, further refine the storage strategy to ensure that data can be stored in the appropriate storage location at the appropriate time and in the appropriate manner. According to the formulated data storage and distribution strategy, business data is stored in the hybrid cloud resource pool and distributed according to business needs. The resource monitoring and analysis module is used to monitor the access traffic of private clouds and public clouds, and monitor and analyze the resource usage status of private clouds and public clouds to understand the resource usage status and performance. The resource monitoring and analysis module includes an access traffic monitoring and analysis unit and a resource usage status perception unit; Among them, the access traffic monitoring and analysis unit is used to monitor the access traffic of private clouds and public clouds in real time, analyze traffic patterns, identify high-concurrency access periods and resource contention hotspots, provide real-time traffic data, help identify potential performance bottlenecks, and deploy traffic monitoring probes (eBPF-based API hooks) in private and public cloud environments to capture traffic data at the network layer (TCP / UDP packet header), application layer (HTTP / database protocol) and storage layer (block / object storage operations) in real time. At the same time, combined with the cloud platform's native monitoring interface (Prometheus), it synchronously collects traffic indicators at the virtual machine / container level, and then standardizes all data to build a traffic view covering the entire link. Based on historical traffic data aggregated by time windows, time series analysis is used to build a traffic baseline model to identify periodic patterns. And use the K-Means clustering algorithm to classify traffic, distinguish normal business access, burst traffic and abnormal traffic, extract traffic features including request rate, data volume, and access source IP entropy value, and generate a traffic feature fingerprint library for real-time matching and anomaly detection. Based on real-time traffic data, the sliding window algorithm is used to calculate the number of concurrent requests per unit time, and high concurrency alarms are triggered in combination with preset threshold rules (historical baseline + 3 times standard deviation). Heat map analysis technology is used to aggregate traffic data by storage device, API interface and user group dimensions to identify resource contention hotspots. At the same time, resource usage data is associated to locate potential performance bottlenecks. The traffic pattern and performance bottleneck information obtained from the analysis are pushed to the hybrid cloud management center through the API, and real-time traffic data is displayed through a visual interface to intuitively understand the system operation status. The resource usage status perception unit is used to dynamically perceive the resource usage status of private and public clouds, form a resource usage view, and analyze resource usage status and performance. Through the lightweight agent (a low-resource usage probe developed based on the Go language) deployed in private and public cloud environments, it collects resource usage status data of CPU (usage rate, load), memory (cache / buffer occupancy), storage (IOPS, latency) and network (bandwidth, packet loss rate) in real time. At the same time, it calls the cloud platform native API (AWS CloudWatch Metrics) to obtain performance data at the virtual machine / container granularity, and clean, align fields (unify timestamps and naming conventions), and convert units of multi-source heterogeneous data to form a standardized JSON-formatted resource base dataset. Based on the standardized data in the resource base dataset, aggregation analysis is performed by cloud platform, business system, and resource type. Resource usage views are generated through time series aggregation, including real-time resource utilization heat maps, resource capacity trend curves, and cross-cloud resource distribution maps. Combined with resource usage views, adaptive thresholds based on historical baselines are used to identify abnormal resource status. Correlation analysis technology is used to cross-validate resource performance data with business load, analyze performance, and locate the root causes of performance bottlenecks. The analyzed resource usage status and performance indicators are pushed to the hybrid cloud management center via RESTful APIs. The resource usage view is displayed through a visual interface to intuitively understand resource usage and performance status. The performance cost evaluation module is used to combine the business access mode, traffic mode and resource usage status to calculate the cost-effectiveness of each business in the private cloud and public cloud storage environments, analyze the matching degree of each business with different cloud environments, and help the system clarify the cost-effectiveness of each business in different cloud storage environments. Through API integration, the access resource monitoring and analysis module is collected to collect data including the access mode, traffic mode and resource usage status of each business, build a business portrait, and mark key feature tags. At the same time, the billing model and cost data of private cloud and public cloud are collected from the cloud platform, including storage fees, network fees and computing resource fees, and integrated to form a complete business performance and cost data set. Among them, the access mode and traffic mode data of the business are obtained from the access traffic monitoring and analysis unit, and the resource usage status data is obtained from the resource usage status perception unit. Based on the data of the integrated business performance and cost data set, the performance of different businesses in private cloud and public cloud storage environments is evaluated, and the performance indicators of the business response time, throughput and I / O operation delay are analyzed. According to the performance indicators, the performance of the private cloud environment and the public cloud environment is compared. Using cloud environment goals, benchmark values for each performance indicator are set. Then, based on the business's access and traffic patterns, a performance difference matrix is generated, identifying performance bottleneck scenarios in the public cloud caused by network latency. For response time, local storage latency in the private cloud is <1ms, while cross-region transmission latency in the public cloud is 50ms. For throughput, the private cloud's all-flash array supports 2GB / s, while public cloud object storage is limited to 500MB / s. For I / O latency, it is <5ms under low load in the private cloud, while >20ms under high concurrency in the public cloud. The cost expenditures of different businesses in private and public cloud storage environments are evaluated, and the resource usage of businesses in different cloud environments, including storage capacity, network bandwidth, and computing resources, is analyzed. Based on the cloud platform's billing model, the cost expenditures of businesses in different cloud environments are calculated. By comparing cost data in different cloud environments, the cost differences between businesses in private and public clouds are determined. Combining the results of performance and cost evaluations, the price / performance ratio of each business in private and public cloud storage environments is calculated. This price / performance ratio calculation is used to analyze the degree of compatibility between each business and different cloud environments. Calculation process for the cost-performance of each business in a private cloud storage environment: For the selected business, traverse all its performance indicators in the private cloud environment, divide each performance indicator value by the corresponding benchmark value to obtain a normalized performance indicator ratio. The benchmark value is set according to the target of the performance indicator in the private cloud environment to make different performance indicators comparable. All performance indicators are normalized to obtain a set of normalized performance indicator ratios. All the normalized performance indicator ratios are squared and the square values are summed. The comprehensive performance of the business in the private cloud environment is analyzed based on the summation result. The smaller the value, the better the performance. The square root of the summation result is taken to obtain a representative value of the comprehensive performance indicator. The inverse of the representative value of the comprehensive performance indicator is taken to obtain the performance measurement indicator, that is, the better the performance, the larger the value. The performance of the selected business in the private cloud environment is calculated. The ratio of the cost in the private cloud environment to the average cost of all services in the private cloud environment. The average cost is obtained by averaging the costs of all services in the private cloud environment. This is used to normalize the cost of a single service to a relative level. The cost measurement index is obtained by adding 1 to the ratio and taking its natural logarithm. The addition of 1 ensures that the input value of the logarithmic function is greater than 0. The use of the logarithmic function compresses the cost to a certain extent, so that the relative difference in cost is more reasonably reflected in the price-performance ratio calculation. The calculated performance measurement index is multiplied by the cost measurement index to obtain the price-performance ratio of the selected service in the private cloud environment. The larger the performance measurement index (better performance) and the smaller the cost measurement index (lower cost), the higher the final price-performance ratio value, indicating that the selected service has a high comprehensive performance and cost advantage in the private cloud environment. The calculation expression for the price / performance ratio of the selected business in the private cloud environment is: ; Where, For the The cost-effectiveness of each business in a private cloud environment, For the Business in a private cloud environment The value of the performance indicator, For the The benchmark values of performance indicators in a private cloud environment, For the The cost of a business in a private cloud environment, is the average cost of all services in a private cloud environment, is the total number of performance indicators, For each business, we conduct performance and cost evaluation in private and public cloud environments. Calculation process for cost-effectiveness of each business in a public cloud storage environment: For the selected business, traverse all its performance indicators in the public cloud environment, divide each performance indicator value by the corresponding benchmark value to obtain a normalized performance indicator ratio. The benchmark value is set according to the target of the performance indicator in the public cloud environment to make different performance indicators comparable. All performance indicators are normalized to obtain a set of normalized performance indicator ratios. All the normalized performance indicator ratios are squared and summed up. The comprehensive performance of the business in the public cloud environment is analyzed based on the summation result. The smaller the value, the better the performance. Then take the square root of the summation result to obtain a representative value of the comprehensive performance indicator. Then take the inverse of the representative value of the comprehensive performance indicator to obtain the performance measurement indicator, that is, the better the performance, the larger the value. Calculate the performance of the selected business in the public cloud environment. The ratio of the cost in the public cloud environment to the average cost of all services in the public cloud environment. The average cost is obtained by averaging the costs of all services in the public cloud environment. This is used to normalize the cost of a single service to a relative level. The ratio is added by 1 and then the natural logarithm is taken to obtain the cost measurement index. The addition of 1 ensures that the input value of the logarithmic function is greater than 0. The use of the logarithmic function compresses the cost to a certain extent, so that the relative difference in cost is more reasonably reflected in the price-performance ratio calculation. The calculated performance measurement index is multiplied by the cost measurement index to obtain the price-performance ratio of the selected service in the public cloud environment. The larger the performance measurement index (better performance) and the smaller the cost measurement index (lower cost), the higher the final price-performance ratio value, indicating that the selected service has a high comprehensive performance and cost advantage in the public cloud environment. The calculation expression for the price / performance ratio of the selected business in the public cloud environment is: ; Where, For the The cost-effectiveness of each business in a public cloud environment, For the Business in the public cloud environment The value of the performance indicator, For the The benchmark values of performance indicators in a public cloud environment, For the The cost of a business in a public cloud environment, The average cost of all services in the public cloud environment. The larger the performance value, the smaller the cost value, and the higher the cost performance. The smaller the performance value, the larger the cost value, and the lower the cost performance. The priority allocation module is used to assign corresponding priorities to different services based on the degree of compatibility between each service and different cloud environments output by the performance cost evaluation module, and define service resource allocation strategies for different priorities to ensure that key services receive sufficient resources during high-concurrency access and prevent low-priority services from occupying too many resources and affecting key service performance. The resource allocation and scheduling module is used to dynamically allocate private cloud and public cloud resources based on access traffic, resource usage status, and priority, optimize resource allocation strategies, balance performance and cost, improve resource utilization, and reduce resource competition.

[0021] Example 2, as Figure 1 、 Figure 2 As shown, based on Example 1, the present invention provides a technical solution: preferably, the priority allocation module specifically includes: The performance cost evaluation module receives the price / performance ratio results of each business in the private cloud and public cloud environments, and assigns weight coefficients to the price / performance ratio of each business in the private cloud and public cloud environments based on the business type. The weighted sum is then used to obtain the matching coefficient, and a matching threshold is preset to preliminarily divide the adaptation relationship between the business and the cloud environment. The business-cloud environment matching matrix is generated. Based on the matching analysis results, the business is divided into three levels of priority, namely high priority, medium priority, and low priority. Among them, high priority refers to the business with a matching degree > 0.8 and strict response time requirements (such as response time < 100ms). Critical businesses are prioritized, medium-priority businesses are prioritized with a matching degree of 0.5 to 0.8 and moderate resource demand elasticity, and are labeled core businesses. Low-priority businesses are prioritized with a matching degree of less than 0.5 or can tolerate performance fluctuations, and are labeled elastic businesses. Resource allocation rules are defined for businesses of different priorities, and the rules are automatically executed through a policy engine. High-priority businesses are allocated dedicated private cloud resource pools, with 30% redundant capacity reserved and QoS enabled. Medium-priority businesses use public cloud elastic resource pools, with resource quota caps set and dynamic scheduling implemented. Low-priority businesses use public cloud on-demand instances or spot instances, and resource preemption policies are configured. The calculation expression of the matching coefficient is: ; Where, is the matching coefficient, For the The cost-effectiveness of each business in a private cloud environment, For the The cost-effectiveness of each business in a public cloud environment, and The weight coefficients of the cost-performance ratio of the business in private cloud and public cloud environments respectively; Multiple priorities correspond to multiple matching thresholds one by one, and the corresponding relationship is as follows: The high priority matching threshold is: ; The matching threshold for medium priority is: ; The low priority matching threshold is: ; in, is the matching coefficient, is the lower threshold of high priority and the upper threshold of medium priority, is the lower threshold of medium priority and the upper threshold of low priority, , ; The resource allocation and scheduling module specifically includes: Through API interface and system integration, access traffic data and resource usage status data are obtained from the resource monitoring and analysis module, and the priority information of each business is obtained from the business priority allocation module, and integrated to form a comprehensive resource allocation decision data set. Based on the integrated data, the resource requirements of each business are analyzed. According to the access traffic data, the changes in the resource requirements of the business in different time periods are analyzed, especially the resource demand peaks during high concurrent access periods. Combined with the resource usage status data, the current resource usage and remaining capacity are determined, and whether there are resource bottlenecks. At the same time, according to the business priority, the degree of urgency of each business's demand for resources is determined. According to the results of the resource demand analysis and the formulated resource allocation strategy, for high-priority businesses, private cloud resources are allocated first, and a certain amount of redundant capacity is reserved to ensure that critical businesses can obtain sufficient resource guarantees during high concurrent access. For medium-priority businesses, public cloud elastic resource pools are used to dynamically adjust resource allocation according to business load, set resource quota limits, and avoid excessive resource occupation. For low-priority businesses, public cloud on-demand instances or Spot For example, configure resource preemption strategies, give priority to meeting the resource needs of high-priority businesses, dynamically allocate private cloud and public cloud resources, and monitor resource usage and business operation status in real time during resource allocation. Dynamically adjust resource allocation according to actual needs. When the resource usage of high-priority businesses approaches the upper limit, automatically allocate more resources from the resource pool. When the load of medium-priority businesses increases, dynamically adjust their resource quotas. When the resources of low-priority businesses are preempted by high-priority businesses, automatically adjust their resource allocation. Through continuous monitoring and optimization, ensure the effective execution of resource allocation strategies, improve resource utilization, reduce resource competition, balance performance and cost, and ensure the normal operation of businesses.

[0022] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A hybrid cloud security storage management system for big data, including a hybrid cloud management center, characterized by: The hybrid cloud management center is connected to the following modules: Hybrid cloud data storage module, used to integrate private cloud and public cloud storage resources to form a unified hybrid cloud resource pool to store data and distribute it according to business needs; Resource monitoring and analysis module, used to monitor and analyze the access traffic and resource usage status of private and public clouds in real time; The performance cost evaluation module is used to calculate the cost-performance ratio of each business in private and public cloud storage environments based on the business's access patterns, traffic patterns, and resource usage, and analyze the degree of compatibility between the business and the cloud environment. Priority allocation module, used to assign priorities to different services and formulate resource allocation strategies based on performance cost evaluation results; The resource allocation and scheduling module is used to dynamically allocate private cloud and public cloud resources based on access traffic, resource usage status, and priority, and optimize resource allocation strategies.

2. The hybrid cloud security storage management system for big data according to claim 1, characterized in that: The hybrid cloud data storage module specifically includes: The system automatically scans storage devices and spaces in private and public cloud environments, accesses storage resources in private and public clouds through standardized interfaces, and automatically identifies storage types, capacities, and performance parameters of different cloud platforms. It also categorizes and labels storage resources, distinguishing between high-performance and large-capacity storage resources. Based on the accessed storage resource information, the storage resources of private cloud and public cloud are integrated to build a unified hybrid cloud resource pool; Develop corresponding data storage and distribution strategies based on business needs and data characteristics. At the same time, further refine the storage strategy based on data access frequency and security requirements; According to the established data storage and distribution strategy, business data is stored in the hybrid cloud resource pool, and data is distributed according to business needs.

3. The hybrid cloud security storage management system for big data according to claim 1, characterized in that: The resource monitoring and analysis module includes an access traffic monitoring and analysis unit and a resource usage status perception unit; The access traffic monitoring and analysis unit is used to monitor the access traffic of private clouds and public clouds in real time, analyze traffic patterns, and identify high concurrent access periods and resource contention hotspots; The resource usage status perception unit is used to dynamically perceive the resource usage status of the private cloud and the public cloud, form a resource usage view, and analyze the resource usage status and performance.

4. The hybrid cloud security storage management system for big data according to claim 3, characterized in that: The access traffic monitoring and analysis unit specifically includes: Deploy traffic monitoring probes in private and public cloud environments to capture real-time traffic data at the network, application, and storage layers. Simultaneously, using the cloud platform's native monitoring interface, collect virtual machine / container-level traffic metrics. Standardize all data and build a comprehensive traffic view. Based on historical traffic data aggregated by time window, a traffic baseline model is constructed using time series analysis to identify periodic patterns. Traffic is then classified using the K-Means clustering algorithm to distinguish between normal business access, burst traffic, and abnormal traffic. Traffic features, including request rate, data volume, and access source IP entropy, are extracted to generate a traffic feature fingerprint library. Based on real-time traffic data, a sliding window algorithm is used to calculate the number of concurrent requests per unit time. This is combined with preset threshold rules to trigger high-concurrency alerts. Heat map analysis technology is used to aggregate traffic data by storage device, API interface, and user group to identify resource contention hotspots. Furthermore, resource usage data is correlated to locate potential performance bottlenecks. The analyzed traffic patterns and performance bottleneck information are pushed to the hybrid cloud management center via API, and real-time traffic data is displayed through a visual interface.

5. The hybrid cloud security storage management system for big data according to claim 4, characterized in that: The resource usage status sensing unit specifically includes: Through lightweight agents deployed in private and public cloud environments, real-time resource usage data for CPU, memory, storage, and network is collected. At the same time, cloud platform native APIs are called to obtain performance data at the virtual machine / container level. Multi-source heterogeneous data is cleaned, fields aligned, and units converted to form a standardized JSON-formatted resource base dataset. Based on the standardized data of the resource base dataset, we conduct aggregate analysis by cloud platform, business system, and resource type. We generate resource usage views through time series aggregation, including real-time resource utilization heat maps, resource capacity trend curves, and cross-cloud resource distribution maps. Combined with resource usage views, adaptive thresholds based on historical baselines are used to identify abnormal resource status. Correlation analysis technology is used to cross-validate resource performance data with business loads, analyzing performance and locating the root causes of performance bottlenecks. The analyzed resource usage status and performance are pushed to the hybrid cloud management center via the RESTful API, and the resource usage view is displayed through a visual interface.

6. The hybrid cloud security storage management system for big data according to claim 3, characterized in that: The performance cost evaluation module specifically includes: Through API integration, access resource monitoring and analysis modules collect data on access patterns, traffic patterns, and resource usage status for each business, build business profiles, and annotate key feature tags. At the same time, private and public cloud billing models and cost data, including storage fees, network fees, and computing resource fees, are collected from the cloud platform to form a complete business performance and cost data set. Based on the integrated business performance and cost data sets, the performance of different businesses in private and public cloud storage environments is evaluated. Performance indicators such as response time, throughput, and I / O operation latency are analyzed. Baseline values for each performance indicator are set based on the targets for each in private and public cloud environments. Furthermore, a performance difference matrix is generated based on the business access and traffic patterns, identifying performance bottlenecks in the public cloud caused by network latency. Evaluate the cost expenditures of different businesses in private and public cloud storage environments, analyze the resource usage of businesses in different cloud environments, including storage capacity, network bandwidth, and computing resources. Combined with the cloud platform's billing model, calculate the cost expenditures of businesses in different cloud environments. By comparing cost data from different cloud environments, determine the cost differences between businesses in private and public clouds. Combining the results of performance and cost evaluation, calculate the cost-performance ratio of each business in private cloud and public cloud storage environments. By calculating the cost-performance ratio, analyze the degree of compatibility between each business and different cloud environments.

7. The hybrid cloud security storage management system for big data according to claim 6, characterized in that: The calculation process of the cost-performance ratio of each business in the private cloud storage environment is as follows: For the selected business, traverse all its performance indicators in the private cloud environment, divide each performance indicator value by the corresponding benchmark value to obtain the normalized performance indicator ratio; All normalized performance index ratios obtained are squared, and the obtained square values are summed. The square root of the sum is taken to obtain a representative value of the comprehensive performance index. The reciprocal of the representative value of the comprehensive performance index is then taken to obtain the performance measurement index. Calculate the ratio of the cost of the selected business in the private cloud environment to the average cost of all businesses in the private cloud environment, add 1 to the ratio and take the natural logarithm to obtain the cost measurement index; Multiply the calculated performance metric by the cost metric to get the price / performance ratio of the selected business in the private cloud environment. The calculation process of the cost-performance ratio of each business in the public cloud storage environment is as follows: For the selected business, traverse all its performance indicators in the public cloud environment, divide each performance indicator value by the corresponding benchmark value, and obtain the normalized performance indicator ratio; All normalized performance index ratios obtained are squared, and the obtained square values are summed. The square root of the sum is taken to obtain a representative value of the comprehensive performance index. The reciprocal of the representative value of the comprehensive performance index is then taken to obtain the performance measurement index. Calculate the ratio of the cost of the selected business in the public cloud environment to the average cost of all businesses in the public cloud environment, add 1 to the ratio and take the natural logarithm to obtain the cost measurement index; Multiply the calculated performance metric by the cost metric to obtain the price / performance ratio of the selected business in the public cloud environment.

8. The hybrid cloud security storage management system for big data according to claim 7, characterized in that: The priority allocation module specifically includes: The performance-cost evaluation module receives the price / performance ratio results of each business in the private cloud and public cloud environments, assigns weight coefficients to the price / performance ratio of each business in the private cloud and public cloud environments based on the business type, and then calculates the weighted sum to obtain the matching coefficient. A matching threshold is preset to preliminarily divide the adaptation relationship between the business and the cloud environment and generate a business-cloud environment matching matrix. Based on the matching analysis results, the business is divided into three levels of priority: high priority, medium priority, and low priority; Define resource allocation rules for businesses of different priorities, and automate the execution of rules through a policy engine. High-priority services are allocated to a private cloud-exclusive resource pool, with 30% redundant capacity reserved and QoS enabled. Medium-priority services use a public cloud elastic resource pool, with resource quota limits set and dynamic scheduling implemented. Low-priority services use public cloud on-demand instances or Spot instances, and resource preemption policies are configured.

9. The hybrid cloud security storage management system for big data according to claim 8, characterized in that: The resource allocation and scheduling module specifically includes: Through API interface and system integration, it obtains access traffic data and resource usage status data from the resource monitoring and analysis module, and obtains the priority information of each business from the business priority allocation module, integrating them to form a comprehensive resource allocation decision data set; Based on the integrated data, we analyze the resource requirements of each business. We analyze changes in resource requirements over different time periods based on access traffic data. Combined with resource usage data, we determine current resource usage and remaining capacity, analyze whether there are resource bottlenecks, and determine the urgency of each business's resource needs based on business priorities. Based on the results of resource demand analysis and the formulated resource allocation strategy, and during the resource allocation process, real-time monitoring of resource usage and business operation status is carried out to dynamically adjust resource allocation according to actual needs.

Images