Elastic storage method, device, electronic device and readable medium

Abstract

Embodiments of the present invention provide an elastically scalable storage method, device, electronic device, and readable medium. The elastically scalable storage method includes: obtaining storage indicator information of Kubernetes within a current period of time, the storage indicator information including the number of copies and time interval of each change; calculating a dynamic weight based on the storage indicator information; determining the dynamic tolerance and expansion speed of the copy based on the dynamic weight; when the dynamic tolerance is greater than the preset tolerance, adjusting the number of copies according to the expansion speed to quickly achieve elastically scalable storage. In peak business scenarios, the technical solution of the present application can quickly break through the tolerance range and adjust the number of copies at a higher expansion speed, thereby meeting the user's sudden large demand for the number of copies, reducing the user's waiting time, and improving the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud native storage, and in particular to an elastically scalable storage method, an elastically scalable storage device based on dynamic weights, an electronic device, and a computer-readable medium. Background Art

[0002] Kubernetes (often referred to as K8S) is an open source container orchestration system from Google Cloud Platform, which is used to automatically deploy, scale, and manage container applications. By deploying their own Pod applications on K8S, users can enjoy the deployment, operation, resource scheduling, service discovery, and dynamic scaling functions provided by K8S. Now more and more services are gradually becoming cloud-native and deploying services on K8S. In actual scenarios, multiple copies are used for deployment. According to business needs, multiple replica Pods are started, and traffic load is imported to each replica Pod in a balanced manner. The orchestration of multiple replica Pods is usually completed through K8S deployment or statefulset.

[0003] The new generation storage engine LAVA deploys external services, such as slot servers, file servers, and objservers, on K8S. When users connect to storage or computing components connect to storage, they complete the storage read and write functions by connecting to the corresponding server. After deployment, it relies on the dynamic expansion and contraction mechanism of K8S to dynamically adjust according to the increase and decrease of the user's business volume. During idle periods, user Pod applications will maintain a lower order of magnitude to avoid consuming unnecessary resources. During periods of high business traffic, the number of user Pod applications will increase to a higher order of magnitude to ensure that the business can normally undertake services. Generally speaking, it is unrealistic to rely solely on manual execution of container replica expansion from idle periods to peak periods. Instead, it relies on K8S's HPA (Horizontal Pod Autoscaler) mechanism to complete it. HPA can dynamically scale the number of pods in workload sets such as statefulSet, deployment, and replicaSet based on certain indicators, so that the business running on it has a certain degree of adaptive ability to changes in indicators.

[0004] The principle of HPA is to calculate the expected number of Pods according to the specified algorithm. Then compare the current number of Pods with the expected value, and calculate the expansion or contraction percentage. If the expansion or contraction percentage exceeds the preset threshold, the expansion or contraction operation is triggered. For example, when the percentage is higher than 100% by a certain amount, it is expanded. When the percentage is less than 100% by a certain amount, it is reduced. However, in order to avoid adjustments as soon as there is an abnormal change, such as frequent small fluctuations in traffic, by setting the tolerance, small fluctuations within a certain range will not cause expansion or contraction.

[0005] Existing technologies react too slowly to rapid traffic growth, usually waiting until the business volume increases to a certain level before taking action, but by then the business may have already been affected to a certain extent. However, if the tolerance is reduced due to the above situation, traffic fluctuations will cause the HPA to frequently expand and shrink. In addition, when performing capacity expansion, the expansion method is step-by-step and will not directly expand to the expected number of Pods. For example, if the current number of Pods is 10 and the expected value is 15, the expansion process will show a step-by-step change: 11→12→13→14→15. With existing technologies, when facing large traffic, the ability to provide services to the outside world will take a long time to reach the expected level. For businesses that are sensitive to storage performance, it is difficult to accept the need to wait a long time to achieve capacity expansion and shrinkage. Summary of the invention

[0006] The embodiments of the present invention provide an elastic storage method, device, electronic device and computer-readable storage medium to solve the problem of too slow increase in the number of replicas.

[0007] An embodiment of the present invention discloses an elastically scalable storage method, which includes:

[0008] Get the storage indicator information of Kubernetes in the current period, including the number of copies and time interval of each change;

[0009] Calculating a dynamic weight based on the storage indicator information;

[0010] Determine the dynamic tolerance and expansion speed of the replica by using the dynamic weight;

[0011] When the dynamic tolerance is greater than the preset tolerance, the number of replicas is adjusted according to the expansion speed to quickly achieve elastic scalable storage.

[0012] In one embodiment, the storage indicator information includes the number of copies and the time interval for each change:

[0013] Analyze the number of copies of each change according to the highest number of copies each time and the preset basic number of copies;

[0014] The time interval of each change is analyzed based on the number of unit times required to reach the maximum number of copies and the preset unit time period.

[0015] In one embodiment, calculating the dynamic weight includes:

[0016] The dynamic weight is calculated based on the maximum number of copies within the preset time interval, the number of unit times required to reach the maximum number of copies, and the preset basic number of copies. The calculation formula is as follows:

[0017]

[0018] The symbol replicas represents the maximum number of replicas, the symbol replicasBase represents the base number of replicas, the symbol intervalCount represents the number of unit times required to reach the maximum number of replicas, and the symbol Ceil represents rounding up.

[0019] In one embodiment, it further includes:

[0020] According to the dynamic weight and the preset weight value strategy form, the virtual copy multiple and expansion speed corresponding to the dynamic weight are matched;

[0021] Determine whether the dynamic tolerance is greater than a preset tolerance, wherein the dynamic tolerance is calculated by the virtual copy multiple and the maximum number of copies;

[0022] When the above judgment is true, the number of replicas is increased at the expansion speed.

[0023] In one embodiment, it further includes:

[0024] If a user-defined dynamic weight is received, the expansion speed of the replica is increased according to the user-defined dynamic weight.

[0025] In one embodiment, it further includes:

[0026] When it is detected that scaling down is required, the scaling down speed of the replica is determined based on the dynamic weight;

[0027] The number of replicas is adjusted according to the shrinking speed to quickly implement elastically scalable storage.

[0028] According to a second aspect of the present invention, there is provided an elastically scalable storage device based on dynamic weights, comprising:

[0029] The acquisition module is used to obtain the storage indicator information of K8S in the current period of time, and the storage indicator information includes the number of copies and time interval of each change;

[0030] A calculation module, used to calculate a dynamic weight according to the stored indicator information;

[0031] A determination module, used to determine the dynamic tolerance and expansion speed of the replica based on the dynamic weight;

[0032] The expansion module is used to adjust the number of replicas according to the expansion speed when the dynamic tolerance is greater than the preset tolerance, so as to quickly realize elastic scalable storage.

[0033] In one embodiment, the acquisition module, the calculation module, the determination module and the expansion module are controlled to execute any one of the elastic scaling storage methods described above.

[0034] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0035] The memory is used to store computer programs;

[0036] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0037] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0038] The embodiments of the present invention include the following advantages: in peak business scenarios, the tolerance range can be quickly exceeded and the number of replicas can be adjusted at a higher expansion speed, thereby meeting the user's sudden large demand for the number of replicas, reducing the user's waiting time and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flowchart of the steps of an elastic storage method provided in an embodiment of the present invention;

[0040] Figure 2 is a flowchart of step S13 of an elastically scalable storage method provided in an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of weight value calculation of an elastically scalable storage method provided in an embodiment of the present invention;

[0042] Figure 4 It is a schematic diagram of the overall structure of an elastically scalable storage method provided in an embodiment of the present invention;

[0043] Figure 5A structural diagram of an elastically scalable storage device based on dynamic weights provided by an embodiment of the present invention;

[0044] Figure 6 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0045] Figure 7 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Reference Figure 1 , shows a flowchart of a method for elastically scalable storage provided in an embodiment of the present invention, which may specifically include the following steps:

[0048] In step S11, the storage indicator information of Kubernetes in the current period is obtained, and the storage indicator information includes the number of copies and the time interval of each change;

[0049] In step S12, a dynamic weight is calculated according to the stored index information;

[0050] In step S13, the dynamic tolerance and expansion speed of the replica are determined by the dynamic weight;

[0051] In step S14, when the dynamic tolerance is greater than the preset tolerance, the number of replicas is adjusted according to the expansion speed to quickly achieve elastic scalable storage.

[0052] In one embodiment, with the widespread application of cloud computing and container technology, the demand for elastic scaling technology in the storage field is becoming increasingly important. Faced with the characteristics of user traffic fluctuations, the traditional fixed resource allocation method is gradually unable to meet business needs, and elastic scaling has become the key to efficient resource management. Especially in the storage system running on the Kubernetes (K8S for short) platform, the dynamic adjustment of replicas is the core means to achieve elastic scaling. Kubernetes is an open source container orchestration platform that allows developers and system administrators to efficiently deploy, scale and manage containerized applications. The core value of this platform lies in its ability to automate the life cycle of containers, including deployment, expansion and operation, thereby greatly simplifying the complexity of microservice architecture. The Kubernetes cluster consists of a control plane (Master) and a group of computing nodes (Node). The Master is responsible for the scheduling and management of the entire cluster, while the Node is where the container actually runs. In Kubernetes, Pods are the basic deployment unit, which can contain one or more containers. Pods are designed to be short-lived and replaceable. They run on Nodes and are scheduled by Masters. In order to manage the number of replicas and update strategies of Pods, Kubernetes introduces Deployments. Deployments use ReplicaSets to ensure that the number of replicas of Pods always remains at the user-defined number. In addition, Kubernetes also provides Services to define the way to access Pods. It provides a unified access interface for a group of Pods that perform the same function, and can provide services to the outside world through ClusterIP, NodePort, or LoadBalancer. Another important feature of Kubernetes is its label system (Labels) and selectors (Selectors). Users can label resources (such as Pods, Services, etc.), and then use selectors to select resources with specific labels, which provides great flexibility for resource management and access. In terms of storage, Kubernetes provides Persistent Volumes (PV) and Persistent Volume Claims (PVC). PV is a storage resource in the cluster, and PVC is a user's request for storage resources. This mechanism allows Pods to request storage of a specific size and access mode, enabling stateful applications to run on Kubernetes. Kubernetes also supports Namespaces, a way to divide cluster resources into multiple logical partitions, allowing different users or teams to run applications on the same physical cluster without interfering with each other.For scenarios that require automatic expansion, Kubernetes provides the Horizontal Pod Autoscaler (HPA), which can automatically expand the number of Pods based on CPU usage or other selected indicators. In addition, Kubernetes also supports ConfigMaps and Secrets, which are used to store configuration data and sensitive information such as passwords and keys, and can be mounted in Pods for application use. Kubernetes also provides StatefulSets to manage stateful applications, which is very important for applications that require persistent storage, network identification, and sequential deployment. At the same time, Kubernetes supports Rolling Updates and Rollback, allowing applications to be updated gradually and rolled back to the previous version when the update fails. Kubernetes' command-line tool Kubectl provides users with an interface for interacting with the cluster and managing resources. With Kubectl, users can easily deploy applications, expand services, and manage the life cycle of containers. Kubernetes is a powerful container orchestration platform that greatly improves the efficiency of development and operation and maintenance by providing the ability to automatically deploy, expand, and manage containerized applications, and makes it easier and more reliable to run and manage containerized applications in a cluster environment.

[0053] The existing technology reacts too slowly when faced with rapid traffic growth, and usually takes action only after the business volume increases to a certain extent, but at this time it may have already had a certain impact on the business. However, if the tolerance is reduced due to the above situation, this will cause traffic fluctuations to prompt the HPA to frequently expand and shrink. In addition, when performing capacity expansion, the expansion method is step-by-step, and it will not directly expand to the expected number of Pods. For example, the current number of Pods is 10, and the expected value is 15. The expansion process presents a step-by-step change: 11→12→13→14→15. Using the existing technology, when faced with large traffic, the ability to provide services to the outside world takes a long time to reach the expected level. For businesses that are sensitive to storage performance, it is difficult to accept the need to wait a long time to achieve expansion and shrinkage. The technical solution in this embodiment can quickly increase the expansion speed of replicas. This embodiment provides a new storage component, which is independently deployed on the K8S cluster where the storage component is located, replacing the original HPA function. Each component actively pushes indicators to the resources defined by CRD, and then Lava HPAPod obtains the indicator information of CRD and combines the new dynamic weight scheme to expand and shrink. LAVAHPA Pod is a mechanism in the Kubernetes cluster for automatically adjusting the number of Pods to adapt to workload changes. In the storage field, HPA allows storage services to dynamically increase or decrease Pod copies according to actual needs to optimize resource utilization and cost-effectiveness. The working principle of HPA is based on monitoring the resource usage of Pods, such as CPU and memory, and other custom indicators. It relies on metrics-server, an API service that aggregates kubelets indicator information. With this data, HPA is able to make scaling decisions to maintain the performance and responsiveness of the service. Users can configure target resources, minimum and maximum Pod numbers, and selectors by defining HPA resource objects, and HPA will automatically adjust the number of Pods based on these configurations. In the storage service, HPA can adjust the number of Pods based on indicators such as the rate of storage I / O operations or the length of the request queue. For example, when the number of requests for storage services increases, HPA can create more Pod replicas to share the load and ensure the responsiveness and throughput of the service. Conversely, when the number of requests decreases, HPA can reduce the number of Pods to save resources. HPA's configuration includes target resources, minimum number of Pods, maximum number of Pods, and selectors. Users can set these parameters based on business needs and resource usage. HPA also supports scaling based on custom indicators, which is particularly important in the storage field because the performance of storage services is often affected by I / O operation rates and data throughput. In addition to automatic scaling, HPA also has automatic scaling capabilities. When the load decreases, HPA will reduce the number of Pods to avoid wasting resources. This process usually has a cool-down period to ensure that scaling operations do not occur frequently due to short-term load fluctuations.Users can monitor the status and events of HPA, as well as the resource usage of Pods, through the kubectl command-line tool. This helps to debug and optimize the configuration of HPA to ensure that it can respond correctly to changes in workloads. Through reasonable configuration and monitoring, HPA can significantly improve the availability and elasticity of storage services. Lava HPA plays the role of dynamic resource management in the storage field. It can automatically adjust the number of Pods of storage services according to the actual workload to achieve efficient resource utilization and cost optimization. Through reasonable configuration and monitoring, HPA can significantly improve the availability and elasticity of storage services, ensuring that storage services can maintain high performance and stability in the face of changing workloads.

[0054] The new dynamic weighting scheme can dynamically adjust the expansion percentage and the incremental steps of expansion according to the number of changes each time and the time interval of each traffic burst, and then actively adjust the number of replicas according to the step changes to ensure that the storage service can respond to traffic peaks more efficiently. The dynamic weighting formula and intelligent control mechanism can improve the response speed of the storage system, reduce resource waste and optimize user experience. Real-time collection of storage indicator information, including changes in the number of replicas and time intervals, provides a basis for the calculation of dynamic weights. Through the storage indicator calculation formula, the weight value is dynamically generated, and it is used to guide the adjustment of the speed and amplitude of elastic scaling. The speed and amplitude of expansion and contraction are determined based on the dynamic weight to quickly respond to peak and valley scenarios. In addition to the automated dynamic adjustment mechanism, the system also supports user-defined weight values. For example, when the prediction model cannot accurately judge the traffic surge, the user can set the weight value based on experience to quickly increase the expansion speed. This user intervention capability makes the system more flexible and can meet business needs in special scenarios. In addition, dynamic weights are combined with intelligent control mechanisms to optimize the execution logic of expansion and contraction. When the expansion or reduction operation is completed, the system will recalculate the dynamic weight value to ensure the continuity and rationality of subsequent operations. This closed-loop design not only improves the response speed of the storage system, but also effectively reduces resource waste, thereby providing a stable service experience in both peak and trough scenarios.

[0055] In order to solve these problems, this embodiment provides an elastically scalable storage solution based on dynamic weights. By introducing a new storage component and deploying it independently in a Kubernetes (K8S) cluster, the component replaces the original HPA (Horizontal Pod Autoscaler) function. Each storage component in the system will actively push operating indicators to the resources defined by CRD (Custom Resource Definition), such as the number of user connections, IOPS, IO bandwidth, and IO latency. Lava HPA Pod quickly calculates the speed and magnitude of expansion and contraction by monitoring the real-time indicator information in CRD and combining it with a dynamic weighting scheme. The dynamic weighting scheme dynamically adjusts the expansion percentage and the incremental expansion steps according to the time interval between each change in the number of replicas and the burst of traffic, and ultimately achieves precise adjustment of the number of replicas. Compared with the prior art, this solution has higher response efficiency when dealing with traffic peaks, and improves resource utilization by optimizing the expansion and contraction strategies.

[0056] Collect core Kubernetes storage-related metrics over a period of time, including changes in the number of replicas and time intervals. Dynamic weight is a key parameter for elastic scaling and is used to adjust the speed of expansion and contraction. Its calculation is based on storage metric information and determines the speed of expansion and contraction. Dynamic tolerance is used to measure the system's ability to withstand load fluctuations, and the expansion speed determines its response efficiency to peak loads. When the dynamic tolerance exceeds the preset threshold, the number of replicas is adjusted according to the calculated expansion speed to quickly adapt to traffic changes. By introducing dynamic tolerance and expansion speed, the system can achieve flexible response to load fluctuations, rather than relying on static thresholds or manual adjustments.

[0057] The difference between the maximum number of copies and the basic number of copies is used to analyze each change in the number of copies. For example, assuming the maximum number of copies is 200 and the basic number of copies is 100, the change in the number of copies is 200-100=100. The time interval between each copy change is calculated by combining the time unit required to reach the maximum number of copies with the preset time period. For example, if it takes 5 time units to reach the maximum number of copies and each time period is 60 seconds, the time interval is 300 seconds. By breaking down the storage indicator information into specific changes in the number of copies and time intervals, the input source for dynamic weight calculation is clarified.

[0058] The calculation formula of the weight value is as follows:

[0059]

[0060] The symbol replicas indicates the maximum number of replicas, the symbol replicasBase indicates the base number of replicas, the symbol intervalCount indicates the number of unit times required to reach the maximum number of replicas, and the symbol Ceil indicates rounding up. The calculation formula for dynamic weights can comprehensively derive the strength of elastic adjustment based on storage indicators in multiple time periods. For example, according to the attached Figure 3 The data shown can be used to obtain the following calculation formula, and the weight value is 9 after calculation.

[0061]

[0062] In certain scenarios, users can manually set dynamic weight values. Custom weight values ​​are directly used to guide the adjustment of expansion speed. For example, when traffic surges and the prediction model cannot accurately judge, users can quickly respond by manually adjusting the weights. The introduction of user customization enhances the flexibility of the method, which can not only meet the needs of automatic adjustment, but also allow users to intervene in special circumstances to ensure that system performance is always in the best state.

[0063] When the need for scaling down is detected, the dynamic weight is used to calculate a reasonable scaling speed. The number of replicas is adjusted according to the scaling speed, with the goal of quickly reducing resource consumption while ensuring business continuity. The closed-loop design of elastic scaling has been improved, so that the system can save resources during business downturns while efficiently expanding capacity.

[0064] During the "Double 11" promotion, a surge in orders on an e-commerce platform caused a rapid increase in the storage system load. Traditional expansion solutions are difficult to meet business needs in a short period of time due to the limitations of trigger conditions and expansion speed. In this embodiment, efficient expansion operations are achieved through a dynamic weight mechanism.

[0065] Real-time collection and dynamic weight calculation: the system collects the replica change amount and traffic burst time interval every 1 minute, and calculates the dynamic weight value to be 2.5. The expansion speed of the weight value matching is to increase 10 replicas each time.

[0066] The current number of replicas starts from 50 and gradually expands to 100. The expansion process is: 50→60→70→80→90→100. The whole process takes less than 3 minutes. There is no order delay or storage performance degradation during peak traffic, which significantly improves the user experience.

[0067] The number of online users on a video streaming platform dropped significantly during the late night hours, and the system load dropped below the normal threshold. Through the dynamic weight mechanism, the system optimized resource utilization while ensuring service quality.

[0068] It is detected that the IOPS and IO bandwidth indicators are continuously lower than the set minimum values, and the dynamic weight calculation shows that the scaling-down speed is to reduce 5 replicas each time.

[0069] The number of replicas was reduced from 50 to 20, and the scaling process was: 50→45→40→35→30→25→20. During the scaling process, sufficient replicas were always retained to support the core business, successfully reducing resource consumption by 60%.

[0070] In view of the particularity of storage and the need for self-customization, when deploying cloud native storage (taking LAVA as an example), LAVA HPAPod and LAVA CRD are deployed at the same time. The overall structure is as shown in the attached figure. Figure 4 As shown. After each storage deployment is completed, N storage server components, such as slot servers and file servers, will be started by default. When the traffic is low, the order of magnitude of the server components remains unchanged. The server components will regularly report the required indicators, such as the number of user connections, IOPS, IO bandwidth, IO latency, etc., and write them to the K8S HPA CRD through the K8S client. The LAVAHPA Pod will obtain first-hand storage service information in real time by listening to the CRD information, and then calculate the strength of expansion and contraction.

[0071] In the storage field, the application of elastic scaling technology requires not only an efficient automation mechanism, but also full consideration of the diversity and complexity of business scenarios. In specific scenarios, users can manually set dynamic weight values, and directly guide the adjustment of expansion speed through customized weight values. For example, when traffic surges, since the prediction model may not be able to accurately judge the actual load of the system, users can manually adjust the dynamic weight value based on experience or real-time monitoring results, so that the system can quickly respond to sudden business pressures. This user intervention mechanism provides higher flexibility and reliability for elastic scaling, which can not only make up for the shortcomings of automatic adjustment in extreme cases, but also allow users to accurately control system performance in key scenarios.

[0072] The introduction of dynamic weight customization also optimizes the user experience. In high-concurrency scenarios, such as live events on video streaming platforms or peak periods of e-commerce promotions, the speed of resource expansion is directly related to the continuity and quality of the service. By customizing weights, users can break through the tolerance limitations of traditional elastic scaling mechanisms and adjust the number of replicas with higher expansion efficiency, thereby ensuring that system performance is always in the best state. In addition, the dynamic weight mechanism is not only suitable for expansion, but also for optimizing resource utilization in low-load scenarios. When the system detects a need to shrink, the dynamic weight can calculate a reasonable shrinking speed, dynamically adjust the number of replicas according to the actual needs of the business, and achieve resource conservation while ensuring service quality. This design completes the closed loop of elastic scaling, allowing the system to efficiently expand capacity while also responding to the resource conservation needs of business troughs.

[0073] In view of the particularity of storage and the need for self-customization, this technical solution proposes an elastic scaling optimization design based on LAVA in the cloud native storage scenario. Deploy LAVA HPAPod and LAVA CRD as core architecture components (see attached Figure 4 As shown in the figure, an efficient operation system for dynamically adjusting storage resources is formed. After each storage system deployment is completed, \(N\) storage server components are started by default, including but not limited to slot server (for storage node allocation), file server (for file system read and write operations) and IO server (for optimizing IO data transmission). These components interact with CRD through the Kubernetes API client during the entire operation process and report real-time indicators (such as the number of user connections, IOPS, IO bandwidth, IO latency, etc.) to HPACRD.

[0074] In low-traffic scenarios, in order to avoid wasting resources, the number of storage server components will be maintained at the lowest order of magnitude while maintaining support for core business needs. For example, when the system detects that the number of user connections or IO requests is continuously below a certain set threshold, the dynamic weight mechanism will trigger a reduction strategy to reduce the number of replicas to the optimal configuration to save resources. In peak traffic scenarios, such as live events or annual promotions, LAVA HPA Pod quickly adjusts the number of replicas by monitoring the real-time indicators of HPA CRD and combining it with the dynamic weight calculation formula, breaking through the limitations of traditional tolerance and achieving dynamic expansion of resources. This mechanism ensures that the storage system can adapt to rapid fluctuations in load in real time, meeting the sudden needs of users while avoiding system performance degradation due to insufficient resources.

[0075] Example 1: Double 11 promotion activity on e-commerce platform

[0076] During the annual Double 11 event, an e-commerce platform faced a surge in user order requests, and the storage system needed to handle large-scale concurrent access in a short period of time. In this scenario, the specific application of the dynamic weight mechanism, real-time monitoring and expansion triggering, the number of user connections and bandwidth indicators continuously exceeded the preset threshold, triggering the expansion demand. The dynamic weight calculated the current weight value to be 3.5, and the matching virtual replica multiple was 2.0. The expansion process is to gradually expand the number of replicas from the initial 100 to 200, and the expansion speed is 20 replicas each time. The storage system remained stable during the peak load and efficiently responded to all user order requests.

[0077] Example 2: Live streaming event on streaming media platform

[0078] During a live broadcast event with 10,000 people, a streaming media platform needed to ensure efficient operation of video storage and playback functions. Traffic prediction and indicator reporting: Before the live broadcast begins, the storage server component regularly reports IOPS and bandwidth usage. LAVA HPA Pod monitors in real time through CRD, and the initial value of the dynamic weight is 2.0. The expansion strategy is adjusted according to the load growth trend, and the expansion speed is dynamically adjusted to increase by 10 copies each time. The number of copies increased from 50 to 100, meeting the user's simultaneous online needs. The entire expansion process took less than 3 minutes, and the live broadcast event went smoothly.

[0079] Example 3: Resource optimization during business downturn

[0080] During business off-peak periods, such as late at night, the number of visits to a storage system decreases significantly. The indicator detection and scaling down is a dynamic weight calculation with a current weight value of 1.2, and the scaling down speed is to reduce 5 copies each time. The number of copies is gradually reduced from 50 to 20. During the scaling down process, key service copies are always retained to ensure the continuity of core business. While saving resource overhead, the system can still meet the needs of a small number of users at night. The elastic scaling storage method realizes efficient management of storage resources by combining user-defined weights, real-time monitoring and dynamic adjustment mechanisms. In peak load scenarios, the dynamic weight mechanism can quickly break through tolerance limits and increase the speed of expansion; in trough periods, scaling down optimizes resource usage and reduces system costs. The above embodiments fully demonstrate the application effect of the technical solution in a variety of complex business scenarios, reflecting its important value in the field of cloud native storage.

[0081] The elastic storage method of the present invention realizes efficient management of storage resources by introducing a dynamic weight mechanism, user-defined weights and real-time monitoring. The examples show that the method has fast response capability in peak load scenarios and can effectively save resources in low load periods, which is an important technological breakthrough in the field of cloud native storage.

[0082] The technical solution in this embodiment can quickly exceed the tolerance range in peak business scenarios and adjust the number of replicas at a higher expansion speed, thereby meeting the sudden large demand for the number of replicas by users, reducing the user's waiting time and improving the user experience. It solves the problem that the replicas in the original LAVA storage scenario cannot be flexibly expanded according to the actual application scenario. The role of modifying the replica expansion rules in real time ensures that the business can cope with traffic peaks more calmly, not only can it exceed the tolerance range faster, but also can quickly increase the expansion speed by calculating a higher weight value.

[0083] In one embodiment, Figure 2 As shown, step S13 includes the following steps S21-S23:

[0084] In step S21, according to the dynamic weight and the preset weight value strategy form, the virtual copy multiple and expansion speed corresponding to the dynamic weight are matched;

[0085] In step S22, it is determined whether the dynamic tolerance is greater than a preset tolerance, wherein the dynamic tolerance is calculated by the virtual copy multiple and the maximum number of copies;

[0086] In step S23, when the above judgment is true, the number of replicas is increased at the expansion speed.

[0087] In one embodiment, the virtual copy multiple and the expansion speed are determined by querying the dynamic weight strategy form, and the dynamic adjustment of the number of copies is realized in combination with the judgment of the dynamic tolerance. The dynamic tolerance is calculated by the virtual copy multiple and the highest number of copies, and is compared with the preset tolerance to determine whether expansion is needed. If the dynamic tolerance exceeds the preset value, the number of copies is increased according to the calculated expansion speed to cope with the growth of the business load.

[0088] For example, by querying the weight value in the preset weight value strategy form, the virtual copy multiple and expansion speed are obtained. The preset weight value strategy form is as follows:

[0089] Weight value Virtual copy multiples Expansion speed 1 1.0 1 2-5 2.0 2 6-10 3.0 4 10-20 5.0 6

[0090] If the tolerance is 108, the basic number of copies is 100, and the calculated required number of copies is 105. If the weight value is between 2-5, then the virtual copy multiple is 2.0, which means that the number of virtual copies = (required number of copies - basic number of copies) × virtual copy multiple + basic number of copies, the number of virtual copies = (105-100)*2+100=110. Since the number of virtual copies 110 exceeds the tolerance of 108, the rapid expansion mechanism will be triggered and the required copies will be started immediately. In addition, the preset weight value policy form can be dynamically configured according to demand. The expansion speed is determined according to the weight value. For example, if the weight value is 7, the corresponding expansion speed is 4, which means that each increase will increase in steps of 4. For example, 100→104→108. Of course, users can also specify the expansion speed in K8S.

[0091] The role of dynamic weight is to bridge the relationship between system load indicators and expansion strategies. Through the preset weight value policy form, the corresponding parameters, including virtual copy multiples and expansion speed, are quickly matched according to the current dynamic weight. Dynamic tolerance, as a load measurement indicator, determines whether expansion needs to be triggered. Its calculation formula depends on the virtual copy multiples and the maximum number of copies. Compare with the preset tolerance threshold to determine whether the current load exceeds the tolerance range of the system. If the dynamic tolerance is greater than the preset tolerance, expansion is required; if the dynamic tolerance is less than or equal to the preset tolerance, the status quo is maintained. The virtual copy multiple and expansion speed are determined through the matching mechanism of the dynamic weight policy form to adapt to different business load scenarios. Dynamic tolerance avoids the delay of copy expansion actions, resulting in a long wait time to reach the number of copies required by users.

[0092] The technical solution in this embodiment can determine the virtual replica multiple and expansion speed by matching the dynamic weight and the preset policy form; combine the calculation and judgment of the dynamic tolerance to decide whether to trigger the expansion; finally, dynamically adjust the number of replicas according to the expansion speed, effectively solving the problems of slow response speed, waste of resources and insufficient stability in the traditional elastic scaling mechanism, making the system more flexible and efficient when dealing with complex load scenarios.

[0093] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0094] Reference Figure 5, shows a structural block diagram of an elastically scalable storage device based on dynamic weights provided in an embodiment of the present invention, which may specifically include the following modules:

[0095] The acquisition module 51 is used to obtain the storage indicator information of K8S in the current period of time, and the storage indicator information includes the number of copies and time interval of each change;

[0096] The calculation module 52 is used to calculate the dynamic weight according to the storage indicator information;

[0097] The determination module 53 is used to determine the dynamic tolerance and expansion speed of the replica according to the dynamic weight;

[0098] The expansion module 54 is used to adjust the number of replicas according to the expansion speed when the dynamic tolerance is greater than the preset tolerance, so as to quickly realize elastic scalable storage.

[0099] The acquisition module 51, the calculation module 52, the determination module 53 and the expansion module 54 included in the block diagram of the elastically scalable storage device based on dynamic weights are controlled to execute the elastically scalable storage method described in any of the above embodiments.

[0100] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0101] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0102] Memory 603, used for storing computer programs;

[0103] Processor 601, when used to execute the program stored in memory 603, implements the following steps: obtaining storage indicator information of Kubernetes in a current period of time, the storage indicator information including the number of replicas and time interval of each change; calculating a dynamic weight based on the storage indicator information; determining the dynamic tolerance and expansion speed of the replicas based on the dynamic weight; when the dynamic tolerance is greater than the preset tolerance, adjusting the number of replicas according to the expansion speed to quickly achieve elastic scalable storage.

[0104] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0105] The communication interface is used for communication between the above terminal and other devices.

[0106] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0107] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0108] like Figure 7 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 701 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the elastic scaling storage method described in the above embodiment.

[0109] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is executed on a computer, the computer executes the elastic scalable storage method described in the above embodiment.

[0110] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0111] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0112] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An elastic storage method, characterized in that: include: Get the storage indicator information of Kubernetes in the current period, including the number of copies and time interval of each change; Calculating a dynamic weight based on the storage indicator information; Determine the dynamic tolerance and expansion speed of the replica by using the dynamic weight; When the dynamic tolerance is greater than the preset tolerance, the number of replicas is adjusted according to the expansion speed to quickly achieve elastic scalable storage.

2. The elastic storage method according to claim 1, characterized in that: The storage indicator information includes the number of copies and the time interval for each change: Analyze the number of copies of each change according to the highest number of copies each time and the preset basic number of copies; The time interval of each change is analyzed based on the number of unit times required to reach the maximum number of copies and the preset unit time period.

3. The elastic storage method according to claim 2, characterized in that: The calculating of the dynamic weight comprises: The dynamic weight is calculated based on the maximum number of copies within the preset time interval, the number of unit times required to reach the maximum number of copies, and the preset basic number of copies. The calculation formula is as follows: The symbol replicas represents the maximum number of replicas, the symbol replicasBase represents the base number of replicas, the symbol intervalCount represents the number of unit times required to reach the maximum number of replicas, and the symbol Ceil represents rounding up.

4. The elastic storage method according to claim 3, characterized in that: The determining the dynamic tolerance and expansion speed of the replica by using the dynamic weight includes: According to the dynamic weight and the preset weight value strategy form, the virtual copy multiple and expansion speed corresponding to the dynamic weight are matched; Determine whether the dynamic tolerance is greater than a preset tolerance, wherein the dynamic tolerance is calculated by the virtual copy multiple and the maximum number of copies; When the above judgment is true, the number of replicas is increased at the expansion speed.

5. The elastic storage method according to claim 1, characterized in that: Also includes: If a user-defined dynamic weight is received, the expansion speed of the replica is increased according to the user-defined dynamic weight.

6. The elastic storage method according to claim 1, characterized in that: Also includes: When it is detected that scaling down is required, the scaling down speed of the replica is determined based on the dynamic weight; The number of replicas is adjusted according to the shrinking speed to quickly implement elastically scalable storage.

7. A device for elastically scalable storage based on dynamic weights, characterized in that: include: The acquisition module is used to obtain the storage indicator information of K8S in the current period of time, and the storage indicator information includes the number of copies and time interval of each change; A calculation module, used to calculate a dynamic weight according to the stored indicator information; A determination module, used to determine the dynamic tolerance and expansion speed of the replica based on the dynamic weight; The expansion module is used to adjust the number of replicas according to the expansion speed when the dynamic tolerance is greater than the preset tolerance, so as to quickly realize elastic scalable storage.

8. The elastic and scalable storage device based on dynamic weights according to claim 7, characterized in that: The acquisition module, the calculation module, the determination module and the expansion module are controlled to execute the elastically scalable storage method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 6 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 6.

Images