A method and apparatus for adjusting deduplication shunting, and a medium

Abstract

The application discloses a deduplication shunt adjustment method and device and a medium thereof, relates to the technical field of storage, and is used for adjusting a data deduplication strategy. In order to solve the problem that a suitable deduplication strategy cannot be selected according to the advantages and disadvantages of online deduplication and background deduplication, a deduplication shunt adjustment method is provided. According to the characteristic that different deduplication strategies will bring different influences on the performance of a system, the load condition of the current system is determined by collecting the system bandwidth, the current pressure of the system is known, and the proportion of actively giving up shunt in the front end is adjusted, that is, the proportion of data to be handed over to online deduplication and background deduplication, so that the data deduplication strategy of the system is dynamically adjusted, the advantages of online deduplication and background deduplication are fully utilized, the influence of the disadvantages is reduced, and the data deduplication needs of the actual system are better met.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a deduplication and traffic adjustment method, apparatus and medium. Background Technology

[0002] Deduplication is a mainstream and popular storage technology that effectively optimizes storage capacity. It eliminates redundant data by deleting concentrated, duplicate data from the system's database, retaining only one copy. Deduplication technology can help many applications reduce data storage volume, save network bandwidth, improve storage efficiency, reduce backup windows, and save costs.

[0003] Current deduplication technologies can be categorized into online deduplication and background deduplication based on execution time. Online deduplication involves deduplicating data before it is written to disk, while background deduplication involves writing data to disk first, then reading it out for deduplication, and finally writing the deduplicated data back to disk. The advantage of online deduplication is that it can improve the data reduction ratio in a timely manner by processing duplicate data online, but it can lead to system performance degradation. The advantage of background deduplication is that it does not cause system performance degradation, but it cannot perform data deduplication in real time. Therefore, if online deduplication is used, system performance cannot be guaranteed, while if background deduplication is used, the data reduction ratio is less reliable. Currently, there is a lack of an adaptive deduplication strategy.

[0004] Therefore, those skilled in the art urgently need a deduplication and traffic adjustment method to solve the current problem of not being able to choose a suitable deduplication strategy based on the advantages and disadvantages of online deduplication and background deduplication. Summary of the Invention

[0005] The purpose of this application is to provide a deduplication distribution adjustment method, device, and medium to solve the problem that it is currently impossible to select a suitable deduplication strategy based on the advantages and disadvantages of online deduplication and background deduplication.

[0006] To address the aforementioned technical problems, this application provides a deduplication and traffic splitting adjustment method, comprising:

[0007] Register a scheduled task to collect the system's bandwidth value in real time according to a preset sampling period;

[0008] The front-end actively abandons the traffic distribution ratio to at least two different preset ratios in sequence, and records the corresponding bandwidth values; wherein, the front-end actively abandons the traffic distribution ratio is the proportion of data that the system distributes to the back-end for deduplication at the front end relative to all data;

[0009] The current load status of the system is determined based on the bandwidth value collected according to a preset ratio.

[0010] The proportion of traffic to be actively abandoned by the front end is adjusted based on the real-time collected bandwidth value and load status for data deduplication.

[0011] Preferably, the preset ratios include: a first preset ratio, a second preset ratio, and a third preset ratio; wherein the values ​​of the first preset ratio, the second preset ratio, and the third preset ratio decrease sequentially.

[0012] Preferably, determining the current system load status based on the collected bandwidth values ​​according to a preset ratio includes:

[0013] If the bandwidth values ​​corresponding to the first preset ratio, the second preset ratio, and the third preset ratio change less than the preset range or increase sequentially, the current load state of the system is determined to be low load; otherwise, it is determined to be high load.

[0014] Preferably, adjusting the proportion of traffic to be actively abandoned at the front end for data deduplication based on real-time collected bandwidth values ​​and load status includes:

[0015] When the load status is low, adjust the front-end actively abandons the offloading ratio to 0%;

[0016] When the load is high, the front end actively abandons the traffic distribution ratio based on the real-time collected bandwidth value through a control algorithm.

[0017] Preferably, the steps of sequentially adjusting the front-end actively abandoning the traffic splitting ratio to at least two different preset ratios and recording the corresponding bandwidth values ​​are repeated according to a preset learning cycle.

[0018] Preferably, after sequentially adjusting the front-end actively relinquishing traffic ratio to at least two different preset ratios and recording the corresponding bandwidth values, the method further includes:

[0019] The performance benchmark is determined according to the bandwidth value corresponding to the preset ratio and the first formula.

[0020] The first formula is:

[0021] PS = 0.7 × AVG + 0.3 × CV;

[0022] PS is the performance benchmark; AVG is the average value of the bandwidth corresponding to each preset ratio; CV represents the current value, which is the maximum value of the bandwidth corresponding to each preset ratio under low load and 80% of the maximum value of the bandwidth corresponding to each preset ratio under high load.

[0023] Accordingly, based on the real-time collected bandwidth values, the control algorithm adjusts the proportion of traffic that the front end actively relinquishes, including:

[0024] Based on the difference between the real-time collected bandwidth value and the performance benchmark, the control algorithm adjusts the proportion of traffic that the front end actively abandons.

[0025] Preferred options also include:

[0026] Based on the real-time collected bandwidth value and the performance benchmark, determine whether the host pressure of the system has changed. If it has changed, return to the step of sequentially adjusting the front-end actively abandoning the diversion ratio to at least two different preset ratios and recording the corresponding bandwidth value.

[0027] Preferably, determining whether the host system pressure has changed based on real-time collected bandwidth values ​​and performance benchmarks includes:

[0028] When the load status is low, if the currently collected bandwidth value is greater than 200% of the performance benchmark, it is determined that the host pressure has changed.

[0029] When the load is high, if the currently collected bandwidth value is greater than 150% of the performance benchmark, it is determined that the host pressure has changed.

[0030] Preferably, the control algorithm is the PID control algorithm.

[0031] To address the aforementioned technical problems, this application also provides a deduplication and traffic adjustment device, comprising:

[0032] The bandwidth acquisition module is used to register a timed task to acquire the system's bandwidth value in real time according to a preset sampling period;

[0033] The baseline sampling module is used to sequentially adjust the front-end actively abandons the diversion ratio to at least two different preset ratios and record the corresponding bandwidth values; wherein, the front-end actively abandons the diversion ratio is the proportion of data that the system diverts from the front-end to the back-end for deduplication relative to all data;

[0034] The load determination module is used to determine the current load status of the system based on the collected bandwidth values ​​according to a preset ratio.

[0035] The proportional adjustment module is used to adjust the proportion of traffic that the front end actively abandons for data deduplication based on the real-time collected bandwidth value and load status.

[0036] Preferably, the above-mentioned deduplication and traffic adjustment device further includes:

[0037] The benchmark determination module is used to determine the performance benchmark according to the bandwidth value corresponding to a preset ratio and a first formula; wherein the first formula is:

[0038] PS = 0.7 × AVG + 0.3 × CV;

[0039] PS is the performance benchmark; AVG is the average value of the bandwidth corresponding to each preset ratio; CV represents the current value, which is the maximum value of the bandwidth corresponding to each preset ratio under low load and 80% of the maximum value of the bandwidth corresponding to each preset ratio under high load.

[0040] The stress detection module is used to determine whether the host stress of the system has changed based on the real-time collected bandwidth value and the performance benchmark. If a change occurs, the benchmark sampling module is triggered.

[0041] To address the aforementioned technical problems, this application also provides a deduplication and traffic adjustment device, comprising:

[0042] Memory, used to store computer programs;

[0043] A processor is used to implement the steps of the deduplication and routing adjustment method described above when executing computer programs.

[0044] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the deduplication and traffic adjustment method described above.

[0045] The deduplication offloading adjustment method provided in this application addresses the issue that, when the data system is under low load, its CPU and memory resources are abundant. Therefore, whether the deduplication data is handled by online deduplication or background deduplication has little impact on system performance. This is reflected in the bandwidth value collected according to the preset ratio, which shows little change and may even increase when the proportion of data handled by online deduplication is higher. Similarly, when the data system is under high load, the bandwidth value is collected to determine the current system pressure. Specifically, CPU and memory resources are scarce, so the impact of handling deduplication data by online deduplication or background deduplication differs greatly in terms of performance. That is, all online deduplication has the greatest impact on system performance, while all background deduplication has the least impact. This is reflected in the bandwidth value collected according to the preset ratio, which gradually decreases as the corresponding preset ratio decreases. In addition, there may be cases where the bandwidth value corresponding to the preset ratio does not fall into either of the above situations. In this case, it indicates that the system host pressure fluctuates greatly, and online deduplication will still affect system performance. Therefore, this application determines the current system load by collecting system bandwidth, and adjusts the proportion of front-end actively abandoning traffic distribution, that is, the proportion of data handed over to online deduplication and background deduplication, thereby dynamically adjusting the system's data deduplication strategy, making full use of the advantages of online deduplication and background deduplication, reducing the impact of their disadvantages, and better meeting the data deduplication needs of the actual system.

[0046] The deduplication and routing adjustment device and computer-readable storage medium provided in this application correspond to the above method and have the same effect. Attached Figure Description

[0047] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart of a deduplication and traffic splitting adjustment method provided by the present invention;

[0049] Figure 2 A flowchart of a method for determining system load status provided by the present invention;

[0050] Figure 3 A stage division diagram for a deduplication and traffic splitting adjustment method provided by the present invention;

[0051] Figure 4 A flowchart of another deduplication and traffic splitting adjustment method provided by the present invention;

[0052] Figure 5 This invention provides a structural diagram of a deduplication and diversion adjustment device.

[0053] Figure 6 This is a structural diagram of another deduplication and diversion adjustment device provided by the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0055] The core of this application is to provide a deduplication and splitting adjustment method, device, and medium.

[0056] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] Currently, there are two main implementation methods for data deduplication. One is online deduplication, which performs deduplication before the data is written to disk after being received at the front end. The advantage is that the data deduplication is timely, but the disadvantage is that it inevitably consumes some data system resources, thus affecting system performance. The other is background deduplication, which writes the data to disk after it is received at the front end, and then reads the data through a background program to perform deduplication when the system is idle. The deduplicated data is then rewritten to the disk. The advantage is that it does not consume system resources when the system is busy, thus not affecting system performance, but the data deduplication is not timely.

[0058] Therefore, using online deduplication or background deduplication alone has its own drawbacks and cannot well meet the needs of actual data deduplication application scenarios. Therefore, there is an urgent need for a deduplication method that can adjust the deduplication strategy according to the current data system pressure. Specifically, it is to adjust the proportion of received data distributed to online deduplication and background deduplication according to the system pressure, so as to take into account the advantages of the two deduplication methods and avoid their disadvantages through dynamic adjustment, and better meet the actual needs.

[0059] Based on this, this application provides a deduplication and traffic adjustment method, such as... Figure 1 As shown, it includes:

[0060] S11: Register a scheduled task to collect the system bandwidth value in real time according to a preset sampling period.

[0061] S12: Sequentially adjust the front-end actively abandons the traffic splitting ratio to at least two different preset ratios, and record the corresponding bandwidth values.

[0062] Among them, the proportion of data that the front-end actively abandons for traffic diversion is the proportion of data that the system diverts from the front-end to the back-end for deleting relative to all data.

[0063] S13: Determine the current load status of the system based on the bandwidth value collected according to the preset ratio.

[0064] S14: Adjust the proportion of traffic to be actively abandoned by the front end based on the real-time collected bandwidth value and load status for data deduplication.

[0065] It should be noted that this embodiment does not limit the specific setting value of the sampling period, which should be determined according to actual needs. The value of the sampling period affects the sensitivity of the deduplication and traffic splitting adjustment method. The smaller the sampling period, the higher the sensitivity, but the greater the load on the system. Generally, a sampling period of 200ms can be set to collect bandwidth values ​​once. In addition, the collection of bandwidth values ​​of system, host and other devices is well known to those skilled in the art, so this embodiment will not describe how to collect bandwidth values.

[0066] Specifically, if the front-end actively abandons 100% of the deduplication, meaning that all deduplication data received by the front-end is not processed during online deduplication and is handed over to subsequent background deduplication tasks, this can maximize the performance of the system's front-end. If the front-end actively abandons 50% of the deduplication, meaning that online deduplication processes 50% of the deduplication data and the remaining 50% is handed over to the background deduplication, this is a more balanced deduplication data distribution strategy. If the front-end actively abandons 0% of the deduplication, meaning that all deduplication data is processed by online deduplication, this scenario has the greatest impact on system performance, but the timeliness of deduplication is also the best.

[0067] Since the impact of data deduplication on system performance varies depending on the proportion of active abandonment of traffic distribution at the front end, and the collected bandwidth can reflect the system's pressure status, step S12 can determine the current system's specific pressure status (i.e., load status) by collecting the bandwidth value under the preset proportion.

[0068] It's easy to understand that because different data systems have different performance levels, a single bandwidth value cannot accurately reflect the current stress level of the system. It's necessary to compare multiple bandwidths and determine their changing trends to obtain an accurate system load status. Therefore, the preset ratios in the above steps should be at least two, and there is no limit to the number. However, too many preset ratios would make the sampling process too lengthy and inefficient. Therefore, choosing three different preset ratios is sufficient to meet the requirements, that is:

[0069] The preset ratios include a first preset ratio, a second preset ratio, and a third preset ratio.

[0070] It should also be noted that the first, second, and third preset ratios must be different values. To more accurately determine the current system pressure, the three preset ratios should cover as much of the possible ratio range as possible. That is, the front-end active abandonment of diversion ratios corresponding to the first, second, and third preset ratios are 100%, 50%, and 0%, respectively. However, this embodiment does not limit the preset ratios to only the above three values. In fact, any three different ratio values ​​can be taken within the range of 0-100%. Although it is necessary to adjust the current front-end active abandonment of diversion ratios to the first, second, and third preset ratios in step 12, the specific order is not restricted. Therefore, the values ​​of the first, second, and third preset ratios can be decreased sequentially as a distinguishing condition, and the purpose of simplifying the judgment can be achieved in subsequent embodiments.

[0071] Based on the above explanation, online deduplication and background deduplication will have different effects under different system load conditions (low load and high load), which will be reflected in different trends in the collected bandwidth values. Therefore, by using the first preset ratio, the second preset ratio, and the third preset ratio to correspond to the collected bandwidth values, the current system load can be determined. Specifically, step S13 is as follows:

[0072] If the bandwidth values ​​corresponding to the first preset ratio, the second preset ratio, and the third preset ratio change less than the preset range or increase sequentially, the current load state of the system is determined to be low load; otherwise, it is determined to be high load.

[0073] It should be noted that, since the order of the first, second, and third preset ratios is not restricted in the above embodiments, the system load status needs to be determined based on the correspondence with the preset ratios when judging the system load status based on the collected bandwidth values. However, it is preferable to adjust the order of the preset ratios (from first to last) to be fixed as: first preset ratio, second preset ratio, and third preset ratio. Then, it can be directly determined whether the changes of the three bandwidth values ​​collected in sequence are less than the preset range or increase sequentially. The value of the preset range also needs to be determined based on how to define a large fluctuation in system host pressure in actual applications. This embodiment does not impose restrictions on this, but generally, the preset range is taken as within 5%, that is, a change in bandwidth value of less than 5% can be considered a small change.

[0074] After determining the system load status, the deduplication strategy can be adjusted accordingly based on the current system pressure, that is, the value of the proportion of traffic actively abandoned by the front end can be adjusted. For step S14, this embodiment also provides a specific implementation plan, which further includes:

[0075] S141: When the load status is low, adjust the front-end actively abandons the offloading ratio to 0%.

[0076] S142: When the load is high, the front end actively abandons the traffic distribution ratio based on the real-time collected bandwidth value through the control algorithm.

[0077] For low-load conditions, using online deduplication will not have a significant impact on system performance. Therefore, in order to improve the timeliness of data deduplication and increase the data reduction ratio as early as possible, we adopt the method of online deduplication for all data deduplication, that is, the front end actively abandons the diversion ratio of 0%.

[0078] Under high load conditions, system resources are strained, making it impossible to handle all deduplicated data using online deduplication. However, relying entirely on background deduplication is also unacceptable. Generally, according to industry testing standards, the requirement for data deduplication is that, after enabling deduplication, the system's performance degradation should not exceed 20% of that in the non-deduplication scenario. In other words, if the system's performance is maintained at 80% of the performance without deduplication, the standard is met. Therefore, in some application scenarios under high load, online deduplication can be used to handle deduplicated data. The specific traffic distribution strategy can be determined based on the current bandwidth value and the 80% performance degradation standard. This can be achieved by using common control algorithms to adjust the proportion of traffic actively abandoned by the front end in real time based on the current bandwidth value and the 80% performance degradation standard, realizing dynamic adjustment of the deduplication strategy. This fully utilizes the advantages of both online and background deduplication while reducing their adverse effects. However, it should be noted that the 80% standard mentioned above is a minimum performance standard, applicable to application scenarios where the primary goal is to reduce redundant data in a timely manner. For other application scenarios, such as those prioritizing system performance, the standard for adjusting the front-end's proactive abandonment of traffic distribution ratio through control algorithms needs to be adapted accordingly.

[0079] This embodiment does not limit the control method used to adjust the front end to actively give up the diversion ratio. Control algorithms such as the binary method and PID control algorithm (Proportion Integral Differential) can be used. Generally, PID control algorithm can achieve better control effect.

[0080] Based on the above embodiments, the process for determining the current load status of the system is as follows: Figure 2 As shown:

[0081] By collecting bandwidth values ​​when the front-end actively abandons the traffic splitting ratio and adjusts it to 100%, 50%, and 0% in sequence, determine which of the following four conditions it meets;

[0082] First, if the bandwidth value changes by less than 5%, it is considered that the bandwidth value remains unchanged. This indicates that the system load is low and the system performance is sufficient to support host input / output (IO). Therefore, all deduplication data is kept online for deletion, which means that the front-end actively abandons the diversion ratio to 0%.

[0083] Secondly, the bandwidth value increases sequentially, indicating that the system load is low and the system performance is sufficient to support host input / output (IO). Therefore, all deduplication data is kept online for deletion, which means that the front end actively abandons the diversion ratio to 0%.

[0084] Third, the bandwidth values ​​decrease sequentially, indicating that the system is under high load and the system performance is insufficient to support host input / output (IO). Therefore, the control algorithm adjusts the proportion of traffic that the front end actively abandons at this time.

[0085] Fourth, if the trend of bandwidth value changes to other conditions, it indicates that the system load is in an oscillating scenario and the system performance is insufficient to support host input / output (IO). Therefore, the front end actively abandons the diversion ratio at this time through the control algorithm.

[0086] Furthermore, regarding step S14, as explained above, since different systems have different hardware configurations, it is unrealistic to determine the current system load based solely on a bandwidth value. Correspondingly, in step S14, it is also difficult to adjust the front-end's proactive traffic relinquishment ratio using only the real-time collected bandwidth value through the control algorithm. The above embodiment explains that the industry's maximum permissible performance degradation of 20% can be used as a standard for adjustment. However, regarding how to determine the current system performance, this embodiment provides a possible implementation scheme: a deduplication traffic relinquishment adjustment method further includes:

[0087] S15: Determine the performance benchmark according to the first formula based on the bandwidth value corresponding to the preset ratio;

[0088] The first formula is:

[0089] PS = 0.7 × AVG + 0.3 × CV;

[0090] PS is the performance benchmark; AVG is the average value of the bandwidth corresponding to each preset ratio; CV represents the current value, which is the maximum value of the bandwidth corresponding to each preset ratio under low load and 80% of the maximum value of the bandwidth corresponding to each preset ratio under high load.

[0091] Accordingly, step S142: Based on the real-time collected bandwidth value, the control algorithm adjusts the proportion of traffic actively abandoned by the front end, which specifically includes:

[0092] S142: Based on the difference between the real-time collected bandwidth value and the performance benchmark, the control algorithm adjusts the proportion of traffic that the front end actively abandons.

[0093] The performance benchmark determined by the first formula, which is the expected value representing the current system performance, is determined by the different values ​​of the current value under high and low load conditions. This is because the expected value of the system performance is different under high and low load conditions. If this error is considered negligible in actual application, the current value under high and low load conditions can be taken in the same way. This embodiment does not impose any restrictions. Similarly, the current value under high load scenario can be taken as 80% of the maximum bandwidth value, which can also be adjusted according to actual needs.

[0094] The deduplication and traffic splitting adjustment method provided in this embodiment utilizes the characteristic that different deduplication strategies will have different impacts on system performance. By collecting bandwidth values ​​at a preset ratio to understand the current system load status, and then dynamically adjusting the front-end actively abandoning the splitting ratio based on the real-time collected bandwidth values, that is, adjusting the ratio of deduplicated data splitting to online deduplication and background deduplication, making full use of the advantages of online deduplication and background deduplication and reducing their adverse effects, so as to meet the needs of actual data deduplication application scenarios. For example, when the system is under low load and resources are sufficient, using online deduplication to process deduplicated data will not significantly impact system performance. Therefore, all deduplicated data can be offloaded to online deduplication to ensure timely data reduction. However, when the system is under high load, resources are insufficient, and using online deduplication will affect system performance. Considering that the industry generally allows a 20% performance degradation when deduplication is enabled, the front-end proactive abandonment ratio can be adjusted based on this standard. This allows for the offloading of as much deduplicated data as possible to online deduplication without compromising system performance by 20%, thereby maximizing the timeliness of data deduplication while ensuring system performance. For application scenarios prioritizing system performance, the front-end proactive abandonment ratio can also be adjusted based on other standards to adapt to more diverse data deduplication needs. Furthermore, this embodiment also conveniently determines the current system's expected performance value using the bandwidth value obtained by downsampling at a preset ratio and the first formula. This provides a basis for adjusting the front-end proactive abandonment ratio, enabling more accurate and dynamically adjusted deduplication strategies that better meet actual system performance needs.

[0095] As can be seen from the above embodiments, the deduplication and traffic splitting adjustment method provided in this application determines the current load status of the system by collecting bandwidth values ​​based on a preset ratio in the previous few times. However, the load status of the system is not static, so the determination of the system load status in the above steps should also be a dynamic process to ensure the accuracy of the deduplication and traffic splitting adjustment method.

[0096] Therefore, this embodiment provides a preferred implementation scheme. The above step S12, which collects the bandwidth value at a preset ratio, should be a periodic repetition process, repeated at a preset learning cycle.

[0097] The learning cycle, distinct from the sampling cycle mentioned above, is the smallest unit representing a single deduplication and traffic splitting adjustment process. A learning cycle includes the aforementioned sampling cycle, as well as the subsequent front-end proactive abandonment of traffic splitting ratio adjustments. Specifically, it can be seen as follows... Figure 3 As shown:

[0098] The learning cycle is a large cycle, which can generally be 1 minute. Depending on the purpose of the steps being performed, the learning cycle can be divided into 3 smaller cycles: the acquisition cycle, the adjustment cycle, and the application cycle.

[0099] The sampling period is the process of collecting bandwidth values ​​at a preset ratio to determine the current system load. In the preferred embodiment above, three different preset ratios are used, meaning the sampling period includes at least three sampling periods to collect bandwidth values ​​at three preset ratios. In this scenario, the sampling period should be 600ms (taking a sampling period of 200ms as an example).

[0100] Secondly, the adjustment cycle and application cycle are the processes of adjusting the front-end's proactive abandonment of traffic sharing ratio based on the system load determined by the sampling cycle and the real-time collected bandwidth value. The difference lies in that, in this embodiment, the stage where the adjustment of the front-end's proactive abandonment of traffic sharing ratio changes significantly is called the adjustment stage. An adjustment stage should also include multiple adjustment actions (because achieving the expected system performance value in a single adjustment is extremely difficult in practical applications; each adjustment process is based on the real-time collected bandwidth value, meaning the duration of each adjustment action should be consistent with the sampling cycle, i.e., an adjustment action is performed every 200ms). The purpose of the adjustment stage is to quickly and stably adjust the system performance value to near the expected value (exceeding 80% of the system performance when the deduplication function is not enabled). The stage where the changes are smaller and the system data deduplication application is relatively stable is called the application stage (similar to the adjustment cycle, the application stage also includes multiple adjustments of the front-end's proactive abandonment of traffic sharing ratio, and the duration of each adjustment action should be consistent with the sampling cycle). The purpose of the application stage is to keep the system running near the current expected performance value. Therefore, the adjustment cycle and application cycle do not have a specific time range and are determined based on the actual adjustment situation.

[0101] In other words, each smallest unit of adjusting the system's deduplication strategy includes the following three stages: the data acquisition stage to determine the system's stress conditions, the adjustment stage to quickly adjust the system performance to the desired value, and the application stage to smoothly maintain the system performance to near the desired value for use.

[0102] The purpose of establishing this long learning cycle is to adapt to constantly changing system load conditions. Therefore, by repeatedly sampling the phase, the load state determined in the deduplication and traffic splitting adjustment method and used as the basis for the subsequent adjustment of the front-end's active abandonment of traffic splitting ratios is adjusted. Regarding the value of the learning cycle, a possible implementation of 1 minute is given above. In reality, an excessively large learning cycle will affect the sensitivity of the deduplication and traffic splitting adjustment method to changes in system load conditions, thus affecting the accuracy of the deduplication strategy adjustment. However, if the learning cycle is too small, the time allocated to the sampling phase within a learning cycle will be too large, while the time allocated to the adjustment phase and application phase—which are essentially the parts where the deduplication strategy is dynamically adjusted—will be reduced, which is not conducive to the implementation of the deduplication function. Therefore, a sampling cycle of 200ms and a learning cycle of 1 minute (60000ms) are selected as a preferred implementation method.

[0103] It should be noted that the above embodiment provides a preferred solution for repeatedly triggering step S12 at the end of a learning cycle to adapt the determined system load state to the constantly changing actual situation. Accordingly, step S13, as a specific step for determining the system load state, should also be triggered after step S12. Specifically, in steps S11-S14, step S11 continuously and in real-time collects system bandwidth values ​​by registering a timed task, which is a continuous process; steps S12 and S13 are a method for determining the system load state in a learning cycle, which is a periodically repeated process; step S14 is a method for adjusting the front-end actively abandoning traffic distribution ratio based on the results obtained from steps S11, S12, and S13 as parameters in a learning cycle. Although the parameters used for adjustment differ in each learning cycle, it is still a continuous process.

[0104] Furthermore, in the above embodiments, a preferred implementation scheme is provided for determining the expected performance value (i.e., performance benchmark) of the current system based on the bandwidth value collected under a preset ratio and the first formula, providing a basis for the front end to actively abandon the adjustment of the traffic splitting ratio during the adjustment and application phases. However, as mentioned above, the system's operating conditions are constantly changing, sometimes rapidly and drastically, and may have already undergone drastic changes before the end of a learning cycle. At this time, the accuracy of the deduplication traffic splitting strategy adjustment can no longer be guaranteed. While simply reducing the value of the learning cycle can solve the above problem, it will bring a new problem that affects the efficiency of the deduplication traffic splitting strategy adjustment, so it is also not advisable.

[0105] Based on this, this embodiment also provides a preferred implementation scheme, wherein the above method further includes:

[0106] S16: Determine whether the host pressure of the system has changed based on the real-time collected bandwidth value and performance benchmark. If it has changed, return to the step of sequentially adjusting the front-end actively abandoning the diversion ratio to at least two different preset ratios and recording the corresponding bandwidth value (i.e., return to step S12).

[0107] Furthermore, this embodiment also provides a preferred implementation scheme for determining whether the host pressure of the system has changed, including:

[0108] When the load is low, if the currently collected bandwidth value is greater than 200% of the performance benchmark, then it is determined that the host pressure has changed.

[0109] When the load is high, if the currently collected bandwidth value is greater than 150% of the performance benchmark, then the host pressure is considered to have changed.

[0110] That is, the complete step S16 includes:

[0111] S161: When the load status is low, determine whether the currently collected bandwidth value is greater than 200% of the performance benchmark. If yes, proceed to step S12; otherwise, proceed to step S141.

[0112] S162: When the load status is high load, determine whether the currently collected bandwidth value is greater than 150% of the performance benchmark. If yes, proceed to step S12; otherwise, proceed to step S142.

[0113] It is easy to understand that, under normal circumstances, the collected system bandwidth value should fluctuate within a certain range. The performance benchmark, as a value reflecting the expected performance value of the system under the current state, should not exceed the performance benchmark by too much in real time (200% of the performance benchmark under low load and 150% of the performance benchmark under high load, and the values ​​of 200% and 150% are adjustable). Therefore, if it exceeds the benchmark, it indicates that the pressure on the system host has changed significantly and has seriously affected the system performance (reflected in the abnormal increase in the bandwidth value). At this time, the load state and performance benchmark determined in the current learning cycle are no longer applicable, and there may have been significant changes. Therefore, it is necessary to end the current learning cycle and jump to the adjustment process of the next learning cycle, that is, return to step S12.

[0114] This embodiment periodically repeats the determination of system load status to adapt the deduplication and traffic splitting strategy to constantly changing system operating pressure, thereby improving the accuracy of the deduplication and splitting strategy. Furthermore, this embodiment also uses the difference between the real-time collected bandwidth value and the performance benchmark to determine if there are changes in host pressure. If so, it indicates that the current system load status and performance benchmark parameters have changed, requiring the current learning cycle to end and a new learning cycle to adjust the deduplication strategy to adapt to the changing actual situation, thus ensuring the accuracy of the deduplication and traffic splitting adjustment method.

[0115] To further illustrate the preferred implementation schemes provided in the above embodiments, the deduplication and traffic adjustment method will be further explained below with reference to examples, such as... Figure 4 As shown, it includes:

[0116] S201: Initialization phase parameter is sampling, start timer to sample real-time bandwidth value.

[0117] S202: Determine whether the stage parameter is a sample. If yes, proceed to step S203; otherwise, proceed to step S209.

[0118] S203: Adjust the front-end active traffic abandonment ratio to 100%, 50%, and 0% in sequence, and record the corresponding three bandwidth values.

[0119] S204: Determine whether the three bandwidth values ​​meet the requirement that the change range is less than 5% or increases sequentially. If yes, proceed to step S205; otherwise, proceed to step S206.

[0120] S205: Set the load flag to low load.

[0121] S206: Set the load flag to high load.

[0122] S207: Determine the performance benchmark based on three bandwidth values ​​and the first formula.

[0123] The first formula is:

[0124] PS = 0.7 × AVG + 0.3 × CV;

[0125] PS is the performance baseline; AVG is the average value of the bandwidth corresponding to each preset ratio; CV represents the current value, which is the maximum value of the bandwidth corresponding to each preset ratio under low load and 80% of the maximum value of the bandwidth corresponding to each preset ratio under high load.

[0126] S208: Set the stage parameter to adjustment, and return to step S202.

[0127] S209: Determine whether the number of samplings is less than 300. If yes, proceed to step S210; otherwise, proceed to step S215.

[0128] The purpose of this step is to set the learning cycle. Since the learning cycle is 1 min = 200 ms * 300, the bandwidth value is sampled 300 times in one learning cycle. Therefore, the number of samplings triggered by the timer is used to determine whether a learning cycle has ended.

[0129] S210: Determine whether the load flag is low load. If yes, proceed to step S211; otherwise, proceed to step S212.

[0130] S211: Determine whether the bandwidth value obtained in this acquisition is less than 200% of the performance benchmark. If so, proceed to step S213; otherwise, proceed to step S215.

[0131] S212: Determine whether the bandwidth value obtained in this acquisition is less than 150% of the performance benchmark. If yes, proceed to step S214; otherwise, proceed to step S215.

[0132] S213: Adjust the front-end actively abandons the diversion ratio to 0%.

[0133] S214: The PID controller adjusts the proportion of the front end actively abandoning the shunt based on the currently acquired bandwidth value and performance benchmark.

[0134] S215: Set the stage parameter to sampling, and return to step S202.

[0135] In the above embodiments, a deduplication and traffic adjustment method has been described in detail. This application also provides an embodiment corresponding to a deduplication and traffic adjustment device. It should be noted that this application describes the device embodiment from two perspectives: one is based on functional modules, and the other is based on hardware.

[0136] From the perspective of functional modules, such as Figure 5 As shown, this embodiment provides a deduplication and traffic adjustment device, including:

[0137] The bandwidth acquisition module 21 is used to register a timed task to acquire the system's bandwidth value in real time according to a preset sampling period;

[0138] The baseline sampling module 22 is used to sequentially adjust the front-end actively abandons the diversion ratio to at least two different preset ratios and record the corresponding bandwidth values; wherein, the front-end actively abandons the diversion ratio is the ratio of the data that the system diverts from the front-end to the back-end for deduplication relative to all data;

[0139] The load determination module 23 is used to determine the current load status of the system based on the collected bandwidth values ​​according to a preset ratio.

[0140] The proportional adjustment module 24 is used to adjust the proportion of the front end actively abandoning traffic splitting for data deduplication based on the real-time collected bandwidth value and load status.

[0141] Preferably, the above-mentioned deduplication and traffic adjustment device further includes:

[0142] The benchmark determination module is used to determine the performance benchmark according to the bandwidth value corresponding to a preset ratio and a first formula; wherein the first formula is:

[0143] PS = 0.7 × AVG + 0.3 × CV;

[0144] PS is the performance baseline; AVG is the average value of the bandwidth corresponding to each preset ratio; CV represents the current value, which is the maximum value of the bandwidth corresponding to each preset ratio under low load and 80% of the maximum value of the bandwidth corresponding to each preset ratio under high load.

[0145] The stress detection module is used to determine whether the host stress of the system has changed based on the real-time collected bandwidth value and performance benchmark. If a change occurs, the benchmark sampling module is triggered.

[0146] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0147] This embodiment provides a deduplication and traffic adjustment device that leverages the fact that different deduplication strategies have varying impacts on system performance. It uses a bandwidth acquisition module to collect bandwidth values ​​in real time; a reference sampling module to collect bandwidth values ​​at a preset ratio; a load determination module to determine the current system load status; and finally, a ratio adjustment module dynamically adjusts the proportion of front-end data to be actively abandoned based on the real-time collected bandwidth values. This means adjusting the ratio of deduplicated data to online deduplication and background deduplication, fully utilizing the advantages of online and background deduplication while minimizing their negative impacts, thus meeting the needs of actual data deduplication application scenarios. For example, when the system is under low load and resources are sufficient, using online deduplication to process deduplicated data will not significantly impact system performance. Therefore, all deduplicated data can be offloaded to online deduplication to ensure timely data reduction. However, when the system is under high load, resources are no longer sufficient, and using online deduplication will affect system performance. Considering that the industry generally allows a 20% performance degradation when deduplication is enabled, the proportion of data actively abandoned by the front end can be adjusted based on this standard. This allows as much deduplicated data as possible to be offloaded to online deduplication without compromising system performance by 20%, thereby maximizing the timeliness of data deduplication while ensuring system performance. For application scenarios that prioritize system performance, the proportion of data actively abandoned by the front end can also be adjusted based on other standards to adapt to data deduplication scenarios with more diverse needs.

[0148] Figure 6 A structural diagram of a deduplication and shunt adjustment device provided in another embodiment of this application is shown below. Figure 6 As shown, a deduplication and traffic adjustment device includes: a memory 30 for storing computer programs;

[0149] The processor 31 is used to implement the steps of a deduplication and traffic adjustment method as described in the above embodiment when executing a computer program.

[0150] The deduplication and traffic adjustment device provided in this embodiment can include, but is not limited to, servers, laptops, or desktop computers.

[0151] The processor 31 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 31 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 31 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 31 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 31 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0152] The memory 30 may include one or more computer-readable storage media, which may be non-transitory. The memory 30 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 30 is used to store at least the following computer program 301, which, after being loaded and executed by the processor 31, is capable of implementing the relevant steps of the deduplication and load balancing method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 30 may also include an operating system 302 and data 303, and the storage method may be temporary or permanent storage. The operating system 302 may include Windows, Unix, Linux, etc. The data 303 may include, but is not limited to, a deduplication and load balancing method.

[0153] In some embodiments, a deduplication and descaling adjustment device may further include a display screen 32, an input / output interface 33, a communication interface 34, a power supply 35, and a communication bus 36.

[0154] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on a deduplication and shunt adjustment device and may include more or fewer components than shown.

[0155] This application provides a deduplication and traffic adjustment device, including a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a deduplication and traffic adjustment method.

[0156] The deduplication and traffic adjustment device provided in this embodiment executes a computer program stored in the memory through a processor to understand the current system load status by collecting bandwidth values ​​at a preset ratio. Then, it dynamically adjusts the proportion of traffic that the front end actively abandons based on the real-time collected bandwidth values. That is, it adjusts the proportion of deduplicated data to be distributed to online deduplication and background deduplication, making full use of the advantages of online deduplication and background deduplication and reducing their adverse effects, so as to meet the needs of actual data deduplication application scenarios. For example, when the system is under low load and resources are sufficient, using online deduplication to process deduplicated data will not significantly impact system performance. Therefore, all deduplicated data can be offloaded to online deduplication to ensure timely data reduction. However, when the system is under high load, resources are no longer sufficient, and using online deduplication will affect system performance. Considering that the industry generally allows a 20% performance degradation when deduplication is enabled, the proportion of data actively abandoned by the front end can be adjusted based on this standard. This allows as much deduplicated data as possible to be offloaded to online deduplication without compromising system performance by 20%, thereby maximizing the timeliness of data deduplication while ensuring system performance. For application scenarios that prioritize system performance, the proportion of data actively abandoned by the front end can also be adjusted based on other standards to adapt to data deduplication scenarios with more diverse needs.

[0157] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0158] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] The computer-readable storage medium provided in this embodiment can, when the computer program stored therein is executed, realize the current system load status by collecting bandwidth values ​​at a preset ratio, and then dynamically adjust the front-end actively abandoning the diversion ratio by collecting bandwidth values ​​in real time. That is, adjust the ratio of deduplicated data diverted to online deduplication and background deduplication, make full use of the advantages of online deduplication and background deduplication, reduce their adverse effects, and meet the needs of actual data deduplication application scenarios. For example, when the system is under low load and resources are sufficient, using online deduplication to process deduplicated data will not significantly impact system performance. Therefore, all deduplicated data can be offloaded to online deduplication to ensure timely data reduction. However, when the system is under high load, resources are no longer sufficient, and using online deduplication will affect system performance. Considering that the industry generally allows a 20% performance degradation when deduplication is enabled, the proportion of data actively abandoned by the front end can be adjusted based on this standard. This allows as much deduplicated data as possible to be offloaded to online deduplication without compromising system performance by 20%, thereby maximizing the timeliness of data deduplication while ensuring system performance. For application scenarios that prioritize system performance, the proportion of data actively abandoned by the front end can also be adjusted based on other standards to adapt to data deduplication scenarios with more diverse needs.

[0160] The foregoing has provided a detailed description of a deduplication and traffic adjustment method, apparatus, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0161] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A deduplication and traffic splitting adjustment method, characterized in that, include: Register a scheduled task to collect the system's bandwidth value in real time according to a preset sampling period; The front-end actively abandons traffic splitting ratio is adjusted sequentially to at least two different preset ratios, and the corresponding bandwidth values ​​are recorded. The front-end actively abandons traffic splitting ratio is the proportion of data that the system splits from the front-end to the back-end for deduplication relative to all data. The preset ratios include: a first preset ratio, a second preset ratio, and a third preset ratio, and the values ​​of the first preset ratio, the second preset ratio, and the third preset ratio decrease sequentially. If the bandwidth values ​​corresponding to the first preset ratio, the second preset ratio, and the third preset ratio change less than the preset range or increase sequentially, the current load state of the system is determined to be low load; otherwise, it is determined to be high load. When the load is low, the proportion of traffic actively abandoned by the front end is adjusted to 0%; when the load is high, the proportion of traffic actively abandoned by the front end is adjusted by a control algorithm based on the real-time collected bandwidth value.

2. The deduplication and traffic adjustment method according to claim 1, characterized in that, The steps of sequentially adjusting the front-end's active abandonment of traffic splitting ratio to at least two different preset ratios and recording the corresponding bandwidth values ​​are repeated according to the preset learning cycle.

3. The deduplication and traffic adjustment method according to claim 2, characterized in that, After sequentially adjusting the front-end actively relinquishing traffic splitting ratio to at least two different preset ratios and recording the corresponding bandwidth values, the process also includes: The performance benchmark is determined according to the bandwidth value corresponding to the preset ratio and the first formula. The first formula is: PS = 0.7 × AVG + 0.3 × CV; PS is the performance baseline; AVG is the average bandwidth value corresponding to each preset ratio; CV represents the current value, which is the maximum bandwidth value corresponding to each preset ratio under low load and 80% of the maximum bandwidth value corresponding to each preset ratio under high load. Accordingly, based on the real-time collected bandwidth values, the control algorithm adjusts the proportion of traffic that the front end actively relinquishes, including: Based on the difference between the real-time collected bandwidth value and the performance benchmark, the control algorithm adjusts the proportion of traffic that the front end actively abandons.

4. The deduplication and traffic adjustment method according to claim 3, characterized in that, Also includes: Based on the real-time collected bandwidth value and the performance benchmark, determine whether the host pressure of the system has changed. If it has changed, return to the step of sequentially adjusting the front-end actively abandoning the diversion ratio to at least two different preset ratios and recording the corresponding bandwidth value.

5. The deduplication and traffic adjustment method according to claim 4, characterized in that, Determining whether the system's host load has changed based on real-time collected bandwidth values ​​and performance benchmarks includes: When the load is low, if the currently collected bandwidth value is greater than 200% of the performance benchmark, it is determined that the host pressure has changed. When the load is high, if the currently collected bandwidth value is greater than 150% of the performance benchmark, it is determined that the host pressure has changed.

6. The deduplication and traffic adjustment method according to any one of claims 1 to 5, characterized in that, The control algorithm is the PID control algorithm.

7. A deduplication and diversion adjustment device, characterized in that, include: The bandwidth acquisition module is used to register a timed task to acquire the system's bandwidth value in real time according to a preset sampling period; The baseline sampling module is used to sequentially adjust the front-end actively abandons the diversion ratio to at least two different preset ratios and record the corresponding bandwidth values; wherein, the front-end actively abandons the diversion ratio is the proportion of data that the system diverts from the front-end to the back-end for deduplication relative to all data; The load determination module is used to determine that the current load state of the system is low if the bandwidth value corresponding to the first preset ratio, the second preset ratio and the third preset ratio changes less than a preset range or increases sequentially; otherwise, it is determined to be high load. The proportional adjustment module is used to adjust the front-end actively abandoning traffic ratio to 0% when the load is low; and to adjust the front-end actively abandoning traffic ratio according to the real-time collected bandwidth value through a control algorithm when the load is high.

8. A deduplication and diversion adjustment device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the steps of the deduplication and traffic adjustment method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the deduplication and traffic adjustment method as described in any one of claims 1 to 6.

Images