A method for improving storage system write performance

Abstract

The present invention discloses a method for improving the write performance of a storage system, and the method for improving the performance includes the following steps: the first step is to send data and a check value; the second step is for the server to verify the check value; the third step is to return an error code; the fourth step is to recalculate the client data check value and perform fault logic processing; the fifth step is to set a lease and enable a data copy option. The present invention provides a method for improving the write performance of a storage system, which realizes automatic recovery of a zero-copy mechanism through multiple check calculations and interactions, as well as a lease, reduces the number of data copies, and improves the overall performance of the storage system. At the same time, compared with the existing method of always closing the zero-copy mechanism by default, or closing the data verification mechanism, the invention judges the accuracy of the data by comparing the check value when in use, has the accuracy of the data, and avoids the stored data being erroneous data.

Description

Technical Field

[0001] The present invention relates to the field of emerging information technology, and in particular to a method for improving the writing performance of a storage system. Background Art

[0002] When the database writes BinLog, it actually calls the write interface to write user-state data into a binary file. The file will be sent to the storage system through the network for persistent storage, which brings challenges to the RDMA zero-copy mechanism. Because the network card directly operates on the user-state memory, there is a situation where the data is modified before it is successfully sent, resulting in inconsistent data verification values. To solve this problem, the first traditional method is to add a data copy action, that is, when the database writes BinLog, its user-state memory data is copied to another user-state memory, and then sent by RDMA. However, in this way, the database's reuse and modification of its own user-state memory will not affect the data transmission, but this method sacrifices the performance of the storage system. The second method is not to verify the data during the data transmission process, but this sacrifices the reliability of the data. When there are defects in the program, network errors, etc., they cannot be identified in time, resulting in the storage of erroneous data. Summary of the invention

[0003] The purpose of the present invention is to provide a method for improving the write performance of a storage system to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for improving the write performance of a storage system, the method for improving the performance comprising the following steps:

[0005] The first step is to send data and checksums. Use the data transmission module to send the client's data and the checksum of the data to the server, and name the checksum value value1.

[0006] In the second step, the server verifies the check value and recalculates the check value based on the data transmitted by the client. It names the check value value2 and compares value1 with value2 to determine whether the data is normal.

[0007] The third step is to send back the error code. According to the data results obtained in the second step, when the data is abnormal, the error code is sent to the client;

[0008] Step 4: Recalculate the client data checksum and perform fault logic processing. Recalculate the client data checksum and compare it with value1 to determine the cause of the problem.

[0009] The fifth step is to set the lease and enable the data copy option. This is to set the lease time to periodically turn off the data copy option and use the zero copy mechanism to improve performance.

[0010] Preferably, the sending of the data and the check value includes the calculation and packaging of the original data check value;

[0011] The calculation of the data check value is to calculate the check value value1 of the data through a checksum algorithm;

[0012] The package transmission is to send the user-side data and the check value obtained by calculating the data check value to the service side together.

[0013] Preferably, the verification value calculation performed by the server includes receiving data, calculating the verification value of the received data, comparing the verification value of the server, and judging the server data.

[0014] Preferably, the data is received by receiving the data and the check value value1 transmitted from the client through the receiving entrance of the server, the calculation of the check value of the received data is to calculate the check value value2 of the data based on the checksum algorithm, the check value comparison of the server is to compare the check value value1 obtained by the client with the check value value2 obtained by the server, and the judgment of the server data is to judge whether the data transmitted to the server is wrong based on the comparison of the two check values.

[0015] Preferably, the judgment of the server data includes the following situations:

[0016] The first one is that if the check value value2 is the same as the check value value1, it means the data is normal;

[0017] The second type is that if the check value value2 is different from the check value value1, it means that the data is abnormal.

[0018] Preferably, the recalculating the client data check value includes retrieving data and recalculating the check value;

[0019] The data retrieval is to extract the data transmitted to the server in the first step;

[0020] The recalculation of the check value is to calculate the check value value3 of the data through a checksum algorithm.

[0021] Preferably, the fault logic processing includes client check value comparison and fault determination;

[0022] The client check value comparison is to compare the check value value1 with the recalculated check value value3;

[0023] The fault is judged by comparing the check value of the client to determine the cause of the fault.

[0024] Preferably, the fault determination includes the following situations:

[0025] The first one is that if the check value value1 is the same as the check value value3, it means that the data has not been modified during the transmission process, and there is a problem in the transmission process;

[0026] The second type is that if the check value value1 is different from the check value value3, it means that the data has been modified.

[0027] Preferably, the step of setting up a lease and enabling the data copy option includes the following steps:

[0028] S1, setting the lease time, is to limit the length of the lease by setting a specified time;

[0029] S2, start data copy and open lease, is to copy and send data, and limit the copy time by opening lease;

[0030] S3, restore the zero-copy mechanism, through the lease time set in S1. After the lease time expires, turn off the copy option and continue to use the zero-copy mechanism.

[0031] Technical effects and advantages of the present invention:

[0032] The method for improving the write performance of the storage system realizes automatic recovery of the zero-copy mechanism through multiple verification calculations and interactions, as well as a lease method, thereby reducing the number of data copies and improving the overall performance of the storage system. At the same time, compared with the existing method of always turning off the zero-copy mechanism or the data verification mechanism by default, the invention judges the accuracy of the data by comparing the verification value during use, thereby ensuring the accuracy of the data and avoiding the storage of erroneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a schematic diagram of the implementation flow of the method for improving performance of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The present invention provides Figure 1A method for improving the write performance of a storage system is shown, and the method for improving the performance includes the following steps:

[0036] The first step is to send data and checksums. Use the data transmission module to send the client's data and the checksum of the data to the server, and name the checksum value value1.

[0037] In the second step, the server verifies the check value and recalculates the check value based on the data transmitted by the client. It names the check value value2 and compares value1 with value2 to determine whether the data is normal.

[0038] The third step is to send back the error code. According to the data results obtained in the second step, when the data is abnormal, the error code is sent to the client;

[0039] Step 4: Recalculate the client data checksum and perform fault logic processing. Recalculate the client data checksum and compare it with value1 to determine the cause of the problem.

[0040] The fifth step is to set the lease and enable the data copy option. This is to set the lease time to periodically turn off the data copy option and use the zero copy mechanism to improve performance.

[0041] It should be noted that the English word value refers to a number that represents a certain quantity, size or value, and thus it is used to represent the size of the data check value; the zero-copy mechanism is an optimization technology for computer data transmission and processing, which aims to reduce the number of times data is copied within the system and improve the efficiency and performance of data transmission. In actual use, the zero-copy mechanism avoids unnecessary data copying and directly transfers data from the source buffer to the target buffer, reducing the number of times data is copied between memories, thereby reducing the burden on the CPU and improving the efficiency of data transmission; the data transmission module is set up to connect the client and the server. When in use, the client's data information is transmitted to the server through the data transmission module, and the server's information is transmitted to the client through the data transmission module.

[0042] Furthermore, automatic recovery of the zero-copy mechanism is achieved through multiple verification calculations and interactions, as well as a lease, thereby reducing the number of data copies and improving the overall performance of the storage system. At the same time, compared with the existing method of always turning off the zero-copy mechanism or the data verification mechanism by default, the invention judges the accuracy of the data by comparing the verification value when in use, thereby ensuring the accuracy of the data and avoiding the storage of erroneous data.

[0043] Specifically, the sending of data and check values ​​includes the calculation and packaging of the original data check values;

[0044] The calculation of the data check value is to calculate the check value value1 of the data through the checksum algorithm;

[0045] Packaging and transmission is to send the user-side data and the check value obtained by calculating the data check value to the server side.

[0046] It should be noted that the checksum algorithm calculates the checksum by adding and inverting the value of each byte in the data and adding the result to the end of the original data, including the following steps: A1, dividing the data into bytes of equal length; A2, adding the byte values; A3, inverting the result to obtain the checksum value.

[0047] Furthermore, before sending the data from the client to the server, the check value of the original data is first calculated by calculating the check value of the original data, and the calculated result is recorded as value1, and then the original data and the calculated check value value1 are sent to the server together.

[0048] Specifically, the verification value calculation performed by the server includes receiving data, calculating the verification value of the received data, comparing the verification value of the server, and judging the server data.

[0049] Specifically, data is received by receiving the data and check value value1 transmitted from the client through the receiving entrance of the server, the calculation of the check value of the received data is to calculate the check value value2 of the data according to the checksum algorithm, the check value comparison of the server is to compare the check value value1 obtained by the client with the check value value2 obtained by the server, and the judgment of the server data is to judge whether the data transmitted to the server is wrong based on the comparison of the two check values.

[0050] Specifically, the judgment of server-side data includes the following situations:

[0051] The first one is that if the check value value2 is the same as the check value value1, it means the data is normal;

[0052] The second type is that if the check value value2 is different from the check value value1, it means that the data is abnormal.

[0053] Furthermore, after the server receives the data and check value input from the user side from the client side, it calculates the obtained data through the same algorithm as the calculation of the data check value to obtain a new check value value2, and then compares the new check value value2 with the check value value1 of the same data received from the user side. When the two values ​​are consistent, it indicates that the data is not wrong. When the two values ​​are inconsistent, it indicates that the data is wrong. When the data is wrong, the error code is returned through the third step to send the data error information back to the user side.

[0054] Specifically, recalculating the client data check value includes retrieving data and recalculating the check value;

[0055] Retrieving data is to extract the data transmitted to the server in the first step;

[0056] The checksum algorithm is used to calculate the checksum value of the data to obtain value 3.

[0057] Specifically, fault logic processing includes client checksum value comparison and fault determination;

[0058] The client checksum comparison is to compare the checksum value value1 with the recalculated checksum value value3;

[0059] The fault is judged based on the result of comparing the client's verification value.

[0060] Specifically, the fault judgment includes the following situations:

[0061] The first one is that if the check value value1 is the same as the check value value3, it means that the data has not been modified during the transmission process, and there is a problem in the transmission process;

[0062] The second type is that if the check value value1 is different from the check value value3, it means that the data has been modified.

[0063] Furthermore, when the client receives the data error, it first retrieves the original data, and then calculates the check value value3 of the original data through the checksum algorithm, and compares the check value value3 with the check value value1. When the check value value3 is compared with the check value value1, when the check value value3 is consistent with the check value value1, it means that there is a problem with the data during the occurrence process. When the check value value3 is different from the check value value1, it means that the data is modified, and then the modified data is resent to the server.

[0064] Specifically, setting up a lease and enabling the data copy option includes the following steps:

[0065] S1, setting the lease time, is to limit the length of the lease by setting a specified time;

[0066] S2, start data copy and open lease, which is to copy and send data, and limit the copy time by opening the lease;

[0067] S3, restore the zero-copy mechanism, through the lease time set in S1. After the lease time expires, turn off the copy option and continue to use the zero-copy mechanism. Specific embodiment 1

[0069] Set the lease to a fixed time. When the lease is opened, the fixed time will be counted down. When the time is up, the data copy will be closed. Specific embodiment 2

[0071] Set the lease to a fixed time and send a zero-copy detection IO at fixed intervals. If the IO has not been modified, the lease time will be shortened; if it has been modified, the lease time will be increased, thereby flexibly changing the lease time. Specific embodiment three

[0073] The lease is set to be valid permanently, but the system is forced to restore the zero-copy mechanism at fixed intervals.

[0074] Furthermore, when the data is modified, data copying is started and the lease is opened, and the action data and checksum are sent to the server. Before the lease expires, the subsequent data will be copied and sent to avoid the problem of inconsistent checksums caused by data modification. When the transmission time reaches the time used by the lease, the data copy option is turned off, and the zero-copy mechanism is continued to be used to improve performance, thereby realizing the on-demand activation of the data copy mechanism, avoiding the problem of long-term low system performance caused by the default or long-term activation of the data copy option, thereby improving the write performance of the storage system.

[0075] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for improving the write performance of a storage system, characterized in that: The method for improving the write performance of the storage system includes the following steps: In the first step, the client sends data and a checksum value. The data transmission module is used to send the client's data and the checksum value of the data to the server, and the checksum value is named value1. In the second step, the server verifies the check value and recalculates the check value according to the data transmitted by the client. The check value is named value2, and value1 is compared with value2 to determine whether the data is normal. When the values ​​of the two are the same, it means that the data is correct and the process ends. When the values ​​of the two are different, it means that the data is wrong and enters the third step. In the third step, the server sends back the error code. According to the data verification result obtained in the second step, if the data is abnormal, the error code is sent to the client. In the fourth step, the client recalculates the data checksum and performs fault logic processing. The checksum of the client data is calculated again, named value3, and compared with value1. If the two are the same, it indicates that the data has not been modified during the transmission process, and there is a problem in the transmission process. If the two are different, it means that the data has been modified, and the process goes to the fifth step. In the fifth step, the client sets a lease and turns on the data copy option. This is done by setting the lease time to periodically turn off the data copy option and using a zero-copy mechanism to improve performance.

2. A method for improving storage system write performance according to claim 1, characterized in that: The sending of the data and the check value includes the calculation and packaging of the original data check value; The calculation of the data check value is to calculate the check value value1 of the data through a checksum algorithm; The package transmission is to send the client's data and the verification value obtained by data verification calculation to the server together.

3. A method for improving storage system write performance according to claim 1, characterized in that: The verification value calculation performed by the server includes receiving data, calculating the verification value of the received data, comparing the verification value of the server, and judging the data by the server.

4. A method for improving storage system write performance according to claim 3, characterized in that: The data is received by receiving the data and the check value value1 transmitted from the client through the receiving entrance of the server, the calculation of the check value of the received data is to calculate the check value value2 of the data according to the checksum algorithm, the check value comparison of the server is to compare the check value value1 obtained by the client with the check value value2 obtained by the server, and the judgment of the data by the server is to judge whether the data transmitted to the server is wrong based on the comparison of the two check values.

5. A method for improving storage system write performance according to claim 4, characterized in that: The data judgment of the server includes the following situations: The first one is that if the check value value2 is the same as the check value value1, it means the data is normal; The second type is that if the check value value2 is different from the check value value1, it means that the data is abnormal.

6. A method for improving storage system write performance according to claim 1, characterized in that: The client recalculates the data check value, including retrieving data and recalculating the check value; The data retrieval is to extract the data transmitted to the server in the first step; The recalculation of the check value is to calculate the check value value3 of the data through a checksum algorithm.

7. A method for improving storage system write performance according to claim 6, characterized in that: The fault logic processing includes client check value comparison and fault determination; The client check value comparison is to compare the check value value1 with the recalculated check value value3; The fault is judged by comparing the check value of the client to determine the cause of the fault.

8. A method for improving storage system write performance according to claim 7, characterized in that: The fault judgment includes the following situations: The first one is that if the check value value1 is the same as the check value value3, it means that the data has not been modified during the transmission process, and there is a problem in the transmission process; The second type is that if the check value value1 is different from the check value value3, it means that the data has been modified.

9. The method for improving the write performance of a storage system according to claim 1, characterized in that: Setting up a lease and enabling the data copy option includes the following steps: S1, setting the lease time, is to limit the length of the lease by setting a specified time; S2, start data copy and open lease, which is to copy and send data, and limit the copy and send time by opening the lease; S3, restore the zero-copy mechanism, through the lease time set in S1. After the lease time expires, turn off the copy option and continue to use the zero-copy mechanism.

Images