Data verification method and device and electronic equipment

Abstract

The invention provides a data verification method and device and electronic equipment, belongs to the technical field of data verification, and aims to ensure the integrity and accuracy of data in a distributed file system. The method comprises the following steps: determining a data verification layer on a data transmission path between a kernel mode client and a storage node; under the condition that a target file is transmitted through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, and the target verification strategy is determined according to the load condition of the system and the importance degree of the target file; and judging whether the target file has a transmission error or not according to the verification data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data verification, and in particular to a data verification method, device and electronic equipment. Background Art

[0002] In today's digital age, data generation and storage are growing explosively. Distributed file systems, as a solution that can effectively cope with large-scale data storage and high concurrent access requirements, are gradually becoming the core infrastructure of many key business scenarios. In this technical context, how to ensure the integrity and accuracy of data in distributed file systems has become a key issue that needs to be solved urgently. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a data verification method, device and electronic device. The technical solution of the present disclosure is as follows: According to a first aspect of an embodiment of the present disclosure, there is provided a data verification method, comprising: Determine a data verification layer on the data transmission path between the kernel-mode client and the storage node; In the case of transmitting the target file through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, and the target verification strategy is determined according to the load of the system and the importance of the target file; According to the verification data, it is determined whether a transmission error occurs in the target file.

[0004] Optionally, the data transmission path includes a transmission process of a target protocol, and determining a data verification layer on the data transmission path between a kernel-mode client and a storage node includes: In combination with the kernel-mode client and the target protocol, a transmission stage before a transmission process from the kernel-mode client to the target protocol and a transmission stage before a transmission process from the target protocol to the kernel-mode client are determined as key positions of the data transmission path; According to the key position of the data transmission path, it is determined to abstract an independent data verification layer in the kernel state client, and the data verification layer is integrated into the kernel state client.

[0005] Optionally, the target verification strategy is determined by: Obtain multiple alternative verification algorithms; Determine the current system load and the importance of the target file; When the load of the current system is higher than a preset load value, selecting a number of alternative verification algorithms less than or equal to a preset number from the plurality of alternative verification algorithms and adding them to the target verification strategy; When the importance of the target file is greater than the importance threshold, a number of candidate verification algorithms greater than a preset number are selected from the plurality of candidate verification algorithms and added to the target verification strategy.

[0006] Optionally, when the target file is written into the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: The target file is prepared by the client and submitted to the kernel-mode client; The kernel state client transmits the target file to the data verification layer; The data verification layer verifies the target file according to the target verification strategy to obtain first verification data; The step of determining whether a transmission error occurs in the target file according to the verification data includes: Adding the first verification data to the target file; Transmitting the target file carrying the first verification data to the storage node; After receiving the target file, the storage node uses the same target verification strategy to obtain second verification data of the target file; At the storage node, by judging whether the first verification data and the second verification data are consistent, it is determined whether a transmission error occurs in the target file.

[0007] Optionally, after determining whether a transmission error occurs in the target file, the method further includes: When the first verification data and the second verification data are consistent, determining that no transmission error occurs to the target file, and writing the target file into a storage medium corresponding to the storage node; When the first verification data and the second verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to resend the target file.

[0008] Optionally, when the target file is read from the storage node through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, including: Initiate a read request for the target file to the storage node through the client; After the storage node receives the read request, the storage node reads the target file from a storage medium; In the data outer layer of the storage node, using the target verification strategy, obtain third verification data of the target file; Adding the third verification data to the target file; Transmitting the target file carrying the third verification data to the data verification layer; After the data verification layer receives the target file, the fourth verification data of the target file is obtained using the same target verification strategy; The step of determining whether a transmission error occurs in the target file according to the verification data includes: At the data verification layer, whether a transmission error occurs in the target file is determined by judging whether the third verification data is consistent with the fourth verification data.

[0009] Optionally, after determining whether a transmission error occurs in the target file, the method further includes: When the third verification data and the fourth verification data are consistent, determining that no transmission error occurs to the target file, and sending the target file to the client through the kernel-mode client, so that the user's application program uses the target file; In the case that the third verification data and the fourth verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to re-initiate a read request for the target file to the storage node.

[0010] Optionally, during data transmission between the verification data layer and the storage node, the data is transmitted through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided a data verification device, comprising: A determination module, used to determine a data verification layer on a data transmission path between a kernel-mode client and a storage node; A verification module, used for, when a target file is transmitted through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, wherein the target verification strategy is determined according to the load of the system and the importance of the target file; The judgment module is used to judge whether a transmission error occurs in the target file according to the verification data.

[0012] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data verification method described in the first aspect are implemented.

[0013] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data verification method described in the first aspect are implemented.

[0014] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the data verification method described in the first aspect are implemented.

[0015] The present disclosure can ensure the data integrity and accuracy of file reading and writing, reduce the risk of errors, timely discover and handle transmission errors, enhance system stability, and use flexible verification strategies to balance performance and reliability. The independent verification layer is easy to maintain and upgrade without affecting other functions of the system. The target verification strategy is determined according to the system load and the importance of the target file, and then the target verification strategy is used to verify the target file for data transmission. This dynamic adjustment of the target verification strategy can realize intelligent and automated verification strategy optimization, ensuring that data security and transmission accuracy are improved while reducing system overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments of the present disclosure will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a schematic diagram of steps of a data verification method shown in an embodiment of the present disclosure; Figure 2 is a target verification strategy adjustment flow chart shown in an embodiment of the present disclosure; Figure 3 is a flow chart showing a method of writing a target file to a storage node according to an embodiment of the present disclosure; Figure 4 is a flow chart of reading a target file from a storage node shown in an embodiment of the present disclosure; Figure 5 is a block diagram of a data verification device shown in an embodiment of the present disclosure; Figure 6 It is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0019] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable when appropriate, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] In today's digital age, the generation and storage of data are growing explosively. Various industries such as finance, medical care, scientific research, and the Internet have put forward extremely high requirements for data security, reliability, and efficient access. Distributed file systems, as a solution that can effectively cope with large-scale data storage and high concurrent access requirements, are gradually becoming the core infrastructure of many key business scenarios. With the continuous development of hardware technology, remote direct memory access (RDMA) technology provides a new way to achieve high-speed data transmission. The use of kernel-mode clients further improves the performance and stability of the system. Distributed file systems are widely used in large-scale data storage and processing scenarios due to their advantages such as high scalability, high reliability, and high performance. However, in the actual application of distributed file systems, there are still many challenges.

[0021] On the one hand, data may be affected by various factors and errors may occur during transmission and storage. For example, network failure, storage medium damage, software errors, etc. may cause data loss or damage. If these errors are not discovered and corrected in time, they will have a serious impact on the integrity and accuracy of the data, and may even cause system failure and data loss.

[0022] On the other hand, existing distributed file systems often have deficiencies in data verification. Some traditional verification methods are inefficient and cannot meet the needs of high-speed data transmission; while some complex verification mechanisms will bring large system overhead and affect system performance. In addition, existing verification mechanisms are usually implemented at a specific level of the file system, lacking flexibility and scalability.

[0023] In order to solve the above technical problems, the present disclosure proposes a data verification method, which can determine an independent data verification layer in the data transmission path, ensure the data integrity and accuracy of file reading and writing, reduce the risk of errors, and promptly detect and handle transmission errors.

[0024] Figure 1 FIG. 1 is a schematic diagram of a data verification method according to an embodiment of the present disclosure. Figure 1 As shown, the data verification method may specifically include the following steps: Step S11: determining a data verification layer on a data transmission path between a kernel-mode client and a storage node.

[0025] The kernel client runs on the client node, which is the entry point for users to access the storage cluster service. Unlike the storage service node, the client node usually does not directly participate in the storage and management of data, but serves as a bridge between the user and the storage cluster. The kernel client has the ability to directly access system resources, allowing users to use the storage cluster service like using a local file system.

[0026] The kernel-mode client directly interacts with the operating system kernel. Once a data error occurs, it may have a serious impact on the system. Therefore, although the use of the kernel-mode client can improve the performance and stability of the system, it also puts forward higher requirements for data verification.

[0027] A data verification layer is determined on the transmission path in which the kernel-mode client participates. Based on data transmission experience, the data verification layer can be determined at a specific location on the transmission path. When data or files are transmitted using the data transmission path, they will pass through the data verification layer. The data verification layer is independent of other functions, which makes it easy to maintain and upgrade the data verification layer without affecting other functions of the system.

[0028] Step S12: When the target file is transmitted through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data. The target verification strategy is determined according to the system load and the importance of the target file.

[0029] When transmitting the target file through the data transmission path, the current transmission situation is first determined, including the system load situation and the importance of the target file. According to the current transmission situation, the target verification strategy is dynamically adjusted to make the target verification strategy meet the current transmission requirements.

[0030] After determining the target verification strategy, the data verification layer is required to verify the file or data before the data is transferred to the kernel-mode client, and before the data is transferred from the kernel-mode client to the outside, to obtain verification data, which is a value calculated by the verification algorithm. The target file is verified by the data verification layer to ensure data stability and avoid data errors caused by data transmission problems.

[0031] Step S13: judging whether a transmission error occurs in the target file according to the verification data.

[0032] Based on the verification data obtained by the data verification layer, it is determined whether there is a transmission error in the target file, so as to timely discover and handle transmission errors, enhance system stability, and flexible verification strategies can balance performance and reliability.

[0033] By adopting the embodiments of the present disclosure, the target verification strategy is determined according to the system load and the importance of the target file, so that the verification strategy can be flexibly adjusted. By determining the data verification layer on the data transmission path between the kernel-mode client and the storage node, the parallel processing of data transmission and verification is realized. By implementing the verification layer on the data transmission path, it can be relatively easy to integrate with other file systems and network transmission protocols. By performing real-time verification during the data transmission process and immediately judging whether the target file has a transmission error based on the verification data, it is helpful to timely discover and correct errors in the transmission process and ensure the integrity and reliability of the data.

[0034] Among them, in an optional embodiment, the data transmission path includes a transmission process of the target protocol, and a data verification layer is determined on the data transmission path between the kernel-mode client and the storage node, including: combining the kernel-mode client and the target protocol, and determining the transmission stage before the transmission process from the kernel-mode client to the target protocol, and the transmission stage before the transmission process from the target protocol to the kernel-mode client, as the key position of the data transmission path; based on the key position of the data transmission path, determine to abstract an independent data verification layer in the kernel-mode client, and integrate the data verification layer in the kernel-mode client.

[0035] The target protocol can be RDMA. The emergence of RDMA technology provides a new idea for solving the data transmission problem in distributed file systems. RDMA allows data to be transmitted directly between memories without the involvement of the CPU, thereby greatly improving the speed and efficiency of data transmission.

[0036] In the case where the transmission path includes the transmission process of the target protocol and the kernel-mode client, the data transmission process between the target protocol and the kernel-mode client is clarified. According to the overall data transmission process, the key position in the transmission path is determined.

[0037] The last transmission stage before the kernel-mode client enters the target protocol transmission process, which may involve operations such as data preparation, encapsulation or formatting. The last transmission stage before the target protocol transmission process returns to the kernel-mode client may involve steps such as data decapsulation and processing.

[0038] Therefore, the transmission stage before the transmission process from the kernel mode client to the target protocol and the transmission stage before the transmission process from the target protocol to the kernel mode client can be determined as the key position of the data transmission path.

[0039] According to the key position, the data verification layer can be arranged in the kernel state client. Specifically, an independent data verification layer can be abstracted in the kernel state client so that the data verification layer is integrated in the kernel state client.

[0040] After abstracting an independent data verification layer in the kernel-mode client, the system can be fully tested to ensure that the data verification layer can work correctly and does not have a significant impact on system performance during data transmission.

[0041] By adopting the embodiments of the present disclosure, by determining and integrating an independent data verification layer on the data transmission path, the integrity and verification capabilities in the data transmission process can be significantly enhanced. The data verification layer is located between the kernel-mode client and the target protocol transmission process, and can ensure that the data is strictly verified when entering and leaving the kernel-mode client, thereby reducing the risk of data corruption or erroneous transmission. The existence of the data verification layer can promptly detect and correct errors in data transmission, preventing these erroneous data from being further processed or stored, thereby improving the security and reliability of the entire system.

[0042] Among them, in an optional embodiment, the method for determining the target verification strategy includes: obtaining multiple alternative verification algorithms; determining the load situation of the current system, and determining the importance of the target file; when the load of the current system is higher than a preset load value, selecting less than or equal to a preset number of alternative verification algorithms from the multiple alternative verification algorithms, and adding them to the target verification strategy; when the importance of the target file is greater than an importance threshold, selecting greater than a preset number of alternative verification algorithms from the multiple alternative verification algorithms, and adding them to the target verification strategy.

[0043] Collect multiple candidate verification algorithms, which may include hash algorithms, cyclic redundancy check algorithms, etc. The performance and applicability of each candidate verification algorithm may be evaluated in advance, for example, calculation speed, resource consumption, error detection rate, etc.

[0044] Real-time monitoring of the current load of the system, including CPU usage, memory usage, network bandwidth usage, etc. A preset load value can be set based on historical data and business rules as a basis for judging the system load. When the current system load is greater than or equal to the preset load value, it means that the system is currently under high load. Conversely, when the current system load is less than the preset load value, it means that the system is currently under low load.

[0045] When the target file is transmitted using the transmission path proposed in the present disclosure, the properties of the target file, such as file type, file size, file source, etc., can be analyzed to determine its importance. An importance threshold can be set according to business needs and industry standards to distinguish important files from non-important files. When the importance of the target file exceeds or equals the importance threshold, it is indicated that the target file is an important file. On the contrary, when the importance of the target file is lower than the importance threshold, it is indicated that the target file is not an important file.

[0046] Dynamically adjust the target verification strategy based on the current system load and the importance of the target file. Determine the verification strength of the target file by the target verification strategy based on the current data transmission situation. The verification strength of the target verification strategy can be reflected by determining the type of verification algorithm used by the target verification strategy. Specifically, if a looser target verification strategy is used based on the current data verification situation, fewer verification algorithms can be selected to add to the target verification strategy. When a stricter target verification strategy is used, more verification algorithms can be selected to add to the target verification strategy. The verification algorithms included in the target verification strategy can also be selected based on the performance and applicability of each verification algorithm to reflect the verification strength of the target verification strategy.

[0047] A preset number can be set in advance, and the preset number needs to be greater than or equal to 1. When the target file needs to be strictly verified, the number of selected verification algorithms needs to be greater than the preset number. When the target file does not need to be strictly verified, the number of selected verification algorithms is less than or equal to the preset number.

[0048] When the load of the current system is higher than the preset load value, the current system is judged to be high-loaded, so a looser target verification strategy is used for the target file, and a number of alternative verification algorithms less than or equal to the preset number are selected from the alternative verification algorithms and added to the target verification strategy. For example, when the alternative verification algorithms include a hash algorithm and a cyclic redundancy check algorithm, the preset number may be 1, and one of the two verification algorithms is selected, and the hash algorithm or the cyclic redundancy check algorithm is added to the target verification strategy.

[0049] When the importance of the target file is greater than the importance threshold, the current target file is determined to be an important file, and a number of alternative verification algorithms greater than a preset number are selected from the multiple alternative verification algorithms and added to the target verification strategy. For example, when the alternative verification algorithms include a hash algorithm and a cyclic redundancy check algorithm, the preset number may be 1, and both verification algorithms are added to the target verification strategy to improve the strictness of the verification.

[0050] In some cases, the system load may be higher than the preset load value and the importance of the target file may be greater than the importance threshold. Since the two situations have opposite effects on the adjustment of the target verification strategy, it is possible to combine and analyze which of the two situations is more in need of consideration at present, and adjust the target verification strategy based on this situation. In a specific implementation, when the system load is higher than the preset load value, a stricter target verification strategy may still be used for verification of important files. In a specific embodiment, when the system load is high, the verification strength of the target verification strategy may be dynamically reduced to reduce system overhead, and any one of the hash algorithm and the cyclic redundancy check algorithm may be selected as the target verification algorithm to calculate the verification data of the target file; when the file to be transmitted is an important file, the target verification algorithm is obtained by combining the hash algorithm and the cyclic redundancy check algorithm, and the verification data of the target file is calculated by the combined target verification algorithm.

[0051] Figure 2 is a target verification strategy adjustment flow chart shown in an embodiment of the present disclosure. Figure 2 As shown, when the current system load is higher than the preset load value, the target verification strategy is dynamically and adaptively adjusted to reduce the verification strength of non-important files; when the target file is an important file, the target verification strategy is dynamically and adaptively adjusted, and a stricter verification algorithm is used to improve the verification strength.

[0052] By adopting the embodiments of the present disclosure, by obtaining multiple alternative verification algorithms, the system can have a wider range of choices, thereby selecting the optimal verification algorithm according to different situations. Taking into account the system load, the system can reduce the number of verification algorithms when resources are tight, thereby freeing up computing resources for other more important tasks. By selecting the number of verification algorithms according to the importance of the target file, it can be ensured that important files receive a higher level of protection. The system can be allowed to dynamically adjust the verification strategy according to real-time conditions, thereby improving the flexibility and adaptability of the system to adapt to different working environments and needs.

[0053] The following are the specific steps for writing data using the data transmission path configured with the data verification layer.

[0054] Among them, in an optional embodiment, a machine learning or prediction model can be introduced to dynamically adjust the verification strategy based on historical data, such as system load change trends, file transfer error rates, etc., so as to achieve more refined resource allocation and higher error detection rates. Specifically, a monitoring and prediction system can be established to collect and analyze data such as system load and file transfer history, and use these data to train models to predict possible future system states and file transfer requirements, thereby dynamically adjusting the strictness of the verification strategy and the verification algorithm used.

[0055] Among them, in an optional embodiment, when the target file is written to the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: preparing the target file through the client and submitting it to the kernel-mode client; the kernel-mode client transmits the target file to the data verification layer; the data verification layer verifies the target file according to the target verification strategy to obtain first verification data; judging whether a transmission error occurs in the target file based on the verification data, including: adding the first verification data to the target file; transmitting the target file carrying the first verification data to the storage node; after the storage node receives the target file, using the same target verification strategy to obtain second verification data of the target file; at the storage node, determining whether a transmission error occurs in the target file by judging whether the first verification data and the second verification data are consistent.

[0056] When the target file is written to the storage node, it is necessary to ensure that the data is not tampered with during the transmission process. When the data transmission path configured with the data verification layer proposed in the present disclosure is used for data transmission, the data verification layer on the data transmission path will be used to verify the transmitted target file.

[0057] First, the target file is prepared by the client, the target file is usually encapsulated in an I / O (Input / Output) request, and the target file is submitted to the kernel-mode client for processing; After receiving the target file, the kernel-mode client transmits the target file to the data verification layer so that the data verification layer verifies the target file. The data verification layer verifies the target file using the verification algorithm included in the target verification strategy to obtain the first verification data of the target file, which can be used to represent the original verification data of the target data. The first verification data is added to the I / O request corresponding to the target file so that the target file can carry the first verification data, and the first verification data is used to verify the integrity of the file during the transmission process.

[0058] The target file carrying the first verification data is transmitted to the storage node through the corresponding transmission path. After receiving the target file, the storage node analyzes the first verification data carried in the target file and the original target file, where the original target file is the target file prepared at the client.

[0059] At the corresponding position of the storage node, the second verification data of the original target file can be re-determined, and the second verification data and the first verification data use the same target verification strategy. When the same data uses the same target verification strategy to calculate the verification data multiple times, the verification data obtained in each calculation is consistent. Therefore, by using the same target verification strategy to calculate the first verification data and the second verification data of the target file, it can be determined whether the target file has changed, thereby determining whether a transmission error has occurred in the target file during the transmission process.

[0060] By judging at the storage node whether the second verification data is consistent with the first verification data carried by the target file, it can be judged whether a transmission error occurs during the transmission of the target file from the data verification layer to the storage node, resulting in changes in the data corresponding to the target file.

[0061] By adopting the embodiments of the present disclosure, by introducing a verification mechanism during the data transmission process, errors that may occur during transmission can be discovered and corrected in a timely manner. Before data transmission, the target file is verified by the data verification layer according to the target verification strategy to generate the first verification data, which can provide a benchmark for subsequent integrity verification. After the storage node receives the file, the second verification data is recalculated using the same verification strategy and compared with the first verification data. This verification mechanism greatly enhances the reliability of data integrity verification. By integrating the verification mechanism in the data transmission path, the system can better deal with potential problems such as network instability and hardware failure.

[0062] Among them, in an optional embodiment, after determining whether a transmission error occurs in the target file, it also includes: when the first verification data and the second verification data are consistent, determining that no transmission error occurs in the target file, and writing the target file to the storage medium corresponding to the storage node; when the first verification data and the second verification data are inconsistent, determining that a transmission error occurs in the target file, and notifying the client to resend the target file.

[0063] The storage node can compare the recalculated second verification data with the western medicine verification data carried by the target file bit by bit. If the comparison result shows that the first verification data is completely consistent with the second verification data, the system confirms that there is no error in the transmission of the target file. If the comparison result shows that the first verification data is not completely consistent with the second verification data, the system confirms that an error occurred in the transmission of the target file.

[0064] When the first verification data is completely consistent with the second verification data, the storage node will start the writing process of the target file, and write the target file and its related metadata to the corresponding storage medium.

[0065] In the case where the first verification data is not completely consistent with the second verification data, the system confirms that an error has occurred in the current target file during transmission, and can record the error, such as the time when the error occurred, the file name involved, verification data differences and other detailed information, to facilitate subsequent analysis and troubleshooting. The system requires the client to resend the target file through the error notification mechanism. When the client resends the target file, it will re-execute the entire transmission process, including recalculating the first verification data corresponding to the target file. After receiving the retransmitted target file, the storage node will recalculate the second verification data of the target file, and re-determine whether the recalculated first verification data and second verification data are consistent, and judge the integrity and reliability of the target file.

[0066] If the verification fails, the client is notified to resend the target file to the storage node until the target file is successfully stored.

[0067] Among them, for the target files that are retransmitted due to verification failure, the system can implement additional monitoring and verification measures to ensure that the files can be correctly transmitted to the storage node.

[0068] By using the embodiments of the present disclosure, by comparing the verification data before writing the data to the storage medium, the system can promptly detect and handle transmission errors, thereby avoiding system instability or failures caused by erroneous data. When the first verification data and the second verification data are consistent, this indicates that no error occurred during the transmission of the target file, so the target file can be safely written to the storage medium corresponding to the storage node, ensuring that only complete and accurate data is stored on the storage medium, thereby avoiding potential problems caused by data errors. When the first verification data and the second verification data are inconsistent, the system can quickly identify the transmission error and notify the client to resend the target file, thereby improving the efficiency and accuracy of error handling.

[0069] Figure 3 This is a flow chart showing a method of writing a target file to a storage node according to an embodiment of the present disclosure. Figure 3 As shown, the client prepares the target file and encapsulates the target file into an input / output request; the target file is transmitted to the data verification layer; the target file is verified by the data verification layer to obtain the first verification data of the target file; the target file is transmitted to the storage node together with the first verification data; the storage node recalculates the second verification data of the target file; it is determined whether the first verification data carried by the target file and the recalculated second verification data are consistent; if they are consistent, it is determined that the target file verification has passed, and the target file is stored in the storage medium corresponding to the storage node; if they are inconsistent, it is determined that the target file verification has failed, triggering the error notification mechanism, so that the client resends the target file.

[0070] Among them, in an optional embodiment, when the target file is read from the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: initiating a read request for the target file to the storage node through the client; after the storage node receives the read request, the storage node reads the target file from the storage medium; in the data outer layer of the storage node, using the target verification strategy to obtain third verification data of the target file; adding the third verification data to the target file; transmitting the target file carrying the third verification data to the data verification layer; after the data verification layer receives the target file, using the same target verification strategy to obtain fourth verification data of the target file; judging whether a transmission error occurs to the target file based on the verification data, including: in the data verification layer, determining whether a transmission error occurs to the target file by judging whether the third verification data and the fourth verification data are consistent.

[0071] The client initiates a read request for the target file, which passes through the kernel client and sends the read request to the storage node. After the storage node receives the read request from the client, it locates the target file in the storage medium based on the file information in the read request, such as the file name and path, and then reads the contents of the target file from the storage medium.

[0072] Before the storage node transmits the target file, it determines the target verification strategy for verifying the target file according to the current transmission situation. The transmission situation includes: the current system load and the importance of the target file.

[0073] In the data outer layer of the storage node, the target file is verified using the target verification strategy to obtain the third verification data of the target file, and the third verification data is added to the target file so that the target file carries the third verification data for data transmission.

[0074] The target file carrying the third verification data is transmitted to the data verification layer using the data transmission path. After receiving the target file carrying the third verification data, the data verification layer uses the same target verification strategy to recalculate the fourth verification data of the target file. The third verification data and the fourth verification data are obtained using the same target verification strategy.

[0075] At the data verification layer, by judging the consistency of the third verification data and the fourth verification data, it is determined whether a transmission error occurs in the target file, that is, whether the data corresponding to the target file changes during the transmission process.

[0076] By adopting the embodiments of the present disclosure, by applying the same target verification strategy to the storage node and the data verification layer respectively, the third verification data and the fourth verification data of the target file can be obtained, and the comparison results of the two data can directly reflect whether the target file has been changed or damaged during the transmission process. The introduction of the data verification layer and the application of the verification strategy provide additional protection for the data transmission process. Even under adverse conditions such as network instability or hardware failure, errors can be discovered and corrected in time through the comparison of verification data, ensuring reliable data transmission.

[0077] Among them, in an optional embodiment, after determining whether a transmission error occurs in the target file, it also includes: when the third verification data and the fourth verification data are consistent, determining that no transmission error occurs in the target file, and sending the target file to the client through the kernel-mode client so that the user's application can use the target file; when the third verification data and the fourth verification data are inconsistent, determining that a transmission error occurs in the target file, and notifying the client to re-initiate a read request for the target file to the storage node.

[0078] At the data verification layer, consistency judgment is performed on the third verification data and the fourth verification data corresponding to the target file.

[0079] When the third verification data and the fourth verification data are consistent, it means that there is no transmission error in the target file during the data transmission process, that is, there is no data change, and therefore it can be trusted. For the target file whose third verification data and the fourth verification data are consistent, it is transmitted to the kernel state client; the kernel state client then passes it to the client so that the user's application can use the target file.

[0080] When the third verification data and the fourth verification data are inconsistent, it means that a transmission error occurred during the data transmission of the target file, and the data corresponding to the target file has changed, so the target file is not trusted. For target files whose third verification data and the fourth verification data are inconsistent, they are refused to be transmitted to the kernel-mode client, and the system notifies the client to re-initiate a read request for the target file to the storage node. After receiving the notification, the client will re-initiate a read request for the target file to the storage node as needed. The storage node then reads the target file and regains the third verification data of the target file, as well as regains the fourth verification data of the target file through the data verification layer, and performs a consistency comparison. In the case of consistency, the target file is eventually transmitted to the client. In the case of inconsistency, the client continues to be notified.

[0081] With the embodiments of the present disclosure, when the third verification data and the fourth verification data are consistent, they are directly sent to the client through the kernel-mode client for use by the user's application, thereby reducing the user's waiting time and improving the user experience. In the case of inconsistent verification data, the system can quickly identify that an error has occurred in the target file during the transmission process, and can prevent the erroneous data from being used by the user's application, thereby avoiding potential data corruption or functional abnormalities. By re-initiating a read request for the target file, the system can ensure that the data ultimately transmitted to the user is complete and correct.

[0082] Figure 4 This is a flow chart of reading a target file from a storage node according to an embodiment of the present disclosure. Figure 4As shown, the client initiates a read request for the target file and transmits the read request to the storage node; after receiving the read request, the storage node reads the target file from the storage medium and calculates the third verification data of the target file; the third verification data and the target file are transmitted to the data verification layer together; the data verification layer obtains the fourth verification data of the target file; it is determined whether the third verification data and the fourth verification data are consistent; if they are consistent, the verification passes and the target file is delivered to the application; if they are inconsistent, the verification fails, and the client re-initiates a read request for the target file to the storage node.

[0083] Among them, in an optional embodiment, during the data transmission between the verification data layer and the storage node, the data is transmitted through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

[0084] During data transmission between the verification data layer and the storage node, RDMA may be used as a target protocol to achieve high throughput and low latency data transmission.

[0085] RDMA supports remote direct memory access, which does not require the intervention of the operating system or CPU, thus greatly reducing the delay of data transmission. Therefore, for target files carrying verification data, the transmission efficiency of the target files will not be affected by the addition of verification data.

[0086] By adopting the embodiments of the present disclosure, the high-throughput target protocol can reduce the time required for transmission, so that the system can complete tasks faster, thereby improving overall work efficiency. The low-latency protocol can quickly respond to data transmission requests, reduce waiting time, ensure the timely arrival of data, and improve user experience.

[0087] Figure 5 is a block diagram of a data verification device shown in an embodiment of the present disclosure. Figure 5 As shown, the data verification device comprises: A determination module 510, configured to determine a data verification layer on a data transmission path between a kernel-mode client and a storage node; Verification module 520, used for, when the target file is transmitted through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, wherein the target verification strategy is determined according to the load of the system and the importance of the target file; The judgment module 530 is used to judge whether a transmission error occurs in the target file according to the verification data.

[0088] Optionally, the data transmission path includes a transmission process of a target protocol, and the determination module is specifically used to execute: In combination with the kernel-mode client and the target protocol, a transmission stage before a transmission process from the kernel-mode client to the target protocol and a transmission stage before a transmission process from the target protocol to the kernel-mode client are determined as key positions of the data transmission path; According to the key position of the data transmission path, it is determined to abstract an independent data verification layer in the kernel state client, and the data verification layer is integrated into the kernel state client.

[0089] Optionally, the data verification device further includes a strategy determination module for Obtain multiple alternative verification algorithms; Determine the current system load and the importance of the target file; When the load of the current system is higher than a preset load value, selecting a number of alternative verification algorithms less than or equal to a preset number from the plurality of alternative verification algorithms and adding them to the target verification strategy; When the importance of the target file is greater than the importance threshold, a number of candidate verification algorithms greater than a preset number are selected from the plurality of candidate verification algorithms and added to the target verification strategy.

[0090] Optionally, when the target file is written into the storage node through the data transmission path, the verification module is specifically configured to execute: The target file is prepared by the client and submitted to the kernel-mode client; The kernel-mode client transmits the target file to the data verification layer; The data verification layer verifies the target file according to the target verification strategy to obtain first verification data; The judgment module is specifically used to execute: Adding the first verification data to the target file; Transmitting the target file carrying the first verification data to the storage node; After receiving the target file, the storage node uses the same target verification strategy to obtain second verification data of the target file; At the storage node, by judging whether the first verification data and the second verification data are consistent, it is determined whether a transmission error occurs in the target file.

[0091] Optionally, after determining whether a transmission error occurs in the target file, the data verification device further includes: A first execution module, configured to determine that no transmission error occurs to the target file when the first verification data and the second verification data are consistent, and write the target file into a storage medium corresponding to the storage node; The second execution module is used to determine that a transmission error occurs in the target file when the first verification data and the second verification data are inconsistent, and to notify the client to resend the target file.

[0092] Optionally, in the case where the target file is read from the storage node through the data transmission path, the verification module is specifically configured to execute: Initiate a read request for the target file to the storage node through the client; After the storage node receives the read request, the storage node reads the target file from a storage medium; In the data outer layer of the storage node, using the target verification strategy, obtain third verification data of the target file; Adding the third verification data to the target file; Transmitting the target file carrying the third verification data to the data verification layer; After the data verification layer receives the target file, the fourth verification data of the target file is obtained using the same target verification strategy; The judgment module is specifically used to execute: At the data verification layer, whether a transmission error occurs in the target file is determined by judging whether the third verification data is consistent with the fourth verification data.

[0093] Optionally, after determining whether a transmission error occurs in the target file, the data verification device further includes: A third execution module is used to determine that no transmission error occurs to the target file when the third verification data and the fourth verification data are consistent, and send the target file to the client through the kernel state client so that the user's application program uses the target file; The fourth execution module is used to determine that a transmission error occurs in the target file when the third verification data and the fourth verification data are inconsistent, and to notify the client to re-initiate a read request for the target file to the storage node.

[0094] Optionally, the data verification device further includes: The protocol determination module is used to transmit data between the verification data layer and the storage node through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

[0095] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0096] The present disclosure also provides an electronic device, referring to Figure 6 , Figure 6 is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 610. The computer program can be run on the processor 620 to implement the steps in the data verification method disclosed in the embodiment of the present disclosure.

[0097] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data verification method disclosed in the embodiment of the present disclosure are implemented.

[0098] The embodiments of the present disclosure also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps in the data verification method disclosed in the embodiments of the present disclosure are implemented.

[0099] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0100] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices or computer program products. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0102] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0104] Although some embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the present disclosure.

[0105] The data verification method, device and electronic device provided by the present disclosure are introduced in detail above. The principles and implementation methods of the present disclosure are explained in this article using specific examples. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present disclosure, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A data verification method, characterized in that: include: Determine a data verification layer on the data transmission path between the kernel-mode client and the storage node; In the case of transmitting the target file through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, and the target verification strategy is determined according to the load of the system and the importance of the target file; According to the verification data, it is determined whether a transmission error occurs in the target file.

2. The method according to claim 1, characterized in that The data transmission path includes a transmission process of a target protocol, and a data verification layer is determined on the data transmission path between a kernel-mode client and a storage node, including: In combination with the kernel-mode client and the target protocol, a transmission stage before a transmission process from the kernel-mode client to the target protocol and a transmission stage before a transmission process from the target protocol to the kernel-mode client are determined as key positions of the data transmission path; According to the key position of the data transmission path, it is determined to abstract an independent data verification layer in the kernel state client, and the data verification layer is integrated into the kernel state client.

3. The method according to claim 1, characterized in that The target verification strategy is determined by: Obtain multiple alternative verification algorithms; Determine the current system load and the importance of the target file; When the load of the current system is higher than a preset load value, selecting a number of alternative verification algorithms less than or equal to a preset number from the plurality of alternative verification algorithms and adding them to the target verification strategy; When the importance of the target file is greater than the importance threshold, a number of candidate verification algorithms greater than a preset number are selected from the plurality of candidate verification algorithms and added to the target verification strategy.

4. The method according to claim 1, characterized in that When the target file is written into the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: The target file is prepared by the client and submitted to the kernel-mode client; The kernel-mode client transmits the target file to the data verification layer; The data verification layer verifies the target file according to the target verification strategy to obtain first verification data; The step of determining whether a transmission error occurs in the target file according to the verification data includes: Adding the first verification data to the target file; Transmitting the target file carrying the first verification data to the storage node; After receiving the target file, the storage node uses the same target verification strategy to obtain second verification data of the target file; At the storage node, by judging whether the first verification data and the second verification data are consistent, it is determined whether a transmission error occurs in the target file.

5. The method according to claim 4, characterized in that After determining whether a transmission error occurs in the target file, the method further includes: When the first verification data and the second verification data are consistent, determining that no transmission error occurs to the target file, and writing the target file into a storage medium corresponding to the storage node; When the first verification data and the second verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to resend the target file.

6. The method according to claim 1, characterized in that In the case where the target file is read from the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: Initiate a read request for the target file to the storage node through the client; After the storage node receives the read request, the storage node reads the target file from a storage medium; In the data outer layer of the storage node, using the target verification strategy, obtain third verification data of the target file; Adding the third verification data to the target file; Transmitting the target file carrying the third verification data to the data verification layer; After the data verification layer receives the target file, the fourth verification data of the target file is obtained using the same target verification strategy; The step of determining whether a transmission error occurs in the target file according to the verification data includes: At the data verification layer, whether a transmission error occurs in the target file is determined by judging whether the third verification data is consistent with the fourth verification data.

7. The method according to claim 6, characterized in that After determining whether a transmission error occurs in the target file, the method further includes: When the third verification data and the fourth verification data are consistent, determining that no transmission error occurs to the target file, and sending the target file to the client through the kernel-mode client, so that the user's application program uses the target file; In the case that the third verification data and the fourth verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to re-initiate a read request for the target file to the storage node.

8. The method according to claim 4 or 5, characterized in that: During the data transmission between the verification data layer and the storage node, the data is transmitted through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

9. A data verification device, characterized in that: include: A determination module, used to determine a data verification layer on a data transmission path between a kernel-mode client and a storage node; A verification module, used for, when a target file is transmitted through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, wherein the target verification strategy is determined according to the load of the system and the importance of the target file; The judgment module is used to judge whether a transmission error occurs in the target file according to the verification data.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data verification method according to any one of claims 1 to 8 are implemented.

Images