A data verification method, device and electronic equipment

Abstract

The present disclosure provides a data verification method and device and electronic equipment, belonging to the technical field of data verification, aiming to ensure the integrity and accuracy of data in a distributed file system. The method comprises: determining a data verification layer on a data transmission path between a kernel state client and a storage node; in the case of transmitting a target file through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, the target verification strategy being determined according to the load condition of the system and the importance of the target file; and determining whether the target file has transmission errors according to the verification data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data verification, and in particular to a data verification method, device, and electronic equipment. Background Art

[0002] In today's digital age, data generation and storage are exploding. Distributed file systems, as a solution that effectively addresses the demands of large-scale data storage and high-concurrency access, are becoming core infrastructure for many critical business scenarios. Against this backdrop, ensuring the integrity and accuracy of data in distributed file systems has become a critical issue that needs to be addressed. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a data verification method, device and electronic device. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a data verification method is provided, comprising:

[0005] Determine the data verification layer on the data transmission path between the kernel-mode client and the storage node;

[0006] When a target file is transmitted through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, wherein the target verification strategy is determined according to the load of the system and the importance of the target file;

[0007] Based on the verification data, it is determined whether a transmission error occurs in the target file.

[0008] Optionally, the data transmission path includes a transmission process of a target protocol, and determining a data verification layer on the data transmission path between the kernel-mode client and the storage node includes:

[0009] In combination with the kernel-mode client and the target protocol, determining a transmission stage before a transmission process from the kernel-mode client to the target protocol, and a transmission stage before a transmission process from the target protocol to the kernel-mode client, as key locations of the data transmission path;

[0010] According to the key position of the data transmission path, it is determined to abstractly obtain an independent data verification layer in the kernel state client, and the data verification layer is integrated into the kernel state client.

[0011] Optionally, the target verification strategy may be determined by:

[0012] Obtain multiple alternative verification algorithms;

[0013] determining a load condition of a current system, and determining an importance level of the target file;

[0014] in a case where the load of the current system is higher than a preset load value, selecting less than or equal to a preset number of the candidate verification algorithms from the plurality of candidate verification algorithms and adding to the target verification strategy;

[0015] in a case where the importance level of the target file is greater than an importance level threshold, selecting more than a preset number of the candidate verification algorithms from the plurality of candidate verification algorithms and adding to the target verification strategy.

[0016] Optionally, in a case where the target file is written to the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, comprising:

[0017] the client prepares the target file and submits to the kernel-mode client;

[0018] the kernel-mode client transmits the target file to the data verification layer;

[0019] the data verification layer verifies the target file according to the target verification strategy to obtain first verification data;

[0020] the verification data is used to determine whether the target file has a transmission error, comprising:

[0021] the first verification data is added to the target file;

[0022] the target file carrying the first verification data is transmitted to the storage node;

[0023] after the storage node receives the target file, the same target verification strategy is used to obtain second verification data of the target file;

[0024] in the storage node, whether the target file has a transmission error is determined by judging whether the first verification data and the second verification data are consistent.

[0025] Optionally, after determining whether the target file has a transmission error, further comprising:

[0026] in a case where the first verification data and the second verification data are consistent, it is determined that the target file has no transmission error, and the target file is written to a storage medium corresponding to the storage node;

[0027] In the case that the first check data and the second check data are inconsistent, it is determined that the target file has transmission error, and the client is informed to resend the target file.

[0028] Optionally, in the case that the target file is read from the storage node through the data transmission path, the data check layer checks the target file according to a target check strategy to obtain check data, comprising:

[0029] The client initiates a read request for the target file to the storage node;

[0030] After the storage node receives the read request, the storage node reads the target file from the storage medium;

[0031] In the data outer layer of the storage node, a target check strategy is used to obtain third check data of the target file;

[0032] The third check data is added to the target file;

[0033] The target file carrying the third check data is transmitted to the data check layer;

[0034] After the data check layer receives the target file, the same target check strategy is used to obtain fourth check data of the target file;

[0035] The determination of whether the target file has transmission error according to the check data comprises:

[0036] In the data check layer, whether the target file has transmission error is determined by judging whether the third check data and the fourth check data are consistent.

[0037] Optionally, after it is determined whether the target file has transmission error, it further comprises:

[0038] In the case that the third check data and the fourth check data are consistent, it is determined that the target file has no transmission error, and the target file is sent to the client through the kernel-mode client, so that the application of the user uses the target file;

[0039] In the case that the third check data and the fourth check data are inconsistent, it is determined that the target file has transmission error, and the client is informed to resend the target file to the storage node.

[0040] Optionally, in the process of data transmission between the check data layer and the storage node, the data is transmitted through a target protocol with high throughput and low delay characteristics, so that the check data carried by the target file does not affect the transmission efficiency of the target file.

[0041] According to a second aspect of the embodiments of the present disclosure, a data check device is provided, comprising:

[0042] A determination module is configured to determine a data check layer on a data transmission path between a kernel-mode client and a storage node.

[0043] A check module is configured to, in a case where a target file is transmitted through the data transmission path, check the target file according to a target check strategy to obtain check data, the target check strategy being determined according to a load condition of a system and an importance level of the target file.

[0044] A judgment module is configured to judge whether the target file has a transmission error according to the check data.

[0045] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the computer program is executed by the processor, the steps of the data check method in the first aspect are implemented.

[0046] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the steps of the data check method in the first aspect are implemented.

[0047] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising a computer program, when the computer program is executed by a processor, the steps of the data check method in the first aspect are implemented.

[0048] The present disclosure can ensure the data integrity and accuracy of file reading and writing, reduce the risk of errors, discover and handle transmission errors in time, enhance the stability of the system, balance the performance and reliability with flexible check strategies, and facilitate maintenance and upgrade of the independent check layer without affecting other functions of the system. The target check strategy is determined according to the load condition of the system and the importance level of the target file, and then the target check strategy is used to check the target file for data transmission. This dynamic adjustment of the target check strategy can realize intelligent and automated check strategy optimization, ensure the reduction of system overhead while improving data security and transmission accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments of the present disclosure will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those of ordinary skill in the art without creative labor under the premise of the drawings.

[0050] Figure 1 is a step schematic diagram of a data verification method shown by an embodiment of the present disclosure;

[0051] Figure 2 is a target verification strategy adjustment flowchart shown by an embodiment of the present disclosure;

[0052] Figure 3 is a flowchart of writing a target file to a storage node shown by an embodiment of the present disclosure;

[0053] Figure 4 is a flowchart of reading a target file from a storage node shown by an embodiment of the present disclosure;

[0054] Figure 5 is a block diagram of a data verification device shown by an embodiment of the present disclosure;

[0055] Figure 6 is a schematic diagram of an electronic device shown by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present disclosure.

[0057] The terms "first", "second", and the like in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually a class, not limited to the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in an "or" relationship.

[0058] In today's digital age, the generation and storage of data are growing explosively, and various industries such as finance, medicine, scientific research, and the Internet have high requirements for data security, reliability, and efficient access. As a solution that can effectively cope with large-scale data storage and high-concurrency access requirements, distributed file systems are gradually becoming the core infrastructure of many key business scenarios. With the continuous development of hardware technology, remote direct memory access (RDMA) technology provides a new way to achieve high-speed data transmission. The use of kernel-level clients further improves the performance and stability of the system. Distributed file systems are widely used in large-scale data storage and processing scenarios due to their high scalability, high reliability, and high performance. However, in the actual application of distributed file systems, there are still many challenges.

[0059] On the one hand, data may be affected by various factors during transmission and storage, resulting in errors. For example, network failures, storage media damage, software errors, etc. can cause data loss or damage. These errors, if not discovered and corrected in time, will have a serious impact on the integrity and accuracy of the data, and may even cause system failure and data loss.

[0060] On the other hand, existing distributed file systems often have deficiencies in data verification. Some traditional verification methods are inefficient and cannot meet the needs of high-speed data transmission; while some complex verification mechanisms will bring a large system overhead, affecting the performance of the system. In addition, existing verification mechanisms are usually implemented at a specific level of the file system, lacking flexibility and scalability.

[0061] To solve the above technical problems, the present disclosure proposes a data verification method that can determine an independent data verification layer in the data transmission path, ensuring the data integrity and accuracy of file reading and writing, reducing the risk of errors, and discovering and processing transmission errors in a timely manner.

[0062] Figure 1 is a step schematic diagram of a data verification method according to an embodiment of the present disclosure. As shown in Figure 1 The data verification method can specifically include the following steps:

[0063] Step S11: Determine a data verification layer on the data transmission path between the kernel-level client and the storage node.

[0064] The kernel client runs on the client node, which is the entry point for users to access the storage cluster service. The client node is different from the storage service node, and the client node usually does not directly participate in the storage and management of data, but serves as a bridge between the user and the storage cluster. The kernel client has the ability to directly access system resources, which can allow users to use the storage cluster service like using a local file system.

[0065] The kernel client directly interacts with the operating system kernel, and once data errors occur, it can have a serious impact on the system. Therefore, although the use of the kernel client can improve the performance and stability of the system, it also puts higher requirements on data verification.

[0066] On the transmission path with the participation of the kernel client, a data verification layer is determined. The data verification layer can be determined at a specific position on the transmission path according to data transmission experience, and the data or file will pass through the data verification layer when it is transmitted using the data transmission path. The data verification layer is relatively independent of other functions, which facilitates the maintenance and upgrade of the data verification layer without affecting other functions of the system.

[0067] Step S12: In the case of transmitting a target file through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, and the target verification strategy is determined according to the load condition of the system and the importance of the target file.

[0068] In the case of transmitting a target file through the data transmission path, the current transmission condition is first determined, including the load condition of the system and the importance of the target file. Through the current transmission condition, the target verification strategy is dynamically adjusted to make the target verification strategy meet the current transmission requirements.

[0069] After determining the target verification strategy, the file being transmitted needs to be verified by the data verification layer before entering the kernel client and before transmitting data from the kernel client, to obtain verification data, which is a value calculated by a verification algorithm. The target file is verified by the data verification layer to ensure the stability of the data and avoid data errors due to data transmission problems.

[0070] Step S13: According to the verification data, it is determined whether the target file has transmission errors.

[0071] According to the verification data obtained by the data verification layer, it is determined whether the target file has transmission errors, so that transmission errors can be discovered and handled in a timely manner, the stability of the system is enhanced, and flexible verification strategies can balance performance and reliability.

[0072] By determining the target verification strategy according to the load condition of the system and the importance of the target file, the flexible adjustment of the verification strategy is realized. By determining the data verification layer on the data transmission path between the kernel-mode client and the storage node, parallel processing of data transmission and verification is realized. By implementing the verification layer on the data transmission path, integration with other file systems and network transmission protocols can be easily achieved. By verifying in real time during data transmission and immediately determining whether the target file has transmission errors according to the verification data, it is helpful to timely discover and correct errors in the transmission process, and to ensure the integrity and reliability of the data.

[0073] In an optional embodiment, the data transmission path includes a transmission process of the target protocol, and the data verification layer is determined on the data transmission path between the kernel-mode client and the storage node, including: in combination with the kernel-mode client and the target protocol, determining the transmission stage before the transmission process of the target protocol from the kernel-mode client and the transmission stage before the transmission process of the target protocol into the kernel-mode client as the key positions of the data transmission path; and determining an independent data verification layer in the kernel-mode client according to the key positions of the data transmission path, and integrating the data verification layer in the kernel-mode client.

[0074] The target protocol can be RDMA, and the emergence of RDMA technology provides a new way to solve the data transmission problem in distributed file systems. RDMA allows direct data transmission between memories without the involvement of CPU, thereby greatly improving the speed and efficiency of data transmission.

[0075] In the case of including the transmission process of the target protocol and the kernel-mode client in the transmission path, the data transmission process between the target protocol and the kernel-mode client is determined. According to the overall data transmission process, the key positions in the transmission path are determined.

[0076] The last transmission stage before the transmission process of the target protocol from the kernel-mode client may involve data preparation, encapsulation or formatting operations. The last transmission stage before the transmission process of the target protocol returns to the kernel-mode client may involve data decapsulation, processing and other steps.

[0077] Therefore, the transmission stage before the transmission process of the target protocol from the kernel-mode client, and the transmission stage before the transmission process of the target protocol into the kernel-mode client can be determined as the key positions of the data transmission path.

[0078] According to the key position, the data verification layer can be arranged in the kernel-mode client, and specifically, an independent data verification layer can be abstracted in the kernel-mode client, so that the data verification layer is integrated in the kernel-mode client.

[0079] After the independent data verification layer is abstracted in the kernel-mode client, the system can be comprehensively tested to ensure that the data verification layer can work correctly and will not have a great impact on the system performance during data transmission.

[0080] By determining and integrating an independent data verification layer on the data transmission path, the embodiments of the present disclosure can significantly enhance the integrity and verification capability during data transmission. The data verification layer is located between the kernel-mode client and the target protocol transmission process, which can ensure that the data is strictly verified when entering and leaving the kernel-mode client, thereby reducing the risk of data damage or incorrect transmission. The existence of the data verification layer can timely discover and correct errors in data transmission, avoid further processing or storage of these error data, and thus improve the security and reliability of the entire system.

[0081] In an optional embodiment, the determination of the target verification strategy includes: obtaining a plurality of candidate verification algorithms; determining the load condition of the current system and the importance of the target file; in the case that the load of the current system is higher than a preset load value, selecting less than or equal to a preset number of candidate verification algorithms from the plurality of candidate verification algorithms and adding them to the target verification strategy; in the case that the importance of the target file is greater than an importance threshold, selecting more than a preset number of candidate verification algorithms from the plurality of candidate verification algorithms and adding them to the target verification strategy.

[0082] A plurality of candidate verification algorithms are collected, which can include hash algorithms, cyclic redundancy check algorithms, etc. The performance and applicability of each candidate verification algorithm can be evaluated in advance, such as calculation speed, resource consumption, error detection rate, etc.

[0083] The current load condition of the system is monitored in real time, including CPU usage, memory occupancy, network bandwidth usage, etc. A preset load value can be set as a judgment benchmark for the system load condition according to historical data and business rules. When the current load of the system is greater than or equal to the preset load value, it means that the current system is under high load, otherwise, when the current load of the system is less than the preset load value, it means that the current system is under low load.

[0084] When the target file is transmitted using the transmission path proposed in the present disclosure, the attributes of the target file, such as the file type, file size, file source, and the like, can be analyzed to determine the importance of the target file. According to business requirements and industry standards, an importance threshold can be set to distinguish important files from non-important files. If the importance of the target file exceeds or equals the importance threshold, the target file is an important file. Otherwise, if the importance of the target file is lower than the importance threshold, the target file is not an important file.

[0085] The target verification strategy is dynamically adjusted in combination with the current load of the system and the importance of the target file. According to the current data transmission situation, the verification strength of the target verification strategy on the target file is determined, which can be embodied by determining the type of verification algorithm used by the target verification strategy. Specifically, if a relatively lenient target verification strategy is used according to the current data verification situation, fewer verification algorithms can be selected to add to the target verification strategy, and if a relatively strict target verification strategy is used, more verification algorithms can be selected to add to the target verification strategy. The performance and applicability of each verification algorithm can also be used to select the verification algorithms included in the target verification strategy to reflect the verification strength of the target verification strategy.

[0086] A preset number can be set in advance, which needs to be greater than or equal to 1. When the target file needs to be strictly verified, the number of selected verification algorithms needs to be greater than the preset number, and when the target file does not need to be strictly verified, the number of selected verification algorithms is less than or equal to the preset number.

[0087] When the current load of the system is higher than the preset load value, it is determined that the current system is in high load, and therefore a relatively lenient target verification strategy is used for the target file. From the selected verification algorithms, less than or equal to the preset number of selected verification algorithms are selected and added to the target verification strategy. For example, when the selected verification algorithms include a hash algorithm and a cyclic redundancy check algorithm, the preset number can be 1, and one of the two verification algorithms, either the hash algorithm or the cyclic redundancy check algorithm, is selected and added to the target verification strategy.

[0088] When the importance of the target file is greater than the importance threshold, it is determined that the current target file is an important file, and more than the preset number of selected verification algorithms are selected from the plurality of selected verification algorithms and added to the target verification strategy. For example, when the selected verification algorithms include a hash algorithm and a cyclic redundancy check algorithm, the preset number can be 1, and both of the two verification algorithms are added to the target verification strategy to improve the strictness of the verification.

[0089] In some cases, the load of the system can be higher than the preset load value and the importance of the target file can be greater than the importance threshold value at the same time. Since the two cases are opposite for the adjustment of the target verification strategy, the target verification strategy can be adjusted based on which case is currently more considered by combining the analysis of the two cases. In one specific implementation, in the case where the load of the system is higher than the preset load value, the more stringent target verification strategy can still be used for verification of important files. In one specific embodiment, when the system load is high, the verification strength of the target verification strategy can be dynamically reduced to reduce system overhead, and any one of the hash algorithm and the cyclic redundancy check algorithm can be selected as the target verification algorithm to calculate the verification data of the target file; when the file being transmitted is an important file, the target verification algorithm is obtained by combining the hash algorithm and the cyclic redundancy check algorithm, and the verification data of the target file is calculated by using the obtained target verification algorithm.

[0090] Figure 2 is a target verification strategy adjustment flowchart shown by an embodiment of the present disclosure. As shown in Figure 2 , in the case where the load of the current system is higher than the preset load value, the target verification strategy is dynamically and adaptively adjusted to reduce the verification strength of non-important files; in the case where the target file is an important file, the target verification strategy is dynamically and adaptively adjusted to use a more stringent verification algorithm to improve the verification strength.

[0091] By using the embodiments of the present disclosure, the system can have a wider selection range by obtaining multiple alternative verification algorithms, so as to select the optimal verification algorithm according to different situations. Considering the system load, the system can reduce the number of verification algorithms when the resources are scarce, thereby releasing the computing resources for other more important tasks. By selecting the number of verification algorithms according to the importance of the target file, the important files can be ensured to be protected at a higher level. The system can be allowed to dynamically adjust the verification strategy according to real-time conditions, thereby improving the flexibility and adaptability of the system to adapt to different working environments and requirements.

[0092] The following are specific steps for data writing using the data transmission path configured with the data verification layer.

[0093] In an optional embodiment, a machine learning or prediction model can be introduced to dynamically adjust the verification strategy based on historical data such as system load trend, file transmission error rate, etc., to achieve more refined resource allocation and higher error detection rate. Specifically, a monitoring and prediction system can be established to collect and analyze system load, file transmission history, etc. data, train the model using these data, predict the future possible system state and file transmission demand, and dynamically adjust the strictness of the verification strategy and the verification algorithm used.

[0094] In an optional embodiment, in the case of writing the target file to the storage node through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, including: preparing the target file through a client and submitting it to the kernel-mode client; the kernel-mode client transmits the target file to the data verification layer; the data verification layer verifies the target file according to a target verification strategy to obtain first verification data; the step of judging whether the target file has a transmission error according to the verification data includes: adding the first verification data to the target file; transmitting the target file carrying the first verification data to the storage node; after the storage node receives the target file, using the same target verification strategy to obtain second verification data of the target file; in the storage node, determining whether the target file has a transmission error by judging whether the first verification data and the second verification data are consistent.

[0095] In the case of writing the target file to the storage node, it is necessary to ensure that the data is not tampered with during transmission. In the case of using the data transmission path configured with the data verification layer proposed by the present disclosure to perform data transmission, the data verification layer on the data transmission path will be used to verify the target file being transmitted.

[0096] First, the client prepares the target file, which is usually encapsulated in an I / O (Input / Output) request, and submits the target file to the kernel-mode client for processing;

[0097] After receiving the target file, the kernel-mode client transmits the target file to the data verification layer to enable the data verification layer to verify the target file. The data verification layer uses the verification algorithm included in the target verification strategy to verify the target file to obtain first verification data of the target file, which can be used to represent the original verification data of the target data. The first verification data is added to the I / O request corresponding to the target file, so that the target file can carry the first verification data, which is used to verify the integrity of the file during transmission.

[0098] The target file carrying the first verification data is transmitted to the storage node through the corresponding transmission path. After receiving the target file, the storage node analyzes the first verification data carried in the target file and the original target file, which is the target file prepared at the client.

[0099] At the corresponding position of the storage node, the original target file second check data can be re-determined, and the second check data and the first check data use the same target check strategy. When the same data uses the same target check strategy to calculate the check data for multiple times, the check data obtained each time is consistent, and therefore, the first check data and the second check data of the target file calculated by using the same target check strategy can be used to determine whether the target file changes, so as to determine whether the target file has transmission errors in the transmission process.

[0100] At the storage node, whether the second check data is consistent with the first check data carried by the target file is determined, and it can be determined whether the target file has transmission errors in the transmission process from the data check layer to the storage node, resulting in changes in the data corresponding to the target file.

[0101] By introducing the check mechanism in the data transmission process, the errors that may occur in the transmission can be found and corrected in time by using the embodiments of the present disclosure. Before data transmission, the data check layer checks the target file according to the target check strategy to generate the first check data, which can provide a reference for subsequent integrity verification. After the storage node receives the file, the second check data is recalculated using the same check strategy, and the first check data is compared. This check mechanism greatly enhances the reliability of data integrity verification. By integrating the check mechanism in the data transmission path, the system can better cope with potential problems such as network instability and hardware failure.

[0102] In an optional embodiment, after determining whether the target file has transmission errors, the method further includes: in the case that the first check data and the second check data are consistent, determining that the target file has no transmission errors, and writing the target file into the corresponding storage medium of the storage node; in the case that the first check data and the second check data are inconsistent, determining that the target file has transmission errors, and notifying the client to resend the target file.

[0103] The storage node can compare the second check data recalculated with the first check data carried by the target file bit by bit. If the comparison result shows that the first check data and the second check data are completely consistent, the system confirms that the target file has no errors in the transmission process. If the comparison result shows that the first check data and the second check data are not completely consistent, the system confirms that the target file has errors in the transmission process.

[0104] In the case that the first check data and the second check data are completely consistent, the storage node starts the writing process of the target file, and writes the target file and related metadata into the corresponding storage medium.

[0105] In the case that the first check data and the second check data are not consistent, the system confirms that an error occurs in the transmission process of the target file, and can record the error, such as the time of the error, the file name involved, the difference in the check data, and other detailed information, for subsequent analysis and troubleshooting. The system requires the client to resend the target file through an error notification mechanism. When the client resends the target file, the complete transmission process is performed again, including the recalculation of the first check data corresponding to the target file. After receiving the retransmitted target file, the storage node recalculates the second check data of the target file, and rejudges whether the recalculated first check data and the second check data are consistent, to judge the integrity and reliability of the target file.

[0106] In the case of failed check, the client is notified to resend the target file to the storage node until the target file is successfully stored.

[0107] Among them, for the target file retransmitted due to failed check, the system can implement additional monitoring and check measures to ensure that the file can be correctly and accurately transmitted to the storage node.

[0108] By comparing the check data before writing the data to the storage medium, the system can timely discover and handle transmission errors, thereby avoiding system instability or failure caused by error data. When the first check data and the second check data are consistent, it indicates that the target file has not occurred error in the transmission process, and therefore the target file can be safely written to the storage medium corresponding to the storage node, ensuring that only complete and accurate data will be stored on the storage medium, thereby avoiding potential problems caused by data errors. When the first check data and the second check data are inconsistent, the system can quickly identify the transmission error and notify the client to resend the target file, improving the efficiency and accuracy of error handling.

[0109] Figure 3 is a flowchart of writing a target file to a storage node according to an embodiment of the present disclosure. As shown in Figure 3 The client prepares the target file and encapsulates the target file into an input / output request; the target file is transmitted to a data check layer; the target file is checked by the data check layer to obtain the first check data of the target file; the target file is transmitted to the storage node together with the first check data; the storage node recalculates the second check data of the target file; it is judged whether the first check data carried by the target file and the recalculated second check data are consistent; in the case of consistency, it is judged that the target file passes the check, and the target file is stored in the storage medium corresponding to the storage node; in the case of inconsistency, it is judged that the target file fails the check, and an error notification mechanism is triggered to make the client resend the target file.

[0110] In an optional embodiment, when the target file is read from the storage node through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, including: initiating, by the client, a read request for the target file to the storage node; after the storage node receives the read request, reading the target file from the storage medium; at the data outer layer of the storage node, using a target verification strategy to obtain third verification data of the target file; adding the third verification data to the target file; transmitting the target file carrying the third verification data to the data verification layer; after the data verification layer receives the target file, using the same target verification strategy to obtain fourth verification data of the target file; and determining, according to the verification data, whether the target file has a transmission error, including: determining, at the data verification layer, whether the target file has a transmission error by judging whether the third verification data and the fourth verification data are consistent.

[0111] The client initiates a read request for the target file, and the read request is sent to the storage node through the kernel client. After the storage node receives the read request from the client, the target file is located in the storage medium according to the file information in the read request, such as the file name, the path, etc., and then the content of the target file is read from the storage medium.

[0112] Before the storage node transmits the target file, a target verification strategy for verifying the target file is determined according to the current transmission situation. The transmission situation includes the load of the current system and the importance of the target file.

[0113] At the data outer layer of the storage node, the target file is verified using the target verification strategy to obtain third verification data of the target file, and the third verification data is added to the target file so that the target file carries the third verification data for data transmission.

[0114] The target file carrying the third verification data is transmitted to the data verification layer using the data transmission path. After the data verification layer receives the target file carrying the third verification data, the fourth verification data of the target file is recalculated using the same target verification strategy. The third verification data and the fourth verification data are obtained using the same target verification strategy.

[0115] At the data verification layer, whether the target file has a transmission error is determined by judging the consistency of the third verification data and the fourth verification data, that is, whether the data corresponding to the target file has changed in the transmission process.

[0116] By applying the same target check strategy to the storage node and the data check layer respectively, the third check data and the fourth check data of the target file can be obtained, and the comparison result of the two data can directly reflect whether the target file has changed or damaged in the transmission process. The introduction of the data check layer and the application of the check strategy provide additional protection for the data transmission process. Even under unfavorable conditions such as unstable network or hardware failure, errors can be discovered and corrected in time through the comparison of the check data, ensuring reliable transmission of data.

[0117] In an optional embodiment, after determining whether the target file has transmission errors, the method further comprises: in the case that the third check data and the fourth check data are consistent, determining that the target file has no transmission errors, sending the target file to the client through the kernel-mode client, so that the application program of the user uses the target file; in the case that the third check data and the fourth check data are inconsistent, determining that the target file has transmission errors, and notifying the client to re-initiate a read request for the target file to the storage node.

[0118] In the data check layer, the consistency of the third check data and the fourth check data corresponding to the target file is judged.

[0119] In the case that the third check data and the fourth check data are consistent, it indicates that the target file has no transmission errors in the data transmission process, that is, no data changes, and therefore it can be trusted. For the target file whose third check data and fourth check data are consistent, it is transmitted to the kernel-mode client; the kernel-mode client is then transmitted to the client, so that the application program of the user uses the target file.

[0120] In the case that the third check data and the fourth check data are inconsistent, it indicates that the target file has transmission errors in the data transmission process, and the data corresponding to the target file has changed, so the target file is not trusted. For the target file whose third check data and fourth check data are inconsistent, it is refused to be transmitted to the kernel-mode client, and the system notifies the client to re-initiate a read request for the target file to the storage node. After receiving the notification, the client will re-initiate a read request for the target file to the storage node as needed, and then the storage node reads the target file and re-obtains the third check data of the target file, and re-obtains the fourth check data of the target file through the data check layer, and performs consistency comparison. In the case of consistency, the target file is finally transmitted to the client, and in the case of inconsistency, the client is notified again.

[0121] Using the embodiments of the present disclosure, when the third verification data and the fourth verification data are consistent, they are sent directly to the client via the kernel-mode client for use by the user's application, reducing the user's waiting time and improving the user experience. In the event of inconsistent verification data, the system can quickly identify that an error has occurred in the target file during transmission, preventing the erroneous data from being used by the user's application, thereby avoiding potential data corruption or functional anomalies. By re-initiating a read request for the target file, the system can ensure that the data ultimately transmitted to the user is complete and correct.

[0122] Figure 4 This is a flow chart showing a method of reading a target file from a storage node according to an embodiment of the present disclosure. Figure 4 As shown, the client initiates a read request for the target file and transmits the read request to the storage node; after receiving the read request, the storage node reads the target file from the storage medium and calculates the third verification data of the target file; the third verification data and the target file are transmitted to the data verification layer; the data verification layer obtains the fourth verification data of the target file; it is determined whether the third verification data and the fourth verification data are consistent; if they are consistent, the verification passes and the target file is delivered to the application; if they are inconsistent, the verification fails, and the client re-initiates a read request for the target file to the storage node.

[0123] Among them, in an optional embodiment, during the process of data transmission between the verification data layer and the storage node, the transmission is carried out through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

[0124] During data transmission between the verification data layer and the storage node, RDMA may be used as a target protocol to achieve high throughput and low latency data transmission.

[0125] RDMA supports remote direct memory access, which eliminates the need for operating system or CPU intervention, significantly reducing data transmission latency. Therefore, for target files that carry checksum data, the addition of checksum data will not affect the transfer efficiency of the target file.

[0126] By using the embodiments of the present disclosure, high-throughput target protocols reduce the time required for transmission, enabling the system to complete tasks more quickly, thereby improving overall work efficiency. Low-latency protocols can quickly respond to data transmission requests, reducing waiting time, ensuring timely data arrival, and improving the user experience.

[0127] Figure 5 This is a block diagram of a data verification device shown in an embodiment of the present disclosure.Figure 5 The data verification device is shown to include:

[0128] The determining module 510 is configured to determine a data verification layer on a data transmission path between the kernel-mode client and the storage node.

[0129] The verification module 520 is configured to, in a case where a target file is transmitted through the data transmission path, verify the target file according to a target verification strategy to obtain verification data, the target verification strategy being determined according to a load condition of a system and an importance level of the target file.

[0130] The judging module 530 is configured to judge whether the target file has a transmission error according to the verification data.

[0131] Optionally, the data transmission path includes a transmission process of a target protocol, and the determining module is specifically configured to perform:

[0132] In combination with the kernel-mode client and the target protocol, a transmission stage before a transmission process of the target protocol enters the kernel-mode client and a transmission stage before a transmission process of the target protocol enters the kernel-mode client are determined as critical positions of the data transmission path.

[0133] According to the critical positions of the data transmission path, an independent data verification layer is abstracted in the kernel-mode client, and the data verification layer is integrated in the kernel-mode client.

[0134] Optionally, the data verification device further includes a strategy determining module configured to:

[0135] obtain a plurality of alternative verification algorithms;

[0136] determine a load condition of a current system and determine an importance level of the target file;

[0137] in a case where the load of the current system is higher than a preset load value, select less than or equal to a preset number of alternative verification algorithms from the plurality of alternative verification algorithms and add the alternative verification algorithms to the target verification strategy;

[0138] in a case where the importance level of the target file is greater than an importance level threshold, select more than a preset number of alternative verification algorithms from the plurality of alternative verification algorithms and add the alternative verification algorithms to the target verification strategy.

[0139] Optionally, in a case where the target file is written into the storage node through the data transmission path, the verification module is specifically configured to perform:

[0140] The target file is prepared by a client and submitted to the kernel-mode client;

[0141] The kernel-mode client transmits the target file to the data verification layer;

[0142] The data verification layer verifies the target file according to a target verification strategy to obtain first verification data;

[0143] The judgment module is specifically configured to perform:

[0144] The first verification data is added to the target file;

[0145] The target file carrying the first verification data is transmitted to the storage node;

[0146] After the storage node receives the target file, the same target verification strategy is used to obtain second verification data of the target file;

[0147] In the storage node, whether the target file has a transmission error is determined by judging whether the first verification data and the second verification data are consistent.

[0148] Optionally, after determining whether the target file has a transmission error, the data verification device further comprises:

[0149] The first execution module is configured to determine that the target file has no transmission error in the case that the first verification data and the second verification data are consistent, and write the target file into a storage medium corresponding to the storage node;

[0150] The second execution module is configured to determine that the target file has a transmission error in the case that the first verification data and the second verification data are inconsistent, and notify the client to resend the target file.

[0151] Optionally, in the case that the target file is read from the storage node through the data transmission path, the verification module is specifically configured to perform:

[0152] The client initiates a read request for a target file to the storage node;

[0153] After the storage node receives the read request, the storage node reads the target file from a storage medium;

[0154] In the data outer layer of the storage node, a target verification strategy is used to obtain third verification data of the target file;

[0155] The third verification data is added to the target file;

[0156] transmitting the target file carrying the third check data to the data check layer;

[0157] after receiving the target file at the data check layer, obtaining fourth check data of the target file using the same target check strategy;

[0158] The judgment module is specifically configured to perform:

[0159] at the data check layer, determining whether the target file has transmission errors by judging whether the third check data and the fourth check data are consistent.

[0160] Optionally, after determining whether the target file has transmission errors, the data check apparatus further comprises:

[0161] The third execution module is configured to, in the case where the third check data and the fourth check data are consistent, determine that the target file has no transmission errors, send the target file to the client through the kernel-mode client, and enable the user's application program to use the target file.

[0162] The fourth execution module is configured to, in the case where the third check data and the fourth check data are inconsistent, determine that the target file has transmission errors, and notify the client to reinitiate a read request for the target file to the storage node.

[0163] Optionally, the data check apparatus further comprises:

[0164] The protocol determination module is configured to, in the process of data transmission between the check data layer and the storage node, perform transmission through a target protocol having high throughput and low delay characteristics, so that the check data carried by the target file does not affect the transmission efficiency of the target file.

[0165] It should be noted that the apparatus embodiment is similar to the method embodiment, and thus is described more simply, and the related parts can be referred to the method embodiment.

[0166] The present disclosure also provides an electronic device, which refers to Figure 6 , Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. As shown in Figure 6 , the electronic device 600 comprises a memory 610 and a processor 620, the memory 610 and the processor 620 are communicatively connected through a bus, and the memory 610 stores a computer program, which can run on the processor 620, and further implement the steps in the data check method disclosed in the present disclosure.

[0167] The embodiment of the present disclosure further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the data verification method disclosed by the embodiment of the present disclosure.

[0168] The embodiment of the present disclosure further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps in the data verification method disclosed by the embodiment of the present disclosure.

[0169] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0170] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, device or computer program product. Therefore, the embodiments of the present disclosure can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present disclosure can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0171] The embodiments of the present disclosure are described with reference to flowcharts and / or block diagrams of the method, device, electronic equipment and computer program product according to the embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal equipment to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal equipment produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the functions specified in one block or multiple blocks.

[0172] These computer program instructions can also be stored in a computer readable storage medium which can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device for implementing the functions specified in one block or multiple blocks.

[0173] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide a process for implementing the functions specified in the flowchart block(s) Figure 1 one flow or multiple flows and / or the functions specified in the block(s) Figure 1 one flow or multiple flows and / or the functions specified in the block(s)

[0174] Although some embodiments of the present disclosure have been described, those skilled in the art who have the basic inventive concept can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present disclosure.

[0175] The above provides a data verification method, device and electronic equipment, and the principles and implementation manners of the present disclosure are described by applying specific examples. The above embodiment descriptions are only used to help understand the method and core idea of the present disclosure; meanwhile, for those skilled in the art, according to the idea of the present disclosure, the specific implementation manners and application ranges can be changed. In summary, the content of the present description should not be understood as a limitation of the present disclosure.

Claims

1. A data verification method, characterized in that: include: Determine the data verification layer on the data transmission path between the kernel-mode client and the storage node; When a target file is transmitted through the data transmission path, the data verification layer verifies the target file according to a target verification strategy to obtain verification data, wherein the target verification strategy is determined according to the load of the system and the importance of the target file; Based on the verification data, it is determined whether a transmission error occurs in the target file.

2. The method according to claim 1, characterized in that The data transmission path includes a transmission process of a target protocol. On the data transmission path between the kernel-mode client and the storage node, a data verification layer is determined, including: In combination with the kernel-mode client and the target protocol, determining a transmission stage before a transmission process from the kernel-mode client to the target protocol, and a transmission stage before a transmission process from the target protocol to the kernel-mode client, as key locations of the data transmission path; According to the key position of the data transmission path, it is determined to abstractly obtain an independent data verification layer in the kernel state client, and the data verification layer is integrated into the kernel state client.

3. The method according to claim 1, characterized in that Methods for determining target verification strategies include: Obtain multiple alternative verification algorithms; Determine the current system load and the importance of the target file; When the load of the current system is higher than a preset load value, selecting less than or equal to a preset number of alternative verification algorithms from the plurality of alternative verification algorithms and adding them to the target verification strategy; When the importance of the target file is greater than an importance threshold, a number of alternative verification algorithms greater than a preset number are selected from the plurality of alternative verification algorithms and added to the target verification strategy.

4. The method according to claim 1, wherein When the target file is written to the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: Prepare the target file through the client and submit it to the kernel-mode client; The kernel-mode client transmits the target file to the data verification layer; The data verification layer verifies the target file according to the target verification strategy to obtain first verification data; The step of determining whether a transmission error occurs in the target file based on the verification data includes: Adding the first verification data to the target file; Transmitting the target file carrying the first verification data to the storage node; After receiving the target file, the storage node uses the same target verification strategy to obtain second verification data of the target file; At the storage node, whether a transmission error occurs in the target file is determined by judging whether the first verification data and the second verification data are consistent.

5. The method according to claim 4, characterized in that After determining whether a transmission error occurs in the target file, the method further includes: If the first verification data and the second verification data are consistent, determining that no transmission error occurs to the target file, and writing the target file to the storage medium corresponding to the storage node; In a case where the first verification data and the second verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to resend the target file.

6. The method according to claim 1, characterized in that In the case where the target file is read from the storage node through the data transmission path, the data verification layer verifies the target file according to the target verification strategy to obtain verification data, including: Initiate a read request for the target file to the storage node through the client; After the storage node receives the read request, the storage node reads the target file from a storage medium; In the data outer layer of the storage node, a target verification strategy is used to obtain third verification data of the target file; Adding the third verification data to the target file; Transmitting the target file carrying the third verification data to the data verification layer; After the data verification layer receives the target file, the data verification layer uses the same target verification strategy to obtain fourth verification data of the target file; The step of determining whether a transmission error occurs in the target file based on the verification data includes: At the data verification layer, whether a transmission error occurs in the target file is determined by judging whether the third verification data and the fourth verification data are consistent.

7. The method according to claim 6, characterized in that After determining whether a transmission error occurs in the target file, the method further includes: If the third verification data and the fourth verification data are consistent, determining that no transmission error occurs in the target file, and sending the target file to the client through the kernel-mode client, so that the user's application program uses the target file; In a case where the third verification data and the fourth verification data are inconsistent, it is determined that a transmission error occurs in the target file, and the client is notified to re-initiate a read request for the target file to the storage node.

8. The method according to claim 4 or 5, characterized in that During data transmission between the verification data layer and the storage node, the data is transmitted through a target protocol with high throughput and low latency characteristics, so that the verification data carried by the target file does not affect the transmission efficiency of the target file.

9. A data verification device, characterized in that: include: A determination module, configured to determine a data verification layer on a data transmission path between a kernel-mode client and a storage node; a verification module configured to, when a target file is transmitted through the data transmission path, verify the target file according to a target verification strategy determined by the data verification layer to obtain verification data, wherein the target verification strategy is determined based on the system load and the importance of the target file; The judgment module is used to judge whether a transmission error occurs in the target file based on the verification data.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data verification method according to any one of claims 1 to 8 are implemented.

Images